It is probably due to a number of people stopping using their alts after some instance hopping.
Also a few people who came to see how it was, and weren’t attracted enough to become regular visitors.
Curious to see at which number we’ll stabilize.
Next peak will probably happen after either major features release (e.g. exhaustive mod tools allowing reluctant communities to move from Reddit) or the next Reddit fuck up (e.g. removing old.reddit)
Stats on each server: https://lemmy.fediverse.observer/list
First optimization is to not fetch every field and prune it down. For example, it gets public key and private key for every community and user account - then does nothing with them. That’s just pushing data between Rust and PostgreSQL for no reason. That kind of thing is pretty obvious… the huge number of things listed after “SELECT”.
The whole approach is what I recently described as: make a JOIN fusion implosion bomb, then wait for null columns to fall out
There are short-term and long-term solutions. Right now there is already a new feature that will add one more JOIN that is pending merge… “instance blocking” by each single user.
Based on the server overloads and resulting crashes, I think some obvious solutions would be to remove post_aggregates table entirely and just throw more columns on the post table… I’ve seen people do stuff like that. But really you have to have a concept of core foundation.
To me the core foundation of Lemmy data is that people want fresh meat, when world events get into a frenzy, they want to F5 and get the LATEST post and the LATEST comments. Data should have a big wall between the most recent 5 days and everything else. It’s the heart of the beast of human events and a platform like this.
From that perspective, that fresh posts and fresh comments mean everything, you can optimize by just doing a INNER SELECT before any JOIN… or partition the database TABLE into recent and non-recent, or some out-of-band steps to prepare recent data before this SELECT even comes up from an API call… and not let PostgreSQL do so much heavy lifting each page refresh.
If I remember, I’m gonna look into that tomorrow when I’m not on a phone screen. Not that I could contribute anything, but this seems like a good opportunity to learn some advanced stuff. Thanks for your answer!