It's been a while, how have you been?

The Lemmy Club has been chugging along just fine, but with one issue - our database is beginning to push the limits of our storage drive. We're pushing 60gb on the DB. That's not photos, just text, vote records, etc. We have a larger drive with plenty of space but I only use it for backups because it's a slow drive and putting the database there would cause the site to crawl.

The last time this happened I just upgraded to the next tier of server up, which gave us a bit more storage and a decent bump in RAM/CPU. I could do the same again but the current server provider only offers an upgrade to a tier with 25gb more storage. We'd also get a bump in CPU/RAM but we're already overpowered in that regard. I like our current server provider however the numbers just don't make sense for us.

Instead, I've secured a server with a different provider. We'll get way more fast storage, more (albeit less powerful) CPU cores, and more RAM. For $7 less per mon

I don't like making new rules, however due to recent events I have unfortunately been forced to create this one.

For further context, please refer to this thread.

If you have any comments or concerns, I would love to hear them.

This release mostly includes some minor fixes in the background, however a major release is imminent in the next few months.

You can see the full changelog here.

As always, thanks for being a member of The Lemmy Club.

If you’d like to help contribute to our ongoing costs, or would like to show some appreciation, you can at OpenCollective: https://opencollective.com/thelemmyclub

This instance is funded primarily out of my own pocket with no return whatsoever. The costs aren’t too high (~$250/yr) but community donations are always very appreciated. At the moment, we’re about 50% community funded. Either way, The Lemmy Club isn’t going anywhere anytime soon.

Skipped over 0.19.6 as I heard a bugfix quick patch was coming and it is now here. 0.19.6 has all the changes, 0.19.7 just has a few bugfixes. Very few user changes. There's now a "controversial" sort mode. Most other changes are back end stuff like improved federation performance.

You can see the full changelog here.

You can follow the upgrade process at https://status.thelemmy.club/maintenance

As always, thanks for being a member of The Lemmy Club.

If you’d like to help contribute to our ongoing costs, or would like to show some appreciation, you can at OpenCollective: https://opencollective.com/thelemmyclub

This instance is funded primarily out of my own pocket with no return whatsoever. The costs aren't too high (~$250/yr) but community donations are always very appreciated. At the moment, we're about 10% community funded. Either way, The Lemmy Club isn't going anywhere anytime soon.

Sorry it took 35 minutes over my promised timeframe. My fault. I should've been keeping up my Ansible config not bumbling around manually. Then I would've known that I needed to uncap pict-rs' memory limit. Ah well we got there in the end.

0.19.4 mostly changes things in the backend, but there are a couple notable user features:

• You can now see all media you've uploaded and delete what you wish (only affects media uploaded after the upgrade)
• Better federation with Mastodon and other similar software, all posts are tagged with the community name (in the background, you won't see it) so that Mastodon users can see them.
• Vote totals now have settings. You can see a total (Upvotes minus downvotes), both seperate, or a percentage.

You can see the full changelog here.

As always, thanks for being a member of The Lemmy Club.

If you’d like to help contribute to our ongoing costs, or would like to show some appreciation, you can at

0.19.4 mostly changes things in the backend, but there are a couple notable user features:

• You can now see all media you've uploaded and delete what you wish (only affects media uploaded after the upgrade)
• Better federation with Mastodon and other similar software, all posts are tagged with the community name (in the background, you won't see it) so that Mastodon users can see them.
• Vote totals now have settings. You can see a total (Upvotes minus downvotes), both seperate, or a percentage.

You can see the full changelog here.

You can follow the upgrade process at https://status.thelemmy.club/maintenance

As always, thanks for being a member of The Lemmy Club.

If you’d like to help contribute to our ongoing costs, or would like to show some appreciation, you can at OpenCollective: https://opencollective.com/thelemmyclub

This instance is funded primarily out of my own pocket with no return whatsoever. The costs aren't too high

Hey all -

Just a quick announcement that we're currently not federating with eviltoast.org - it seems to be the base of the spam bots we've seen recently. I don't think the site administration teams have anything to do with the spam however they also seem to be absent, and are not doing anything to combat it. Until such time that they are actively cleaning up their spam issue they will remain defederated. I'm pretty lax about our federation, but this has been a persistent issue (See rule 9). I'm sure the spammer will probably find another host at some point though.

Thanks for being a member of TheLemmy.Club

Edit: it looks like thier admins have cleaned it up. Removing for now.

0.19 is a major update for Lemmy!

Big changes:

• Instance blocking for users: You can block the posts of users of any instance from showing in your feed.
• New Sort - Scaled: boosts smaller communities
• Better 2FA: actually tests your 2FA before applying, so you won't get locked out. If you had 2FA before, it'll be reset.
• User data export/import: change instances easily!
• Lots of performance and background changes.

Check here for more info: https://thelemmy.club/post/6199772

Well borked some things during the upgrade that required going back to the backup. But we got there eventually. I will have to mess some more with Ansible to bring it in line with my current deployment so future upgrades go a bit more smoothly.

If you were logged in, you'll have to log in again. If you had 2FA set up, you don't anymore and will have to redo it.

We were down for heh, 2.5 hours. Any activity on the Fediverse in that time won't have synced to our instance. However if anyone votes

0.19 is a major update for Lemmy!

Big changes:

• Instance blocking for users: You can block the posts of users of any instance from showing in your feed.
• New Sort - Scaled: boosts smaller communities
• Better 2FA: actually tests your 2FA before applying, so you won't get locked out. If you had 2FA before, it'll be reset.
• User data export/import: change instances easily!
• Lots of performance and background changes.

Check here for more info: https://thelemmy.club/post/6199772

Should take 30 minutes or less. Thanks for being a member of The Lemmy Club!

If you’d like to help contribute to our ongoing costs, or would like to show some appreciation, you can at OpenCollective: https://opencollective.com/thelemmyclub

This instance is funded primarily out of my own pocket with no return whatsoever. The costs aren't too high (~$250/yr) but community donations are always very appreciated. At the moment, we're about 10% community funded. Either way, The Lemmy Club isn't goin

We're up and running now, 30 minutes behind schedule.

I had added 1TB of storage to the server, but it turned out to be too slow and impacted performance hugely.

I was able to get 25GB extra NVMe storage for $7/mo, which should be enough to hold us for a couple months at the rate the DB has been growing. But that's just a bandaid. I like our current hosting provider but I will most likely have to find a new one as HostHatch doesn't have many options for larger SSD storage at prices we can afford. Good news I suppose, in addition to additional storage we also have 2 extra CPU cores and 6GB more RAM (22GB). Of course we don't really need those, but nice to have in any case.

In the meantime, I'll speak to their support team to see if they have any non-standard upgrade options or else research other providers.

Thanks for being a member of The Lemmy Club.

We've run out of space for the database! I've found a solution without increasing our costs too much ($5/mo), though the storage will be slower. I don't anticipate it affecting real world performance at this scale.

This was the cause of the outage earlier today. Backups are suspended until tomorrow to keep free space to keep running.

Additionally, the photo server database was corrupted by the free space running out, a backup had to be deployed and so any photos uploaded after 11am November 7th 2023 (US East) will have been lost.

I don't anticipate this taking more than about an hour, follow along at https://status.thelemmy.club/

If you appreciate what I do, you can contribute towards server hosting costs at https://opencollective.com/thelemmyclub

Thanks for being a user of The Lemmy Club.

See release notes here: https://join-lemmy.org/news/2023-08-08_-_Lemmy_Release_v0.18.4

Pretty minor update, bugfixes etc.

Also our server upgrades earlier this week were succesful and we're no longer running out of RAM.

If you'd like to help contribute to our ongoing costs, or would like to show some appreciation, you can at OpenCollective: https://opencollective.com/thelemmyclub

Thanks for being a member of The Lemmy Club!

Hey, your benevolent dictator server admin here. We've just about reached the limit of the VPS I currently have. So we're upgrading for the very first time!

Old server:
2 virtual cores (shared, Intel)
4GB RAM

New server: 4 cores (2 dedicated, 2 shared AMD EPYC 7402)
16GB RAM

We've mostly outgrown the RAM limitation. In normal operation it hovered at 3.5GB lately. But when backup tasks run or during particularly busy times we'd dip into the swap memory, which significantly slows things down even on NVME storage. We've been alright on CPU but this should futureproof things for a while. We'll also be getting IPv6, so that's cool. Expect max downtime of two hours if all goes well. Follow along at https://status.thelemmy.club/

Also note our IP address will change, and the old IP might be stuck in various caches on your devices.

Now about costs (per month):

• VPS rental: $15
• Backblaze B2 storage for backups (assuming we don't need to download them):

I do try to be fairly hands-off as much as possible, but there are limits. Please see rules below.

Instance Rules:

1. Don't be a dick.
2. Racism/slurs/etc use will not be tolerated.
3. No spamming.
4. Don't harass other users (See rule 1)
5. NSFW content must be marked correctly.
6. All content must comply with US law
7. Loli/etc. will not be tolerated. Suggestive or sexual art must be reasonably recognzable as adult subjects.
8. These rules apply to all content and users that appear on The Lemmy Club. Moderation is on an as noticed/as reported basis. If you see rule breaking content, I likely have just not seen it yet. Please report it.
9. Instances/users/communities that tolerate, repeatedly fail to enforce, or allow content that breaks any of these rules may be banned from The Lemmy Club.
10. The site admin team (well, just @bdonvr really as of now) has final say in interpretations of all rules.

Release notes: https://github.com/LemmyNet/lemmy/blob/main/RELEASES.md

TL;DR it's faster, the database is smaller (my DB backups went from 3.5GB to 570MB!), and it's more secure.

Messed up a simple config and it took about an hour to get the site back up instead of 10 minutes as I had thought. Sorry!

This is not hosted on the same server so it should stay up no matter how bad something goes wrong.

You can also see explanations of what happened during downtimes after I (hopefully) fix it, along with scheduled downtime.

Thanks for being a user at The Lemmy Club!

Check them out:

https://old.thelemmy.club/
https://app.thelemmy.club/

Let me know what you think!

Going to be migrating from storing media directly on the server, to cloud object storage.

To get more storage on the server I'd have to upgrade the CPU and RAM as well, and we really don't need that yet but we are pushing the limits of our storage.

It's quite a large upload to migrate, and pict-rs can't be running during the migration - plus I'm gonna do some full backups first so I've blocked out 2 hours. Could be shorter or slightly longer.

Thanks for being a user of The Lemmy Club!

Welp, just woke up.

Looks like several big instances have been compromised.

We're doing okay at the moment, and based on what I've heard in discussion threads and my instance admin group chat - the vulnerability lies in custom instance emoji something I haven't set up yet, luckily. Looks like we should be alright for now. I'll continue monitoring.