Skip Navigation
sunaurus

Hey folks! I'm responsible for servers at lemm.ee.

For general lemm.ee policy/admin/moderation topics, please contact @[email protected]

Posts
56
Comments
464
Joined
2 yr. ago
  • Most applications are handled automatically, a smaller percentage is sent to manual approval for admins. This is based on different signals and the aim is to potentially reduce some abuse, or at least make it a bit harder.

    Median time for accepting applications in the past month on lemm.ee was 1 minute and 42 seconds. And that includes the time it took for users to manually verify their e-mails first.

  • We briefly had some database issues, it should be solved now, but I will continue to monitor.

    Sorry for the inconvenience!

  • What I mean is that some posters have a bunch of alts on different instances, and if they get awards split between their accounts, then they'll be at a bit of a disadvantage 😄

  • This is cool!

    Any rules for alts? Can people "claim" alt accounts to sum up their votes?

  • It's very unlikely lemm.ee will ever be a suitable host for video content. Please host your videos on other platforms, embedding them on lemm.ee posts will work just fine.

  • Whenever you give your free time and energy to build something open source for the world, there are always some people who say “why aren’t you giving me 10x more, you lazy dev” 😅

  • Could I ask you to try again with the app and let me know if it's working for you now?

  • Hey, sorry for the issues, please see my comment here for explanation: https://lemm.ee/post/57870550/18964440

  • Hey, sorry for the issues, please see my comment here for explanation: https://lemm.ee/post/57870550/18964440

  • Hey, for the past few weeks, we have been tuning different rate limits and other bot prevention mechanisms on lemm.ee. We have had a huge increase in traffic lately, as well as a couple of instances of what was effectively a DDoS.

    In other words, the problems likely have nothing to do with your app or different versions, and everything to do with our own measures.

    For the mark post as read endpoint, I recently made the rate limit a bit more relaxed, so hopefully users won't see issues there anymore. OTOH, we are also presenting a Cloudflare challenge for some IP ranges currently, which may inadvertently affect legit users. This was necessary to mitigate a recent DDoS, but we will continue tuning and hopefully restore things to normal for legit users soon.

  • Definitely not 😄

  • Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    Hey folks!

    For the past few hours, lemm.ee has been bombarded with abnormal (almost definitely automated) traffic from a range of different IP addresses. This managed to overwhelm our servers, and we were offline for the past hour or so.

    I was in the middle of celebrating my birthday, so response was a bit slow, but I believe we are recovering now, with mitigations in place to try and prevent further issues. Some of you may be inconvenienced by some bot checks when you browse lemm.ee, I am sorry about that, but it's necessary for now.

    Sorry for the issues and I hope you have a nice weekend ahead!

  • In my experience, it's mostly a combination of confusing Lemmy UI + user error:

    1. When using the e-mail verification link, Lemmy just shows a blank page with a tiny "e-mail verified" notification in the bottom left corner, which people miss. They then assume that it didn't work, because all they see is a blank page.
    2. When opening the e-mail verification link multiple times, only the first time registers, and further attempts will generate confusing errors. This also makes people assume it didn't work, but in reality, their e-mail got verified successfully.
    3. On lemm.ee, it's very common for people to sign up with typos in their e-mail addresses (happens few times every week recently).
    4. We have a custom question on our sign-up page asking people to state they agree to our rules, and it's relatively common that people just don't read the question and write something random in there - we generally don't accept such applications to try and weed out bots, but I'm pretty sure we also end up rejecting a bunch of legit users this way who just didn't read the instructions.

    Generally I'm happy to provide support to people in such cases if they reach out to me directly, but at the end of the day, the easiest and fastest solution if the account is truly stuck in limbo is just to create another account.

    It's a shame that there is so much potential friction on sign-up, but at the same time, I don't see us reducing the friction on lemm.ee any time soon, because this is the lesser evil compared to bot sign-ups etc that we have seen in the past.

  • I’ve just restored it

  • lemm.ee is growing. Welcome, new members!

  • Hey, there isn’t any default community right now. There are a few different databases that track graphs of such things, for example, fedidb.org.

  • lemm.ee is growing. Welcome, new members!

  • Thank you very much for the support!

    Our infrastructure costs are currently quite stable at around 200€ per month, and considering that the instance is right now quite decently supporting nearly 6000 monthly active users, you could say that you are indeed relatively contributing a ton - you are effectively covering server costs for 60 people!

    The fact that it’s a monthly amount is particularly great, because with recurring income, we will get advanced warning if there is danger of funds starting to run low.

  • lemm.ee is growing. Welcome, new members!

  • We had a few really huge days in 2023, but other than those, it seems like the growth so far in March is definitely outpacing our initial wave of new users in 2023.

  • Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    lemm.ee is growing. Welcome, new members!

    Hi folks!

    Over the past few months, we have started seeing a significant amount of new user sign-ups. I would like to take this opportunity to welcome all of our new members, and to share some useful resources and info about lemm.ee.

    First, some stats

    Here is a bar chart of daily new users (this is only counting users which have been approved by our admins):

    As you can see from the chart, for most of 2024, we were accepting roughly around 10-20 new users every day. Then, from the start of this year, the daily numbers have been constantly growing. Yesterday, we approved a massive 609 new users on lemm.ee.

    The increase in sign-ups is significant enough that I have been taking several steps to improve our monitoring & anti-bot measures, but **so far, it seems the vast majori

  • Hey, could you make a post in that community?

  • Lemmy software resolves it automatically - it marks such instances as dead and stops sending activities to them.

  • Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    When this post is 3 hours old, lemm.ee will go offline for some brief database maintenance

    Hey folks

    Just a quick heads up, we will be performing some database maintenance today. Expected downtime is ~15 minutes.

    Sorry for the inconvenience!


    Update: maintenance complete!

  • The e-mail successfully went out from our side - please check your spam filters etc

  • It's the correct error code (the client is sending a bad request - it's an already used token), the problem is just that Lemmy UI doesn't really display a useful error page to the user.

  • Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    Admin team update

    Hey Folks

    Just a quick note to let you all know about some changes in the lemm.ee admin team. After discussing things with the other admins, we've decided to shuffle around our roles a bit.

    Up until now, I’ve been the head admin at lemm.ee - handling infrastructure, maintaining rules and policies, and acting as the main contact person for the admin team.

    However, I’ve come to realize that this role has taken a toll on me. While I still love the idea of Lemmy and everything it stands for, being an admin has slowly drained the joy I once had for the platform. The occasional negative experiences have been increasingly difficult for me to shake off. For the past several months, I’ve found myself hesitating to check my DMs or the moderation queue, simply because I’m bracing for some new drama that I no longer have the energy to manage.

    After some conversations with the team, we’ve agreed on a plan to ensure my burnout doesn’t negatively impact the instance:
    1. I am st
    Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    When this post is 1 hour old, lemm.ee will go down for an upgrade to 0.19.7 (UPDATE: upgrade complete!)

    Hey folks!

    I'll be updating lemm.ee to 0.19.7 shortly. I'll take it offline & also perform some database maintenance at the same time.

    I have been investigating some incoming federation issues, and I haven't been able to find the actual cause. My current theory is that they are related to some server timeouts. I am hoping that the combination of the Lemmy upgrade and database maintenance may help improve the situation, but if not, then I will try other approaches.

    Edit: the upgrade has been completed!

    Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    Expected lemm.ee downtime when this post is 13 hours old.

    Hey!

    Unfortunately, Hetzner (our hosting provider) is currently experiencing some network issues. They are planning to address this with an emergency maintenance in roughly 13 hours from now, which will cause lemm.ee downtime. Hopefully we'll be fully recovered later tomorrow!


    UPDATE: Sorry for the false alarm, I was on the move when I posted this and missed the fact that the Hetzner notice was actually for next month! So it's not as imminent as I originally understood. As we have a whole month to prepare, I will probably be able to come up with some alternative solution to prevent the downtime while they are conducting this maintenance.

    Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    Seeking feedback: how should lemm.ee move forward with external images? (related to frequent broken images)

    Hey folks!

    I am looking for feedback from active lemm.ee users on what you all value when it comes to images on Lemmy. I'll go into a bit of detail about what our options are, and then I would ask you to voice your opinion about the issue in the comments.

    First, some context for those who don't know. Lemmy software can be configured to handle images in three different ways:

    1. Store images locally - whenever an external image is posted somewhere, lemm.ee will download a permanent local copy. When you view posts, you are seeing our local copy of the image.
    2. Proxy all images - similarly to the first option, lemm.ee will download a local copy of external images, however, this copy is temporary. It will be automatically deleted shortly after, and if users open the relevant post/comment again in the future, there will be another attempt to download a temporary copy at that point.
    3. Pass through external images directly - lemm.ee never downloads any external images, users will al
    Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    Hey folks!

    Unfortunately, roughly 2 hours ago, lemm.ee went offline. The cause was our load balancer: it suddenly decided that all of our servers had become unhealthy, despite all health checks responding successfully when I requested them directly. In such cases, the load balancer stops serving all requests, effectively meaning that lemm.ee is unreachable for all users. I am still not sure what exactly caused the issue, but I will try to investigate more over the weekend.

    For now, we have partially recovered, and I am continuing to work on remaining issues. Hopefully we will be back to 100% very soon. Sorry for the inconvenience!

    Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    Hey folks!

    For anybody stumbling on this post from outside lemm.ee: I am the head admin of lemm.ee, a general purpose Lemmy instance, which recently turned 1 year old. I am writing this post to elaborate on how we approach defederation on lemm.ee.

    Anybody who has been on Lemmy for a while has most likely seen several public defederation drama posts (most recently regarding lemmy.ml, but there have been many many others previously). As an admin, I have probably seen far more than what is visible publicly, as I regularly receive private messages on the topic, ranging from polite questions about federation, to outright demands that I immediately defederate, and even to threats and personal attacks over the fact that I have not defederated some particular instance. It is definitely a topic that will keep coming up for as long as Lemmy exists, which is why I feel it would be useful to condense my current thoughts about it in a single place.

    Note that while I strongly believe everythin

    Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    When this post is 3 hours old, lemm.ee will briefly go offline for database maintenance

    Hey folks

    Just a heads up that I will be doing some minor database maintenance shortly. I expect the downtime to last <5 minutes.

    Have a nice day!

    Update: maintenance is complete!

    Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    Hey, folks!

    Today, we can celebrate the first anniversary of the creation of lemm.ee! I thought it would be cool to write down how lemm.ee was born, as well as collect some stats about our first year. Here goes!

    A quick recap of the beginning of this instance

    As probably many others here, I discovered Lemmy early last summer. I had been aware of the Fediverse previously, and always thought it was an amazing concept, but I had never been super interested in Twitter-style social networks. When I found out that Lemmy combined all the great parts of federation with the best parts of link aggregation, I knew that I had to join immediately.

    As I was trying to find an instance to make my account on, I realized that most instances were struggling to keep up with a massive influx of new users. At the time, there was a big explosion in Lemmy user numbers, and the network wasn’t fully ready for it. I have some experience with building software for scale, so it felt natural to set up a n

    Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    Update about lemm.ee infrastructure & upcoming cakeday

    Hey all!

    Upcoming lemm.ee cakeday

    Can you believe that lemm.ee is almost 1 year old? In just a couple of weeks (specifically, on the 9th of June), we will be able to celebrate our first instance cakeday.

    I am thinking of compiling some stats about how lemm.ee has been used in its first year, if you have any specific stats in particular you would like to see, feel free to comment below. I will try to accommodate any ideas as I start gathering this info!

    Infrastructure updates

    A few weeks ago, I posted about plans to make some changes to our infrastructure in order to deal with different intermittent networking issues.. It took a bit longer than I hoped (just did not manage to get enough free time between then and now), but I am happy to report that this work has now been completed! Additionally, I have decommissioned our stand-alone pict-rs server.

    **With the two changes mentioned above, I believe lemm.ee should now be much more

    Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    Hey folks!

    This is a quick notice about a change to our moderation policy.

    We have had a policy on lemm.ee for administration and federation nearly since the very beginning. This policy has also always included a section about moderator responsibilities. Today, we have made two changes to this policy:

    1. The policy has been renamed to Policy for administration, moderation, federation - this is to make it clear that the policy is also relevant for mods
    2. We have introduced a new responsibility for moderators, they must "Ensure that they only provide accurate and clear reasons for mod actions".

    The reason for the addition is that mod log actions federate out to other instances, and are more or less permanent (due to how Lemmy and federation works right now). This means that users do not really currently have any easy way to clarify or defend themselves against inaccurate accusations in the mod log.

    As always, I am very grateful to all mods for your

    Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    Hey folks!

    We unfortunately had about half an hour of unplanned downtime today. This was caused by an issue with our hosting provider. The issue is solved for now, and I am planning to make some changes to prevent similar issues in the future. Sorry for the inconvenience!


    Technical details

    Our servers are communicating with our database over Hetzner's "vSwitch" service. Unfortunately, this service seems to be quite flaky - over the past few months, I have had to deal with the connection just dropping without recovering many times. Mostly this has not resulted in any noticeable downtime, as we have redundant servers, so even if one of them stops working, it won't affect lemm.ee users. However, in this instance, all of our API servers lost their connection to our database at the same time, which resulted in actual downtime.

    I have now decided to migrate our setup away from the vSwitch in the near future to hopefully stop these issues for good. Should be possible to do t

    lemmy-ui-next @lemm.ee
    sunaurus @lemm.ee

    Seeking feedback on lemmy-ui-next! (Milestone 2 is nearing completion!)

    Hey folks!

    I've been steadily working through the roadmap for lemmy-ui-next (which is a new alternative Lemmy frontend), and it's getting to a point where I think https://next.lemm.ee/ is becoming quite usable. I've been personally using it as my main Lemmy frontend for several weeks now, and I know there are a few other brave users doing the same, so at this point, I'm confident enough to ask the wider lemm.ee population to try it out and share some honest feedback.

    If you're at all interested in this project, I would massively appreciate it if you could spend some time using https://next.lemm.ee/ and letting me know how you feel about it. I'm interested to hear about things like:

    • are you running into any issues or bugs
    • are there any things that generally annoy you
    • are you missing any features
    • what would it take for lemmy-ui-next to become your preferred frontend
    • anything else that comes to mind

    Please keep in mind th

    Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    When this post is 6 hours old, lemm.ee will undergo database maintenance, which will include a few minutes of downtime

    Hey folks

    This is just a quick heads up that I need to perform some maintenance & upgrades on our database server, which unfortunately will require downtime. I don't expect the downtime to last for longer than 2-3 minutes, but just wanted to give a heads up first so you know not to be concerned.

    That's all, hope you have a great week!

    Edit: maintenance complete!

    Test @lemm.ee
    sunaurus @lemm.ee

    First test post from lemmy-ui-next!

    Hello, world!

    Edit: first test edit!

    lemmy-ui-next @lemm.ee
    sunaurus @lemm.ee

    Milestone 1 is complete!

    Milestone 1 complete!

    This is just a mini-announcement & celebration for the fact that I have completed the scope for the first milestone I set for myself in the roadmap.

    Of course, I am still planning to keep improving and tweaking things as I go, but in terms of the raw list of features, the work for milestone 1 is complete. I am now going to take a day or two to clean up the code and work on some performance optimizations, and then in the later half of the week, I will continue working towards milestone 2, starting with commenting features!

    If anybody is interested (and brave), please feel free to check it out at https://next.lemm.ee/, and feel free to share any thoughts and feedback in the comments!

    lemmy-ui-next @lemm.ee
    sunaurus @lemm.ee

    Intro & roadmap

    Intro

    This project is an open source alternative frontend for Lemmy. It is built with Next.js.

    Screenshots (desktop & mobile)

    Goals

    • Drop-in replacement for lemmy-ui
    • Minimalistic design, following in the footsteps of other timeless link aggregator UIs
    • Fast!
    • Super basic NextJS architecture, taking advantage of features like the app router & server actions

    Motivation

    The original lemmy-ui has been extremely important for the growth of Lemmy, and the new lemmy-ui-leptos also looks quite interesting. One issue with both of these is that they are built using

    Meta (lemm.ee) @lemm.ee
    sunaurus @lemm.ee

    Announcing lemmy-ui-next, an alternative Lemmy frontend built with NextJS!

    Hello, friends!

    TL;DR: I am working on a new Lemmy frontend in nextJS. There is still much work to be done, but you can already have an early look at https://next.lemm.ee

    First of all, quick note to lemm.ee users: I am making this announcement post in [email protected], as this is also a notice that I will be hosting an alternative frontend (lemmy-ui-next) for the first time on lemm.ee. Going forward, I will post updates about lemmy-ui-next in a separate dedicated community: [email protected]. If you're interested in future updates, please subscribe there!

    What is lemmy-ui-next?

    Lemmy is generally accessed through some kind of frontend UI. By default, Lemmy provides its own web interface (lemmy-ui), which you can find on the front page of most Lemmy instances (including lemm.ee). There are also several other independent frontends, for both the web and different mobile platforms, which I'm sure many of you are familiar with.

    Lemmy-ui-next is a brand new alternative front