Skip Navigation
Crashdoom (he/him)
Crashdoom (he/him) @ crashdoom @pawb.social

Network Administrator for Pawb.Social, furry, and a programmer

Mastodon: @crashdoom@furry.engineer

Posts
59
Comments
105
Joined
2 yr. ago
  • I'm not able to replicate this myself as it's showing on both the Create Post and editing modes. I've cleared the site cache on Cloudflare as it could be it's caching an old version of the UI. I'd also encourage clearing your browser cache / force refreshing.

  • We've got a fix deployed that'll allow the NSFW box to be toggled again for new and existing posts -- Is there any chance you'd be able to facilitate that on your existing posts? We're hoping that'll alleviate reports and complaints in the interim while we wait for the permanent backend fix x3

  • Update, Apr 27, 2025

    We've updated Lemmy UI to a hotfix version of 0.19.11 that re-enables the NSFW field on the Create Post form for NSFW communities. This should once again allow users to correctly mark posts explicitly with the NSFW tag.

    While a permanent fix is in development by the Lemmy team, please ensure you check this NSFW toggle if you're posting to a NSFW community.

    Thank you for your understanding and patience.

  • Pawb.Social Announcements @pawb.social
    Crashdoom (he/him) @pawb.social
    PSA

    Known Issue: Posts in NSFW Communities

    We’ve been made aware that it is no longer possible to identify posts as NSFW in a NSFW community as of Lemmy 0.19.11 (https://github.com/LemmyNet/lemmy-ui/pull/2887), and as a result some posts appear to be not marked as NSFW.

    While some apps (Voyager is the only one I have personally verified) do correctly flag the posts in NSFW communities as NSFW by default.

    We’ve verified on 0.19.11 Lemmy will blur the image and not tag the post as NSFW, but we haven’t verified behavior from older instances viewing our communities / posts.

    Due to the current time as of writing this, I’m not able to investigate further until the morning but I hope this provides some basic information on the situation for now.

    Am I The Asshole? @pawb.social
    Crashdoom (he/him) @pawb.social

    AITA: Welcome, FAQ, and How to Post

    Welcome to Pawb.Social's c/AITA!

    While furry-themed, this is a general purpose Am I The Asshole community intended to help posters see their situations from another perspective and grow.

    How to Post
    • Ensure your title starts with "AITA" and explains (concisely) what happened (e.g. "AITA for downing all the punch?")
    • Describe the situation as objectively as you can, ensuring you anonymize other participants in the situation
    • Include how the other participants are upset with you or believe you to be the asshole (as much as you're aware)
    • Include why you believe you might be the asshole at the end of your post
    • Try to keep posts reasonably concise. Only provide backstory information where it's absolutely necessary and cannot be summarized.
    FAQs
    Voting Guide / Terminology
    • 🟢 NTA – Not The Asshole
    • 🟡 ESH – Everyone Sucks Here
    • 🔴 YTA – You’re The Asshole
    • 🔵 NAH – No Assholes Here
    • 🟣 NMI - Need More Information
  • I believe there was a change to some Lemmy communities where if the community itself is marked as "NSFW" there's no ability to toggle it off anymore: https://github.com/LemmyNet/lemmy-ui/pull/2887 (https://join-lemmy.org/news/2025-04-08_-_Lemmy_Release_v0.19.11).

    If the post isn't flagging automatically as NSFW in a NSFW community, then there may be a backend bug that needs to be reported to the Lemmy developers via GitHub.

  • Update: Turns out a security feature we enabled to block "AI Bots" (like ChatGPT) was blocking federation traffic... Since it's affecting stuff right now, I've disabled it again until I can get time after LVFC to take a look into building a comprehensive "allow list" of different federation traffic headers.

    tl;dr Give it a little bit and things should hopefully catch up with new posts, but I don't know if Lemmy post-syncs old posts without someone searching for them on our instance.

  • Restarting the federation pods appears to have at least lowered the failure queue a little, but it's showing that while we've sent things to lemmit.online (as the example), it hasn't seen anything from them since 4/14/2025, 10:14:27 PM. So I'm presuming the next time they try to sync and send something to us it SHOULD update...

    The federation state checker seems to be indicating that we're not behind in sending them posts anymore so outgoing should be fixed, but I'm unsure how to troubleshoot incoming further. I'm wondering if there's a bug with the latest version of Lemmy?

  • Morning all! It does appear like we're not getting some instances posts (you can find a list by using https://phiresky.github.io/lemmy-federation-state/site?domain=pawb.social). I'm currently investigating and I appreciate folks poking on this :3

  • 😂 We've given it a stern talking to and helped it recover through a restart, and image uploads appear to be working as normal again!

  • via here (I'm setting up an RSS feed monitor), via email (network[at]pawb.social), or with a DM message to highlight it. It seems like mentions in the original post don't alert the mentioned person, so I only got a notice when you replied to a message from me / mentioned me in a comment. >~<;

  • Media uploads should be working again. My apologies.

  • Media uploads should now work again!

  • We're investigating! Appreciate the report!

  • Taking a peek now, appreciate the heads up!

  • The server hosting the database crashed for a still unknown reason. Sometime around 5 am MT, the database server had a kernel panic and locked up, it’s been like that since then until we woke up to the alert this morning.

    We’ve got a watchdog on the server and an auto-restart in case it hangs, and we’re working to migrate it away to a known stable server while we troubleshoot.

  • Honestly, like you said, I genuinely have no idea what the US will do; Currently things are being so shaken up that anything is plausible right now.

    For outside the US, I don't really know what reach other countries have. The UK's Online Safety Act is the current one I'm watching for since that would require "highly effective age assurance" which... if the folks writing the law had been online like, ever, they'll have realized is an utterly stupid statement unless they plan to enforce integrating ID Verification via a Credit Agency (Experian, Equifax) which does personal question-based verification (which imo is a huge breach of user privacy since while WE might not know who you are still, the credit agency does and where you signed up).

  • Yes, several US states along with other nations have been considering porn ban laws, along with the rather blatant anti-LGBTQ stance of the currently elected US party.

  • In the works! We're planning to have some info very soon, though we're still building out an improvement for the NAS to allow us to expand the storage capacity for long term, high size file storage specifically for Pixelfed!

  • Time (if it's been persistent for a while), what you were trying to do, what content you were interacting with, etc.

    If you've got network logs from it, that'd be a cherry on top! Though for the most part, it'll be our local logs I'll need to correlate your issues with in order to figure out what happened.

  • Previously, we were just operating under myself as a sole proprietor. I hadn't formalized things as I hadn't expected the big growth, but we're big enough to justify it to ensure the long-term security of all of our instances, as well as my own personal protection against the changing political climate.

  • Yeah, it's been on my plate for a while, and honestly I really should have

  • Pawb.Social Announcements @pawb.social
    Crashdoom (he/him) @pawb.social
    IMPORTANT

    Pawb.Social Organizational Changes

    This notice is also being sent by email to all users with a verified email on Pawb Social services. (Completed as of 02/13/2025)

    UPDATE (02/13/2025)

    ℹ️ If you received the "Important - Pawb.Social is re-structuring as an LLC!" email from us about an account you don't recognize, or no longer wish to have, on our services and would like the account deleted:

    Please email network@pawb.social from the SAME email address, and we'll be able to lookup your accounts and remove them.

    If your account has any content, we will issue a backup to the email on file and wait 7 days before deleting the account, unless you ask us to delete it immediately after we inform you that content exists.


    Begin Original Post

    As of Friday, February 14th, we will be transitioning the management and ownership of our instances to PawbSocial LLC, a new limited liability company (LLC) registered in the State of Colorado.

    This change is primarily driven by the desire to ensure legal protecti

    Pawb.Social Announcements @pawb.social
    Crashdoom (he/him) @pawb.social

    Lemmy 0.19.6-beta.13 and "Toastify is awesome!" (which it is.)

    Firstly, the maintenance for Pawb.Social services has been completed and we've upgraded Lemmy to 0.19.6-beta.13, and our Mastodon instances to v4.3.0.

    Thanks for your patience.


    and now, onto the "Toastify is awesome" (because it is. Totally.) postmortem:

    Short summary: Crash did a stupid with the config and emails weren’t getting sent (oops). Emails should now be working again, and registration shouldn't require admin intervention.

    Longer story:

    Lemmy has several configuration options that need to be updated to correspond to the specific instance that's running it. For example, here's a snippet of Pawb.Social's config:

     undefined
        
    {
      database: {
        host: "the.best.db.url.internal"
        user: "pawbsocial_lemmy"
        database: "pawbsocial_lemmy"
        password: "pawbsocial_do_you_really_think_i_would_leak_this"
        pool_size: 128
      }
      hostname: "pawb.social"
      pictrs: {
        url: "http://pawb-social-pictrs:8080/"
        api_key: "sekrit"
      }
      email: {
        smtp_server: "email-smtp.us-east
      
    Pawb.Social Announcements @pawb.social
    Crashdoom (he/him) @pawb.social

    Lemmy v0.19.4 / v0.19.5 Updates

    We've successfully upgraded Lemmy from v0.19.3 to v0.19.5 and have a bunch of changes to share!

    Bug Fixes

    • Previously users were reporting that the web client kept logging them out. This was suspected to be due to a misconfiguration of the cookie storing the session information and should be fixed.
    • Some communities were slow to load. Federation changes in v0.19.4 and v0.19.5 should rectify some federation issues. Additionally, we increased the number of database connections for Lemmy and will gradually increase this if we identify any further slowdowns.

    (Longer term, we're planning to upgrade the database hardware which should improve the performance for all services)

    Local Only Communities

    When creating or editing your communities, you'll now see a "Visibility" option that allows you to choose between "Public" and "Local Only".

    The "Public" setting will continue to operate as normal and federate your community to all other instances we federate with.

    "Local Only" wi

    Pawb.Social Announcements @pawb.social
    Crashdoom (he/him) @pawb.social

    Sept 22 - Login Issues on Pawb.Social

    We’re aware of an issue affecting our Lemmy service (pawb.social) that seems to be causing users to be randomly logged out. We believe we’ve identified the issue thanks to a bug fix in a newer version of the UI code.

    We’re needing to verify there are no other breaking changes for the upgrade process, and expect to perform maintenance this Sunday, Sept 22nd at Noon US Mountain Time (https://everytimezone.com/s/7abd4955). We’ll provide an update tomorrow to confirm the maintenance window.

    Thank you to everyone who brought this to our attention and we apologize for the inconvenience.

    Pawb.Social Administrative Actions @pawb.social
    Crashdoom (he/him) @pawb.social

    tenforward.social

    After reviewing a report regarding the personal account of the admin of tenforward.social, and discussing with other fediverse admins, we're deciding to suspend the admin and limit tenforward.social until this blows over. I'll explain why on both, starting with the admin themselves.

    Suspending packetcat@tenforward.social

    Rule 1: Threats of Violence

    packetcat made several posts[1][2] that were perceived as a threat of violence, metered by it being "theoretical". This is not acceptable behavior for any user, especially an instance administrator, irrespective of the intended target of their annoyance.

    This comes across as extremely volatile and dangerous, and on any other platform (except the cesspool that is X / Twatter) would be immediately removed.

    Rule 6: No Spam

    packetcat appears to have been going on a run of posts using several hashtags excessively in ways we, and other instances, have previously actioned as abuse / spam.

    In the first case, the #fediblock hasht

    Test Community @pawb.social
    Crashdoom (he/him) @pawb.social

    test

    test

    Pawb.Social Feedback @pawb.social
    Crashdoom (he/him) @pawb.social

    Maintenance Windows

    Given the size the community has grown to and concerns around our current approach to maintenance, I’d like to get some feedback on when we should look to schedule maintenance going forward.

    There’s two cases covered by the poll, so please choose the all options that fit your preferences.

    Case 1: Non-interactive Maintenance This covers any maintenance where we don’t actively need to monitor the upgrade process; Copying data from one location to another, etc. (Usually, this could be left overnight and brought back up in the morning)

    Case 2: Interactive Maintenance This covers any maintenance that requires us to perform a series of actions and continually monitor the process to ensure no issues; Upgrading Mastodon or Lemmy, database migration, or upgrading the Kubernetes cluster. (Usually, this would need to occur during the day due to time constraints)

    To vote, please find the corresponding pinned comments and upvote them!

    If you’ve got any questions, concerns, etc. please leav

    Pawb.Social Feedback @pawb.social
    Crashdoom (he/him) @pawb.social

    Use of Automated Moderation Tools

    Due to the recent spam waves affecting the Fediverse, we'd like to open requests for comment on the use of automated moderation tools across Pawb.Social services.

    We have a few ideas on what we'd like to do, but want to make sure users would feel comfortable with this before we go ahead with anything.

    For each of these, please let us know if you believe each use-case is acceptable or not acceptable in your opinion, and if you feel like sharing additional info, we'd appreciate it.


    1. Monitoring of Public Streaming Feed

    We would like to set up a bot that monitors the public feed (all posts with Public visibility that appears in the Federated timeline) to flag any posts that meet our internally defined heuristic rules.

    Flagged posts would be reported per normal from a special system-user account, but reports would not be forwarded to remote instances to avoid false-positives.

    These rules would be fixed based on metadata from the posts (account indicators, mentio

    Pawb.Social Feedback @pawb.social
    Crashdoom (he/him) @pawb.social

    Pawb.Social Migration Bugs / Issues Megathread

    On Feb 14th we migrated Lemmy from its standalone Docker setup to the same Kubernetes cluster operating furry.engineer and pawb.fun, discussed in https://pawb.social/post/6591445.

    As of 5:09 PM MT on Feb 14th, we are still transferring the media to the new storage, which may result in broken images. Please do still reply to this thread if your issue is media related, but please check again after a few hours and edit your comment to say "resolved" if it's rectified by the transfer.

    As of 11:02 AM MT on Feb 15th, we have migrated all media and are awaiting the media service coming back online and performing a hash check of all files. Once this is completed, uploads should work per normal.


    To make it easier for us to go through your issues, please include the following information:

    • Time / Date Occurred
    • Page URL where you encountered the issue
    • What you were trying to do at the time you encountered the issue
    • Any other info you think might be imp
    Pawb.Social Announcements @pawb.social
    Crashdoom (he/him) @pawb.social

    Feb 7 - Mastodon Data Migration

    tl;dr summary furry.engineer and pawb.fun will be down for several hours this evening (5 PM Mountain Time onward) as we migrate data from the cloud to local storage. We'll post updates via our announcements channel at https://t.me/pawbsocial.


    In order to reduce costs and expand our storage pool, we'll be migrating data from our existing Cloudflare R2 buckets to local replicated network storage, and from Proxmox-based LXC containers to Kubernetes pods.

    Currently, according to Mastodon, we're using about 1 TB of media storage, but according to Cloudflare, we're using near 6 TB. This appears to be due to Cloudflare R2's implementation of the underlying S3 protocol that Mastodon uses for cloud-based media storage, which is preventing Mastodon from properly cleaning up no longer used files.

    As part of the move, we'll be creating / using new Docker-based images for Glitch-SOC (the fork of Mastodon we use) and hooking that up to a dedicated set of database nodes and replicated st

    Pawb.Social Announcements @pawb.social
    Crashdoom (he/him) @pawb.social

    Lemmy v0.19.3

    We've updated to Lemmy v0.19.3!

    For a full change log, see the updates below:

    Major changes

    Improved Post Ranking

    There is a new scaled sort which takes into account the number of active users in a community, and boosts posts from less-active communities to the top. Additionally there is a new controversial sort which brings posts and comments to the top that have similar amounts of upvotes and downvotes. Lemmy’s sorts are detailed here.

    Instance Blocks for Users

    Users can now [block instanc

    Pawb.Social Administrative Actions @pawb.social
    Crashdoom (he/him) @pawb.social
    • Instance: pisskey.io
    • Type: Defederation
    • Affects: Pawb.Social, furry.engineer, pawb.fun
    • Reason: Nazi imagery, affiliated with poa.st and other known abusive instances
    • Fediseer Action: Censured
    Pawb.Social Administrative Actions @pawb.social
    Crashdoom (he/him) @pawb.social

    the.asbestos.cafe

    • Instance: the.asbestos.cafe
    • Type: Defederation
    • Affects: Pawb.Social, furry.engineer, pawb.fun
    • Reason: Homophobia, harassment, trolling, admin involved abuse
    • Fediseer Action: Censured

    Block has been applied to the entire domain.

    Pawb.Social Administrative Actions @pawb.social
    Crashdoom (he/him) @pawb.social
    • Instance: lab.nyanide.com
    • Type: Defederation
    • Affects: Pawb.Social, furry.engineer, pawb.fun
    • Reason: Trolling, harassment, homophobia, nazi imagery, admin / mod engaged abuse
    • Fediseer Action: Censured

    Block has been applied to the entire domain.

    Fediblock @lemmy.dbzer0.com
    Crashdoom (he/him) @pawb.social

    cunnyborea.space

    cross-posted from: https://pawb.social/post/3393854

    • Instance: cunnyborea.space
    • Type: Defederation
    • Affects: Pawb.Social, furry.engineer, pawb.fun
    • Reason: Racism, antisemitism, homophobia, abusive admin, nazi imagery
    • Fediseer Action: Censured
    Pawb.Social Administrative Actions @pawb.social
    Crashdoom (he/him) @pawb.social

    cunnyborea.space

    • Instance: cunnyborea.space
    • Type: Defederation
    • Affects: Pawb.Social, furry.engineer, pawb.fun
    • Reason: Racism, antisemitism, homophobia, abusive admin, nazi imagery
    • Fediseer Action: Censured
    Fediblock @lemmy.dbzer0.com
    Crashdoom (he/him) @pawb.social

    cross-posted from: https://pawb.social/post/3337642

    • Instance: bv.umbrellix.org
    • Type: Defederation
    • Affects: Pawb.Social, furry.engineer, pawb.fun
    • Reason: Toxicity, abusive admin, death threats, emotional abuse
    • Fediseer Action: Censured

    Admin Note: This block applies to the root domain (umbrellix.org) and all associated sub-domains.

    Pawb.Social Administrative Actions @pawb.social
    Crashdoom (he/him) @pawb.social
    • Instance: bv.umbrellix.org
    • Type: Defederation
    • Affects: Pawb.Social, furry.engineer, pawb.fun
    • Reason: Toxicity, abusive admin, death threats, emotional abuse
    • Fediseer Action: Censured

    Admin Note: This block applies to the root domain (umbrellix.org) and all associated sub-domains.