Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
Reddit is Being Manipulated by Professional Shills Every Day (reupload) - YouTube
Request to add TOTP 2FA authenticator support in Piefed
It's best to use either Cloudflare (best IMO) or Anubis.
If you don't want any AI bots, then you can setup Anubis (open source; requires JavaScript to be enabled by the end user): https://github.com/TecharoHQ/anubis
Cloudflare automatically setups robots.txt file to block "AI crawlers" (but you can setup to allow "AI search" for better SEO). Eg: https://blog.cloudflare.com/control-content-use-for-ai-training/#putting-up-a-guardrail-with-cloudflares-managed-robots-txt
Cloudflare also has an option of "AI labyrinth" to serve maze of fake data to AI bots who don't respect robots.txt file.