Drew DeVault put it nicely https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html

@carl we literally had to put a proof-of-work reverse proxy in front of the GNOME gitlab instance to mitigate AI scrapers leaving about 10% of the available resources to the people actually using gitlab. It's a disaster.

@ebassi May I ask you for some details and numbers about this? I'd like to write an article about this problem!

@niccolove you want to talk to @barthalion who does the system administration of the GNOME infrastructure; he'll be able to give you more precise numbers

@ebassi @niccolove I don't think I can share any hard numbers off-hand. The tangible change is that the number of webservice pods for our GitLab instance went down from the maximum 6 to the minimum 3, while becoming more responsive to what users are doing. People can also run `git clone/pull/push` and expect it will finish relatively quickly instead of getting errors.

The logs I'm looking at right now are ephemeral and the earliest line starts today at 17:34 UTC and ends about now. Anubis explicitly denied 81366 clients (bots and abusive scrappers with recognizable user-agent patterns), while 2690 clients passed the proof of work challenge. That's only the last 2.5 hours!

For anything more I would need to dive into our log collection system, and I'm afraid I don't have much time for that.