Potential DDoS detection #24

New issue

Open

opened 2026-06-20 17:29:00 -04:00 by chipmunkmc · 1 comment

chipmunkmc commented

2026-06-20 17:29:00 -04:00

Owner

The de-facto workaround for this turned out to be to reverse-proxy the targeted service through a piece of software called Anubis, which filters out some requests. However, simply removing the sub-string Mozilla from one's User-Agent string bypasses this filter under default behavior, so Anubis is not real DDoS protection, only protection against such crawlers.
It also relies on JavaScript and Cookies as its near-only means of filtering requests, and also requires support for HTTPS and modern JS APIs, which greatly harms compatibility. Again this doesn't truly prevent attacks, but merely shakes off a few specific clients. All in all, it's quite an ugly hack akin to a using a flamethrower that can only serve the purpose of a fly-swatter.

There's also Cloudflare (aka cuckflare) which is a service that acts as a reverse-proxy and intercepts all requests coming to the site behind it; since it is not ran on the site administrator's machine, it is effectively a MITM attack in my book.
It most infamously filters requests in a similar style as Anubis (granted it came first), except it's more complex and worse. However, it can protect endpoints of sites usable to non-navigator clients (like git repository contents and API endpoints). Getting hit with challenges on even some hypertext endpoints rarely to never happens. So it seems, they don't strictly need challenges to do their filtering, and this gives me inspiration...
Something I noticed after I began writing this: I read that someone had to configure CF a bunch to mitigate an attack? Perhaps it's not as powerful as I thought, though I dunno.

I'd like to note sometimes some of the "oldest tricks in the book" are still used today due to websites still being vulnerable.

A while ago, https://files.chipmunk.land/ got flooded with a bunch of POST requests.
These came from many IPs (in the 80.0.0.0/8 and 81.0.0.0/8 subnets), overloading nginx and even lagging my home network until I filtered it out those IPs from the VPS side
I found this page once, which describes a Slowloris attack.
This type of attack has been around since 2009, about 10 years prior to the article being written, and even has a Wikipedia page.
- Of course, we should block this type of attack for instance.

I'd also like to note that while blocking these crawlers, I noticed their requests are often fingerprintable (see #17). While fingerprints can vary and sometimes be shared by legitimate clients, I still find this quite notable.
This approach alone seems to not be enough, which made me think about automatically detecting relevant fingerprints and networks. And well, perhaps we could catch other attacks this way?

So perhaps we could make our own DDoS detector for HTTP?
Considerations

Detection
- The rate at which requests are made can be useful, but could lead to false positives.
  - (Relates to identification) but what if it's a bunch of different fingerprints making slower requests?
  - How much is too much can be subjective as described later.
- Requests to the underlying service taking a long time to be served could be a sign?
  - If the whole sample is going to slow endpoints / the server is slow for other reasons / etc this could cause false positives.
  - Detection would take a while though, and this could be abused if someone were to start/stop attacking a bunch maybe? We could cache the detection perhaps.
- Of course, what causes denial of service partly depends on the service (do we just serve a file or do we do a bunch of processing?), the data being accessed through it (i.e. git objects for a repo), the underlying hardware (i.e. whether the CPU is good/bad/shit/etc or how many reads/writes the disk can take at once), etc so this is sorta hard. Request rates differ by popularity and service (federated?) too.
Identification
- As hinted before, we could look for similar fingerprints and networks across requests
- This could certainly beat some primitive attacks.
- Whether or not this would scale well for an attack spanning a bunch of fingerprints and/or networks a bunch is unknown though. (sort of relates to a previous remark but goes beyond those crawlers; think like huge DDoSes)
- How do we fingerprint?
  - Headers!
    - UA strings can sometimes help
    - Accept-Encoding shows what the client can decode
    - What if we see, say, a Chrome 132 Accept with an IE 5.0 User-Agent?
  - SSL/TLS stuff perhaps (for HTTPS clients only)
  - Minor details like HTTP version, protocol used, spacing between commas in headers, etc.
  - Networks - subnets or ASNs
    - We should make sure to not be overzealous though (i.e. if a single customer's IPv6 subnet is being used, just block that)
  - We'd have to avoid false positives though, somehow...

Perhaps a generic DDoS filter is something too ambitious or just unattainable. Still, I think it's something worth thinking about. I think we can, at the very least, kill a bunch of primitive attacks though if we try, at least maybe.

As has been seen all over the web (and documented in #17), recently a bunch of crawlers (suspected by many to be AI-related) have been making frequent HTTP requests that overload various websites, including many git forges (like the one you're on now). However, this is effectively an accidental DDoS attack, and in theory said sites were *always* vulnerable to being taken down in such a way; this simply brought said vulnerabilities to light, at least on sites less targeted by deliberate attacks. The de-facto workaround for this turned out to be to reverse-proxy the targeted service through a piece of software called Anubis, which filters out some requests. However, simply removing the sub-string `Mozilla` from one's User-Agent string bypasses this filter under default behavior, so Anubis is not real DDoS protection, only protection against such crawlers. It also relies on JavaScript and Cookies as its near-only means of filtering requests, and also requires support for HTTPS and modern JS APIs, which greatly harms compatibility. Again this doesn't truly prevent attacks, but merely shakes off a few specific clients. All in all, it's quite an ugly hack akin to a using a flamethrower that can only serve the purpose of a fly-swatter. There's also Cloudflare (aka cuckflare) which is a *service* that acts as a reverse-proxy and intercepts all requests coming to the site behind it; since it is not ran on the site administrator's machine, it is effectively a MITM attack in my book. It most infamously filters requests in a similar style as Anubis (granted it came first), except it's more complex and worse. However, it can protect endpoints of sites usable to non-navigator clients (like git repository contents and API endpoints). Getting hit with challenges on even some hypertext endpoints rarely to never happens. So it seems, they don't strictly *need* challenges to do their filtering, and this gives me inspiration... Something I noticed after I began writing this: I read that someone had to *configure* CF a bunch to mitigate an attack? Perhaps it's not as powerful as I thought, though I dunno. I'd like to note sometimes some of the "oldest tricks in the book" are still used today due to websites still being vulnerable. * A while ago, https://files.chipmunk.land/ got flooded with a bunch of POST requests. These came from many IPs (in the 80.0.0.0/8 and 81.0.0.0/8 subnets), overloading nginx and even lagging my home network until I filtered it out those IPs from the VPS side * I found [this](https://coffee-and-dreams.uk/security/2019/10/20/mitigating-a-ddos.html) page once, which describes a Slowloris attack. This type of attack has been around since 2009, about 10 years prior to the article being written, and even has a Wikipedia page. * Of course, we should block this type of attack for instance. I'd also like to note that while blocking these crawlers, I noticed their requests are often fingerprintable (see #17). While fingerprints can vary and sometimes be shared by legitimate clients, I still find this quite notable. This approach alone seems to not be enough, which made me think about automatically detecting relevant fingerprints and networks. And well, perhaps we could catch other attacks this way? So perhaps we could make our own DDoS detector for HTTP? Considerations * Detection * The rate at which requests are made can be useful, but could lead to false positives. * (Relates to identification) but what if it's a bunch of different fingerprints making slower requests? * How much is too much can be subjective as described later. * Requests to the underlying service taking a long time to be served could be a sign? * If the whole sample is going to slow endpoints / the server is slow for other reasons / etc this could cause false positives. * Detection would take a while though, and this could be abused if someone were to start/stop attacking a bunch maybe? We could cache the detection perhaps. * Of course, what causes denial of service partly depends on the service (do we just serve a file or do we do a bunch of processing?), the data being accessed through it (i.e. git objects for a repo), the underlying hardware (i.e. whether the CPU is good/bad/shit/etc or how many reads/writes the disk can take at once), etc so this is sorta hard. Request rates differ by popularity and service (federated?) too. * Identification * As hinted before, we could look for similar fingerprints and networks across requests * This could certainly beat some primitive attacks. * Whether or not this would scale well for an attack spanning a bunch of fingerprints and/or networks a bunch is unknown though. (sort of relates to a previous remark but goes beyond those crawlers; think like huge DDoSes) * How do we fingerprint? * Headers! * UA strings can _sometimes_ help * Accept-Encoding shows what the client can decode * What if we see, say, a Chrome 132 `Accept` with an IE 5.0 `User-Agent`? * SSL/TLS stuff perhaps (for HTTPS clients only) * Minor details like HTTP version, protocol used, spacing between commas in headers, etc. * Networks - subnets or ASNs * We should make sure to not be overzealous though (i.e. if a single customer's IPv6 subnet is being used, just block that) * We'd have to avoid false positives though, somehow... Perhaps a generic DDoS filter is something too ambitious or just unattainable. Still, I think it's something worth thinking about. I think we can, at the *very* least, kill a bunch of primitive attacks though if we try, at least maybe.

chipmunkmc referenced this issue

2026-06-20 17:36:16 -04:00

Filter out crawlers that lag Forgejo #17

chipmunkmc commented

2026-06-21 01:17:39 -04:00

Author

Owner

Perhaps this filter could be written in C as a library, so it can neatly be made into modules into various web servers such as nginx. This means it would neatly fit into existing sites instead of being another reverse proxy that has to complicate things.
Other languages can probably also do the trick, but C is so standard (not that we strictly need this but literally everything has a compiler for it) and minimal (has been called portable assembly) that it's appealing, not to mention many web servers (nginx, lighttpd, apache) are written in it.
Scripting environments (njs, lua, etc) might help with testing filtering techniques early on though.

I'd also like to note that it should be plug-and-play, like Anubis. No one likes manually filtering stuff, which is probably why said program got so heavily adopted across different sites.

Also for a name, I thought of hyperwall. I got this idea when I read the definition for "Web App Firewall" which I guess this would fit but that term places emphasis on apps which...

Was originally a cellphone term
It makes me think of walled gardens and proprietary garbage associated with those devices
- Don't get me wrong, I love my phones, however I treat them as either hobby devices or workstations, while most would use them as thin clients these days
  (srsly, even technical people do, at least now in the SafetyNet era)
Some people call every service an app these days... Likely in reference to those social media apps on phones
I liked calling things programs and sites more
I don't want to be like those proprietary social media clients! I don't want to be like their web counterparts!!!
If app = program then I guess they're not wrong for some types of sites but the terminology just feels off

So I felt like looking at a lower level instead.

At the heart of the web is HTTP, HyperText Transfer Protocol
HTTP mainly used to transmit HTML, HyperText Markup Language, documents
Hypertext is essentially text with hyperlinks contained
It's what pretty much every true website is at its core.
Back to Forgejo as an example of a service, it utilizes links heavily, in some places more obviously than others
To me, it places an emphasis on the simpler, less bloated place the web was meant to be...

Oh yeah, "hyperwall" just sounds cool in my mind. That could just be me though.

Perhaps if an attack rotates some aspects of its request fingerprints, we can detect the patterns of how it does this? Like, how frequently requests with very different headers come in? Of course this is easier said than done and would make identification much harder. Just a thought I had. Also there's probably something we haven't thought of. Even if we catch all potential attacks, we should do so in a modular way so we can add even more filters later. Perhaps this filter could be written in C as a library, so it can neatly be made into modules into various web servers such as nginx. This means it would neatly fit into existing sites instead of being another reverse proxy that has to complicate things. Other languages can probably also do the trick, but C is so standard (not that we strictly need this but literally everything has a compiler for it) and minimal (has been called portable assembly) that it's appealing, not to mention many web servers (nginx, lighttpd, apache) are written in it. Scripting environments (njs, lua, etc) might help with testing filtering techniques early on though. I'd also like to note that it should be plug-and-play, like Anubis. No one likes manually filtering stuff, which is probably why said program got so heavily adopted across different sites. Also for a name, I thought of _hyperwall_. I got this idea when I read the definition for "Web App Firewall" which I guess this would fit but that term places emphasis on apps which... * Was originally a cellphone term * It makes me think of walled gardens and proprietary garbage associated with those devices * Don't get me wrong, I love my phones, however I treat them as either hobby devices or workstations, while most would use them as thin clients these days (srsly, even technical people do, at least now in the SafetyNet era) * Some people call every service an app these days... Likely in reference to those social media apps on phones * I liked calling things programs and sites more * I don't want to be like those proprietary social media clients! I don't want to be like their web counterparts!!! * If app = program then I guess they're not wrong for some types of sites but the terminology just feels off So I felt like looking at a lower level instead. * At the heart of the web is HTTP, HyperText Transfer Protocol * HTTP mainly used to transmit HTML, HyperText Markup Language, documents * Hypertext is essentially text with hyperlinks contained * It's what pretty much every true website is at its core. Back to Forgejo as an example of a service, it utilizes links *heavily*, in some places more obviously than others * To me, it places an emphasis on the simpler, less bloated place the web was meant to be... Oh yeah, "hyperwall" just sounds cool in my mind. That could just be me though.