Rate limit HTTP clients with nginx
Nginx has many options, with one of them the option to rate limit requests. This is a very helpful option ensure HTTP clients behave themselves a little bit. If they don’t, they can be quickly discovered and actions taken. In this article we have a look at how to set this up.
Setting up a rate limited zone
The first step it so set up a zone where rate limiting is applied. With nginx we need to do this in a few steps.
- Match IP address to determine if a client is restricted or unrestricted
- Map the IP address to leverage its binary notation of the IP address
- Define the rate limited zone
- Optionally define the error code to use
- Apply zone in virtual host
Steps 1-4 can be defined within the http context, so that we can use the rate limiting functionality in multiple virtual hosts. This is normally done in your main nginx configuration file (nginx.conf).
The first steps are split to give a little bit more insights, with a full configuration piece at the end of the section.
Define geo
First we define if a client needs to be rate limited or not. We use the variable $unrestricted_ip for this. If it contains a zero (0), then it is a normal client that should be rate limited. If we define the value 1 to it, then no rate limiting is applied.
http {
# ... other configuration options ...
# Define if a client IP is unrestricted (by default it is not)
geo $unrestricted_ip {
default 0; # Default set to 0, normal clients are definitely not unrestricted
proxy 1.2.3.4/24; # Use this if traffic comes in via a proxy, otherwise remove
proxy abc:abc:abcd::/48; # Same, but for IPv6
::1 1; # localhost
1.2.3.4/24 1; # Our proxies should never be limited
66.249.64.0/19 1; # We allow Google bot to go nuts
127.0.0.1 1; # localhost
}
}
In this example we defined our default (0), we added proxies (optional), and defined some hosts and network ranges that should not be rate limited.
Mapping restricted clients for accounting purposes
Next step is to store the IP addresses of normal client in a binary format, so that nginx can process them more efficiently. This is done by using the value from $binary_remote_addr, which is a built-in variable. We tell nginx that if a client is restricted, that the binary representation of the client IP address should be used for accounting purposes. If the client is unrestricted, then empty the value. The result will be stored in variable $restricted_ip_key.
http {
geo $unrestricted_ip { ...; }
# Next step: map client IP to the binary notation of the IP address, otherwise empty it
map $unrestricted_ip $restricted_ip_key {
0 $binary_remote_addr; # key set, so rate-limiting applies
1 '';
}
}
Create a rate limited zone
Now it is time to create the zone itself. You can give it any name that you want, for this example we call it ratelimitedzone.
For this zone we reserve 10 megabytes of memory, enough to hold a lot of IP addresses. If you have a very busy server, tune this to your needs.
The rate limit is defined with 60 requests per minute, which translated to 1 every second. That is not much, especially considering that HTTP clients typically make multiple requests shortly to pull in the HTML, CSS, JavaScript, fonts, etc. So this needs some fine-tuning when we apply the zone, by allowing a ‘burst’ of requests. If your average web page has around 15 requests, this rate limit of 60 would mean the visitor could browse to 4 pages within 1 minute. Typically a visitor takes the time to read the page and won’t switch that quickly between multiple pages.
The last line defines what error code we want to return. Unfortunately, nginx returns by default a HTTP 503 error, indicating it is a server issue. There is a better response code and that is HTTP 429 or Too Many Requests, exactly what happens when a client performs too many requests in some amount of time. So we use that status code when rate limiting is active.
http {
geo $unrestricted_ip { ...; }
map $unrestricted_ip $restricted_ip_key { ...; }
# Define our zone with the binary notation of the IP address
limit_req_zone $restricted_ip_key zone=ratelimitedzone:10m rate=60r/m;
limit_req_status 429; # Return HTTP/429 = Too Many Requests (instead of default 503)
}
Full example of the rate limited zone
So if we combine these parts, you get something like this:
http {
# ... other configuration options ...
# Define if a client IP is unrestricted (by default it is not)
geo $unrestricted_ip {
default 0; # Default set to 0, normal clients are definitely not unrestricted
proxy 1.2.3.4/24; # Use this if traffic comes in via a proxy, otherwise remove
proxy abc:abc:abcd::/48; # Same, but for IPv6
::1 1; # Do not restrict localhost
1.2.3.4/24 1; # Our proxies should never be limited
66.249.64.0/19 1; # We allow Google bot to go nuts
127.0.0.1 1; # Do not restrict localhost
}
# Next step: map client IP to the binary notation of the IP address, otherwise empty it
map $unrestricted_ip $restricted_ip_key {
0 $binary_remote_addr; # key set, so rate-limiting applies
1 '';
}
# Define our zone with the binary notation of the IP address
limit_req_zone $restricted_ip_key zone=ratelimitedzone:10m rate=60r/m;
limit_req_status 429; # Return HTTP/429 = Too Many Requests (instead of default 503)
}
This configuration could be simplified by just using the last two lines (with $binary_remote_addr). The downside of that is that all HTTP clients will be rate limited. That may get you into troubles if you have a fairly strict rate defined and “good” clients get rate limited as well. For example, the Google bot typically behaves well, but now and then it wants to do a quick update of many pages and may hit the rate limits. Also you might have services like a link checker that you want to grant full speed. For this reason, it is useful to allow exceptions for those systems.
Configuring the virtual host
With this initial piece of configuration, the next step is applying this configuration within a virtual host. This way the zone will actually be used.
server {
listen ...;
server ...;
# Apply our rate limited zone which is defined within the HTTP context (in /etc/nginx/nginx.conf)
limit_req zone=ratelimitedzone burst=90 nodelay;
}
Within our virtual host configuration we have now defined that we want to apply rate limiting. We refer to our zone (ratelimitedzone) and can give it some additional options. In this case we allow a burst, meaning that clients can temporarily go beyond the related of 60 requests per minute (= 1 per second). This is useful to allow legitimate clients to pull in all required files for the first time (e.g. CSS, JavaScript) to serve a page. In this example we will not delay any requests (nodelay). See the related module
for details about fine-tuning this to your needs.
Not sure if your chosen rate limit is correct to properly catch bad-behaving clients while still allowing legitimate clients? Use the limit_req_dry_run option to do accounting and logging, but not enforcing the limits.
limit_req_dry_run on;
Restart and testing
After implementing the changes, confirm that your nginx configuration looks fine with the nginx command and the -t option:
nginx -t
All good? Restart the server, for Linux that is usually done with systemctl.
systemctl restart nginx.service
Next step is testing the rate limit. You could do this by setting the rate limit values very low and send some requests with curl. Another option is to fire up the ab tool (Apache Bench) and do a short stress test.
Blocking repeating offenders
If you have repeating offenders, then they will be showing up with a 429 status code in the log file. If a client receives this message and back offs, then it actually understood that it was going too quick. But some clients simply ignore it and will continue to perform their requests. This is especially the case for crawlers that want to index as much pages as quickly as possible.
One option is to consider to block the IP addresses that keep hitting the rate limit values. One option is to count the number of 429s per IP address and it goes over a threshold, to block them.
Need more ideas how to do this? Let it know!
