Protecting Your Site from Abusive Bot Traffic

Prev Next

Overview

Automated traffic (“bots”) has increased across municipal websites and, in some cases, has slowed or interrupted sites. CivicPlus has strengthened the protections on our side. If your site is served through your own CDN, DNS or security service, part of that protection happens in your service, and only your team can set it up.

What to Do

  1. Share this article with your IT or web team.

  2. Before turning on any rule that challenges or blocks automated traffic, make sure it still lets through CivicPlus’s own system-to-system requests and the CivicPlus tools you use. Try a new rule on one page first.

  3. Contact Support if anything stops working.

A General Approach

  1. Target the rule at the whole site or only at the path being abused.

  2. Add a condition based on how likely the request is to be automated, and use a challenge before a block.

  3. Choose what to limit on: per visitor address, per network, or per page. Each has cautions (shared addresses can catch real visitors; never apply a network rule broadly to a local internet provider).

  4. Set the limit from normal traffic for that page, not a guess, and give it headroom. A site-wide rule needs a much higher limit than a single-page rule.

  5. Check your numbers: analytics that already include bot traffic inflate the baseline, and one visit can make several requests.

What CivicPlus Does

We keep monitoring and maintaining our protections. We don’t configure your service for you. If we pause a feature on your site to keep it stable, we will tell you why and what we need from you to turn it back on.


5 Tips for Preventing Site Abuse and Overload

High-level guidance for protecting an entire site, or a specific module, from abusive traffic.

1. Create a Targeted Rule

Create a rule that targets either the whole site or only the path being abused.

  • Whole site: Path = /*

  • Specific module: Path starts with /blog.aspx

2. Add a Bot Score Condition

Add a condition such as “Bot Score < 30.” This assumes a scale where 1 means near-certain bot and 99 means near-certain human. The threshold is a starting point; tune it to your platform and to how well the rule performs. Use a managed challenge when a request scores below the threshold.

3. Choose What to Rate Limit On

Method

Best when

Cautions

Per IP

A large volume of requests comes from a small number of IPs.

Shared IPs (NAT or Network Address Translation, offices, mobile carriers) can catch legitimate users.

Per ASN (Autonomous System Number)

A large volume comes from many IPs that share one AS (Autonomous System) number.

Use only against a known set of ASNs. Never apply broadly to your primary local ISP (for example, Comcast) without excluding it.

Per path

Specific paths are being abused.

Usually the easiest option to make strict without hurting real users.

The broader the criteria, the more room the limit needs. A site-wide rule needs a much higher threshold than a single-path rule.

4. Set the Threshold

Base the limit on normal traffic for the target module. This example uses blog.aspx.

Per-user (per-IP) limit. A human is unlikely to request one path more than 2 to 3 times a minute, and may reach about 10 in bursts. A per-IP limit of 10 requests per minute would catch most abuse and rarely affect real users.

Per-path (aggregate) limit, from daily analytics:

  • Take the daily visit count of the page, for example, 1200.

  • Assume the traffic falls within a busy window, for example, 12 hours.

  • 1200 ÷ 12 = 100 requests per hour.

  • 100 ÷ 60 ≈ 1.67 requests per minute on average.

  • Apply a multiplier for headroom. At 3 to 5 times the average, the limit is roughly 4 to 10 requests per minute for the path as a whole.

If you have minute-level analytics, use the peak minute observed during known-good traffic as the baseline instead, and add headroom on top of it.

With a challenge instead of a block, you can set a stricter limit: it slows bots more, and at worst it only delays a real visitor rather than blocking them.

5. Caveats

  • Confirm the analytics reflect real users. If they already include bot traffic, the baseline will be inflated.

  • Visits and requests are not the same. One visit can generate several requests: AJAX (Asynchronous JavaScript and XML) calls, pagination, refreshes.