HTTP User Agents

The User-Agent request header is a free-text string a client sends to identify itself — nominally the browser, operating system, and rendering engine making the request. It is one of the oldest and most widely inspected signals in web security tooling, and also one of the weakest: the client fully controls its content, and nothing about HTTP requires it to be truthful. A WAF that trusts the User-Agent at face value is trusting the attacker to accurately label their own attack traffic.

This article defines the recommended WAF action — Allow, Delay, Challenge, or Block — for the categories of User-Agent a WAF will see in practice, and the reasoning behind each. It assumes a baseline WAF is deployed; see Web Application Firewall for foundational concepts and Advanced Web Application Firewall for the fingerprinting techniques (JA3/TLS, HTTP/2, behavioural) referenced throughout as cross-checks.

What a User-Agent Is

A typical browser User-Agent looks like:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36

The format is a historical accretion rather than a clean specification — every major browser claims to be Mozilla/5.0 and lists the engines of its ancestors and competitors for legacy compatibility. Non-browser clients follow their own convention, typically product/version:

python-requests/2.31.0
curl/8.4.0
PostmanRuntime/7.36.0
Googlebot/2.1 (+http://www.google.com/bot.html)
com.example.myapp/2.1.0 (iOS 17.4; iPhone15,3)

Why It Cannot Be Trusted Alone

• It is entirely client-supplied. Any HTTP client — curl, a Python script, a browser devtools override, a mobile proxy tool — can set this header to any value.
• Spoofing tooling is trivial and ubiquitous. Setting a Chrome on Windows User-Agent from a curl one-liner takes a single flag: curl -A "Mozilla/5.0 ...".
• Legitimate clients change it too. Browser vendors rotate version numbers constantly, and privacy-focused browsers/extensions deliberately randomise or genericise the string.
• It does not prove the claimed software actually produced the request. A UA string claiming Chrome tells you nothing about whether the TLS handshake, HTTP/2 frame ordering, or JavaScript execution environment actually matches real Chrome — see the fingerprinting layers in Advanced Web Application Firewall.

The User-Agent is useful as a routing and prioritisation signal, and as one input into a broader risk score — never as a standalone authorisation decision, and never as a substitute for verifying identity (reverse DNS/ASN for bots, TLS/HTTP fingerprint for browsers, authentication for users).

The Four WAF Actions

Challenge and Delay exist because Block is a blunt instrument: false positives on Block affect real users and real revenue, while Delay and Challenge let low-confidence signals accumulate more evidence (does it solve the JS challenge? does it back off when throttled?) before an irreversible decision is made.

Callout — Challenge assumes a human, or at least a browser, is present. M2M (machine-to-machine) traffic — API integrations and webhook deliveries in particular — is automated non-human traffic by design, but that does not make Challenge the right control for it. A webhook sender (Stripe, GitHub, Twilio, etc.) or a server-side API client has no browser engine to execute a JS challenge and no human to solve a CAPTCHA. Routing that traffic through Challenge does not slow an attacker down — it simply fails the legitimate integration outright, and most webhook senders will retry a fixed number of times with backoff before marking the delivery permanently failed and disabling it, which looks identical to an outage from the sender's side.

For known M2M endpoints (webhook receivers, first- and third-party API integrations), replace Challenge with:
- Strong authentication instead of interactive proof: HMAC payload signature verification (e.g. Stripe-Signature, GitHub's X-Hub-Signature-256), mTLS, or OAuth client-credentials/API keys — see API Security. A verified signature is a stronger identity signal than anything a JS challenge produces.
- Allow once authentication passes, Block once it fails — Challenge is not a valid middle state for these endpoints.
- Delay (rate limiting) is still appropriate and does not break M2M clients the way Challenge does, since a well-behaved client backs off and retries rather than failing to respond at all.
- Explicitly exempt known webhook paths and confirmed partner source IPs/ASNs from any cloud WAF or bot-management product's default Challenge behaviour — the default is tuned for browser traffic and will otherwise silently break integrations.

Recommended Action by User-Agent Category

Implementation Notes

Corroborate, Don't Trust in Isolation

Every row above pairs the User-Agent with a second signal — reverse DNS, IP range, TLS fingerprint, authentication state, or app version registry. Treat the UA as the first filter that determines which corroboration check runs, not the final verdict.

# ModSecurity: challenge requests claiming to be Googlebot that don't
# resolve via reverse DNS to a googlebot.com / google.com host
SecRule REQUEST_HEADERS:User-Agent "@contains Googlebot" \
    "id:2001,phase:1,pass,nolog,\
     setvar:tx.claimed_bot=googlebot"

# Follow-up (application/edge layer): perform reverse DNS + forward-confirm
# on tx.claimed_bot before granting Allow; otherwise route to Challenge/Block.

# Example rate-based rule: delay (throttle) generic scripting clients
# on public-facing routes rather than blocking outright
- Name: ThrottleGenericClients
  Statement:
    AndStatement:
      Statements:
        - ByteMatchStatement:
            FieldToMatch: { SingleHeader: { Name: "user-agent" } }
            SearchString: "python-requests"
            PositionalConstraint: CONTAINS
        - NotStatement:
            Statement:
              ByteMatchStatement:
                FieldToMatch: { UriPath: {} }
                SearchString: "/api/"
                PositionalConstraint: STARTS_WITH
  Action:
    Challenge: {}

Log the Raw String, Always

Regardless of the action taken, log the full raw User-Agent alongside the request. It is frequently the first artefact reviewed during incident response to cluster related requests, identify a specific attack tool version, or confirm whether a false positive affected a legitimate but unusual client.

Re-tune After Every Bot-Landscape Shift

User-Agent-based rules decay faster than most WAF rules — new AI crawlers, new headless framework defaults, and new legitimate SDKs appear continuously. Review and retune UA-based rules on the same cadence as CRS rule updates (see Web Application Firewall), not as a one-time configuration.

Never Let UA Filtering Replace Authentication or Input Validation

A User-Agent rule is a triage and cost-raising control, not an access control. An attacker who solves a challenge or waits out a delay still needs to fail on authentication, authorisation, and input validation to be stopped — UA handling reduces noise and raises the cost of automation, it does not substitute for those controls.

Hardening Checklist

References

• Web Application Firewall
• Advanced Web Application Firewall
• API Security
• OWASP Automated Threat Handbook — Web Applications
• MDN — User-Agent header
• Google — Verifying Googlebot
• OWASP Core Rule Set