“429 Too Many Requests” on the Wayback Machine — and how to actually fix it.

What HTTP 429 actually means

HTTP status code 429 Too Many Requests is defined in RFC 6585. Short version: the server received more requests from your client than it’s willing to answer right now, and is asking you to slow down.

The response usually looks like this:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
Content-Type: text/html

<html>...rate limit exceeded...</html>

That Retry-After: 30 is the server telling you: wait at least 30 seconds before trying again. If you ignore it and keep hammering, you’ll stay rate-limited and may escalate to a longer timeout.

Why the Internet Archive rate-limits downloads

The Wayback Machine serves an enormous volume of traffic on infrastructure paid for by donations. If any one client could pull a full-site copy at line speed, the archive would spend most of its bandwidth on bulk downloads instead of actual visitors.

So the archive enforces per-IP and per-user-agent limits. The exact thresholds aren’t published and they change — which is exactly what you’d expect for a defensive rate limit. The practical upper bound for a polite single-IP client is roughly one request per second, sustained. Fire ten parallel requests and you’ll hit the limit instantly.

The Retry-After header is not a suggestion

If you’re writing your own script, two rules will get you 90 % of the way:

1. If the response is 429 and includes Retry-After: N, sleep for at least N seconds before retrying the same request.
2. If there’s no header (or you don’t want to parse it), back off exponentially starting at 2 s and cap at about 60 s: 2, 4, 8, 16, 32, 60, 60, 60. Give up after ~6 attempts on the same URL and log the failure rather than spinning forever.

Minimal pseudocode:

fetch(url):
    delay = 2 seconds
    repeat up to 6 times:
        response = GET url
        if response status is not 429:
            return response
        wait = Retry-After header, or delay
        sleep for wait
        delay = the smaller of (delay × 2) and 60 seconds
    note the URL as unfinished
    return nothing

Fix A: you’re driving the archive from a browser

If you’re just clicking around web.archive.org in Chrome or Firefox and keep seeing “Too Many Requests”:

• Close any Wayback Machine tabs that are auto-refreshing (the calendar view, year overviews).
• If you have a browser extension that pre-fetches linked pages (for example certain reader-mode or ad-blocker “speed” features), disable it on web.archive.org.
• Wait 60–90 seconds and try again. The archive forgets about you fairly quickly if you stop sending requests.
• If you’re on a shared IP (an office, a VPN, a university), a roommate or a colleague may be doing the pounding. Switch to a different network or wait it out.

Fix B: you’re driving it from a CLI tool or script

This is where most 429 complaints come from. Three concrete fixes:

1. Lower concurrency to one or two requests at a time

The single biggest unlock. If your downloader exposes a “parallel”, “threads”, or “concurrency” flag, drop it to 1 or 2. Don’t try to be clever — the archive doesn’t care that your laptop can handle 32 threads.

For wget, add --wait=1 --random-wait --limit-rate=200k:

wget --mirror \
     --wait=1 --random-wait \
     --limit-rate=200k \
     --retry-connrefused --waitretry=5 \
     --tries=4 \
     https://web.archive.org/web/20130425054636/http://example.com/

2. Honor Retry-After explicitly

Many older tools ignore Retry-After and retry immediately, which accomplishes nothing. If the tool is yours, add the two-rule logic from the previous section. If it’s a third-party CLI, check its issue tracker; often there’s a --retry-429 or similar flag.

3. Use a realistic User-Agent and don’t spoof

Set a clear, honest User-Agent string that identifies your tool and contact e-mail. The Internet Archive does not block polite, honest clients — they block floods. Impersonating “Googlebot” is a great way to get an IP-level block and ethically a terrible idea.

Fix C: hand it to a hosted downloader

If you’re downloading a real site and the DIY approach keeps choking, the simplest answer is to stop hammering the archive from your own IP and let a server-side tool do it. A well-built hosted downloader:

• Honors Retry-After instead of retrying immediately.
• Deduplicates requests — if two pages reference the same stylesheet, it’s fetched once.
• Queues jobs politely so it never floods the archive during traffic spikes.

That’s exactly what we built wayback.download to do. Paste your timestamped URL, add an e-mail, and the job runs on our side with the polite-rate logic above baked in. You get the ZIP without having to babysit the log.

Errors that look like 429 but aren’t

Before you spend an hour fighting rate limits, check that the error really is 429:

• 403 Forbidden — the archive has specifically refused this request (for example because the original site’s robots.txt asked to exclude it after the fact). No amount of backoff fixes this.
• 404 Not Found — the archive never captured this URL at that timestamp. Pick a different capture date or a different URL.
• 503 Service Unavailable — the archive is under temporary load from everyone, not just you. Wait a minute and retry; this usually clears itself quickly.
• Timeouts with no status code — often a network issue on your side, or the archive dropping the connection because the capture is enormous (think a big PDF). Retry once; if it still times out, split the job.

When nothing works

If you’ve lowered concurrency, honored Retry-After, switched to a hosted tool, and still can’t get a clean download, two realistic paths:

• Pick a different capture date. Sometimes a specific snapshot has a huge asset the archive serves slowly. A capture from a week before or after is often fine.
• Hand it to the restoration team. We do this work by hand at waybackrevive.com — HTML from $30, WordPress from $110. If the archive doesn’t have what you need, we tell you before you pay.

Try the free downloader →   Next chapter: Ruby gem alternatives