TaxKernelBot

What it fetches

Every document TaxKernelBot fetches is named in a public manifest before it is fetched, one per jurisdiction. The manifests are checked into a public repository and versioned. They are short: a few dozen URLs per jurisdiction, all of them rates pages, tax tables, withholding schedules or consolidated legislation. Nothing is discovered by crawling; a URL enters the list by a human adding it.

• The manifests - what we intend to read.
• Every source and its current state - including every failure.
• What we encoded from it - including the gaps.

How it behaves

These are the operating rules, and they are enforced by code rather than by intention. The gate that implements them is a Durable Object, one per host, so that one request at a time holds across the whole fleet rather than within a single process.

robots.txt is obeyed

Fetched once a day per host. We honour Disallow and Crawl-delay and match the most specific group that names us. A path you disallow is not fetched, and we record it as a permanent gap in our own public coverage report rather than routing around it. If robots.txt cannot be read at all we treat that as a full disallow, per RFC 9309, and stop.

One request at a time, per host

Never two concurrently, whatever else our infrastructure is doing.

Three to five seconds between requests

Measured from the moment a request is dispatched rather than from when it returns, so a slow response is never an excuse to fire the next one immediately. Where a host publishes a longer Crawl-delay, the longer figure applies.

A hard daily cap per host

Counted on dispatch, so failures spend budget too. Once it is reached we stop until the next UTC day. For most authorities the cap is between 100 and 300 requests a day, and normal operation uses a small fraction of it.

Conditional requests

We store ETag and Last-Modified and send If-None-Match on every re-check. An unchanged page costs a 304 and no body at all, which is what makes a daily freshness check on a rates page cheap enough to be polite.

Exponential backoff with jitter

A 429 or a 5xx doubles the wait, up to thirty minutes, and we honour a Retry-After header when it asks for longer than our own backoff would. A 404 does not trigger backoff: it is information about a document, not a sign of strain.

No rendering, no authentication, no forms

Raw HTTP is preferred everywhere it works, which is nearly everywhere. TaxKernelBot fetches public documents anonymously by GET and nothing else.

What it does with what it reads

Raw bytes go to content-addressed storage, keyed by their SHA-256, and are never modified. A citation in a released rule names that hash, so a figure can be traced back to the exact bytes an authority served on a given day.

Verifying it is us

TaxKernelBot requests carry the User-Agent above and a From header with our contact address. We do not spoof a browser, and we do not use residential proxies. If a request claims to be TaxKernelBot and is not behaving as described here, it is not us, and we would like to know.

Asking us to stop

Any of these work, and none of them needs a conversation first.

• Add a Disallow for the paths in your robots.txt, for TaxKernelBot or for *. It takes effect within a day.
• Set a Crawl-delay. We will use it.
• Email contact@taxkernel.com and we will remove the source, block the host, or slow down to whatever rate you name.