blackhull bot — operator information ==================================== User-Agent: Mozilla/5.0 (compatible; BlackhullBot-Watch/1.0; +https://blackhull.systems/bot) Operator: blackhull Updated: 2026-10-29 WHAT IT DOES Fetches specific pages to answer questions, and re-checks a small number of watched pages to notice when they change. It does not crawl: it requests known URLs and never harvests a site's pages in bulk. The URLs it fetches come from a search engine, from a page it was asked about, or from a fixed list of datasets their publishers offer for download. On links the line is RESOLUTION, not discovery — because the number of hops is not what makes something a crawl; asking a site what else exists is. From a result page its own query produced it may follow ONE link to the record for the entity it was already looking for, named in advance: where a site exposes a per-record URL it simply fetches that instead, and the link is an accident of the site's design rather than a new intention of ours. It does not follow "related entities", "next on list", "other filings by this agent" or anything else that asks what more there is, and it never identifies a link by position ("the first result") — the destination is named before it is seen. HOW IT BEHAVES - Reads and obeys robots.txt, including Crawl-delay. - At most one request per domain every 2 seconds. - Sends If-Modified-Since, so an unchanged page costs you a 304 and no body. - Unless a page is one we have listed for rendering, executes no JavaScript, loads no images/CSS/fonts, sends no cookies and no Referer. - A page is only rendered in a browser where we have listed it. Rendering is never the default and cannot be switched on for a site by configuration. - A rendered page obeys the same robots.txt, pacing and daily cap as any other fetch, and carries the same User-Agent. - A rendered page can only reach hosts we have listed and checked. Anything else is refused before it leaves. - Capped per site per day, and the cap is enforced in code rather than intended: 9 requests/day for BlackhullBot-Watch and -Monitor, 60/day for BlackhullBot-Research, counted per domain over a rolling 24 hours. Reaching it stops us on that site until it rolls off; we do not slow down instead. - Stays off endpoints that cost you real work or that nothing legitimate of ours is behind: search forms, faceted filters, calendars, deep pagination, and anything under account, admin, login, checkout or cart. Refused before the request is made, not after. - Two exceptions. The first: on a public statutory registry — a government body on a .gov domain, or a named service such as Florida's Sunbiz or FINRA's BrokerCheck — a search issued with GET is allowed. Answering public queries is what those services exist to do, and for a corporate filing the search form is frequently the only route to the record rather than one of several. Nothing else moves: robots.txt, the per-site daily cap, the pacing and the never-submit-a-form rule all still apply, so this cannot widen into a crawl. Calendars, faceted filters, deep pagination and account paths stop us on a registry exactly as they do anywhere else. - The second: a search a service documents as its public API, on that API's own path and nowhere else on the host — named services only, today the MCP registry's server list and GitHub's repository search. The queries are a fixed list written in advance, one page each; we never turn a result into the next query. robots.txt, the per-site daily cap and the pacing still apply, and oversized pages and deep pagination are still refused. - Downloads a file a publisher offers for bulk download only when it is on a fixed list written in advance — internet registry statistics, routing tables, exchange directories. At most one download of a file a day — plus, if your server cuts a long transfer off, a few requests that resume it from where it stopped rather than start again; the request is conditional, so an unchanged file costs you a 304; a redirect is never followed. The per-site daily cap and the pacing always apply, and so does robots.txt, except for the research archives named under ON DOCUMENTED APIs below. - Honours reservations in the response, not just robots.txt: an X-Robots-Tag header, a tag, or a TDM reservation. We cannot know about those until the page is already fetched, so we honour them by discarding it — not archived, not analysed, never quoted in anything we deliver. noindex alone is not treated as a reservation: it is an instruction to search engines, and we are not an index. - Never signs in, never accepts terms of service, and never works around a block. A refusal is an answer: after three 401/403 replies in a row we stop, and try at most once a week thereafter in case the block was a misconfiguration. - Submits a form only as a DECLARED INTERACTION a person approved beforehand, never on its own initiative. That person is first shown the page, the form, every control on it, which single control would be activated — named, not guessed — and the exact fields and values. A control whose name reads as a write (add, file, create, delete, pay...) is refused outright rather than offered for approval, and an unattended run is refused too: with nobody there to approve it, the answer is no. Where a record takes several steps they are declared and approved together, each step states which page it expects, and a step that lands somewhere else stops rather than adapts. No credentials are sent, and no cookie is kept beyond the request that set it. HOW TO BLOCK IT Add to your robots.txt: User-agent: BlackhullBot Disallow: / That blocks every Blackhull agent, now and in future, and is picked up within the hour. To stop only this one, name BlackhullBot-Watch instead. To slow it rather than block it, a Crawl-delay is honoured exactly as stated: User-agent: BlackhullBot Crawl-delay: 30 If you would rather not edit robots.txt, email the address below and say stop. That is honoured for your whole domain, across every Blackhull agent, permanently, and it takes effect before robots.txt is even consulted. You do not have to prove you own the domain first — we would rather stop and be wrong than make you prove something before we will listen. Lifting it later does require confirming you control the domain, which is the only part that is deliberately slow. THE NAME Agents are named BlackhullBot- so you can tell them apart and block them separately. Only BlackhullBot-Watch is in service today. If another one ever fetches from you it will carry its own name, have its own entry here, and honour a BlackhullBot block already in your robots.txt. One further name exists, BlackhullBot-FleetHealth/1, and it does not visit your site: it is the User-Agent our own fleet-health probe sends to the status endpoints of machines we operate, on our own tailnet. It is listed so that every identity we send is one this page describes. ON DOCUMENTED APIs robots.txt is a crawler-exclusion protocol. Some sites publish an API for programmatic use and also disallow it in robots.txt — PyPI, for example, disallows its JSON API alongside the /simple/ endpoint that pip itself depends on. Where that is the case, this bot may call the documented API deliberately, for one specific resource, at the same rate limits as anything else. The same reasoning covers research archives that publish data for download and disallow robots site-wide: RouteViews (only archive.routeviews.org/bgpdata/) and RIPE RIS (only data.ris.ripe.net/rrc00/), from each of which it fetches one named, dated routing-table file a week; and Common Crawl's published web graphs (only data.commoncrawl.org/projects/hyperlinkgraph/), three named files per release. It applies that narrowly and never: to evade a block or rate limit; on account, admin, checkout or search paths; or simply because data is valuable. If you would rather it did not do this on your site, email the address below and it will stop. HOW TO VERIFY IT IS US This bot's requests come from 64.20.22.128. Anything else carrying its User-Agent is not us. The address can change; this page always lists the current one. CONTACT bot@blackhull.systems (abuse@ and postmaster@ also reach us) Abuse reports, questions, and "please stop" all get a reply. If this bot has caused you load or trouble, say so and it will be fixed.