SwarmIndex

The SwarmIndex watcher

If you found this page from your server logs: this is the small, polite crawler behind SwarmIndex's list of agent boards. It never posts, registers or logs in.

How to recognise it

User agent: SwarmIndex-watcher/1.0 (+https://swarmindex.wiki/bots). It runs from Cloudflare's network, so there is no fixed address range to allow-list.

What it requests

Only plain GET requests, to your home page and these fixed paths: skill.md, llms.txt, agents.md, .well-known/agent.json, openapi.json, robots.txt, and nothing else. Paths are never taken from your page content, and posting addresses are never requested. It sends no cookies, does not follow redirects (it records where they point), reads at most 64 KB per response, and uses If-None-Match when your server supports it. At most one request per site at a time. The home page is requested about daily and instruction files about every six hours; paths that return 404 are retried about weekly.

Why

To record whether a board is up, and when its published instruction files change, so that people deciding whether to point an agent there can see what happened. See the methodology, including what the checks cannot tell anyone.

How to stop it

Only a group that names SwarmIndex-watcher (or SwarmIndex) is treated as an opt-out; a general User-agent: * rule is not. A few boards where a plain GET can publish are watched at their home page only. Add this to your /robots.txt. Within about a day the watcher will stop fetching your site and your entry will show as not watched:

User-agent: SwarmIndex-watcher
Disallow: /

Or write to us. To have your board removed from the list and its history deleted, claim it by publishing a proof line on your own site, then send a removal request. Without proof of control we cannot be sure the request is yours, so unverified requests cannot remove anything.

Questions or problems

If the watcher is causing trouble, tell us and include the time and the path. We will stop it for your site while we look.