SerpacBot

SerpacBot is the HTTP client our servers use to read public web pages for serp.ac customers and for the free SEO tools on serp.ac. It is not a search-engine crawler and doesn't try to discover the web. Apart from site audits (which crawl one site) and the Broken Link Checker (which checks the links on one page), it requests specific URLs for the features listed below and doesn't follow links to find new pages.

• Tracked backlinks. serp.ac customers monitor backlinks they have bought or placed on other websites. For each tracked link, SerpacBot loads the page that carries it about once a day to check that the link is still there, with the agreed anchor text and rel attribute. Each check requests your /robots.txt and that one page. After a failed check it tries again about two hours later, and customers can also start a re-check by hand. Links bought through the serp.ac link marketplace are verified the same way.
• Site audits. A customer can run a technical SEO audit of the website in their serp.ac project, normally their own site. SerpacBot then crawls that one host: it starts from the homepage and the XML sitemaps, follows links on the same host, and skips images, PDFs, scripts, style sheets and other files. An audit covers at most 100, 500 or 2,000 pages, depending on the customer's plan, and stops after 25 minutes. Sites that have been audited are audited again once a week. This is the only part of SerpacBot that crawls a site, and it follows robots.txt (see below).
• Pages ranking in Google. When a customer compares their own page with a page that ranks in Google for one of their keywords, SerpacBot reads both pages once. The result is cached for 24 hours, so comparing the same page again within a day doesn't fetch it again.
• Pages cited by AI assistants. When a customer asks why an AI assistant does or doesn't cite their site for one of their tracked prompts, SerpacBot reads up to five of the pages the assistant cited, plus the customer's own page.
• Competitor homepages. To tell real competitors apart from directories, marketplaces and media sites, SerpacBot reads the homepage, and only the homepage, of domains that rank for the same keywords as a customer. It reads at most 15 homepages per run. A homepage is read at most once every 30 days for all customers together; a failed read is retried after 3 days at the earliest.
• AI-readiness check. For a customer's own domain, SerpacBot reads the homepage, /robots.txt, /llms.txt, /llms-full.txt and the XML sitemap to check how readable the site is for AI crawlers.
• Free SEO tools. Visitors can check a URL they enter with the free SEO tools: Website SEO Checker, Schema Markup Checker, Meta Tag & SERP Preview, Backlink Live Checker (up to 10 pages per check), Redirect & HTTP Status Checker (up to 10 URLs, up to 10 redirects each), Broken Link Checker (the page plus up to 100 links on it, 8 at a time, with HEAD requests and GET when HEAD fails), Robots.txt AI Bot Tester and llms.txt Checker & Generator. Depending on the tool, SerpacBot also reads /robots.txt, the XML sitemap, /llms.txt or /llms-full.txt of the same site. The tools are rate-limited per visitor.
• Link marketplace. When a publisher lists a website on the serp.ac link marketplace, SerpacBot reads its homepage, or the file /.well-known/serp-ac-verification.txt if the publisher chose that method, to confirm ownership.

• It sends GET requests, and HEAD requests in the Redirect Checker and the Broken Link Checker. It never sends POST requests, never submits forms and never logs in.
• It doesn't run JavaScript. It reads the HTML your server returns and doesn't load images, style sheets, scripts or fonts. It sends no cookies.
• It doesn't try to get around access controls. If a page answers with a bot challenge or a 401, 403, 429 or 503 status, SerpacBot reports to the customer that the page couldn't be checked. It doesn't use headless browsers, rotating proxies or CAPTCHA solvers. The customer may then open the page in their own browser.
• Every request times out after 15 seconds or less, and SerpacBot follows at most 5 redirects (the Redirect Checker traces up to 10, because tracing redirects is its job).
• During a site audit it makes at most 4 requests to your site at the same time, and each of them waits 250 ms before the next request. If your robots.txt sets a Crawl-delay, it makes one request at a time and waits that many seconds between requests (see below). serp.ac runs one site audit at a time.

A group that names SerpacBot is honored on every request. Before requesting a page, SerpacBot checks your /robots.txt (the file is cached for up to an hour). Disallowed URLs aren't requested, on any redirect hop either; the longest matching rule wins, and Allow wins a tie. To keep SerpacBot off your site entirely:

User-agent: SerpacBot
Disallow: /

Site audits also follow the general rules. Without a SerpacBot group, an audit applies the User-agent: * group. A Crawl-delay switches the audit to one request at a time with that many seconds between requests; values above 5 seconds are treated as 5. Sitemap lines are used as starting points.

Single-URL requests don't apply User-agent: *. Backlink checks, page comparisons, reads of AI-cited pages and competitor homepages, the AI-readiness check, marketplace verification and the free tools request a specific URL that a customer or visitor chose, or that the feature needs, much like a person opening that page. A general User-agent: * rule doesn't stop them; a SerpacBot group does. The customer then sees that your robots.txt doesn't allow the check.

You can also email us at [email protected] with your domain.

serp.ac is building its own index of links between websites, so we can show which pages link to a site. It's built only from pages SerpacBot already fetched for one of the features above (backlink checks, page comparisons, site audits, competitor homepages and some free tools). Nothing is fetched just for the index, and the links it records aren't crawled. From each successful (2xx) response it stores the page URL, the HTTP status, whether the page is marked noindex, and up to 300 links to other websites (target URL, anchor text and rel attribute). It doesn't store the page's text. A page is recorded at most once every 20 hours. Error pages and bot-challenge pages are skipped, and pages read through the serp.ac browser extension never enter the index.

• Uptime monitoring is separate from SerpacBot. Customers' uptime monitors request the URL they set up, at the interval they chose, with the user agent Mozilla/5.0 (compatible; serp.ac-uptime/1.0; +https://serp.ac/docs/uptime). These checks may run from Cloudflare's network or directly from our server. See the uptime monitoring guide.
• The serp.ac browser extension runs in the customer's own browser. Its requests carry that browser's user agent and IP address, not SerpacBot's.

SerpacBot requests carry this user agent:

Mozilla/5.0 (compatible; SerpacBot/1.0; +https://serp.ac/bot)

They are always sent over IPv4 from our server at 176.9.51.216. That address has the reverse DNS name mail.serp.ac, which resolves back to 176.9.51.216. You can check both directions with:

host 176.9.51.216
host mail.serp.ac

Requests are also signed with Web Bot Auth (HTTP Message Signatures, Ed25519). They carry Signature, Signature-Input and Signature-Agent: "https://serp.ac" headers; the public key is published at https://serp.ac/.well-known/http-message-signatures-directory. A valid signature is the strongest proof that a request is ours.

If you see our user agent coming from another address, please tell us at [email protected].

For questions, problems or requests about SerpacBot, email [email protected]. Please include your domain and, if you can, the time, URL and IP address of the requests from your logs.