Crawler

SerpexBot

SerpexBot is the web crawler that builds the Serpex search index — an independent, quality-selective index that answers questions for AI agents. This page explains how it identifies itself, how to verify it, and how to control what it crawls.

How SerpexBot identifies itself

Every request SerpexBot makes sends this exact User-Agent:

SerpexBot/1.0 (+https://serpex.ai/bot)

We never disguise our crawler as a browser or another company's bot. If a request claims to be SerpexBot, you can confirm it (below) — anything that fails verification is an impostor, not us.

Verifying SerpexBot

SerpexBot crawls from a fixed range with forward-confirmed reverse DNS under serpex.ai. To confirm a visitor is genuinely SerpexBot, reverse-lookup its IP and check it resolves back:

# 1. Reverse-lookup the crawler IP -> it resolves under serpex.ai
$ host 46.4.31.29
46.4.31.29.in-addr.arpa domain name pointer crawler.serpex.ai

# 2. Forward-confirm: crawler.serpex.ai resolves back to the same IP
$ host crawler.serpex.ai
crawler.serpex.ai has address 46.4.31.29

We additionally sign requests with Web Bot Auth (see below), giving you a cryptographic guarantee independent of IP.

Controlling SerpexBot with robots.txt

SerpexBot fully honors the Robots Exclusion Protocol. To allow it:

# Allow SerpexBot to crawl your whole site
User-agent: SerpexBot
Allow: /

To block or throttle it:

# Block SerpexBot entirely
User-agent: SerpexBot
Disallow: /

# Or slow it down / limit sections
User-agent: SerpexBot
Crawl-delay: 10
Disallow: /private/

We fetch and cache your robots.txt, honor Disallow and Crawl-delay, and respect noindex directives. Changes take effect within 24 hours.

How we crawl — our principles

Polite. Rate limited per domain; we back off on 429/503 and never overload a site.
Honest. One truthful User-Agent, no browser or identity spoofing, no anti-bot evasion.
Selective. We index quality content only — no spam, no adult material.
Respectful. We obey robots.txt, noindex, and remove content on request.

Web Bot Auth

SerpexBot supports Web Bot Auth — cryptographically signed HTTP requests (Ed25519, per RFC 9421 HTTP Message Signatures). Our public keys are published at:

https://serpex.ai/.well-known/http-message-signatures-directory

This lets Cloudflare and other providers verify SerpexBot without maintaining IP allowlists. We are registered as a search crawler and welcome verified-bot allowlisting.

Contact

Questions, crawl-rate requests, or content removal: bot@serpex.ai. We respond within two business days.