GISTBot — how gist crawls your site

gist audits and monitors websites by crawling public pages, the same way a search engine does. This page explains what GISTBot is, what it collects, and how to control it.

Who is GISTBot?

GISTBot is the crawler used by the gist product to analyse a website you have asked us to audit. It identifies itself in the User-Agent header as GISTBot/1.0 with this page as its contact address.

What it fetches

GISTBot fetches public web pages from the domain you asked us to analyse: HTML, linked assets relevant to technical SEO, robots.txt, and sitemaps. It does not bypass login walls or attempt to access non-public content, and it never submits forms.

Rate and behaviour

By default GISTBot makes at most one request per second per domain, respects robots.txt (including Disallow and Crawl-delay), and caches aggressively so it does not re-fetch pages that have not changed.

How to stop GISTBot

You can block GISTBot in your robots.txt:

User-agent: GISTBot
Disallow: /

gist honours robots.txt for every crawl, so this stops future fetches. You can also contact us at [email protected] to request that a domain be removed from our crawler's path.