# myintelbrief.com/robots.txt # Last updated: 2026-07-27 # Block commercial SEO/CI scrapers — they sell our data to competitors. # These bots respect robots.txt because their business depends on appearing ethical. # # 2026-07-27 — AhrefsBot + SemrushBot were REMOVED from this block list # (also unblocked at the Caddy edge, @wbi_bad_ua). The original block was # defensive; at this stage the trade runs the other way: # 1. Backlinks are the active growth strategy and blocking these two # means we cannot measure our own backlink profile at all. # 2. Journalists/directory editors/partners check Ahrefs+Semrush before # linking to a site; no data reads as "no history". # 3. The crawl surface we would be protecting is small and public anyway. # 4. Ahrefs Webmaster Tools (free DR for verified owners) REQUIRES # AhrefsBot to be able to crawl us. # The pure-extraction crawlers below (DotBot/MJ12/SERanking/Petal/etc.) # stay blocked — we use no tool of theirs, so there is no reciprocal value. User-agent: MJ12bot User-agent: DotBot User-agent: rogerbot User-agent: PetalBot User-agent: Bytespider User-agent: DataForSeoBot User-agent: MegaIndex User-agent: serpstatbot User-agent: BLEXBot User-agent: SeznamBot User-agent: Barkrowler # 2026-07-20: competitor-intelligence change-monitoring crawlers caught # enumerating our sitemaps (robots.txt + sitemap.xml + blog/sitemap.xml) # to track our pages — exactly what a rival CI tool does. They fetch # robots.txt FIRST, so the Disallow stops them; a fail2ban jail # (wbi-ci-scraper) hard-blocks any IP that ignores it or spoofs the UA. User-agent: talos-watch User-agent: SitemapMonitor Disallow: / # 2026-09-12 (Chuck: "lets open up crawling to yandex, baidu and 360"): # Yandex / YandexBot / Baiduspider / 360Spider were UNBLOCKED — their # `Disallow: /` group was removed and they now fall through to the # wildcard `User-agent: *` group below (Allow: / plus the standard # private-path Disallows). Same reasoning that unblocked Ahrefs+Semrush # on 2026-07-27: the crawl surface is small and public, and being # absent from an index reads as "no history". The 71id block was a # crawl-budget argument, and crawl budget is not the constraint here. # # ⚠️ Yandex is unblocked HERE but is still stopped at the EDGE: the # site-wide geo block (451, OFAC list in refresh-geo-blocklist.sh) # includes Russia, and Yandex crawls from RU ranges — verified # 2026-09-12 that 5.255.253.x / 87.250.224.x / 95.108.213.x all sit # inside the blocked CIDRs. So robots.txt now says yes and Caddy still # says 451. Baidu (180.76.x, 220.181.x) and 360 (101.226.x) are CN and # NOT geo-blocked, so those two are genuinely open as of this change. # Opening Yandex for real means carving RU out of the geo block, which # is a separate compliance decision and deliberately NOT done here. # # The 2026-06-19 note on 360Spider still stands as a watch item: it was # seen scraping with a spoofed Chrome UA and not reading robots first. # If volume becomes a problem, a UA-403 is the fallback. # 71hu — AI training + AI search scrapers are explicitly welcomed. # WBI competes on price against Crayon/Kompyte/Klue; being citeable # in ChatGPT/Claude/Perplexity answers is a real distribution # channel. Each gets its own User-agent block with the same # Allow as the wildcard rule below — the duplication is intentional # so each scraper sees an EXPLICIT permission line in its log # (some operators treat "fell through to *" as ambiguous). Order: # OpenAI, Anthropic, Google, Perplexity, Cohere, Apple, Common # Crawl (powers many open-source models), Meta, Diffbot. User-agent: GPTBot User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Claude-Web User-agent: Google-Extended User-agent: PerplexityBot User-agent: Perplexity-User User-agent: cohere-ai User-agent: Applebot User-agent: Applebot-Extended User-agent: CCBot User-agent: FacebookBot User-agent: Meta-ExternalAgent User-agent: Diffbot User-agent: omgilibot User-agent: xAI-Bot User-agent: xai-bot User-agent: Grok Allow: / Disallow: /dashboard Disallow: /account Disallow: /admin Disallow: /api/ Disallow: /health Disallow: /internal/ Disallow: /register Disallow: /login Disallow: /cdn-cgi/ Disallow: /brief/ Disallow: /signup/*/start # 71hu — AI-readable summary file. The /llms.txt convention # (proposed by Jeremy Howard, adopted by Anthropic + Vercel + many # others) is the AI-equivalent of robots.txt: a single text file # describing what the site offers in LLM-friendly prose. Listing # it here makes it discoverable by agents that look in robots.txt # first. Not all AI agents look for it yet, but the cost of # advertising is zero and the benefit grows with adoption. # See https://llmstxt.org for the spec. # 2026-06-06: kept as a COMMENT, not a bare directive. `LLM-Content:` # is not part of the robots.txt spec, so Google/Lighthouse flagged the # bare line as "robots.txt is not valid" (docked the SEO score). As a # comment it is 100% valid syntax AND still discoverable: agents that # scan robots.txt for an llms.txt reference read the URL from this line. # LLM-Content: https://myintelbrief.com/llms.txt # Default for everyone else (Google, Bing, other AI bots, etc.): # Allow public content, block private/transactional paths. User-agent: * Allow: / Disallow: /dashboard Disallow: /account Disallow: /admin Disallow: /api/ Disallow: /health Disallow: /internal/ Disallow: /register Disallow: /login Disallow: /cdn-cgi/ Disallow: /brief/ Disallow: /signup/*/start Disallow: /ar/ Disallow: /de/ Disallow: /es/ Disallow: /fr/ Disallow: /he/ Disallow: /hi/ Disallow: /id/ Disallow: /it/ Disallow: /ja/ Disallow: /ko/ Disallow: /ms/ Disallow: /nl/ Disallow: /pl/ Disallow: /pt/ Disallow: /ru/ Disallow: /sv/ Disallow: /th/ Disallow: /tr/ Disallow: /vi/ Disallow: /zh/ Sitemap: https://myintelbrief.com/sitemap.xml Sitemap: https://myintelbrief.com/blog/sitemap.xml