Articles Bot & WAF

AI Training Bots vs. AI Search Bots: Stop Accidentally Erasing Your Brand from ChatGPT

Many webmasters block all AI user-agents to protect training data, inadvertently blocking OAI-SearchBot and ChatGPT-User and wiping out their live search citations.

Brian Ho · Founder & CEO, Brian Ho Marketing · 8 min read

One of the most damaging mistakes we uncover during Layer 3 (WAF) and Layer 5 (robots.txt) audits is treating every AI user-agent identically. OpenAI, Anthropic, and Perplexity separate their bots by function.

1. Three Distinct AI Bot Categories

  • Model Training Crawlers: e.g., GPTBot, CCBot, Google-Extended. You may disallow these if you want to opt out of foundational model training.
  • AI Search Indexing Crawlers: e.g., OAI-SearchBot, PerplexityBot. These bots index pages specifically to surface live citations and links in AI search answers.
  • User-Triggered Fetchers: e.g., ChatGPT-User, Claude-User. Triggered in real time when a human user asks the assistant to read or verify your page.

2. Recommended Baseline Configuration for GEO

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

You can test how your domain handles 196 AI bots using our AI Crawler Access Check and AI Log Analyzer inside Share · Tools.

Technical GEO · 10 min read

The 12-Layer Technical GEO Stack & 7-Step AI Search Funnel from Discover to Mention: Why You Must Audit Bottom-Up

Many teams rush to publish llms.txt or rewrite content while their WAF silently blocks OAI-SearchBot or critical pages rely on client-side JS. Here is why bottom-up engineering is mandatory.

GEO Methodology · 9 min read

4-Entity Model, EEAT-A & Internal Wikipedia: Turning Your Site into AI’s Primary Grounding Source

On-page GEO differs fundamentally from classic On-page SEO. Instead of keyword density, you must engineer unambiguous entity graphs and an Internal Wikipedia structure.

Agentic & MCP · 7 min read

Turning Claude, ChatGPT, and Cursor into Technical GEO Specialists with BHM-MCP

Instead of pasting fragmented prompts, Model Context Protocol (MCP) lets your AI assistant dynamically invoke Brian Ho’s verified technical skills and audit checklists.