Are you blocking the AI bots?

If AI bots cannot crawl your website, your content will never appear in answers generated by ChatGPT, Claude, or Gemini. This audit verifies that your 'digital front door' is open to these AI models, which is the first step in Generative Engine Optimization (GEO).

Our comprehensive audit covers 4 key areas: 1) Robots.txt Analysis: validating permissions for AI crawlers. 2) LLMs.txt Check: verifying the presence of this new standard file. 3) Meta Tag Inspection: ensuring no hidden 'noindex' or bot-specific blocking tags exist. 4) Bot Access Simulation: we simulate real visits from GPTBot, ClaudeBot, and Gemini to test firewall/WAF rules.

Many website checkers only look at your main landing page. However, it is common for the homepage to be accessible while specific product pages are blocked by firewalls or misconfigured settings. Testing the exact URL where your content lives ensures that AI bots can actually access and read the details about your product. This ability to test deep links is a key advantage of our tool.

The score (0-100) measures your site's accessibility to AI. It is calculated based on: Robots.txt permissions (30%), existence of an llms.txt file (20%), and successful access by AI agents like GPTBot, ClaudeBot, and Gemini (50%). A score below 90 implies potential visibility issues.

ChatGPT (GPTBot) respects your robots.txt file. If you have 'User-agent: GPTBot' followed by 'Disallow: /', it will not crawl your site. Additionally, strict Web Application Firewalls (WAFs) like Cloudflare may identify and block AI bots as 'bot traffic' unless explicitly whitelisted.

Traditional SEO focuses on ranking blue links on Google. GEO (Generative Engine Optimization) focuses on being cited in direct answers provided by AI models. While SEO relies on keywords and backlinks, GEO requires technical transparency (crawlability) and high-context content formats (like llms.txt).

Googlebot has been around for decades and is whitelisted by almost everyone. Newer AI bots like 'GPTBot' or 'ClaudeBot' are often blocked by older security rules, default WAF settings, or outdated robots.txt files that haven't been updated for the AI era.

A 403 Forbidden error means your server blocked the request. This is usually caused by a security plugin (like Wordfence) or a WAF (like Cloudflare or AWS WAF). You need to whitelist the user agents 'GPTBot', 'ClaudeBot', and 'Google-Extended' in your firewall settings.

An llms.txt file is a new standard for AI optimization. It's a markdown file placed at the root of your domain (like robots.txt) that provides a condensed, clean summary of your website specifically for Large Language Models to read. This improves accuracy when AI answers questions about your brand.

No. Our audit tool only checks public-facing signals that any web crawler can see, such as your robots.txt file, meta tags, and server response codes. We do not store or use your website's private content.

BobUpAI provides actionable insights and recommendations based on our analysis. This includes optimizing product data, online content, building authoritative content, and ensuring consistent information across platforms that LLMs use for training.

You can download the report directly in the application (csv and pdf).