Is Your Website Blocking ChatGPT and Claude? The 10-Minute robots.txt Check for Agents

Jerad Larkin

1 hour ago

7 min read

How do I check whether my real estate website is blocking ChatGPT, Claude, and Perplexity?

Open yourdomain.com/robots.txt and look for Disallow rules naming OAI-SearchBot, Claude-SearchBot, or PerplexityBot. No rule means they can read your site. A Disallow rule means you are invisible to that tool.

This morning I opened my own robots.txt file for the first time in months. It took about 20 seconds, and it answered a question I keep hearing in my classes: is my website even letting ChatGPT, Claude, and Perplexity read it?

Most Denver agents I talk to are putting real effort into getting found in AI search. They are cleaning up their Google Business Profile, writing neighborhood pages, and adding schema. Almost none of them have checked the one file that decides whether AI crawlers are allowed through the front door.

As an Account Executive with Chicago Title of Colorado, I spend a good part of every week teaching agents how to get found online, and this is the fastest check I know. It takes about 10 minutes, it costs nothing, and you do not need a developer.

What robots.txt Is, in Plain English

Every website can publish a small text file at yourdomain.com/robots.txt. It is a set of instructions for automated visitors. Each rule names a bot (called the user-agent) and says which parts of your site it may or may not read.

Google's crawler, Bing's crawler, and the bots behind AI tools check this file before they read your pages. A rule that says Disallow: / under a bot's name asks that bot to stay out of your entire site.

One caveat before we go further. robots.txt is a request, not a lock. The major crawlers follow it. Some bots ignore it, and Wix says in its own help center that blocking is not an airtight solution.

What I Found in My Own robots.txt

Here is the file from milehightitleguy.com, which I pulled on November 3, 2026:

User-agent: *
Allow: /
Disallow: *?lightbox=

# Optimization for Google Ads Bot
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google
Disallow: /_partials*
Disallow: /pro-gallery-webapp/v1/galleries/*

# Block PetalBot
User-agent: PetalBot
Disallow: /

# Crawl delay for overly enthusiastic bots
User-agent: dotbot
Crawl-delay: 10
User-agent: AhrefsBot
Crawl-delay: 10

Sitemap: https://www.milehightitleguy.com/sitemap.xml

Here is what that says in plain English:

User-agent: * with Allow: / means every bot that is not named below is welcome to read the whole site. That wildcard is what lets OAI-SearchBot, Claude-SearchBot, and PerplexityBot in.
The lightbox line only keeps bots away from pop-up image views, not from pages.
PetalBot is blocked. The file's own comment says it is auto generated by Wix, so this came with the platform.
dotbot and AhrefsBot are two SEO tool crawlers that are asked to slow down with a 10 second crawl delay.
No AI bot is named anywhere. No block and no special allow.

That last point is the one that matters. My site is not blocking AI search crawlers, and I did not have to do anything to get there. Your site may not be so lucky, and you will not know until you look.

The Bots That Matter for Agents

Every AI company runs more than one bot, and the bots do different jobs. Mixing them up is the most common mistake I see. Here is what each company's own documentation says.

OpenAI (ChatGPT)

OAI-SearchBot decides whether your site can appear in ChatGPT's search answers. OpenAI says opted-out sites will not appear in those answers, though they can still show up as plain links.
GPTBot collects content that may be used to train OpenAI's models. Blocking it is about training, not about search.
ChatGPT-User is used when a person in ChatGPT asks it to visit a page. OpenAI says robots.txt rules may not apply to these user-initiated visits.

OpenAI also says each setting is independent, so you can allow OAI-SearchBot and block GPTBot at the same time. Changes to robots.txt can take about 24 hours to affect search results. Source: OpenAI's bot documentation.

Anthropic (Claude)

ClaudeBot collects web content that could contribute to model training.
Claude-SearchBot analyzes online content to improve search result relevance. Anthropic says blocking it may reduce your visibility in user search results.
Claude-User visits a page when a Claude user asks a question that needs it. Anthropic says blocking it may reduce visibility in user-directed web search.

Anthropic also recommends against blocking its IP addresses, because that can stop the bots from reading your robots.txt in the first place. Use robots.txt rules instead. Source: Anthropic's crawler help article.

Perplexity

PerplexityBot surfaces and links websites in Perplexity results. Perplexity says it is not used to crawl content for AI foundation models, and recommends allowing it.
Perplexity-User supports user-initiated visits and generally ignores robots.txt, since a person asked for the page.

Source: Perplexity's bot guide.

The pattern across all three companies is the same: one bot for training, one for search, and one for live user requests. If your goal is to get found, the search bots are the ones to protect.

The 10-Minute robots.txt Check

Type your domain followed by /robots.txt into a browser. If you get a plain text page, you are in the right place.
Search the page (Ctrl+F or Cmd+F) for OAI-SearchBot, Claude-SearchBot, and PerplexityBot. Then search for GPTBot and ClaudeBot.
Read the rule under each name. Disallow: / means blocked. Allow: / or an empty Disallow means welcome.
If a bot is not named at all, find the User-agent: * rule. If that rule says Disallow: /, every bot without its own rule is blocked, including the search bots.
Scan for Disallow lines that name your blog, post, or neighborhood page folders. One stray line there can hide the pages you most want read.
Write down what you found. Fix only what contradicts your goal. If everything reads Allow, you are done.

If the page returns an error or comes back blank, ask your web provider. A missing file generally means no restrictions, but a server error can be treated differently, so do not assume.

How to Fix It on a Wix Site

Wix is what my own site runs on, so these are the steps from the Wix help center, which I checked today. Wix now labels this area SEO & GEO.

Open your site dashboard and go to SEO & GEO.
Scroll to Tools and settings and click Robots.txt Editor.
Click View File, then click into the field under Your robots.txt file.
Delete any Disallow rule aimed at a search bot, or add the allow rules from the starter block below.
Click Save Changes, then click Save.
Reload yourdomain.com/robots.txt to confirm the change shows up.

Two warnings from Wix. Editing robots.txt is an advanced feature, and Wix Customer Care cannot help with changes you make to it. If something goes wrong, Reset to Default in the same screen restores the original file.

One more thing worth knowing. Wix also has a help article on blocking AI crawlers, and its example blocks CCBot, GPTBot, ChatGPT-User, and BingAI. If you or a vendor copied that list because it sounded protective, remember that blocking is a choice about visibility, not a free safety upgrade. Decide on purpose.

If Your Site Is on an IDX, CRM, or Template Platform

Not every agent website lets you edit robots.txt. Many sit on platforms where the vendor controls the file. In that case, send your vendor a short message. Here is one you can paste:

Hi, can you confirm whether our robots.txt or any firewall setting blocks OAI-SearchBot, Claude-SearchBot, or PerplexityBot? If it does, please allow them. Our goal is for AI search tools to be able to read our public pages. Thank you.

Ask about firewalls and CDNs too. In July 2025, Cloudflare announced it would block AI crawlers by default for new domains on its network, according to MIT Technology Review. A setting like that can stop a bot without a single line in your robots.txt, so a clean-looking file does not settle the question if your site sits behind a service like that.

A Starter Block You Can Paste

If you want to be explicit instead of relying on the wildcard, this is the allow-only block I would add for the three search bots:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

If your file already says User-agent: * with Allow: /, these lines change nothing. They just make your intent visible to anyone reading the file.

Whether to block the training bots, GPTBot and ClaudeBot, is a separate decision. Blocking them is a reasonable choice if you do not want your content used for model training, and per the documentation above it does not remove you from ChatGPT or Claude search, because those run on different bots. I leave them alone on my own site because I want my content readable. That is my preference, not a rule.

What This Will and Will Not Do

It removes a possible barrier. It does not make ChatGPT or Claude recommend you. Being readable is the entry ticket. Your content still has to be the best answer.
I have not measured a citation lift from changing robots.txt, and I do not know of a controlled study that has. Treat this as hygiene, not a growth hack.
Some visits ignore it. User-initiated visits like ChatGPT-User and Perplexity-User may not follow robots.txt.
Give it time. OpenAI says about 24 hours for search results to reflect a change, and Wix notes search engines update their copy only when they next crawl your site.

Once the door is open, the work that gets you recommended is the work you may already be doing. These posts cover the next steps: why most Denver agents are invisible in AI search, how to get found in ChatGPT, Gemini, and AI search, schema markup for agent websites, and fixing your Google Business Profile for AI Overviews.

Frequently Asked Questions

Should real estate agents block ChatGPT from their website?

For most agents, no. If you want to appear in ChatGPT search answers, OAI-SearchBot needs access. Blocking the training bot (GPTBot) is a separate choice and does not affect search, per OpenAI's documentation.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot is what lets your site surface in ChatGPT's search features. You can allow one and block the other.

Does blocking AI crawlers hurt my Google rankings?

Wix's help center says blocking specific AI crawlers should not significantly affect SEO, but it recommends checking your analytics afterward. GPTBot and ClaudeBot are not Google's crawlers.

How long does a robots.txt change take to work?

OpenAI says about 24 hours for search results. Other crawlers update when they next visit your site, so timing varies.

Where do I find my robots.txt file?

Type your domain followed by /robots.txt into a browser. On Wix, edit it under SEO & GEO, then Tools and settings, then Robots.txt Editor.

Next Step

If you want a second set of eyes on your website's AI search setup, or you are looking for marketing ideas to get your listings in front of the right people, reach out to me.

Want more tools, tactics, and resources like this? Subscribe to my weekly emails at milehightitleguy.com. I share real estate marketing ideas, AI tools, and invites to upcoming classes and events across Colorado.