What Your Website Needs to Get Found by AI Assistants

Getting found by AI assistants starts with a website they can read, and most of what makes a website readable is old and cheap. Three things do most of the work: a robots.txt file that lets the AI search crawlers in, words that sit in the page itself instead of loading later through JavaScript, and a sitemap that lists your pages. The newer AI files you may have heard pitched are optional, and Google says its own search doesn't use them.

The share of people using AI assistants keeps climbing. A Pew Research Center survey fielded in February 2026 found that 44% of US adults say they use ChatGPT, up from 34% a year earlier.

Website work is part of what we do at Pioneer when a business needs it, so while writing this post we ran the same checks against both of our own websites. Our business site passed every check. Our softball app failed two, and we built that one too.

AI assistants find your business by reading your website

AI assistants find businesses the way search engines always have. A crawler, the program that visits and reads web pages, fetches your site, and the assistant draws on what it read when someone asks a question. Google's guide to its generative AI search features says so directly: "The best practices for SEO continue to be relevant because our generative AI features on Google Search are rooted in our core Search ranking and quality systems." SEO, short for search engine optimization, is the work of making a site easy for search engines to find and understand.

Google's AI answers use the same bar as regular search. Its documentation for AI Overviews and AI Mode, the AI-written answers Google shows in search, says a page only has to be indexed and eligible to show in regular Google Search with a text preview to appear as a supporting link. In Google's words, "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary."

The other assistants send their own crawlers. OpenAI's is called OAI-SearchBot and surfaces websites in ChatGPT search, Anthropic runs Claude-SearchBot for search results in Claude, and Perplexity's PerplexityBot surfaces and links websites in its answers. All three companies also say their assistants may visit a web page directly when a user asks a question.

Readability matters more when fewer people click through to your site. In a Pew Research Center study of 68,879 Google searches made by 900 US adults in March 2025, 18% of the searches produced an AI summary. Users clicked a regular search result on 8% of visits with an AI summary and on 15% of visits without one. They clicked a link inside the summary itself on 1% of visits.

Your robots.txt file decides which AI crawlers can read your site

A robots.txt file is a short text file at yoursite.com/robots.txt that tells crawlers which parts of a website they may read, and it is the first thing to check. The AI companies split their crawlers by job, so one file can let an assistant find your business without handing your content over for model training.

OpenAI's crawler documentation says GPTBot collects content that may be used to train its models. OAI-SearchBot is the one that surfaces websites in ChatGPT search, and each setting is independent. OpenAI spells out the tradeoff: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." Anthropic splits its crawlers the same way, with ClaudeBot for training and Claude-SearchBot for search, and warns that blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results." Perplexity says PerplexityBot is not used to crawl content for AI models at all.

An owner who doesn't want their content used for training can block GPTBot and ClaudeBot and still let OAI-SearchBot and Claude-SearchBot in. Blocking every crawler with an AI name on it blocks the search crawlers along with the training ones.

Check the file even if nobody at your business has ever touched it. Cloudflare, a network company that says it manages and protects traffic for 20% of the web, began asking every new domain at sign-up in July 2025 whether to allow AI crawlers. If your site runs through Cloudflare, find out what that answer was.

Our own robots.txt on pioneeraidata.com has five lines of text. It lets every crawler read everything, lists our two sitemaps, and carries a Content-Signal line saying our content may be used for search, as input to AI answers, and for training. Content-Signal is a robots.txt line Cloudflare introduced in September 2025, and Cloudflare's own policy says the signals "express preferences; they are not technical countermeasures against scraping."

One robots.txt mistake has nothing to do with AI and shuts out every crawler that follows the rules. On a client website we built this summer, the preview version of robots.txt told every crawler to stay out, which is the right rule while pages are half-finished. Launching with that rule still in place would hide the finished site from search engines and AI assistants alike. The script we run before that site's launch fails if the stay-out rule is still in the file, and it also fails unless the file names GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended.

Most AI crawlers read your raw page and never run its JavaScript

The major AI crawlers read the raw page file your server sends and don't run its JavaScript, the code that builds or changes a page inside the browser, so words that only appear after a script runs are invisible to them. Vercel, a web hosting platform, published a study in December 2024 that tracked 569 million fetches by OpenAI's GPTBot and 370 million by Anthropic's crawler. The study found that "none of the major AI crawlers currently render JavaScript." The crawlers it named came from OpenAI, Anthropic, Meta, ByteDance, and Perplexity. Google is the exception: the same study says Gemini uses Googlebot's infrastructure, which renders JavaScript fully.

We checked what that means on our own two sites. The home page of pioneeraidata.com sends its words inside the page itself, and before any script runs it hands a crawler 1,399 words of readable text. The home page of pioneerpennant.com, the softball analytics platform on our work page, is a JavaScript app. Its raw page is 3,528 bytes, and the body holds zero words until a script runs. A crawler that doesn't run JavaScript gets a page title and an empty box where the site should be.

The softball app fails a second check, one that Google's own documentation says can be hard to avoid in apps built this way. Type a made-up address on pioneerpennant.com and the server answers with a success code and the same 3,528-byte page. Google calls that a soft 404: a page that returns a success status without the content that should be there. On pioneeraidata.com, a made-up address gets a proper 404, the "page not found" code, which is what a missing page should return. Google's suggested fix for apps is to send missing addresses to a page that returns a real 404, or to mark error pages so search engines skip them.

The softball platform isn't invisible, because its charts and data pages have their words built in and sit outside the app. In August 2026 we gave those pages their own sitemap, and it lists 365 of them. The app's own sitemap lists three. As of September 14, 2026, the app's home page still reads as blank to any crawler that doesn't run JavaScript.

To check your own site, open your home page, right-click, choose View Page Source, and search for a sentence you can see on the screen. If that sentence isn't in the source, the crawlers in Vercel's study couldn't have read it either.

A sitemap lists your pages, and Google has to hear about it separately

A sitemap is a file, usually at yoursite.com/sitemap.xml, that lists the pages you want crawlers to find, and a new small business website should have one. Google's sitemap documentation says a site of "about 500 pages or fewer" that is "comprehensively linked internally" might not need one, and it lists a new site with few outside links pointing to it as a site that might. A sitemap costs nothing to keep either way. The sitemap on pioneeraidata.com lists 35 pages, and our robots.txt points to it.

Telling search engines about a new or changed page is a separate job. IndexNow is a protocol for notifying search engines when a page is added or updated, and a notice sent to one participating engine is shared with all of them: Amazon, Bing, Naver, Seznam.cz, Yandex, and Yep. Google is not on that list. We send an IndexNow notice every time we publish a blog post. For Google, the notice is the sitemap itself, listed in robots.txt or submitted in Google Search Console, and Google notes that a submitted sitemap is "merely a hint" about what to crawl.

Structured data and a Google Business Profile help, but neither is required

Structured data is a block of code on a page that states facts like a business's name, services, hours, and address in a format software can read without guessing, and Google says its AI search doesn't require it. The generative AI guide puts it this way: "Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add." The same guide says it's still a good idea to keep using structured data, because it helps a page qualify for rich results, the enhanced listings in regular Google Search.

For a local business, Google's local business structured data lets a business tell Google about "business hours, different departments within a business," and more. When people search for a business on Google Search or Maps, Google says the results may show "a prominent Google knowledge panel with details about a business." Google's AI features documentation adds one rule: structured data should match the visible text on the page. Structured data appears on 23 of the 28 top-level pages of pioneeraidata.com, and every blog post on the site gets it automatically when it's published.

A Google Business Profile is your business's listing on Google. Google's generative AI guide says products like Google Business Profiles "can help your products and services to be visible in both AI responses and other Google Search results."

An llms.txt file is optional, and Google Search doesn't read it

An llms.txt file is a plain-language summary of a website written for AI tools, and Google says Google Search doesn't use one. The file was proposed in September 2024 by Jeremy Howard as a standard place for a website to give AI agents the information they need to use it. Google's generative AI guide says you don't need "AI text files, markup, or Markdown" to appear in Google Search, including its AI features, "as Google Search itself doesn't use them."

We publish one anyway. The llms.txt on pioneeraidata.com runs 55 lines: what we do, the builds that are live, the blog, and our full price list. Keeping it current costs almost nothing, and assistants other than Google's are free to read it. On the client site, the script we run before launch fails if the llms.txt is missing the owner's license number, because a summary written for AI tools is a good place for the facts an assistant should repeat exactly. An llms.txt file is still the smallest item on this list, so be wary of any "AI search optimization" quote that treats one as the centerpiece.

Five checks to see whether AI assistants can read your website

You can check the basics with nothing but a web browser.

1. Open yoursite.com/robots.txt. A Disallow: / line under User-agent: * tells every crawler to stay out. Also look for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, or PerplexityBot with a Disallow line under the name.
2. View the source of your home page and search for a sentence you can see on the screen. If the sentence isn't in the source, the AI crawlers in Vercel's study couldn't read it.
3. Type a made-up address on your site, like yoursite.com/zzz-test. You should land on a "page not found" page. If your home page loads instead, that is the soft 404 pattern.
4. Open yoursite.com/sitemap.xml. It should list your real pages on your real domain, not a preview address or a page that no longer exists.
5. Look up your Google Business Profile and confirm the hours, services, and phone number match your website.

The robots.txt and Business Profile fixes are small edits. The JavaScript one can mean rebuilding pages, which is worth knowing before anyone quotes you for it. If you are weighing whether to pay for help, we published what AI actually costs a small business, our own prices included.

None of this is new, and that's the good news. The work that gets a small business found by search engines is most of the work that gets it found by AI assistants.