Cloudflare and OpenAI Discovery Pilot, October 2026: 6 Fixes

Cloudflare and OpenAI's discovery pilot filters out dead addresses, catch-all redirects and duplicate pages. Six fixes so AI search crawlers find your site.

By Yannis Spanenburg · 2026-10-16 · 6 min read

Cloudflare and OpenAI have tested since July 2026 whether Cloudflare's network data can help ChatGPT find pages its search crawler would otherwise miss. There is no way to join the pilot. What you control is how your site answers crawlers.

What Is the Cloudflare and OpenAI Discovery Pilot?

The discovery pilot is a research project in which Cloudflare passes signals from its network to OpenAI's search crawler, so ChatGPT can find new and updated pages with less wasted crawling. Cloudflare and OpenAI launched it in July 2026, and Cloudflare published the first results on 15 October 2026. OpenAI still evaluates and ranks every page it finds.

• The pilot started in July 2026 and runs on a cohort of about 1% of Cloudflare's network.
• OpenAI discovered and indexed hundreds of millions of new pages through it, according to Cloudflare.
• Cloudflare shares metadata about pages for search and discovery. It does not share page content for model training.

What Signals Does Cloudflare Share With OpenAI?

Cloudflare describes four patterns its network signals pick up: dead addresses, catch-all redirects, duplicate pages and real page changes. Each signal tells OpenAI's crawler something about an address before it fetches the page, so the crawler spends its visits on pages that exist, carry their own content and have changed since the last visit.

• Dead addresses: bots keep requesting files that are not there, and the signals let crawlers skip them.
• Catch-all redirects: some sites send almost any address to the homepage, and the signals filter those redirects out.
• Duplicates: different addresses that lead to copies of the same content.
• Changes: change signals tell a crawler when a page is worth another visit.

Cloudflare's July 2026 press release lists content freshness, traffic quality and actual page changes as examples of these network insights.

Expect more crawling at first: Cloudflare writes that newly found pages can raise crawl volume while OpenAI evaluates and indexes them.

Can You Opt In to the Pilot?

You cannot opt in. Cloudflare's October post and the July press release describe no setting to switch on and no way to apply. The one tool Cloudflare points site owners to is AI Crawl Control, available on all Cloudflare plans, which shows which AI services access your site and lets you allow or block each crawler.

The six fixes below remove the patterns the pilot's signals filter out, and they help Google's crawler too. Our guide to optimizing a website for ChatGPT and Perplexity covers the rest of AI search.

How to Prepare Your Site for AI Discovery in 6 Steps

Fix status codes first, then addresses, then dates, then access. Steps 1 to 3 make missing and moved pages answer with the right code, steps 4 and 5 give each page one address and an honest change date, and step 6 lets OpenAI's search crawler in.

1. Return a real 404 for addresses that do not exist. Type a made-up address on your domain, such as /this-page-does-not-exist. If the server answers 200 and shows the homepage or an empty template, you have a soft 404, which Google's documentation on HTTP status codes reports as an error in Search Console. Serve a 404 or 410 status with a short page that links to your main sections.
2. Remove any rule that redirects unknown addresses to the homepage. Look in your server config, your CMS redirect plugin and your Cloudflare redirect rules. Keep redirects only for addresses that once held a page.
3. Use a 301 for every page that moved. Point each old address at the closest matching new page in one hop, with no chains and no temporary 302s left over from a migration. When we rebuilt the Life Experts website, all 147 old addresses were kept or redirected.
4. Give each page one address. Versions with and without a trailing slash, with tracking parameters, on http and https, or on www and the bare domain hand a crawler several copies. Pick one, 301 the others to it, and point a rel="canonical" tag at it on every page.
5. Keep sitemap lastmod dates honest. Set lastmod to the date of the last real change to the page. Google's sitemap documentation says Google uses lastmod only when the value is consistently and verifiably accurate, so a sitemap that stamps every page with today's date gives crawlers no usable signal.
6. Allow OAI-SearchBot and check your Cloudflare AI settings. OAI-SearchBot surfaces sites in ChatGPT search, and GPTBot collects content that may train OpenAI's models. OpenAI's crawler documentation states the two settings are independent, and a site that blocks OAI-SearchBot is left out of ChatGPT search answers, apart from navigational links. Then open AI Crawl Control and confirm no rule blocks OAI-SearchBot.

We found step 1 broken on our own site. In September 2026 a retired Dutch address on proofofpixel.agency answered with a 200 status, served the homepage and pointed its canonical tag at the homepage, and Google still listed it under its old title. That address now returns a 301 to the English page, and unknown addresses return a 404.

How Do You Check AI Crawl Control on Cloudflare?

Open AI Crawl Control in the Cloudflare dashboard for your domain. It works on every plan without setup and shows which AI crawlers visit, how often, and whether a rule allows or blocks each one. Three of its features matter for discovery:

• It shows crawler activity and request patterns, so you can see whether OAI-SearchBot visits at all.
• It sets an allow or block rule per crawler, so you can allow OAI-SearchBot and block GPTBot.
• It tracks which crawlers follow your robots.txt and lets you enforce the rules.

Check it after every change to your Cloudflare security settings. We have seen domains where robots.txt allowed GPTBot and ClaudeBot while Cloudflare answered both with a 403. In our 2026 study of 9,066 business websites, 2.8% had a firewall that turned an AI search crawler away.

Proof of Pixel's position: a website should answer every crawler with the correct status, a 404 for a page that does not exist, a 301 for a page that moved and one address for each page.

Proof of Pixel is a marketing and consulting agency for businesses that live on enquiries and orders, building the website, the tracking and the follow-up as one system, with offices in Dubai, New York, London, Antwerp and Malaysia. Canonicals and 301 redirects are on the checklist we run at every launch in our SEO service.

Start With One Made-Up Address

Open a made-up address on your domain with your browser's developer tools on the Network tab. A 404 status means step 1 is done. A 200, or a redirect to the homepage, means it needs fixing.

Our free AI crawler checker tests whether AI crawlers can read your site and whether unknown addresses return a real 404.

If it finds a problem you want fixed, talk to our team.