If you've been making websites or producing content over the past ten years, then you'll be familiar with the process: choose a keyword, write a long-form article, get some backlinks, and then hope that the Google algorithm grants your site a position among the top 'ten blue links'.
However, the internet is experiencing a major change in its structure since people are now going around the traditional search engines and are instead asking questions directly of AI tools such as ChatGPT, Perplexity, Google Gemini, and Microsoft Copilot.
They are not traditional search engines; instead, they are Answer Engines, and in order to optimize for them a totally new approach known as Generative Engine Optimization (GEO) or Answer Engine Optimization (AEO) is needed.
If you want ChatGPT to recommend your web development clients (or your own TechMitten projects) rather than your competitors, then you need to understand how these AI systems browse the web. Let's look at the workings of AI search and examine how you can design your site for the AI age.
How AI Answer Engines Actually Work
If you want an AI to recommend your business, then you need to understand the way in which it obtains information. AI models have two methods of "knowing" things:
- Parametric Memory: This makes up the AI's internal brain; it is the static data which the AI acquired during its initial training (for example, from sources such as Wikipedia and through large-scale data scraping). In the case of a brand that was not well known before the AI's training cut-off date, it will not know who you are.
- Retrieval-Augmented Generation (RAG): This is the method by which modern AI links with the live internet. For example, if a user asks ChatGPT to name the best web developers in Michigan, the AI does not depend on its own memory. Instead, it stops, carries out a live web search, reads through the top search results, and then produces a tailored response.
In order for your website to be cited by an AI, you have to win the RAG pipeline.
The AI Speed Limit: Why Web Performance Is Critical
When an AI bot (for example, OpenAI's search crawler) is looking for answers, it assesses around 40 to 50 URLs and then attempts to retrieve the raw HTML from the most promising ones.
The catch is that the AI allows only two seconds per page.
The AI will drop your page before it has even read one word if your origin server is slow, if you have complicated redirect chains, or if your host is slow. Moreover, bots such as OAI-SearchBot parse the raw HTML; they don't execute client-side JavaScript.
Lesson for web development: When you create Single-Page Applications (SPAs) using React, Angular, or Vue without incorporating hydration, the AI bot will simply see a blank page. Server-Side Rendering (SSR) or Static Site Generation (SSG) is not merely a performance advantage any longer; it has become a necessary requirement for achieving AI visibility.
Writing for AI: The 128-Token Rule
Generative models do not read web pages in the way that humans do, nor do they rank whole pages as Google does; instead, they break your document down into small sections (typically about 128 tokens) and then assess those particular sections for relevance.
A recent academic study (known as the "GEO-bench" study) looked at 10,000 queries and discovered that if you make intentional changes to the structure of your content it is possible to increase your visibility in AI responses by as much as 40%.
Here is how to optimize your content for AI bots:
1. Apply the Inverted Pyramid
Don't hide the answer at the bottom of the page since research has found that 44.2% of all AI citations are taken directly from the first 30% of a web page.
Under each of your main headings, provide a direct and concise answer (about 40 to 60 words). This effectively packs your answer into one highly scannable 128-token segment.
2. Introduce Specific Data and Statistics
AI models prefer verifiable facts and, if you include empirical figures, percentages, and financial metrics once every 100 words in your informative copy, your AI visibility can increase by as much as 41%.
3. Make Use of Quoted Material and Attributable Sources
Statements in the field such as "We are the best in the business" are ignored. AI systems require authoritative evidence. Visibility can be increased by 40% by including inline source attribution (that is, linking your claims to primary research), and an additional 30% in visibility can be achieved by incorporating direct quotes from named subject matter experts.
The Technical Setup: Directing the Bots
As a web developer you have complete control over the way AI bots interact with your server, but it's easy to remove your site from AI search because of a simple error.
The robots.txt Pitfall
Since many businesses are frightened that AI will take their intellectual property, they block AI bots in their robots.txt file; but it is important to make a distinction between training bots and search bots.
-
Bots used for training (for example,
GPTBot): These work by scraping the web in order to train future AI models; you can safely block them if you want to protect your IP. -
Bots such as
OAI-SearchBotandPerplexityBot: These are the RAG bots which search for up-to-date answers to user queries. If you block them, you'll not show up in the search results on ChatGPT or Perplexity.
Tip: When you specifically permit
OAI-SearchBot, be sure to also repeat your disallow rules for sensitive directories (for example,/admin/), since the bot will disregard the global wildcard rules the moment it comes across its own specific name.
Enter llms.txt
Just as there is a robots.txt file for giving permissions and a sitemap.xml file for URLs, so too has the AI era brought about an llms.txt file.
It's a clean, machine-parsable Markdown file that is located at the root of your domain (for example, at techmitten.com/llms.txt). The file gets rid of all the navigational clutter, popups, and layout formatting, thus giving AI reasoning models a clear text outline of what your business does, who it serves, and where your main documentation can be found. So add it to your site now.
Entity Grounding: Proving You Exist
AI does not reason in terms of keywords; instead it works with Entities (which are distinct people, places, organisations, or concepts). If the AI is unable to confirm that your business is a real and legitimate entity then it will automatically suggest your competitors.
To prove you are real, you need to establish a solid "knowledge graph" presence:
-
Use Schema.org JSON-LD. Place the Organization schema in the
<head>section of your homepage and make sure that the@idURI is included, using thesameAsarray to link your website directly to your official Wikidata item, Crunchbase, LinkedIn and your social profiles. -
Declare your expertise. Make it clear to the AI which topics and technical fields you are qualified to discuss by using the
knowsAboutproperty in your schema. - Keep your NAP consistent. For local businesses, ChatGPT makes heavy use of Yelp, whereas Microsoft Copilot depends on Bing Places and Facebook. The name, address, and phone number (NAP) must be exactly the same on all these platforms. The AI doesn't make guesses; if the address of your business differs on Yelp and Google, then the AI will reject you as a possible match.
Off-Page SEO: Mentions Over Backlinks
In traditional SEO the hyperlink held the position of king, but in Generative Engine Optimization unlinked brand mentions are now taking over that role.
Evaluation based on 75,000 brands reveals that unlinked references to a brand are related three times as strongly to AI visibility as traditional backlinks. Moreover, 82% of all AI citations are from earned third-party media (for example, digital public relations, industry news, and discussions on Reddit), whereas just 6% come from the company's own website.
The AI will not believe a major claim that you make on your website unless other credible sites are also talking about it; therefore, your public relations should centre on getting your brand name cited in reputable industry publications, even if they don't provide you with a hyperlink.
How to Track AI Traffic in GA4
A major source of annoyance with AI search is the fact that it functions as a "zero-click" environment: users obtain their answers without having to go to your website. Furthermore, when they do click through, mobile apps tend to remove the referral data, which causes Google Analytics 4 (GA4) to treat AI traffic as general "Direct" traffic.
In order to fix this, you have to create a Custom Channel Group in GA4.
In the GA4 Admin section, go to Data Display and set up a rule based on a regular expression, positioning it above the standard Referral channel. Use this precise RE2-compliant regular expression to capture traffic from AI:
^.*(chatgpt\.com|chat\.openai\.com|openai\.com|claude\.ai|anthropic\.com|perplexity\.ai|gemini\.google\.com|bard\.google\.com|copilot\.microsoft\.com|bing\.com\/chat|grok\.com|x\.ai|grok\.x\.com).*
You'll probably be amazed when you filter this traffic since, even though the amount of AI search is less than that of Google organic search, the conversion rate is usually much higher because the AI has already pre-qualified the user's intent.
Your 90-Day GEO Action Plan
If you'd like to optimize your site for ChatGPT and Perplexity, then follow this simple roadmap:
Days 1–30: The Tech Fix
- Check your
robots.txtfile to make sure thatOAI-SearchBot,Claude-SearchBot, andPerplexityBotare permitted. - Audit your server response times (keep them below 2 seconds!).
- Implement Server-Side Rendering if it's necessary.
- Create a file called
llms.txt. - Set up your GA4 regex filter.
Days 31–60: Content Re-Architecture
- Rewrite the top 30% of your service pages by including a direct summary of about 50 words right at the beginning.
- Add some hard statistics and check your claims against reliable sources.
- Incorporate comprehensive Organization schema into your headers.
Days 61–90: Off-Page and PR Efforts
- Audit the listings of your local directory to make sure the NAP information is consistent.
- Begin carrying out digital PR campaigns with the aim of getting your brand name mentioned on industry blogs and forums such as Reddit.
The move to Answer Engines has already taken place, and by using these Generative Engine Optimization techniques you can make it so that when the AI is asked to provide the best web solutions it will directly point to you.
