Many of Google’s AI rivals will have been disappointed to see the tech giant escape largely unscathed from its first major antitrust battle with the U.S. government. Despite concluding Google had operated an illegal monopoly, U.S. district judge Amit Mehta did not force the company to take remedies such as spinning off its Chrome browser or to stop paying hardware vendors for prime positioning on their platform. Nor did the judge say Google must give users explicit “choice screens” that could have encouraged them to make AI rivals like ChatGPT or Perplexity their default search option.
But Mehta did order Google to undertake one measure that could be a major win for its rivals: The judge ordered the tech giant to share its search index, along with certain user data, with competitors. Appeals by Google and the Justice Department could mean it may take years for the remedy to take effect but, if it goes forward, the order could shake up the competitive landscape for AI.
A search engine uses something called a search index to find the best web pages to answer a query. That index is an organized list of web addresses, usually ranked by how relevant they are to any particular query. And, by all accounts, Google has the best search index in existence. That’s because Google scrapes data from more of the web, more often, than most competitors. But the company’s dominance also stems from the fact it has been in the search game for so long and sees many more queries than any other company—as many as 13 billion searches every day. This has enabled Google to build a highly-accurate algorithm for ranking pages based on their relevancy and other factors, such as how authoritative a particular source of information is.
This data advantage is particularly profound for less common searches. Google’s search index is in many ways the crown jewel of its tech stack. And, until now, the company has not shared its search index with competitors outside a few specific partnerships, such as ones Google has struck with Apple and Meta.
A search index is vital for more than just search engines. It turns out to be essential to companies building AI chatbots too. Most of today’s chatbots have the ability to search the web in response to a prompt that calls for up-to-date information, such as news, or sports scores, or information about local shopping options. But finding the best web pages for the chatbot to use when formulating a response requires a good search index. A search index can also help AI companies better curate internet data that they might want to feed an AI model during training.
When OpenAI launched ChatGPT in November 2022, it put Google on the defensive. Suddenly there was a new, more conversational way to find information—one that many people preferred to a traditional Google search. At the time, many thought AI chatbots such as ChatGPT and Perplexity, might quickly erode Google’s market position.
That hasn’t happened. But Google has been forced to respond to the advent of AI chatbots by rolling out its own AI chatbot, Gemini, and incorporating AI into its traditional search product. The company now provides AI-generated capsule “overviews” for most search queries and also, in the U.S. and many other geographies, offers an “AI mode” where people can have fuller back-and-forth, chatbot-like conversations to find information from the web. Google’s search index gives Gemini and its AI search tools a powerful advantage over rivals in being able to find the most relevant, authoritative, or accurate web pages from which to draw information.
In fact, OpenAI approached Google in 2024 about a partnership that would have seen the search giant share its search index with the AI company to power searches within ChatGPT, but Google rejected the idea, according to evidence presented during the antitrust trial.
Because Google has so far refused to make its search index available, most AI vendors, such as OpenAI, Anthropic, Meta, and Perplexity, have been forced to look elsewhere. Many have turned to Microsoft, which does allow other companies to pay to use the search index that powers its Bing search engine. But Bing, because it only has about 4% marketshare in search, sees many fewer queries than Google does, and its search index is not considered to be as accurate, especially for those less common queries.
The AI companies have also looked for other ways to enhance their search accuracy. Many have bought a service from an Austin, Texas-based company called SerpiApi that runs Google searches and scrapes the results to create a kind of approximation of the Google search index. OpenAI and Apple were both listed as SerpiApi customers previously, although their logos no longer appear on the company’s website. Perplexity and Meta are listed as current SerpiApi customers.
In addition, many AI companies have been investing millions of dollars to build their own, in-house search indices. OpenAI is among them. But Nick Turley, the company’s vice president and head of ChatGPT, testified at Google’s antitrust trial, that it was not easy to create a search index that could match Google’s. “Our goal—which was a lofty goal, and we’re nowhere near close—is to serve about 80% of our traffic from our own first-party index,” Turley told the court. “We think 100% is long-term attainable, but so far away and so uncertain that it’s not an operationalizable goal, even for a set of smart people who are ambitious and think they can do the impossible.”
Meta has also been reportedly working on building its own search index and search engine, in an effort to reduce its dependence on its current arrangement with Google and its use of the Bing API. The social media giant has also struck a deal with the news agency Reuters to use its content for Meta AI prompts that require information about news and current affairs.
Perplexity, the AI “answer engine” that has tried to position itself as an alternative to Google as a way to find factual information, has also created its own search index. And, it too, has struck partnership deals with a number of news publishers to use their content to answer queries requiring information about current events. (Disclosure: Fortune is one of Perplexity’s media partners.)
If Judge Mehta’s ruling stands, the path for these AI rivals to match Google’s search performance will have gotten significantly easier. That’s because Mehta told Google it would have to share its search index with others “at marginal cost.”
Google is only required to share a one-time snap-shot of the index, not allow continual access to it, as Bing does with its API. Mehta specifically rejected the idea of ongoing access because he said he wants people to build rival search indexes and was worried that if competitors had continual access to Google’s they would become “free riders,” and not build their own. But even having that snap-shot could give competitors a major leg up in matching Google’s performance.
Mehta also ruled that Google will need to share certain key data on how users interact with search results. This includes information such as where the user is located, what type of device they are using, which links they clicked on, and which links they hovered over and for how long. The judge said Google would need to turn over this data to competitors “at least twice” during the five-year period during which the court’s ordered remedies would apply. Again, this information could help AI companies get closer to building Google-level search indices.
In the end, both aspects of the antitrust ruling could wind up saving the AI companies significant time and money. So at least there’s some silver lining for Google’s AI competitors, in what was otherwise a gloomy day for those hoping to end Google’s dominance as the primary way people find information digitally.
This story was originally featured on Fortune.com
