Study: AI Search Engines Provide Incorrect Answers 60% of the Time

AI search tools like Google’s Gemini and Grok 3 lead users to syndicated content, fabricated URLs, and broken links, causing tension for publishers. Chief operating officer Mark Howard from Time magazine emphasizes the need for transparency and improvement in AI-generated searches. OpenAI and Microsoft acknowledge the issues but focus on supporting publishers and adhering to protocols. The report highlights accuracy problems previously identified in ChatGPT by the Tow Center.

“

AI search tools often directed users to syndicated versions of content instead of original publisher sites, even when formal licensing agreements were in place. This issue highlighted URL fabrication problems, with a significant number of citations leading to fabricated or broken URLs, resulting in error pages. Testing 200 citations from Grok 3 revealed that 154 of them resulted in broken links.

Publishers faced a dilemma as blocking AI crawlers could lead to a loss of attribution, while allowing them to crawl potentially meant widespread reuse without driving traffic back to the publishers’ websites. Mark Howard, Time magazine’s chief operating officer, emphasized the importance of transparency and control over how Time’s content appears in AI-generated searches. Despite acknowledging current shortcomings, Howard expressed optimism about future improvements, stating that today represents the worst state of these AI tools with ongoing investments and engineering efforts aimed at enhancing them.

On the other hand, Howard also pointed out user responsibility, suggesting that consumers should be skeptical of the accuracy of free AI tools. He remarked, “If anybody as a consumer is right now believing that any of these free products are going to be 100 percent accurate, then shame on them.”

OpenAI and Microsoft acknowledged the report’s findings but did not directly address the specific issues raised. OpenAI highlighted its commitment to supporting publishers by driving traffic through summaries, quotes, clear links, and attribution. Microsoft, on the other hand, stated its compliance with Robot Exclusion Protocols and publisher directives.

This latest report from the Tow Center builds upon previous findings from November 2024, which identified similar accuracy issues in how ChatGPT handled news-related content. For a more comprehensive review of the report, visit Columbia Journalism Review’s website.

🔗 Source