Amazon quietly blocks more of OpenAI’s ChatGPT web crawlers from accessing its site – Modern Retail
When economist Tyler Cowen sat down with OpenAI CEO Sam Altman for a fireside chat during a four-day conference in Berkeley, California last month, the conversation eventually turned to the ChatGPT maker’s buzzy new deal with America’s biggest retailer: With OpenAI now letting Walmart shoppers buy products directly inside ChatGPT, would Amazon ever agree to a similar deal?
“Do you think Amazon will fold and join that, or are they going to fight back and try to do their own thing?” Cowen asked.
“I have no idea,” Altman replied. “If I were them, I would fight back.”
Amazon appears to be doing just that.
The e-commerce giant has quietly blocked more OpenAI-related bots from crawling Amazon.com, according to updates in its publicly visible robots.txt file. The change was first spotted by independent e-commerce analyst Juozas Kaziukėnas, who posted a screenshot on LinkedIn showing new disallow rules for several ChatGPT-associated crawlers responsible for model training, web browsing and search. Modern Retail also confirmed the change by reviewing the code that underpins Amazon’s e-commerce site.
The update marks Amazon’s latest move to ward off AI agents that could scrape its product pages, monitor prices or even attempt automated purchases. While retailers like Walmart, Target and Etsy are seeing more referral traffic from ChatGPT — on top of inking partnerships with the AI-powered chatbot — Amazon continues to restrict what external AI agents can access, opting instead to steer shoppers toward its own AI offerings.
Amazon has a financial reason to keep shoppers on its own site, as the company generates roughly $56 billion a year from advertising, a business that depends on people browsing Amazon.com rather than shopping through ChatGPT, where ads can be bypassed.
Amazon “doesn’t want to just be the back end of the internet,” said Scot Wingo, author of the Substack Retailgentic and founder of ReFiBuy, a company that helps brands and retailers optimize for agentic AI. “They want to be the front door, and they’re going to fight this for quite a while, would be my guess.”
Neither Amazon nor OpenAI responded to requests for comment.
Amazon has taken multiple steps in recent months to prevent third-party crawlers from scraping its website, including those tied to Meta, Google, Perplexity and more, Modern Retail previously reported. Amazon had also already blocked OpenAI’s GPT-bot, which is the web crawler that scrapes content for its ChatGPT model training.
But the newest additions go further by targeting two more types of OpenAI crawlers. One rule blocks “ChatGPT-User,” the agent ChatGPT uses to fetch live information from the web when a user asks a question, and another targets “OAI-SearchBot,” which powers OpenAI’s new SearchGPT search engine.
According to a working paper authored by OpenAI’s Economic Research team, those agents help power around 50 million shopping-related queries per day — everything from “recommend a laptop under $1,000” to “how much are running shoes.”
The timing of Amazon’s latest act against third-party AI agents is notable given that Amazon says it hasn’t ruled out further partnerships. On the company’s latest earnings call, CEO Andy Jassy said Amazon is “having conversations” with third-party shopping agents and expects it will “find ways to partner” over time. But he also said that many existing AI shopping agents provide inaccurate pricing, delivery and inventory information.
“We have to find a way, though, that makes the customer experience good,” he said.
Amazon’s clash with third-party AI tools isn’t entirely limited to web scraping. Earlier this month, the company sent a cease-and-desist letter to Perplexity AI over its new Comet browser, which lets users ask an AI agent to find and buy items on Amazon. Perplexity accused Amazon of “bullying” after receiving what it described as an “aggressive legal threat” demanding the startup stop allowing Comet to complete purchases on behalf of users. In the letter, Amazon alleged that Perplexity was committing computer fraud and using “disguised or obscured” agents to access Amazon’s store without authorization.
In a statement, Amazon said third-party shopping agents should “operate openly and respect service provider decisions” on whether or not to participate.
Instead of opening its doors to outside agents, Amazon is investing in its own AI tools. For instance, Amazon has rolled out features like “Auto Buy,” which automatically purchases items for customers when prices drop. Amazon also began testing an agent earlier this year called “Buy For Me” that lets shoppers buy products from other websites without leaving its app. Amazon says shoppers who use Rufus are 60% more likely to finish a purchase, and it expects the tool to generate more than $10 billion in yearly sales.
Other retailers are betting that AI agents will become a bigger part of online shopping. In recent weeks, OpenAI has inked partnerships with Walmart, Etsy and Shopify merchants that make their wares available for purchase through ChatGPT’s Instant Checkout feature. On Wednesday, OpenAI and Target also announced plans to launch the retailer’s app inside ChatGPT.
Many of those same retailers are seeing a surge of referral traffic coming from ChatGPT, giving them a fresh source of online shoppers. Walmart, for instance, now gets around 36% of referral traffic from Walmart, up from 20% when Modern Retail first reported on the trend, according to data from web traffic analytics company Similarweb. Referral clicks account for less than 5% of total site visits, but it could have major implications for brands and retailers if more shopping takes place within AI agents like ChatGPT.
Amazon has increasingly taken “an agnostic approach” to whether consumers buy a product on Amazon or off Amazon, as long as Amazon gets a cut of the sale somewhere, according to Sky Canaves, a retail analyst at eMarketer. This has lead to some surprising partnerships, like when Amazon announced in September that its third-party logistics service would expand to handle orders for competitors including Walmart, Shein and Shopify. Amazon also partnered with social media sites including TikTok and Pinterest to enable users to buy items from Amazon via shoppable ads on those platforms. That posture has prompted some, like Cowen, to wonder whether Amazon could eventually take a similar approach with AI agents.
But because Amazon controls nearly 40% of all U.S. e-commerce spending, and 90% of its shoppers begin their searches directly on Amazon.com, it’s likely the retail giant will continue to block AI agents, at least for the time being, according to Kaziukėnas. As he put it, “There’s no benefit for a platform like Amazon to be part of something like that.”
The e-commerce giant has quietly blocked more OpenAI-related bots from crawling Amazon.com, according to updates in its publicly visible robots.txt file.
Kroger has launched a weekly AI-generated insights report for brand executives, analysts or category managers to analyze their performance.
In partnership with OpenAI, Target plans to launch a new Target app within ChatGPT beginning the week of Thanksgiving and Black Friday.
Join now to gain access to exclusive content, unlimited articles and more.
Visit your account page to modify or renew your subscription.
Get Modern Retail's top stories every morning in your email inbox
Follow @ModernRetail for the latest news, insider access to events and more.
