Few travel websites explicitly allow AI crawlers, study finds

Skift reported that a review of 50 major travel websites found only 10 explicitly direct AI crawlers through their robots.txt files, which tell automated programs whether they are welcome on the site.

The majority of these travel sites either lump AI bots together with other automated programs or leave their crawler policies at default settings, according to Skift's audit.

This gap in explicit AI directives matters as AI increasingly becomes a tool for travelers to shop, with AI visits to travel sites surging 119% year over year, Adobe reported in July.

Adobe analyst Vivek Pandya noted a 'major bottleneck' because many U.S. travel sites remain unreadable to AI, limiting their presence in AI search results and potentially leaving revenue opportunities unclaimed.

Skift highlighted that when AI language models cannot easily access key brand content such as amenities, pricing, or availability, travel brands may lose out on potential customer engagement and sales.