From AI Crawlers to Headless Bots, Organizations Face New Challenges in Managing Automated Traffic

Fastly, Inc. (NASDAQ: FSLY), a leader in global edge cloud platforms, today released its latest Threat Insights Report, offering a detailed analysis of the evolving bot landscape and its implications for businesses. Drawing from trillions of application and API requests across Fastly’s global network, the report highlights how automated traffic is reshaping industries, forcing organizations to adapt their strategies to manage both the risks and opportunities bots present.

The findings reveal that bots now account for nearly one third (29%) of all web traffic, with the majority of this bot traffic (about 25% of all web traffic) classified as unwanted. Additionally, bot requests increased by 2% compared to the prior quarter. This increase represents billions of requests and is driving organizations to respond, particularly in industries like Media and Entertainment, Commerce, and Financial Services.

Key findings from the report:

• Wanted bots act as a double-edged sword: Organizations are increasingly scrutinizing the value of wanted bots, like AI crawlers and fetchers. While these bots play a critical role in indexing and optimizing content, billions of their requests were blocked (4%). This trend reflects growing concerns about data usage and the impact on revenue, particularly in industries like Media and Entertainment, where AI fetchers are reducing referral traffic to source websites.
• Headless bots become a threat to transaction-heavy industries: Headless bots, which mimic human behavior at machine speed, are targeting transaction-heavy industries like Financial Services and Commerce. 89% of headless bot traffic in Q3 targeted these two industries, reinforcing the need for businesses to implement advanced detection strategies that can distinguish between nefarious headless traffic and intended headless usage so as to avoid disrupting software delivery.
• AI crawlers and fetchers dominate the digital landscape: Meta’s AI crawler and OpenAI’s ChatGPT fetcher emerged as the dominant players, accounting for 60% and 68% of their respective traffic categories. This dominance highlights the growing influence of AI-driven automation on web traffic and the critical need for organizations to balance innovation with control over content.

“Fastly’s latest Threat Insights Report highlights the urgency with which organizations should take a proactive approach to bot management as it’s increasingly becoming a critical business issue,” said Marshall Erwin, Chief Information Security Officer at Fastly. “By understanding the nuances of bot traffic, businesses can safeguard their infrastructure, optimize operations, increase brand visibility, and drive innovation.”

The report sheds light on bot industry preferences, and their impact across sectors as organizations reconsider how to remain competitive in the AI era. Findings highlight Commerce (88%) and Public Sector (96%) receiving the highest distribution of crawler traffic, compared to Education (77%) with the highest fetcher volume, suggesting an indication of crawler prioritization or ‘fresh’ yet to be crawled content.

The report also highlights how businesses must adapt with the rise of AI. Media and Entertainment publishers are struggling as AI fetchers scrape their content for generative AI models, leading to diminished referral traffic – only 1% of users click through to source websites that have an AI summary. Commerce saw the highest crawler traffic, driven by demand for real-time inventory and pricing data, underscoring the need to balance automation with competitive protection.

As organizations consider how to effectively operate amid rising bot volume, the latest report provides macro insights to fuel decisions. For more information and to access the full Fastly Threat Insights Report, visit www.fastly.com.

About Fastly:

Fastly’s powerful and programmable edge cloud platform helps the world’s top brands deliver online experiences that are fast, safe, and engaging through edge compute, delivery, security, and observability offerings that improve site performance, enhance security, and empower innovation at global scale. Compared to other providers, Fastly’s powerful, high-performance, and modern platform architecture empowers developers to deliver secure websites and apps with rapid time-to-market and demonstrated, industry-leading cost savings. Organizations around the world trust Fastly to help them upgrade the internet experience, including Reddit, Neiman Marcus, Universal Music Group, and SeatGeek. Learn more about Fastly at https://www.fastly.com, and follow us @fastly.

Source: Fastly, Inc.

View source version on businesswire.com: https://www.businesswire.com/news/home/20251218074042/en/