The new F5 Labs report finds that over half of page requests for web content are now automated as use of LLM scrapers intensifies.
A new report from F5 Labs shows that bots, driven by the rise of generative AI, are now accessing certain types of online content more than humans. According to the 2025 Advanced Persistent Bots Report, which analysed over 207bn web and API transactions between November 2023 and September 2024, automated systems—many used by AI companies—now dominate web activity in some categories.
The data reveals that over half (50.04%) of all page requests for web content came from bots, outpacing human traffic in this area. In comparison, only 22.3% of search requests and 21.5% of “Add to Cart” actions were bot-driven. Much of this surge is attributed to content scrapers used by large language model developers like OpenAI, Anthropic and Perplexity, many of which persist in sending requests even after being blocked.
Out of the total traffic, 10.2%—or 21.22bn transactions—were automated, including 4.8% from malicious bots.
David Warburton, Director, F5 Labs, said: “For years, bot traffic has primarily been targeted at Search flows, as well as aspects of the user journey where someone signs up or logs in to use a service, adds an item to their basket, checks out, or seeks to change their password. The huge upsurge in content scraping, undoubtedly associated with the explosion of generative AI and LLMs, underlines how dynamic bot traffic is and the need for organisations to be constantly on watch for changes in attack patterns.”
The report also highlighted varying impacts across industries. On the web, hospitality (44.6%), healthcare (32.6%), and eCommerce (22.7%) were the most targeted sectors, while on mobile, entertainment led with 23% of its traffic coming from bots. Credential stuffing attacks were most common in the tech industry, with 33.5% of login attempts flagged as potential account takeovers.
Bot sophistication also differed by sector. Healthcare faced mostly basic bot traffic, whereas advanced bots increasingly targeted retail, banking, and airlines—industries that have already deployed stronger defences.
Interestingly, while most sectors saw a year-on-year decline in bot traffic, hospitality and quick-service restaurants (QSR) bucked the trend, with increases of 18.3% and 11.2% respectively.
The report examined mitigation efforts as well. While mobile platforms showed significant success in reducing bot activity through active defences, the web presented a paradox: organisations actively mitigating bots often recorded higher levels of bot traffic. F5 suggests this is due to the persistence of operators whose business models rely on scraping valuable content and data.
Warburton added: “Certain industries are perennial targets for unwanted bot traffic. Hospitality experiences high volumes, because aggregators want to scrape hotel room rate and availability data, or malicious actors, are trying to steal loyalty points. Similarly, eCommerce providers are targeted by resellers and bots trained to exploit voucher and gift card details.”
On mobile, the trend was clear and expected. Organisations mitigating traffic saw a significantly lower share of automated activity in their search traffic (0.9% compared with 24.8% for those just monitoring), a pattern matched in login (5% for mitigators compared with 21.7% for those monitoring) and sign-up (2.4% compared with 21.7%).
On the web, it was a different story. In most workflows, automated traffic was higher for organizations that were actively mitigating bots. These customers saw 20.9% of automated traffic in search versus 14.9% for those simply monitoring, and the equation was the same in Add to Cart (19.2% vs. 18.2%), Checkout (8.6% vs. 7.4%) and Account Recovery (6.6% vs. 4.6%).
Warburton concluded: “Typically, we’d expect mitigation to lead to a decline in bot traffic, as operators that are blocked move on in search of weaker targets. However, there are now whole business models built around the scraping of data, prices, and intellectual property: those operators are not going to give up easily when they are deterred. An increase in traffic can mean these actors are trying harder, and in more ways—not necessarily that they are succeeding. The consistent trend of this research, and all of our experience at F5, is that mitigation works, and deterrence makes a difference.”
