k
keynews.ai
keynews.ai
WorkThe story, in brief

The rise of the AI crawler

569 million requests in one month. OpenAI's GPTBot is now 1 in 8 web requests—and nobody's talking about what that means for your infrastructure.

Reporting by Vercel Blog 1 min brief
Illustration of two anonymous hands arranging task cards around an amber tool on a shared desk.
People, judgement and the changing nature of work.AI illustration by KeyNews
The KeyNews take

Why it matters

AI crawlers are reshaping web infrastructure and data consumption patterns at scale. Understanding crawler behavior and impact is becoming critical for founders building on the web stack and for policy discussions around AI data sourcing.

The key facts

5 to know

  1. OpenAI GPTBot: 569M requests/month across Vercel network

  2. Anthropic Claude crawler: 370M requests/month

  3. Combined AI crawler volume = ~20% of Googlebot's 4.5B requests/month

  4. Data source: Vercel infrastructure analysis (Dec 2024)

  5. Focus: JavaScript handling, content prioritization, web navigation patterns

Go to the source

Vercel Blogvercel.com

Publisher excerpt: AI crawlers have become a significant presence on the web. OpenAI's GPTBot generated 569 million requests across Vercel's network in the past month, while Anthropic's Claude followed with 370 million. For perspective, this combined volume represents about 20% of Googlebot's 4.5 billion requests…
Read original report
Back to today's editionMore work news

The wider picture

View all
Illustration of two anonymous hands arranging task cards around an amber tool on a shared desk.
AI illustration by KeyNews
Work01

Trump now says he wants to form an ‘AI Force’

A major political signal on AI governance: the administration is positioning itself to accelerate rather than constrain AI development, with formal institutional backing (czar + task force). Practitioners and policy-watchers need to know the regulatory stance is shifting toward facilitation.

The Verge AI
Illustration of two anonymous hands arranging task cards around an amber tool on a shared desk.
AI illustration by KeyNews
Work02

The 'robot relations' department may become reality in workplace of the future

As corporations deploy autonomous systems across operations, workers face real changes to pay, autonomy, and job structure. The organizational and policy implications of managing human-AI work dynamics are becoming immediate workplace issues.

CNBC Technology
Paper-cut illustration of an amber microchip with circuit paths extending into a row of data-center cabinets.
AI illustration by KeyNews
Work03

AI, data center alarms dominate Congressional Black Caucus week in Washington

AI regulation and data-center expansion are now front-and-center in a major political forum, signaling emerging consensus-building around policy that will affect enterprise AI deployment and the communities hosting compute infrastructure.

CNBC Technology