What happened
On July 1, 2025, Cloudflare — infrastructure for roughly 20% of the web — declared "Content Independence Day": AI crawlers are now blocked by default, and without permission or compensation there is no access to the content (Cloudflare).
Launched alongside it is Pay Per Crawl (private beta) — a marketplace where a publisher sets a price for each page crawl. A crawler gets three options: Allow (free), Charge (paid access at the domain's price) or Block. AI companies are required to declare the purpose of the crawl: training, inference or search (Nieman Lab).
Why it matters
The era of free "if I can see it, it's mine" scraping ended administratively: not by law, but by an infrastructure switch. The economics of model training gained a new line item, and publishers gained leverage the courts never handed them.
What it means for vibe coding
For tools with web access (research agents, RAG over the live web) it means part of the content becomes paid or unavailable, and the quality of "models with search" starts to depend on the vendor's deals with publishers. If you're building an agent that crawls — respect the new rules, or your IP reputation will burn fast.