Meta Website Crawling Terms Paused: Nothing Changes on November 20, but Check Your robots.txt
Meta has paused its new commercial terms that would have let its crawlers collect content from websites linked to business accounts, so nothing changes on November 20, 2026. Meta posted the pause on November 5, and the terms that apply to advertisers, Pages owners and publishers today stay in force.
Key facts
- October 15, 2026: Nieman Lab first reported the terms, which touch four documents: the Meta Commercial Terms, Product Catalog Terms, Self-Serve Ad Terms and Ad Creative Generative AI Terms.
- October 16: Meta removed the word "train" and limited future use of scraped content to "advertising and business products."
- October 23: the effective date moved from October 30 to November 20.
- October 30: the UK Competition and Markets Authority opened a Competition Act 1998 investigation into whether the terms abuse a dominant position.
- November 5: Meta said the changes are paused and will not take effect on November 20, with existing terms remaining in effect.
- Courts in Norway and Brazil had issued preliminary injunctions against the terms after lawsuits from media organizations.
What the terms would have done
Businesses accept Meta's terms by using Meta tools such as ads, Facebook Pages and Instagram business accounts. The update would have let Meta's automated crawlers visit websites linked to those accounts and reuse the content in advertising and business products. The CMA says the terms could let Meta collect content from business and publisher websites to build and run its own products, including AI features, with no choice for the site owner.
Publishers objected that the narrowed wording still allows AI training. The News/Media Alliance, which represents more than 2,000 US publishers, wrote to Mark Zuckerberg demanding withdrawal. It argued that blocking Meta is no real option for most publishers because their ad revenue depends on Meta.
Why a pause is no cancellation
Meta says it is "pausing the changes as we continue to engage with stakeholders." Andy Stone, Meta's vice president of communications, said earlier that the consultation "could extend beyond November 20." Meta has not said it will drop the terms, and it has given no new date. A revised version could come back after consultation.
Regulators are not waiting for that. The CMA has made no finding, but it can impose interim measures if temporary action is needed to prevent significant damage. CMA chief executive Sarah Cardell said "where a company holds a dominant position, it must not use that position to impose unfair terms." In the US, Senators Adam Schiff and Marsha Blackburn each wrote to Meta asking it to withdraw the terms.
What robots.txt covers here
A Meta spokesperson said website owners can still use robots.txt to signal their preferences. That is the only control Meta has pointed to, and publishers dispute that it works in practice. If you manage AI crawler access, the lesson is that the file that blocks a crawler can also hide you from AI answers, so decide per bot rather than blocking everything.
What to do this week
- Treat November 20 as cancelled. The date no longer applies.
- Check your robots.txt and decide whether you want Meta's crawlers on your site. Keep a dated copy of what you had in place.
- If you run ads or a business account with Meta, save the current version of the terms now. If revised terms return, you can compare them line by line.
- If you publish content and want a control beyond robots.txt, look at network-level blocking. Cloudflare now blocks AI training crawlers by default on many sites with ads.
- Follow the CMA case page for updates, and read our piece on AI transparency laws in the EU and California for the wider rules.
More analysis is in the AI SaaS Radar article archive.
Questions people ask
Did Meta cancel its website crawling terms?
No. Meta paused them on November 5 and said they will not take effect on November 20. It has not withdrawn them or set a new date, and existing terms remain in effect.
Does robots.txt stop Meta from crawling my site?
Meta's spokesperson says site owners can use robots.txt to signal their preferences. The News/Media Alliance says that is not a real option for most publishers, so treat it as a policy statement and not a guarantee.
What is the CMA investigating?
Since October 30, 2026, the CMA has been examining under the Competition Act 1998 whether Meta holds a dominant position in a relevant UK market and, if so, whether its terms for publishers and advertisers are exploitative. It has made no decision.
Why did courts in Norway and Brazil get involved?
Media organizations in both countries sued, and the courts issued preliminary injunctions blocking the terms.
Sources