Meta's updated Commercial Terms take effect Oct. 30 and grant the company the right to crawl websites associated with any business using its products — not just advertisers — and to use that content to develop, improve, research and train its products. Nieman Lab first reported the change on Oct. 15.

Three legal professionals who reviewed the terms for the outlet believe the language could support an argument by Meta that it is permitted to scrape news sites and use the content to train its AI products. That is legal-analysis speculation, not a confirmed Meta practice.

Meta disputes that reading. On Oct. 16, spokesperson Andy Stone responded on X to a post summarizing the Nieman Lab report and called the summary false. He wrote that the update exists so Meta can pull accurate information about products for sale on third-party websites when it makes ad and content recommendations, and that it is "not about scraping news sites."

What does the new clause allow?

The new crawling paragraph permits Meta, or anyone acting for it, to use web scrapers, bots and other automated tools to crawl and collect content from domains and pages associated with a business's products or services. It also covers URLs made available through the business's use of Meta's products — or through users' use of them.

The clause also lets Meta "create test credentials" to reach those pages.

The scope is broad. Meta's own list of business uses includes:

• Advertising
• Marketing messages
• Listing products on Meta platforms
• Tools such as Ads Manager and Meta Business Suite

The same crawl permission appears in the revised Self-Serve Ad Terms, also effective Oct. 30. Continued business use of Meta's products will be interpreted as acceptance of the new terms. Operators who disagree must stop using Meta's products for business purposes.

Matt Topic, a media and intellectual property attorney representing The Intercept in its copyright case against OpenAI, told Nieman Lab there is no assurance a court will enforce the provision — but publishers should assume it will.

How much of this is actually new?

The current Commercial Terms, dated March 6, 2024, contain no crawling clause at all. The current Self-Serve Ad Terms, dated Aug. 22, 2025, already grant Meta the right to crawl advertiser domains, webpages or URLs when an ad order is placed, with listed uses including "research and training."

Two other existing agreements already carried similar permissions:

• The Ad Creative Generative AI Terms (May 6, 2024) permit crawling URLs advertisers provide for use with Meta's generative AI ad tools.
• The Product Catalog Terms allow periodic catalog updates from catalog URLs.

Meta's Seller Agreement, last updated May 26, 2026, already permits automated software — including creating test credentials — to obtain information from seller-provided URLs.

The October change is the expansion of scope. Prior ad agreements tied crawl permission to placing an ad purchase and covered the advertiser's domain or advertiser-supplied URLs. The new terms tie permission to any business use, and cover URLs related to a business's products or otherwise made available through users' activity on Meta's products.

Separately, Meta announced in its Business Help Center that Product Catalog Terms will move into the Commercial Terms and Ad Creative Generative AI Terms will move into the Self-Serve Ad Terms, to "reduce overlap." That announcement did not reference crawling or training.

Where does robots.txt fit in?

Stone told Nieman Lab the new terms will not change how Meta crawls publishers and that "content owners can still manage how their content is accessed by using robots.txt." Neither updated terms document mentions robots.txt.

The legal experts told the outlet the new language gives Meta an opening to argue its commercial terms supersede other blocking instructions. Cornell law professor James Grimmelmann said test credentials could technically let Meta sign up as a fake user to get past registration walls. Meta declined to answer whether test credentials would bypass paywalls or other paid content.

Meta's crawler documentation names four bots, with different rules:

• Meta-ExternalAgent crawls for training foundation AI models and product improvement by indexing content directly.
• Meta-WebIndexer improves Meta AI search results; Meta says allowing it helps the company cite and link to content in Meta AI responses.
• Meta-ExternalFetcher may bypass robots.txt when fetching pages at a user's request.
• FacebookExternalHit, which creates link previews, may also bypass robots.txt during security or integrity checks.

Because the crawlers are distinct, site operators can block Meta-ExternalAgent while still allowing Meta-WebIndexer. The new terms do not indicate whether a business's robots.txt settings restrict what it agrees to share by accepting the terms.

Who is affected?

The Oct. 30 grant attaches to regular business use of Meta's products. That means it covers brands and clients with a Facebook Page or a Business Suite account — even those that never buy ads. Publishers face the sharpest exposure, since the terms cover URLs users share, not only URLs the business itself supplies.

What happens next?

Website operators have roughly two weeks from the Oct. 15 report to decide whether to continue business use of Meta products and to review how their robots.txt files treat each Meta crawler. The European Publishers Council is also writing to the European Commission about the terms, specifically concerned that Meta may claim permission to use an article whenever a user shares a link to that story on social media — a dispute worth monitoring as the Oct. 30 effective date passes.