Search crawler and model-training access settings
Documentation / Search visibility and sharing
Search crawling and model-training collection are different uses of website content. A policy for one should not be assumed to express the other, and neither replaces authentication for material that must remain private.
Understand the implemented control
The engine serves /robots.txt. When a custom robotsTxt value is configured, it serves that text directly. Otherwise, a private site receives a general disallow rule. The normal public default permits crawling while disallowing admin and API paths and identifying the sitemap.
The verified current CMS general settings and page SEO tab do not expose dedicated search-crawler and model-training switches. Do not assume those independent controls are available because broader reference material mentions them. Ask an administrator to inspect the effective configuration and response when a specific crawler policy is required.
Custom robots text replaces the generated default rather than merging with it. An administrator preparing a policy must therefore preserve any required sitemap and route guidance deliberately. This article does not establish a native editor for that raw setting or automatic management of every crawler identity.
Keep access decisions separate
Robots rules are requests to cooperating crawlers. They do not stop a visitor who knows the address, remove all existing copies, or guarantee that a training dataset excludes previously collected material.
Use page availability and site access controls for protection. Use Exclude this page from search indexing for a page-level indexing request, understanding that it does not restrict access or automatically define a model-training policy.
Validate the effective policy
Open the site's actual /robots.txt and inspect page robots metadata separately. Confirm the intended crawler-specific behavior with your administrator before publication.