# Site AI Access Check (`site.ai_access`)

Shows which AI crawlers and agents a site lets in: robots.txt rules per agent for a given path, llms.txt, noai page tags and AI usage signals, with a plain verdict for each.

Docs: https://primaryapi.com/docs/api-reference/site/ai_access

- Endpoint: `POST https://api.primaryapi.com/v1/run/site.ai_access`
- Category: Web Intelligence · Platform: `site` · Kind: lookup · Status: live
- Price: 1 credit per call. Every call costs the same, including calls that were retried automatically.
- Billing unit: per successful call
- Reliability: routed to a healthy upstream and retried automatically when an attempt fails; live 30-day status: `GET https://api.primaryapi.com/v1/status/apis?sku=site.ai_access`
- Execution: synchronous (answers 202 only if it outlives the wait window)

## Input

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `url` | string | yes |  | max 2048 chars | Site or page to check: a domain ("nytimes.com") or a URL. The URL path (default "/") is what the rules are matched against. |
| `agents` | string[] | no |  | min 1 items; max 40 items; items pattern: ^[A-Za-z_-]{1,64}$ | Agent tokens to report (for example ["GPTBot", "ClaudeBot"]). Default: the built-in list of about 25 documented AI crawlers and fetchers. Unknown tokens are evaluated too. |
| `checkPage` | boolean | no | true |  | Also fetch the page to read its robots meta tag and X-Robots-Tag header (noai, noimageai, noindex). |

Unknown fields are rejected with `400 invalid_input`. The routing controls `preferLatencyUnderMs` and `maxCredits` may be sent in the same JSON body.

## Output

When `output.found` is `true`, `output.data` has this shape:

| Field | Type | Description |
|---|---|---|
| `url` | string | URL that was evaluated (scheme and host normalised). |
| `origin` | string |  |
| `path` | string | Path plus query that the rules were matched against. |
| `robotsTxt` | object |  |
| `robotsTxt.url` | string |  |
| `robotsTxt.status` | enum | ok: parsed. not_found: a 4xx answer, which means no rules apply. unreachable: 5xx or a network error, which means complete disallow. redirect_limit: more than 5 redirects (treated as not found). too_large: over 2 MiB, not evaluated. |
| `robotsTxt.httpStatus` | integer \| null |  |
| `robotsTxt.groups` | integer | User-agent groups found. |
| `robotsTxt.sitemaps` | string[] | Sitemap URLs listed in the file, sorted. |
| `robotsTxt.contentSignal` | object \| null | Content-Signal line of the * group (search, ai-input, ai-train = yes/no); null when absent. |
| `robotsTxt.contentSignal.search` | boolean \| null |  |
| `robotsTxt.contentSignal.aiInput` | boolean \| null |  |
| `robotsTxt.contentSignal.aiTrain` | boolean \| null |  |
| `robotsTxt.aiTrainingPreference` | enum \| null | Preference for the path from a "User-Agent: __usage_ai-training" group (IETF draft, honoured by some vendors); null when the file has no such group. The * group never counts. |
| `llmsTxt` | object |  |
| `llmsTxt.url` | string |  |
| `llmsTxt.present` | boolean |  |
| `llmsTxt.title` | string \| null | The file's H1 title (the only required part of llms.txt). |
| `llmsFullTxt` | object |  |
| `llmsFullTxt.url` | string |  |
| `llmsFullTxt.present` | boolean |  |
| `page` | object | Page-level signals of the URL itself. |
| `page.checked` | boolean | false when the page was not fetched (robots.txt disallows us, or checkPage=false). |
| `page.httpStatus` | integer \| null |  |
| `page.robotsMeta` | string[] | Directives from <meta name="robots"> and X-Robots-Tag, lower-cased and sorted. |
| `page.noai` | boolean | "noai" is set (a voluntary opt-out first published by DeviantArt; not a standard). |
| `page.noimageai` | boolean |  |
| `page.noindex` | boolean |  |
| `summary` | object |  |
| `summary.agents` | integer |  |
| `summary.allowed` | integer |  |
| `summary.blocked` | integer |  |
| `summary.unknown` | integer |  |
| `agents` | object[] | One row per agent, in the order requested (default: the built-in list). |
| `agents[].token` | string | robots.txt user-agent token, as the vendor documents it. |
| `agents[].operator` | string \| null | Company that runs the agent; null for a token you supplied. |
| `agents[].purpose` | enum | training: collects data for model training. training_control: a token that does not crawl, only controls training use of what another crawler fetched. search: indexes for AI search answers. user_fetch: fetches a page when a person asks. legacy: retired token still common in robots.txt files. custom: a token you supplied. |
| `agents[].respectsRobots` | enum | What the operator's own docs say about obeying robots.txt (partial: may skip it for user requests). |
| `agents[].docsUrl` | string \| null | Operator's documentation of the agent. |
| `agents[].group` | enum | Which robots.txt group applied: one naming this agent, the * group, or none. |
| `agents[].allowed` | boolean \| null | RFC 9309 verdict for the path; null when robots.txt could not be read. |
| `agents[].matchedRule` | string \| null | The deciding rule, for example "Disallow: /private/" (longest match wins, Allow wins ties). |
| `agents[].siteBlocked` | boolean \| null | true when the agent may not fetch the home page "/" either. |
| `agents[].crawlDelay` | number \| null | Crawl-delay seconds in the applying group (non-standard, informative). |
| `agents[].contentUsage` | string \| null | Content-Usage preference that applies to the path in this group (IETF aipref draft), e.g. "train-ai=n". |
| `agents[].verdict` | enum | allowed: robots.txt lets this agent fetch the path. blocked: a rule (or an unreachable robots.txt, which RFC 9309 treats as disallow-all) stops it. unknown: robots.txt could not be read in full. |
| `agents[].explanation` | string | One plain sentence that explains the verdict. |

When the thing you asked about does not exist (or a search matches nothing) the call still succeeds: `output.found` is `false`, `output.data` is `null` and `output.reason` says why (`not_found`, `private`, `no_results`, …). Misses are free unless the notes below say misses are billed.

## Example request

```bash
curl -sS -X POST "https://api.primaryapi.com/v1/run/site.ai_access" \
  -H "Authorization: Bearer $PRIMARYAPI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://www.nytimes.com/section/technology"}'
```

## Example response

_Values are placeholders that follow the output schema; `credits` is what this call would cost at the usual price._

```json
{
  "output": {
    "found": true,
    "data": {
      "url": "https://www.nytimes.com/section/technology",
      "origin": "string",
      "path": "string",
      "robotsTxt": {
        "url": "https://www.nytimes.com/section/technology",
        "status": "ok",
        "httpStatus": 0,
        "groups": 0,
        "sitemaps": [
          "string"
        ],
        "contentSignal": {
          "search": true,
          "aiInput": true,
          "aiTrain": true
        },
        "aiTrainingPreference": "disallowed"
      },
      "llmsTxt": {
        "url": "https://www.nytimes.com/section/technology",
        "present": true,
        "title": "string"
      },
      "llmsFullTxt": {
        "url": "https://www.nytimes.com/section/technology",
        "present": true
      },
      "page": {
        "checked": true,
        "httpStatus": 0,
        "robotsMeta": [
          "string"
        ],
        "noai": true,
        "noimageai": true,
        "noindex": true
      },
      "summary": {
        "agents": 0,
        "allowed": 0,
        "blocked": 0,
        "unknown": 0
      },
      "agents": [
        {
          "token": "string",
          "operator": "string",
          "purpose": "training",
          "respectsRobots": "yes",
          "docsUrl": "https://example.com",
          "group": "agent",
          "allowed": true,
          "matchedRule": "string",
          "siteBlocked": true,
          "crawlDelay": 0,
          "contentUsage": "string",
          "verdict": "allowed",
          "explanation": "string"
        }
      ]
    }
  },
  "provider": "PrimaryAPI",
  "credits": 1,
  "items": 1,
  "requestId": "req_01JBXK6Y2N9V8T4R3Q2P1M0A7Z"
}
```

## Pricing and reliability

- Usual price: 1 credit per call
- Price ceiling: the usual price, even when a call is retried. A quote’s `maxCredits` is the most a given call can cost; the `maxCredits` routing control caps it per call
- Live 30-day status, success rate, uptime and latency: `GET https://api.primaryapi.com/v1/status/apis?sku=site.ai_access`

## Errors to handle

`invalid_input`, `unauthorized`, `insufficient_balance`, `key_limit_reached`, `trial_cap_reached`, `forbidden`, `not_found`, `idempotency_conflict`, `rate_limited`, `all_sources_failed`, `service_unavailable`, `timeout`. See https://primaryapi.com/docs/errors.

## Notes

Follows RFC 9309: the most specific matching rule wins (Allow on a tie), the * group applies only to agents without their own group, a 4xx robots.txt means no restrictions and a 5xx or unreachable one means complete disallow. A host that does not resolve is a free miss (reason domain_not_found). robots.txt is a request, not an access control: several vendors document user-triggered fetchers that may ignore it (see respectsRobots). Output is stable between runs, so a monitor fires only when a rule, file or tag changes.

