Site AI Access Check
Shows which AI crawlers and agents a site lets in: robots.txt rules per agent for a given path, llms.txt, noai page tags and AI usage signals, with a plain verdict for each.
- Price per call
- 1 credit
- per successful call
- Price ceiling
- 1 credit
- every call, same price
- Failed calls
- Free
- retried automatically
- Billed
- Per call
- only on success
Endpoint
api.primaryapi.com/ v1/ run/ site. ai_accessAuth: API key with run or full access, or a dashboard session
Send the input below as a JSON body. The call is routed to a healthy upstream; if an attempt fails, it is retried automatically inside the same request and you pay only for the answer you get. To check an input and see the exact price without running anything, call POST /v1/quote/site.ai_access with the same body (free).
Example request
curl -sS -X POST "https://api.primaryapi.com/v1/run/site.ai_access" \
-H "Authorization: Bearer $PRIMARYAPI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://www.nytimes.com/section/technology"}'const res = await fetch('https://api.primaryapi.com/v1/run/site.ai_access', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.PRIMARYAPI_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
url: 'https://www.nytimes.com/section/technology',
}),
});
const body = await res.json();
if (!res.ok) throw new Error(`${body.code}: ${body.error}`);
// found:false is a successful answer ("does not exist"), not an error
console.log(body.output.found ? body.output.data : body.output.reason, body.credits);import os
import requests
res = requests.post(
"https://api.primaryapi.com/v1/run/site.ai_access",
headers={"Authorization": f"Bearer {os.environ['PRIMARYAPI_API_KEY']}"},
json={
"url": "https://www.nytimes.com/section/technology",
},
timeout=60,
)
body = res.json()
if not res.ok:
raise RuntimeError(f"{body['code']}: {body['error']}")
# found=False is a successful answer ("does not exist"), not an error
print(body["output"]["data"] if body["output"]["found"] else body["output"]["reason"], body["credits"]){
"name": "run_api",
"arguments": {
"sku": "site.ai_access",
"input": {
"url": "https://www.nytimes.com/section/technology"
}
}
}Price this exact input first (free)
curl -sS -X POST "https://api.primaryapi.com/v1/quote/site.ai_access" \
-H "Authorization: Bearer $PRIMARYAPI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://www.nytimes.com/section/technology"}'Returns credits (what the call is expected to cost) and maxCredits (the most it can cost, which is set aside while it runs). Invalid input fails here the same way it would on run.
Input
JSON body fields. Unknown fields are rejected with 400 invalid_input (never silently ignored). You may add the routing controls preferLatencyUnderMs and maxCredits to the same object.
urlstringrequired- Site or page to check: a domain ("nytimes.com") or a URL. The URL path (default "/") is what the rules are matched against.max 2048 chars
agentsstring[]- Agent tokens to report (for example ["GPTBot", "ClaudeBot"]). Default: the built-in list of about 25 documented AI crawlers and fetchers. Unknown tokens are evaluated too.min 1 items · max 40 items · items pattern: ^[A-Za-z_-]{1,64}$
checkPagebooleandefaulttrue- Also fetch the page to read its robots meta tag and X-Robots-Tag header (noai, noimageai, noindex).
Output
When output.found is true, output.data has these fields.
urlstring- URL that was evaluated (scheme and host normalised).
originstringpathstring- Path plus query that the rules were matched against.
robotsTxtobjectrobotsTxt.urlstringrobotsTxt.statusenum- ok: parsed. not_found: a 4xx answer, which means no rules apply. unreachable: 5xx or a network error, which means complete disallow. redirect_limit: more than 5 redirects (treated as not found). too_large: over 2 MiB, not evaluated.one of: ok, not_found, unreachable, redirect_limit, too_large
robotsTxt.httpStatusinteger | nullrobotsTxt.groupsinteger- User-agent groups found.
robotsTxt.sitemapsstring[]- Sitemap URLs listed in the file, sorted.
robotsTxt.contentSignalobject | null- Content-Signal line of the * group (search, ai-input, ai-train = yes/no); null when absent.
robotsTxt.contentSignal.searchboolean | nullrobotsTxt.contentSignal.aiInputboolean | nullrobotsTxt.contentSignal.aiTrainboolean | nullrobotsTxt.aiTrainingPreferenceenum | null- Preference for the path from a "User-Agent: __usage_ai-training" group (IETF draft, honoured by some vendors); null when the file has no such group. The * group never counts.one of: disallowed, allowed
llmsTxtobjectllmsTxt.urlstringllmsTxt.presentbooleanllmsTxt.titlestring | null- The file's H1 title (the only required part of llms.txt).
llmsFullTxtobjectllmsFullTxt.urlstringllmsFullTxt.presentbooleanpageobject- Page-level signals of the URL itself.
page.checkedboolean- false when the page was not fetched (robots.txt disallows us, or checkPage=false).
page.httpStatusinteger | nullpage.robotsMetastring[]- Directives from <meta name="robots"> and X-Robots-Tag, lower-cased and sorted.
page.noaiboolean- "noai" is set (a voluntary opt-out first published by DeviantArt; not a standard).
page.noimageaibooleanpage.noindexbooleansummaryobjectsummary.agentsintegersummary.allowedintegersummary.blockedintegersummary.unknownintegeragentsobject[]- One row per agent, in the order requested (default: the built-in list).
agents[].tokenstring- robots.txt user-agent token, as the vendor documents it.
agents[].operatorstring | null- Company that runs the agent; null for a token you supplied.
agents[].purposeenum- training: collects data for model training. training_control: a token that does not crawl, only controls training use of what another crawler fetched. search: indexes for AI search answers. user_fetch: fetches a page when a person asks. legacy: retired token still common in robots.txt files. custom: a token you supplied.one of: training, training_control, search, user_fetch, legacy, custom
agents[].respectsRobotsenum- What the operator's own docs say about obeying robots.txt (partial: may skip it for user requests).one of: yes, partial, no, unknown
agents[].docsUrlstring | null- Operator's documentation of the agent.
agents[].groupenum- Which robots.txt group applied: one naming this agent, the * group, or none.one of: agent, wildcard, none
agents[].allowedboolean | null- RFC 9309 verdict for the path; null when robots.txt could not be read.
agents[].matchedRulestring | null- The deciding rule, for example "Disallow: /private/" (longest match wins, Allow wins ties).
agents[].siteBlockedboolean | null- true when the agent may not fetch the home page "/" either.
agents[].crawlDelaynumber | null- Crawl-delay seconds in the applying group (non-standard, informative).
agents[].contentUsagestring | null- Content-Usage preference that applies to the path in this group (IETF aipref draft), e.g. "train-ai=n".
agents[].verdictenum- allowed: robots.txt lets this agent fetch the path. blocked: a rule (or an unreachable robots.txt, which RFC 9309 treats as disallow-all) stops it. unknown: robots.txt could not be read in full.one of: allowed, blocked, unknown
agents[].explanationstring- One plain sentence that explains the verdict.
TypeScript type
/** output.data when output.found is true */
type SiteAiAccess = {
/** URL that was evaluated (scheme and host normalised). */
url: string;
origin: string;
/** Path plus query that the rules were matched against. */
path: string;
robotsTxt: {
url: string;
/** ok: parsed. not_found: a 4xx answer, which means no rules apply. unreachable: 5xx or a network error, which means complete disallow. redirect_limit: more than 5 redirects (treated as not found). too_large: over 2 MiB, not evaluated. */
status: "ok" | "not_found" | "unreachable" | "redirect_limit" | "too_large";
httpStatus: number | null;
/** User-agent groups found. */
groups: number;
/** Sitemap URLs listed in the file, sorted. */
sitemaps: string[];
/** Content-Signal line of the * group (search, ai-input, ai-train = yes/no); null when absent. */
contentSignal: {
search: boolean | null;
aiInput: boolean | null;
aiTrain: boolean | null;
} | null;
/** Preference for the path from a "User-Agent: __usage_ai-training" group (IETF draft, honoured by some vendors); null when the file has no such group. The * group never counts. */
aiTrainingPreference: "disallowed" | "allowed" | null;
};
llmsTxt: {
url: string;
present: boolean;
/** The file's H1 title (the only required part of llms.txt). */
title: string | null;
};
llmsFullTxt: {
url: string;
present: boolean;
};
/** Page-level signals of the URL itself. */
page: {
/** false when the page was not fetched (robots.txt disallows us, or checkPage=false). */
checked: boolean;
httpStatus: number | null;
/** Directives from <meta name="robots"> and X-Robots-Tag, lower-cased and sorted. */
robotsMeta: string[];
/** "noai" is set (a voluntary opt-out first published by DeviantArt; not a standard). */
noai: boolean;
noimageai: boolean;
noindex: boolean;
};
summary: {
agents: number;
allowed: number;
blocked: number;
unknown: number;
};
/** One row per agent, in the order requested (default: the built-in list). */
agents: {
/** robots.txt user-agent token, as the vendor documents it. */
token: string;
/** Company that runs the agent; null for a token you supplied. */
operator: string | null;
/** training: collects data for model training. training_control: a token that does not crawl, only controls training use of what another crawler fetched. search: indexes for AI search answers. user_fetch: fetches a page when a person asks. legacy: retired token still common in robots.txt files. custom: a token you supplied. */
purpose: "training" | "training_control" | "search" | "user_fetch" | "legacy" | "custom";
/** What the operator's own docs say about obeying robots.txt (partial: may skip it for user requests). */
respectsRobots: "yes" | "partial" | "no" | "unknown";
/** Operator's documentation of the agent. */
docsUrl: string | null;
/** Which robots.txt group applied: one naming this agent, the * group, or none. */
group: "agent" | "wildcard" | "none";
/** RFC 9309 verdict for the path; null when robots.txt could not be read. */
allowed: boolean | null;
/** The deciding rule, for example "Disallow: /private/" (longest match wins, Allow wins ties). */
matchedRule: string | null;
/** true when the agent may not fetch the home page "/" either. */
siteBlocked: boolean | null;
/** Crawl-delay seconds in the applying group (non-standard, informative). */
crawlDelay: number | null;
/** Content-Usage preference that applies to the path in this group (IETF aipref draft), e.g. "train-ai=n". */
contentUsage: string | null;
/** allowed: robots.txt lets this agent fetch the path. blocked: a rule (or an unreachable robots.txt, which RFC 9309 treats as disallow-all) stops it. unknown: robots.txt could not be read in full. */
verdict: "allowed" | "blocked" | "unknown";
/** One plain sentence that explains the verdict. */
explanation: string;
}[];
};If what you asked about does not exist, is private, or a search matches nothing, the call still succeeds: output.found is false, output.data is null and output.reason says why. Such misses are free unless the notes below say misses are billed, in which case a miss costs the price of one call.
Example response
Values are placeholders that follow the output schema (strings as "string", numbers as 0); credits shows what this call would cost at the usual price.
{
"output": {
"found": true,
"data": {
"url": "https://www.nytimes.com/section/technology",
"origin": "string",
"path": "string",
"robotsTxt": {
"url": "https://www.nytimes.com/section/technology",
"status": "ok",
"httpStatus": 0,
"groups": 0,
"sitemaps": [
"string"
],
"contentSignal": {
"search": true,
"aiInput": true,
"aiTrain": true
},
"aiTrainingPreference": "disallowed"
},
"llmsTxt": {
"url": "https://www.nytimes.com/section/technology",
"present": true,
"title": "string"
},
"llmsFullTxt": {
"url": "https://www.nytimes.com/section/technology",
"present": true
},
"page": {
"checked": true,
"httpStatus": 0,
"robotsMeta": [
"string"
],
"noai": true,
"noimageai": true,
"noindex": true
},
"summary": {
"agents": 0,
"allowed": 0,
"blocked": 0,
"unknown": 0
},
"agents": [
{
"token": "string",
"operator": "string",
"purpose": "training",
"respectsRobots": "yes",
"docsUrl": "https://example.com",
"group": "agent",
"allowed": true,
"matchedRule": "string",
"siteBlocked": true,
"crawlDelay": 0,
"contentUsage": "string",
"verdict": "allowed",
"explanation": "string"
}
]
}
},
"provider": "PrimaryAPI",
"credits": 1,
"items": 1,
"requestId": "req_01JBXK6Y2N9V8T4R3Q2P1M0A7Z"
}Pricing and reliability
Calls go only to upstreams that are healthy right now. If an attempt fails, the call is retried automatically inside the same request; a failed attempt costs nothing, and a call never costs more than its quote’s maxCredits. Live figures below are measured on real traffic.
Success rate, uptime, latency and call counts are measured live from real traffic over 30 days and show “—” until there is data.
Errors
Errors are JSON { code, error, requestId } and are never charged. The ones to handle for this endpoint:
- invalid_inputThe request failed validation before anything ran.
- unauthorizedNo valid API key or session was presented.
- insufficient_balanceThe balance cannot cover the most this call could cost.
- key_limit_reachedThe key’s own spend limit is used up for the current window.
- trial_cap_reachedAn agent trial key has used its whole trial budget.
- forbiddenThe credential is valid but may not do this.
- not_foundThe thing addressed by the URL does not exist for you.
- idempotency_conflictThe
Idempotency-Keyis already bound to a different request. - rate_limitedToo many requests in the current window.
- all_sources_failedThe call could not be answered, even after automatic retries.
- service_unavailableThis API cannot run in this deployment right now.
- timeoutThe request passed its deadline before it finished.
Notes
Follows RFC 9309: the most specific matching rule wins (Allow on a tie), the * group applies only to agents without their own group, a 4xx robots.txt means no restrictions and a 5xx or unreachable one means complete disallow. A host that does not resolve is a free miss (reason domain_not_found). robots.txt is a request, not an access control: several vendors document user-triggered fetchers that may ignore it (see respectsRobots). Output is stable between runs, so a monitor fires only when a rule, file or tag changes.
Generated from the PrimaryAPI catalog. Machine-readable contract: GET https://api.primaryapi.com/v1/apis/site.ai_access.