Best practices for the Extract Structured API

Some Extract Structured integrations generate errors or reprocess files that already completed successfully. That consumes monthly AI unit quotas without producing new metadata. Use this guide to pace jobs, persist results, and design schemas so each file is extracted once. For parameter details and SDK samples, see the Extract metadata (structured) API reference and the Extract metadata from file (structured) guide.

This guide covers the Extract Structured API (POST /2.0/ai/extract_structured). For folder-based extraction without custom code, configure Box Extract in the Box web app.

How extraction works

Box AI runs conversion, OCR and layout parsing, then an agentic extraction loop against your schema. You send one file per request. Typical processing time is 10 to 45 seconds, depending on page count and layout density.

When to use the API

Use the Extract Structured API when you need programmatic control: agent overrides, token limits, inline fields, confidence scores, or integration into an existing pipeline.
Use Box Extract in the product UI when users upload files to a folder and you want extraction without API orchestration. Box Extract can reduce development complexity because you don’t build or maintain the integration yourself. To learn more, see Box Extract.

Configure Extract Agent panel in Box Extract with a metadata template, AI agent, and extraction policy

Before you start

Confirm Box AI is enabled for the enterprise.
Confirm monthly processing capacity matches the workload. For a large backfill, request an allowance increase before you run the job.
For a large migration or archive backfill, notify your Box account team so capacity can be planned.
Enable Manage AI on the app, plus file read (and write if you attach metadata to files).
Validate the schema and expected output on a small sample of files before a production run.

Design for one extraction per file

Extraction jobs run independently. Store extracted metadata on the file with a Box metadata template, or in your own store. Do not call extract again for a file that already has a successful result. Use a metadata template when the schema is stable and you want values indexed in Box. Use inline fields for one-off or changing schemas.

Configure the request

Choose an agent

The standard agent is enough for straightforward documents. For multi-column layouts, scanned files, dense tables, or nested struct and table fields, use the Enhanced Extract Agent:

{
  "ai_agent": {
    "type": "ai_agent_id",
    "id": "enhanced_extract_agent"
  }
}

Successful responses include ai_agent_info with the model and processor that ran. Check it during development to confirm the override is active.

Set client timeouts

Set the HTTP client timeout to 120 seconds or higher so multi-page documents are not cancelled while the model is still working.

Keep schemas focused

More fields increase processing time. Group related attributes into smaller templates when you have a large set of fields, for example billing fields in one template and vendor contact details in another.

Raise completion tokens for long answers

If the schema asks for long summaries, clause excerpts, or many table rows, raise num_tokens_for_completion in the agent override:

{
  "items": [{ "id": "123456789", "type": "file" }],
  "fields": [
    {
      "key": "contract_summary",
      "type": "string",
      "prompt": "Summarize the key obligations."
    }
  ],
  "ai_agent": {
    "type": "ai_agent_extract_structured",
    "basic_text": { "num_tokens_for_completion": 4096 },
    "long_text": { "num_tokens_for_completion": 4096 }
  }
}

Check confidence and citations

Set include_confidence_score and include_reference to true when you need per-field confidence and document citations for automated review. Outputs can vary slightly across runs. Validate against the schema and use confidence scores before writing metadata or triggering downstream work.

Pace and concurrency

Run 8 or more parallel workers.
In each worker, send the next request only after the previous one finishes.
Start with a small batch, then raise concurrency.
When you can, process shorter documents first. Group 50-page or heavily scanned files for off-peak runs.

A burst of hundreds of requests at once performs worse than a steady stream. If the API returns 412 or 429, wait and resume at a slightly lower rate so cluster capacity can catch up.

Example: paced batch runner

This example uses concurrent workers, exponential backoff, and box_sdk_gen for JWT auth. Replace the file IDs and field list with your own.

import json
import logging
import time
from concurrent.futures import ThreadPoolExecutor, as_completed

import requests
from box_sdk_gen import BoxJWTAuth, JWTConfig

logging.basicConfig(level=logging.INFO)
LOGGER = logging.getLogger(__name__)

JWT_CONFIG_PATH = "config.json"
USER_ID = ""
FILE_IDS = []

NUM_PARALLEL_REQUESTS = 8
TIMEOUT_SECONDS = 120
MAX_RETRIES = 6
BACKOFF_INITIAL_SECONDS = 30
BACKOFF_MAX_SECONDS = 480

RETRYABLE_STATUS_CODES = [412, 429, 503, 544]

FIELDS = [
    {
        "type": "string",
        "key": "name",
        "displayName": "Name",
        "description": "The document name",
        "prompt": "What is the name of this document?",
    },
    {
        "type": "string",
        "key": "document_number",
        "displayName": "Document Number",
        "description": "The document number",
        "prompt": "The document number",
    },
]


def send_request(auth: BoxJWTAuth, file_id: str):
    response = None

    for attempt in range(MAX_RETRIES + 1):
        try:
            response = requests.post(
                "https://api.box.com/2.0/ai/extract_structured",
                headers={
                    "content-type": "application/json",
                    "authorization": f"Bearer {auth.retrieve_token().access_token}",
                },
                json={
                    "items": [{"id": file_id, "type": "file"}],
                    "fields": FIELDS,
                },
                timeout=TIMEOUT_SECONDS,
            )
        except Exception as exc:
            LOGGER.info("Transport error for file %s: %s", file_id, exc)
            response = None

        if response is not None and response.status_code == 200:
            break

        status = None if response is None else response.status_code
        if status is not None and status not in RETRYABLE_STATUS_CODES:
            break

        if attempt < MAX_RETRIES:
            wait_seconds = min(
                BACKOFF_INITIAL_SECONDS * (2 ** attempt),
                BACKOFF_MAX_SECONDS,
            )
            LOGGER.info(
                "Pacing retry for file %s in %ss (status %s)",
                file_id,
                wait_seconds,
                status,
            )
            time.sleep(wait_seconds)

    return response


if __name__ == "__main__":
    with open(JWT_CONFIG_PATH, "rt") as file:
        config_json = json.dumps(json.load(file))

    auth = BoxJWTAuth(
        config=JWTConfig.from_config_json_string(config_json_string=config_json)
    )
    if USER_ID:
        auth = auth.with_user_subject(USER_ID)

    results = []
    with ThreadPoolExecutor(max_workers=NUM_PARALLEL_REQUESTS) as executor:
        futures = {
            executor.submit(send_request, auth, fid): fid for fid in FILE_IDS
        }
        for future in as_completed(futures):
            try:
                results.append(future.result())
            except Exception as exc:
                LOGGER.info(
                    "Failed for file %s", futures[future], exc_info=exc
                )
                results.append(None)

    succeeded = sum(
        1 for r in results if r is not None and r.status_code == 200
    )
    LOGGER.info(
        "Batch complete — %s/%s files successfully processed.",
        succeeded,
        len(FILE_IDS),
    )

Status codes

Status	Action	Description
`412`, `429`	Exponential backoff and retry	Cluster capacity is scaling. Wait, then resume.
`503`, `544`	Retry with backoff	Service or processing timed out.
Other `4xx`	Fix the request	Permissions, invalid file ID, or malformed JSON.
Other `5xx`	Investigate	Unexpected server error. Check status or contact support.

Support

When you contact Box Support about a failed or slow extraction job, include:

request_id from the response
HTTP status and response body
File ID and enterprise ID
Request timestamp and time zone
Field definitions or metadata template key
Concurrency and client timeout settings

​How extraction works

​When to use the API

​Before you start

​Design for one extraction per file

​Configure the request

​Choose an agent

​Set client timeouts

​Keep schemas focused

​Raise completion tokens for long answers

​Check confidence and citations

​Pace and concurrency

​Example: paced batch runner

​Status codes

​Support

How extraction works

When to use the API

Before you start

Design for one extraction per file

Configure the request

Choose an agent

Set client timeouts

Keep schemas focused

Raise completion tokens for long answers

Check confidence and citations

Pace and concurrency

Example: paced batch runner

Status codes

Support