Batch scanner runs without duplicate work

Bulk video analysis should behave like a durable queue, not a loop that fires thousands of requests and hopes for the best. Create one scanner, submit one run per authorized recording, and make your database the source of truth for submission and terminal state.

1. Pin the scanner version you intend to operate

Create the scanner once with POST /v1/scanners. Save the returned scanner_id and version beside your batch definition. If you later patch the scanner, treat the new version as a new processing cohort rather than silently mixing outputs.

POST /v1/scanners
Authorization: Bearer $CCT_API_KEY
Idempotency-Key: onboarding-friction-scanner-v1

{
  "name": "Onboarding friction",
  "type": "find_moments",
  "instructions": "Find repeated setup actions, visible errors, or abandonment before the next step.",
  "profile": "conversation",
  "max_observations": 20,
  "min_seconds": 10,
  "max_seconds": 60
}

2. Give every source a durable queue row

Store your own recording ID, ready upload ID, scanner ID, intended scanner version, submission state, idempotency key, and returned run ID. Never put signed PUT targets or bearer credentials in logs. Add bounded scalar metadata so the resulting observations can return to the right tenant, cohort, or research study.

idempotency_key = f"scan:{recording.id}:{scanner_id}:v{scanner_version}"

POST /v1/scanners/{scanner_id}/runs
Idempotency-Key: {idempotency_key}

{
  "upload_id": "upl_YOUR_READY_UPLOAD",
  "metadata": {
    "recording_id": "rec_1842",
    "cohort": "new-user",
    "workspace_id": "ws_81"
  }
}

A retry with the same idempotency key returns the original operation rather than intentionally creating duplicate work. Use a new key only when you truly intend a new run.

3. Bound submissions to the account limit

Read current concurrency and monthly allowances in Console → Usage. Keep a smaller client-side worker pool, pause new submissions on admission or quota responses, and retry transient failures with capped exponential backoff and jitter. Do not retry validation, authentication, source-host, or insufficient-credit responses until the underlying request or account state changes.

while queue.has_ready_rows():
    capacity = configured_worker_limit - queue.active_count()
    for row in queue.claim(capacity):
        submit_run(row)
    poll_due_runs()
    sleep(with_jitter(5, 10))

4. Poll terminal state and preserve zero results

Poll GET /v1/runs/{run_id} every 5–10 seconds with jitter. Stop on completed or failed. For a completed run, fetch GET /v1/runs/{run_id}/observations once and store the observation IDs plus scanner and run provenance.

A completed run with zero observations is not a processing failure. It is an unanswerable result: sampled visual evidence could not support an answer. Keep it separate from an answered monitor no-match, which contains one explicit boolean false observation.

5. Aggregate only compatible records

Query GET /v1/observations?scanner_id=scn_...&limit=100 for cross-run retrieval, but group by scanner version and semantics before aggregating. Boolean monitor output, a classifier label, a scorer value, and a timestamped moment are different contracts. relevance_score helps rank support for the scanner brief; it is not calibrated confidence.

Failure checklist

• Persist the run ID before claiming submission succeeded.
• Use a completed, account-owned private upload_id; production visual scanners do not run arbitrary source URLs.
• Separate terminal failed, unanswerable zero-observation, and explicit monitor-false outcomes.
• Retain scanner_version, source_metadata, and run_id with downstream results.
• Use webhooks only when the console confirms delivery is enabled; keep reconciliation polling even then.