Async API

An API design pattern where long-running operations return a job ID immediately and deliver results via polling or webhook when complete.

What Is an Async API?

An async (asynchronous) API is a design pattern where an endpoint accepts a request, starts a background operation, and immediately returns a reference — typically a job ID — rather than waiting for the work to complete. The client then retrieves the result later, either by polling a status endpoint or by receiving a webhook notification.

Async APIs decouple request submission from result delivery, enabling operations that would otherwise exceed HTTP timeout limits.

Sync vs. Async: When to Choose Each

Characteristic	Synchronous	Asynchronous
Response time	Milliseconds to seconds	Seconds to minutes
HTTP timeout risk	Low	Eliminated
Client complexity	Simple	Requires polling or webhook
Best for	Scraping, search, classification	Full extraction, bulk jobs

KnowledgeSDK exposes both patterns:

Sync: POST /v1/scrape, POST /v1/search, POST /v1/classify — return results directly in the response body, typically within a few seconds.
Async: POST /v1/extract/async — returns a jobId and delivers the full knowledge extraction result (which can take 1–3 minutes) via webhook or polling.

The Async API Workflow

Step 1 — Submit the Job

const res = await fetch("https://api.knowledgesdk.com/v1/extract/async", {
  method: "POST",
  headers: {
    "x-api-key": process.env.KNOWLEDGE_API_KEY!,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com",
    callbackUrl: "https://yourapp.com/webhooks/knowledge",
  }),
});

const { jobId } = await res.json();
// jobId: "job_abc123"

Step 2a — Poll for the Result

async function waitForJob(jobId: string, apiKey: string) {
  while (true) {
    const res = await fetch(`https://api.knowledgesdk.com/v1/jobs/${jobId}`, {
      headers: { "x-api-key": apiKey },
    });
    const job = await res.json();

    if (job.status === "completed") return job.result;
    if (job.status === "failed") throw new Error(job.error);

    // Wait before polling again
    await new Promise((r) => setTimeout(r, 5000));
  }
}

Step 2b — Receive via Webhook (Recommended)

app.post("/webhooks/knowledge", (req, res) => {
  const { jobId, status, result } = req.body;
  if (status === "completed") {
    saveKnowledgeItem(result);
  }
  res.status(200).json({ ok: true });
});

Designing Robust Async Clients

Persist the job ID immediately. If your process crashes before receiving the result, you need the job ID to reconcile state on restart.
Use exponential backoff for polling. Start at 2 seconds, double each attempt, cap at 30 seconds. Constant polling wastes quota and degrades shared infrastructure.
Prefer webhooks in production. Polling is fine for scripts and CLIs; webhooks are more efficient for web applications handling multiple concurrent jobs.
Handle idempotency. Webhook delivery is "at least once." Use the jobId to deduplicate repeated deliveries of the same result.
Set a timeout. If a job has not completed within a reasonable window (e.g., 15 minutes), treat it as stalled and alert.

Async API Standards

The AsyncAPI specification is an open standard for documenting event-driven and asynchronous APIs, analogous to OpenAPI for synchronous REST APIs. While KnowledgeSDK does not currently publish an AsyncAPI spec, the job-based pattern it uses is fully compatible with the specification's channel and message model.