Idempotency

What Is Idempotency?

An operation is idempotent if performing it multiple times has the same effect as performing it once. In API design, idempotency is a contract: calling the same endpoint with the same inputs repeatedly leaves the system in the same state as a single call.

The term comes from mathematics: applying a function twice should yield the same result as applying it once — f(f(x)) = f(x).

Why Idempotency Matters

Networks are unreliable. Requests time out, connections drop, and servers restart mid-operation. When a client does not receive a response, it cannot know whether the server processed the request or not. Without idempotency, retrying a failed request can cause:

• Duplicate records — two knowledge items created for the same URL.
• Double charges — a payment processed twice.
• Conflicting state — a job triggered multiple times running in parallel.

Idempotency gives you a safe retry guarantee: try as many times as you need, the outcome is always the same.

HTTP Methods and Idempotency

By convention, HTTP methods carry idempotency semantics:

POST is not inherently idempotent — that is why APIs that need idempotent POST behavior use an explicit mechanism.

Idempotency Keys

The standard solution is an idempotency key: a unique string (typically a UUID) the client generates and passes in a header:

Idempotency-Key: a3f8c2d1-9e47-4b6e-8c12-0d5f7a3b1e29

The server stores the key alongside the operation result. If the same key arrives again, the server returns the stored result instead of executing the operation a second time.

Idempotency in KnowledgeSDK Async Workflows

When you call POST /v1/extract/async, KnowledgeSDK returns a jobId:

{ "jobId": "job_xyz789", "status": "pending" }

That jobId serves as an idempotency identifier for the job. If your webhook receiver at callbackUrl receives the same completed event twice (webhook delivery is "at least once"), you should deduplicate using the jobId:

const processed = new Set<string>();

app.post("/webhooks/knowledge", (req, res) => {
  const { jobId, result } = req.body;

  if (processed.has(jobId)) {
    return res.status(200).json({ ok: true, duplicate: true });
  }

  processed.add(jobId);
  saveToDatabase(result);
  res.status(200).json({ ok: true });
});

In production, use a persistent store (Redis, Postgres) rather than an in-memory Set.

Designing Idempotent Operations

• Use natural keys. For extraction jobs, the source URL is a natural deduplication key.
• Return consistent responses. A duplicate request should return the same response body as the original, including the same resource ID.
• Set key expiry. Idempotency keys do not need to live forever — 24 hours is a common TTL.
• Scope keys per operation type. The same UUID used for an extract request and a search request should be treated as separate keys.