Developer Glossary

The iterative perceive–think–act cycle that AI agents execute until a goal is achieved or a stopping condition is met.

The framework or runtime that wires together the agent loop, memory, tool routing, and LLM calls to run an AI agent.

A RAG architecture where an AI agent autonomously decides when and how to retrieve information, often across multiple retrieval steps.

An AI system that perceives its environment, reasons about it, and takes autonomous actions to complete goals.

Techniques websites use to detect and block automated scrapers, including CAPTCHAs, fingerprinting, and behavioral analysis.

A secret token passed in HTTP headers or query parameters to authenticate requests to an API service.

Extracting data by calling a website's internal or undocumented APIs rather than parsing its HTML.

A class of algorithms that find vectors approximately closest to a query vector, trading perfect accuracy for massive speed gains.

An API design pattern where long-running operations return a job ID immediately and deliver results via polling or webhook when complete.

An AI agent that can complete multi-step tasks end-to-end without requiring human input at each decision point.

An asynchronous task that runs independently of the main request-response cycle, allowing long-running operations like web extraction to run without blocking.

A probabilistic ranking function used in information retrieval that scores documents based on term frequency and inverse document frequency.

A prompting technique that encourages LLMs to reason step-by-step before producing a final answer, improving accuracy on complex tasks.

Monitoring web pages over time and detecting when their content has been updated, added, or removed.

The process of splitting long documents into smaller, overlapping or non-overlapping segments before embedding and indexing.

The process of identifying and removing duplicate or near-duplicate documents in a scraped dataset.

The practice of carefully designing and managing the information provided to an LLM's context window to maximize response quality.

The maximum number of tokens an LLM can process in a single inference call, including both input and output.

A metric that measures the angle between two vectors, commonly used to compare how semantically similar two embeddings are.

An AI agent that autonomously browses the web, synthesizes findings from multiple sources, and produces comprehensive research reports.

A retrieval method that represents both queries and documents as dense vectors and finds matches via nearest-neighbor search.

A database that stores semi-structured or unstructured documents (JSON, markdown, text) and supports retrieval by ID or metadata filters.

Traversing and extracting content from a browser's Document Object Model tree using selectors like CSS or XPath.

A dense numerical vector representation of text, images, or other data that captures semantic meaning in a high-dimensional space.

The NLP task of identifying and classifying named entities — people, organizations, locations, concepts — in unstructured text.

An AI agent's memory of specific past events and interactions, stored and retrieved to inform future decisions.

A prompting technique that provides a small number of input-output examples in the prompt to guide the LLM toward the desired response format.

The process of further training a pre-trained LLM on a smaller domain-specific dataset to adapt its behavior for a particular task.

Capturing all visible and structured content from a web page — text, links, metadata, and media references — in a single API call.

A structured mechanism that allows LLMs to output structured JSON specifying a function name and arguments for external execution.

A RAG variant that retrieves from a knowledge graph rather than a flat vector store, enabling multi-hop reasoning across connected entities.

Safety and policy constraints applied to agent inputs and outputs to prevent harmful, off-topic, or undesired behaviors.

When an LLM generates plausible-sounding but factually incorrect or fabricated information.

A web browser that runs without a graphical user interface, used to render JavaScript-heavy pages for scraping.

Hash-based Message Authentication Code — a cryptographic signature used to verify that webhook payloads are authentic and untampered.

Hierarchical Navigable Small World — a graph-based algorithm for fast approximate nearest-neighbor search in high-dimensional vector spaces.

A retrieval strategy that combines dense vector search with sparse keyword search (like BM25) to improve recall and precision.

The property of an API operation where making the same request multiple times produces the same result as making it once.

The process of transforming raw content into a searchable structure — embeddings, inverted indexes, or graph nodes — that enables fast retrieval.

The process of running a trained LLM to generate output from a given input prompt, as opposed to training or fine-tuning the model.

Using AI or LLMs to understand and extract meaningful content from web pages without manually writing CSS selectors or XPath rules.

The process of executing a page's JavaScript in a real or headless browser to capture the fully rendered DOM before extraction.

A vocabulary for describing and validating the structure of JSON data, widely used to define the expected output format for LLM function calls.

A structured or unstructured collection of information that an AI system can query to answer questions or complete tasks.

The process of automatically deriving structured facts, entities, and relationships from unstructured text or web content.

A graph-structured database that represents real-world entities as nodes and their relationships as edges, enabling structured reasoning.

A single indexed unit of knowledge — a document chunk with title, content, category, and embedding — stored in KnowledgeSDK's search index.

A neural network trained on vast text corpora that can generate, summarize, translate, and reason about language.

The time delay between sending an API request and receiving the response, a critical metric for real-time AI applications.

The capability of modern LLMs to process very large input texts — from tens of thousands to millions of tokens — in a single call.

Persistent storage of information across agent sessions, enabling recall of facts, preferences, or past interactions beyond the context window.

Converting raw HTML web pages into clean, structured Markdown text, removing navigation, ads, and boilerplate.

The mechanisms by which an AI agent stores and retrieves information across turns, sessions, or tasks to maintain continuity.

The process of transferring important information from an agent's working memory or episodic store into long-term semantic memory.

An open protocol by Anthropic that standardizes how AI applications provide context — tools, resources, and prompts — to language models.

An architecture where multiple specialized AI agents collaborate, each handling a sub-task and communicating results to an orchestrator.

A formal representation of a domain's concepts, categories, and relationships used to structure knowledge bases and improve reasoning.

A top-level agent that coordinates multiple sub-agents or tools, routes tasks, and assembles final outputs from their results.

A hierarchical chunking strategy that indexes small child chunks for retrieval but returns their larger parent context to the LLM.

A reasoning component that decomposes a high-level goal into a sequence of sub-tasks or tool calls for an agent to execute.

Following web crawling best practices such as respecting robots.txt, adding crawl delays, and identifying your crawler in the user agent.

An information retrieval metric measuring the fraction of retrieved documents that are actually relevant to the query.

The practice of crafting and optimizing instructions given to an LLM to elicit accurate, relevant, and well-formatted responses.

Automatically cycling through a pool of IP addresses when scraping to avoid rate limits and IP-based blocking.

A retrieval technique that augments the original query with synonyms, related terms, or generated sub-questions to improve recall.

A control mechanism that restricts how many API requests a client can make within a given time window.

A post-retrieval step that re-scores and reorders retrieved documents using a more powerful cross-encoder model to improve relevance.

A prompting framework that interleaves Reasoning traces and Action steps, enabling agents to plan while gathering information.

An information retrieval metric measuring the fraction of all relevant documents that were successfully retrieved.

A self-evaluation step in which an agent critiques its own outputs or reasoning and revises them to improve quality.

The end-to-end sequence of steps — query processing, search, re-ranking, and context assembly — that retrieves relevant documents for an LLM.

A technique that grounds LLM responses by retrieving relevant documents from an external knowledge base before generation.

Reinforcement Learning from Human Feedback — a training technique that uses human preference ratings to align LLM outputs with human values.

A text file at the root of a website that instructs web crawlers which pages or sections they are allowed or disallowed from accessing.

An end-to-end workflow that orchestrates URL discovery, fetching, parsing, deduplication, and storage of scraped web data.

An API that captures a full-page or viewport screenshot of a URL as an image, enabling visual monitoring and multimodal AI workflows.

An agent's persistent store of general facts and knowledge about the world, distinct from memories of specific events.

A search approach that finds results based on meaning and intent rather than exact keyword matching.

An XML or HTML file listing all discoverable URLs on a website, used by crawlers to efficiently find and index pages.

A discrete, reusable capability or tool that an agent can invoke to perform a specific action, such as web search or code execution.

A chunking strategy where consecutive chunks overlap by a fixed number of tokens to preserve context at chunk boundaries.

A retrieval method that represents documents as sparse term-frequency vectors, enabling fast keyword-based matching.

Pulling specific fields — prices, names, dates — from web pages into structured formats like JSON or CSV.

LLM responses constrained to a specific format — typically JSON — by using function calling, grammar constraints, or guided generation.

Instructions placed at the start of an LLM conversation that define the model's role, persona, constraints, and output format.

A sampling parameter that controls the randomness of LLM outputs — lower values make responses more deterministic, higher values more creative.

The number of requests or operations a system can process per unit of time, a key performance metric for scraping and search APIs.

The basic unit of text processed by an LLM — roughly 3/4 of a word in English — that models use to read and generate language.

A rate limiting algorithm that allows bursts of traffic up to a bucket capacity while enforcing a sustained average request rate.

The process of converting raw text into a sequence of tokens that an LLM can process using a vocabulary-based algorithm like BPE.

A catalog of available tools and their schemas that an agent or orchestrator can consult to discover and invoke capabilities.

The ability of an LLM-powered agent to call external functions, APIs, or services to gather information or take actions.

A decoding strategy that restricts token selection to the K most probable next tokens, balancing coherence and diversity.

A database optimized for storing subject-predicate-object triples (RDF), the fundamental unit of knowledge in semantic web and knowledge graphs.

Setting a custom HTTP User-Agent header to make a scraper appear as a real browser or specific client to the target server.

A specialized database that stores high-dimensional embedding vectors and enables fast similarity search.

The systematic traversal of websites by following links to discover and fetch pages at scale.

The automated extraction of data from websites by programmatically fetching and parsing HTML content.

An HTTP callback that sends real-time event notifications from a server to a client-specified URL when something happens.

The information held in an AI agent's active context window during a single task or conversation turn.