KnowledgeSDK MCP Server: Give Claude and Cursor Live Web Access

The Model Context Protocol (MCP) is the open standard that lets AI tools like Claude Desktop and Cursor connect to external capabilities. Instead of copying and pasting content into your AI chat, you describe what you need and the AI calls the right tool automatically.

KnowledgeSDK ships an official MCP server — @knowledgesdk/mcp — that exposes scrape, search, and extract as MCP tools. Once installed, Claude can browse competitor websites on your behalf, Cursor can read library documentation while you code, and any MCP-compatible host can query your private knowledge base semantically.

This guide covers installation, configuration for both Claude Desktop and Cursor, and several real-world use cases that become immediately possible.

What is MCP and Why Does it Matter?

MCP is an open protocol created by Anthropic that standardizes how AI models communicate with external tools and data sources. Think of it as a USB standard for AI capabilities: one protocol, many compatible devices.

Before MCP, extending an AI assistant with external data meant either fine-tuning (expensive, slow to update), RAG pipelines (complex infrastructure), or manually pasting content into context (tedious, error-prone). MCP replaces all of that with a clean tool-calling interface that the AI model itself orchestrates.

When you install the KnowledgeSDK MCP server, Claude and Cursor gain three new tools:

• scrape — fetch any URL as clean markdown, handling JavaScript, anti-bot, and pagination
• search — semantic + keyword search over previously scraped content
• extract — AI-structured data extraction from any URL with a custom schema

The AI decides when and how to call these tools based on your request. You never have to think about the API.

Installation

The KnowledgeSDK MCP server is available on npm:

npm install -g @knowledgesdk/mcp

Or use it without installing via npx (recommended for Claude Desktop):

npx @knowledgesdk/mcp

You'll need a KnowledgeSDK API key from knowledgesdk.com/setup.

Configuring Claude Desktop

Claude Desktop reads MCP server configurations from a JSON file. Open or create ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "knowledgesdk": {
      "command": "npx",
      "args": ["@knowledgesdk/mcp"],
      "env": {
        "KNOWLEDGESDK_API_KEY": "sk_ks_your_key_here"
      }
    }
  }
}

Restart Claude Desktop. You should see a tool icon in the chat interface — click it to verify that scrape, search, and extract are listed.

Verifying the connection

Open a new Claude conversation and type:

Can you scrape https://knowledgesdk.com and tell me what the main features are?

Claude will automatically call the scrape tool, receive the markdown, and summarize it for you. No copy-pasting required.

Configuring Cursor

Cursor supports MCP through its settings panel. Open Cursor → Settings → Extensions → MCP Servers and add:

{
  "knowledgesdk": {
    "command": "npx",
    "args": ["@knowledgesdk/mcp"],
    "env": {
      "KNOWLEDGESDK_API_KEY": "sk_ks_your_key_here"
    }
  }
}

Alternatively, create a .cursor/mcp.json file in your project root for project-scoped configuration:

{
  "mcpServers": {
    "knowledgesdk": {
      "command": "npx",
      "args": ["@knowledgesdk/mcp"],
      "env": {
        "KNOWLEDGESDK_API_KEY": "sk_ks_your_key_here"
      }
    }
  }
}

This is useful when you want Cursor to automatically read documentation for the libraries your project uses.

Use Case 1: Research Competitor Pricing with Claude

One of the most immediately useful applications is competitive research. Instead of opening five browser tabs, copying pricing tables, and pasting them into Claude, you just ask:

Research the pricing for Firecrawl, Apify, Browserless, and ScrapingBee.
Extract: plan names, monthly prices, page/credit limits, and whether they offer
a free tier. Present as a comparison table.

Claude will:

1. Call scrape on each competitor's pricing page
2. Call extract with a schema matching your requested fields
3. Synthesize the results into a formatted comparison table

The entire research task that used to take 30 minutes takes 90 seconds. And because KnowledgeSDK handles JavaScript rendering and anti-bot measures, it works even on sites that block simple HTTP requests.

Use Case 2: Read Library Documentation While Coding in Cursor

When you're working with a library you haven't used before, Cursor can now read its documentation live rather than relying on training data that may be months out of date.

In Cursor's composer, try:

I'm trying to use the Drizzle ORM with Neon Postgres. Read the latest Drizzle
docs at https://orm.drizzle.team/docs/get-started-postgresql and help me write
a schema for a users table with id, email, createdAt, and a JSONB metadata field.

Cursor will scrape the documentation, read the current API, and generate code that matches the actual current library — not a stale version from its training data. This is especially valuable for:

• Rapidly evolving frameworks (Next.js App Router, React 19 APIs)
• Libraries with breaking changes between major versions
• Internal or private documentation hosted on your company's intranet

Use Case 3: Build a Knowledge Base and Search It

The search tool becomes powerful after you've scraped multiple pages. Claude and Cursor can search across all previously scraped content semantically.

A typical session might look like:

1. Ask Claude to scrape your competitor's entire documentation site
2. Later, ask: "What does Competitor X say about their rate limiting policy?"
3. Claude calls search with that query and returns the relevant passages

This turns Claude into a persistent research assistant with memory across sessions. You build the knowledge base once, then query it freely.

Scrape all pages at https://docs.competitor.com/api and index them.

Search the knowledge base: "what authentication methods do they support?"

Claude manages the scrape-then-search flow automatically.

Use Case 4: Monitor Changes and Summarize Diffs

When combined with KnowledgeSDK's webhook capability (configured separately via the API), the MCP server lets Claude analyze what changed on a monitored page:

The webhook says https://competitor.com/pricing changed.
Scrape the current version and compare it to this previous version: [paste text].
What changed? Focus on price changes.

Claude scrapes the live page via MCP, diffs it against the pasted previous version, and gives you a natural-language change summary.

Available MCP Tools in Detail

scrape

Description: Fetches a URL and returns clean markdown content.

Parameters:

• url (string, required) — the URL to scrape
• includeLinks (boolean, optional) — whether to include hyperlinks in output (default: false)
• waitFor (number, optional) — milliseconds to wait for dynamic content to load

Example invocation (how Claude sees it):

{
  "tool": "scrape",
  "parameters": {
    "url": "https://example.com/pricing",
    "includeLinks": false
  }
}

search

Description: Semantic and keyword search over the knowledge base built from previous scrapes.

Parameters:

• query (string, required) — natural language search query
• limit (number, optional) — number of results to return (default: 5, max: 20)
• hybrid (boolean, optional) — combine semantic and keyword ranking (default: true)

extract

Description: AI-structured extraction from a URL using a custom schema.

Parameters:

• url (string, required) — the URL to extract from
• schema (object, required) — JSON schema describing the fields to extract
• prompt (string, optional) — additional instructions for the extraction AI

Example:

{
  "tool": "extract",
  "parameters": {
    "url": "https://startup.com",
    "schema": {
      "companyName": "string",
      "fundingStage": "string",
      "teamSize": "number",
      "techStack": "array"
    }
  }
}

MCP vs. Manual Copy-Paste: Comparison

Workflow	Manual	With KnowledgeSDK MCP
Research competitor pricing (5 sites)	30 min	2 min
Read library docs while coding	Switch tabs, search	Ask Cursor directly
Extract structured data from a page	Write scraper + parser	One natural language request
Search across 50 scraped pages	Impossible manually	Sub-second semantic search
Get latest version of changing docs	Outdated training data	Always live
Monitor competitor page for changes	Manual periodic checks	Automatic webhook + Claude analysis

Security and API Key Management

Your KnowledgeSDK API key is stored in the MCP server configuration as an environment variable. It is never sent to Claude or Cursor — it stays on your local machine and is used only to authenticate calls from the MCP server to the KnowledgeSDK API.

For team environments where multiple developers share a Cursor or Claude Desktop config, consider:

• Using a shared API key with read-only permissions (if your plan supports it)
• Storing the key in a .env file and referencing it with $KNOWLEDGESDK_API_KEY in the config
• Rotating keys periodically through the KnowledgeSDK dashboard

Never commit your claude_desktop_config.json or .cursor/mcp.json files to version control if they contain hardcoded API keys.

Debugging MCP Connection Issues

If the tools don't appear in Claude Desktop after configuration:

1. Check that npx @knowledgesdk/mcp runs without errors from your terminal
2. Verify the JSON syntax in your config file — trailing commas cause silent parse failures
3. Confirm your API key is valid by testing it directly: curl -H "x-api-key: sk_ks_your_key" https://api.knowledgesdk.com/v1/account
4. Check Claude Desktop logs at ~/Library/Logs/Claude/ (macOS)

For Cursor, open the Output panel and select "MCP" from the dropdown to see connection logs.

Using the MCP Server Programmatically

The MCP server can also be integrated into your own applications that implement the MCP client protocol. If you're building an agent framework or AI application, you can spawn the MCP server as a subprocess and communicate with it over stdio:

import { spawn } from 'child_process';

const mcp = spawn('npx', ['@knowledgesdk/mcp'], {
  env: {
    ...process.env,
    KNOWLEDGESDK_API_KEY: 'sk_ks_your_key'
  },
  stdio: ['pipe', 'pipe', 'pipe']
});

// Send MCP protocol messages over stdin/stdout

This pattern is useful for building custom AI orchestration systems that need live web access as one of many available tools.

What's Next for KnowledgeSDK MCP

The roadmap includes additional MCP tools:

• screenshot — capture a URL as an image for vision-capable models
• sitemap — get all URLs from a domain for bulk scraping
• watch — register a URL for change monitoring directly from Claude

These will be available in upcoming releases. Update with npm update -g @knowledgesdk/mcp to get new tools as they ship.

FAQ

Does the MCP server work with other AI tools besides Claude Desktop and Cursor? Any MCP-compatible host works. This includes Continue.dev, Zed, and custom applications built with the MCP SDK. The protocol is open and standardized.

Will Claude's responses include live data every time? Claude calls the scrape tool when it determines live data is needed. For queries that don't reference external URLs or search, it won't call the tool. You can explicitly instruct Claude to scrape by mentioning a URL.

Is there a cost per MCP tool call? Each tool call to scrape, search, or extract counts against your KnowledgeSDK plan's usage limits, the same as direct API calls. MCP is just a protocol layer — the underlying API calls are identical.

Can I restrict which URLs the MCP server is allowed to scrape? Not at the MCP server level, but you can use KnowledgeSDK's API key permissions (when available) to restrict access. For now, use separate API keys for different environments or use cases.

Does the MCP server cache responses? Yes. KnowledgeSDK has a built-in cache layer. If the same URL is scraped twice within a short window, the second call returns cached content and doesn't re-fetch the page.

Can I use the MCP server with the free tier? Yes. The free tier works with the MCP server. Usage limits apply based on your plan.

---

Give your AI tools live eyes on the web. Install the KnowledgeSDK MCP server in minutes at knowledgesdk.com/setup.