When a competitor changes their pricing, every hour you don't know about it is an hour you're potentially losing deals. Sales teams quote the wrong comparison. Marketing runs outdated competitive battlecards. Product roadmaps miss the signal that a competitor just commoditized a feature you're charging premium for.
The solution is a pricing monitor that runs continuously and alerts your team the moment any competitor pricing page changes. This tutorial builds exactly that — a production-ready system using KnowledgeSDK webhooks, Node.js, and Slack.
Architecture Overview
The system has three phases:
Setup (run once):
- Scrape the initial version of each competitor pricing page
- Store baseline content in your database
- Register webhook subscriptions with KnowledgeSDK
Runtime (continuous, zero cost when nothing changes): 4. KnowledgeSDK monitors the subscribed URLs 5. When content changes, KnowledgeSDK sends a webhook to your server 6. Your server receives the diff, re-scrapes for the latest content, and parses prices
Alerting: 7. Compare old and new pricing 8. Send a Slack message with before/after context 9. Update the baseline in your database
This is fundamentally more efficient than polling. Instead of scraping 5 URLs every hour (120 scrape calls/day), you register subscriptions once and only process pages when they actually change.
Prerequisites
- Node.js 18+
- A KnowledgeSDK API key from knowledgesdk.com/setup
- A Postgres database (or swap for SQLite locally)
- A Slack incoming webhook URL
- A publicly accessible server or a tunneling tool like ngrok for local development
Project Setup
mkdir pricing-monitor && cd pricing-monitor
npm init -y
npm install @knowledgesdk/node pg dotenv express
npm install -D typescript @types/node @types/express tsx
Create .env:
KNOWLEDGESDK_API_KEY=sk_ks_your_key
DATABASE_URL=postgresql://user:pass@localhost:5432/pricing_monitor
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T.../B.../...
WEBHOOK_SECRET=a_random_secret_string_here
SERVER_URL=https://your-server.com # or ngrok URL for local dev
Database Schema
-- migrations/001_create_tables.sql
CREATE TABLE IF NOT EXISTS pricing_baselines (
id SERIAL PRIMARY KEY,
url TEXT UNIQUE NOT NULL,
competitor_name TEXT NOT NULL,
markdown TEXT NOT NULL,
scraped_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE TABLE IF NOT EXISTS pricing_changes (
id SERIAL PRIMARY KEY,
url TEXT NOT NULL,
competitor_name TEXT NOT NULL,
old_markdown TEXT NOT NULL,
new_markdown TEXT NOT NULL,
diff_added TEXT[],
diff_removed TEXT[],
detected_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
slack_notified BOOLEAN DEFAULT FALSE
);
CREATE TABLE IF NOT EXISTS webhook_subscriptions (
id SERIAL PRIMARY KEY,
knowledgesdk_webhook_id TEXT UNIQUE NOT NULL,
watch_urls TEXT[] NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
Step 1: Define Your Competitors
// src/competitors.ts
export interface Competitor {
name: string;
pricingUrl: string;
}
export const COMPETITORS: Competitor[] = [
{
name: "Firecrawl",
pricingUrl: "https://firecrawl.dev/pricing",
},
{
name: "Apify",
pricingUrl: "https://apify.com/pricing",
},
{
name: "Browserless",
pricingUrl: "https://www.browserless.io/pricing",
},
{
name: "ScrapingBee",
pricingUrl: "https://www.scrapingbee.com/#pricing",
},
{
name: "Zyte",
pricingUrl: "https://www.zyte.com/pricing",
},
];
Step 2: Scrape Baselines and Register Webhooks
// src/setup.ts
import KnowledgeSDK from "@knowledgesdk/node";
import { Pool } from "pg";
import { COMPETITORS } from "./competitors";
import "dotenv/config";
const ks = new KnowledgeSDK({ apiKey: process.env.KNOWLEDGESDK_API_KEY! });
const db = new Pool({ connectionString: process.env.DATABASE_URL });
async function scrapeAndStore(
competitorName: string,
url: string
): Promise<string> {
console.log(`Scraping baseline for ${competitorName}: ${url}`);
const result = await ks.scrape({ url });
await db.query(
`INSERT INTO pricing_baselines (url, competitor_name, markdown)
VALUES ($1, $2, $3)
ON CONFLICT (url) DO UPDATE
SET markdown = EXCLUDED.markdown,
scraped_at = NOW()`,
[url, competitorName, result.markdown]
);
console.log(` Stored: ${result.markdown.split(" ").length} words`);
return result.markdown;
}
async function registerWebhooks(): Promise<void> {
const watchUrls = COMPETITORS.map((c) => c.pricingUrl);
const webhook = await ks.webhooks.create({
url: `${process.env.SERVER_URL}/webhooks/pricing-change`,
watchUrls,
events: ["content.changed"],
secret: process.env.WEBHOOK_SECRET,
});
await db.query(
`INSERT INTO webhook_subscriptions (knowledgesdk_webhook_id, watch_urls)
VALUES ($1, $2)
ON CONFLICT (knowledgesdk_webhook_id) DO NOTHING`,
[webhook.id, watchUrls]
);
console.log(`Registered webhook: ${webhook.id}`);
console.log(`Watching ${watchUrls.length} URLs`);
}
async function setup(): Promise<void> {
// Scrape all competitors with a 1s delay between requests
for (const competitor of COMPETITORS) {
await scrapeAndStore(competitor.name, competitor.pricingUrl);
await new Promise((r) => setTimeout(r, 1000));
}
await registerWebhooks();
console.log("Setup complete. Monitoring is active.");
await db.end();
}
setup().catch(console.error);
Run this once:
npx tsx src/setup.ts
Step 3: Parse Pricing from Markdown
This is where you extract actual prices from the scraped content. KnowledgeSDK's extract endpoint handles this cleanly:
// src/priceParser.ts
import KnowledgeSDK from "@knowledgesdk/node";
const ks = new KnowledgeSDK({ apiKey: process.env.KNOWLEDGESDK_API_KEY! });
export interface PricingPlan {
name: string;
monthlyPrice: string;
features: string[];
}
export interface ParsedPricing {
plans: PricingPlan[];
hasFreeTier: boolean;
enterprisePricing: string;
}
export async function extractPricing(url: string): Promise<ParsedPricing> {
const result = await ks.extract({
url,
schema: {
plans: "array",
hasFreeTier: "boolean",
enterprisePricing: "string",
},
prompt:
"Extract all pricing plans. For each plan include: name, monthly price (as a string like '$29/month'), and top 3 features.",
});
return result.data as ParsedPricing;
}
export function diffPricing(
oldContent: string,
newContent: string
): { added: string[]; removed: string[]; priceChanges: string[] } {
const oldLines = new Set(oldContent.split("\n").map((l) => l.trim()).filter(Boolean));
const newLines = new Set(newContent.split("\n").map((l) => l.trim()).filter(Boolean));
const added = [...newLines].filter((line) => !oldLines.has(line));
const removed = [...oldLines].filter((line) => !newLines.has(line));
// Look for price-related changes specifically
const priceRegex = /\$[\d,]+(?:\.\d{2})?(?:\/(?:mo|month|year|yr))?/gi;
const priceChanges: string[] = [];
for (const line of removed) {
if (priceRegex.test(line)) {
const matchingAdded = added.find((addedLine) =>
// Find a similar line that also has a price (likely the updated version)
addedLine.toLowerCase().includes(line.toLowerCase().slice(0, 20))
);
if (matchingAdded) {
priceChanges.push(`Changed: "${line}" → "${matchingAdded}"`);
} else {
priceChanges.push(`Removed: "${line}"`);
}
}
}
for (const line of added) {
if (priceRegex.test(line)) {
const wasChanged = priceChanges.some((c) => c.includes(line));
if (!wasChanged) {
priceChanges.push(`Added: "${line}"`);
}
}
}
return { added, removed, priceChanges };
}
Step 4: Slack Notification
// src/slack.ts
import "dotenv/config";
export interface PriceChangeAlert {
competitorName: string;
url: string;
priceChanges: string[];
addedLines: string[];
removedLines: string[];
detectedAt: Date;
}
export async function sendSlackAlert(alert: PriceChangeAlert): Promise<void> {
const changes =
alert.priceChanges.length > 0
? alert.priceChanges.join("\n")
: "Content changed but no direct price changes detected";
const addedSample = alert.addedLines
.slice(0, 5)
.map((l) => `+ ${l}`)
.join("\n");
const removedSample = alert.removedLines
.slice(0, 5)
.map((l) => `- ${l}`)
.join("\n");
const message = {
blocks: [
{
type: "header",
text: {
type: "plain_text",
text: `Pricing Change Detected: ${alert.competitorName}`,
},
},
{
type: "section",
fields: [
{
type: "mrkdwn",
text: `*URL:*\n<${alert.url}|View Page>`,
},
{
type: "mrkdwn",
text: `*Detected:*\n${alert.detectedAt.toISOString()}`,
},
],
},
{
type: "section",
text: {
type: "mrkdwn",
text: `*Price Changes:*\n\`\`\`${changes}\`\`\``,
},
},
...(addedSample || removedSample
? [
{
type: "section",
text: {
type: "mrkdwn",
text: `*Content Diff (sample):*\n\`\`\`${removedSample}\n${addedSample}\`\`\``,
},
},
]
: []),
{
type: "actions",
elements: [
{
type: "button",
text: { type: "plain_text", text: "View Current Page" },
url: alert.url,
},
],
},
],
};
const response = await fetch(process.env.SLACK_WEBHOOK_URL!, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(message),
});
if (!response.ok) {
throw new Error(`Slack webhook failed: ${response.status}`);
}
}
Step 5: Webhook Handler Server
// src/server.ts
import express from "express";
import { Pool } from "pg";
import KnowledgeSDK from "@knowledgesdk/node";
import { diffPricing, extractPricing } from "./priceParser";
import { sendSlackAlert } from "./slack";
import crypto from "crypto";
import "dotenv/config";
const app = express();
const db = new Pool({ connectionString: process.env.DATABASE_URL });
const ks = new KnowledgeSDK({ apiKey: process.env.KNOWLEDGESDK_API_KEY! });
// Parse raw body for signature verification
app.use(
"/webhooks",
express.raw({ type: "application/json" }),
(req, _res, next) => {
(req as any).rawBody = req.body;
(req as any).body = JSON.parse(req.body.toString());
next();
}
);
app.use(express.json());
function verifyWebhookSignature(
rawBody: Buffer,
signature: string,
secret: string
): boolean {
const expected = crypto
.createHmac("sha256", secret)
.update(rawBody)
.digest("hex");
return crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(expected)
);
}
app.post("/webhooks/pricing-change", async (req, res) => {
// Verify signature
const signature = req.headers["x-knowledgesdk-signature"] as string;
if (
!signature ||
!verifyWebhookSignature(
(req as any).rawBody,
signature,
process.env.WEBHOOK_SECRET!
)
) {
console.warn("Invalid webhook signature");
return res.status(401).json({ error: "Unauthorized" });
}
// Acknowledge immediately — process asynchronously
res.status(200).json({ received: true });
const { url, diff, newContent } = req.body;
try {
// Get baseline from database
const { rows } = await db.query(
"SELECT competitor_name, markdown FROM pricing_baselines WHERE url = $1",
[url]
);
if (rows.length === 0) {
console.error(`No baseline found for URL: ${url}`);
return;
}
const { competitor_name, markdown: oldMarkdown } = rows[0];
// Get the detailed diff
const priceDiff = diffPricing(oldMarkdown, newContent);
// Store the change
await db.query(
`INSERT INTO pricing_changes
(url, competitor_name, old_markdown, new_markdown, diff_added, diff_removed)
VALUES ($1, $2, $3, $4, $5, $6)`,
[
url,
competitor_name,
oldMarkdown,
newContent,
diff.added,
diff.removed,
]
);
// Send Slack alert
await sendSlackAlert({
competitorName: competitor_name,
url,
priceChanges: priceDiff.priceChanges,
addedLines: priceDiff.added.slice(0, 10),
removedLines: priceDiff.removed.slice(0, 10),
detectedAt: new Date(),
});
// Update baseline to the new content
await db.query(
`UPDATE pricing_baselines
SET markdown = $1, scraped_at = NOW()
WHERE url = $2`,
[newContent, url]
);
console.log(`Processed change for ${competitor_name}: ${url}`);
} catch (error) {
console.error(`Error processing webhook for ${url}:`, error);
}
});
// Health check
app.get("/health", (_req, res) => res.json({ status: "ok" }));
const PORT = process.env.PORT ?? 3000;
app.listen(PORT, () => {
console.log(`Pricing monitor listening on port ${PORT}`);
});
Running the System
# Run setup once
npx tsx src/setup.ts
# Start the webhook server
npx tsx src/server.ts
For local development, use ngrok to expose your server:
ngrok http 3000
# Copy the https URL and set it as SERVER_URL in .env
# Re-run setup.ts to register the new webhook URL
Production Considerations
Rate Limits and Deduplication
If multiple competitors change their pricing simultaneously (e.g., after a major market event), your webhook server may receive several requests in quick succession. Add a simple deduplication check to avoid processing the same URL twice within a short window:
const processingUrls = new Set<string>();
// In the webhook handler, before processing:
if (processingUrls.has(url)) {
console.log(`Already processing ${url}, skipping duplicate`);
return;
}
processingUrls.add(url);
// After processing:
processingUrls.delete(url);
Database Connection Pooling
For production, tune your Postgres connection pool:
const db = new Pool({
connectionString: process.env.DATABASE_URL,
max: 10,
idleTimeoutMillis: 30000,
connectionTimeoutMillis: 2000,
});
Alerting Fatigue
If a competitor has a dynamic page (A/B tests, countdown timers, rotating testimonials), you may get frequent false-positive change notifications. Add a minimum change threshold:
const MINIMUM_CHANGE_RATIO = 0.05; // Only alert if 5%+ of content changed
const changeRatio =
(priceDiff.added.length + priceDiff.removed.length) /
oldMarkdown.split("\n").length;
if (changeRatio < MINIMUM_CHANGE_RATIO) {
console.log(`Change too small for ${url}: ${(changeRatio * 100).toFixed(1)}%`);
return;
}
Retry Logic for Slack
Slack webhooks occasionally fail transiently. Add a simple retry:
async function sendSlackWithRetry(alert: PriceChangeAlert, retries = 3) {
for (let i = 0; i < retries; i++) {
try {
await sendSlackAlert(alert);
return;
} catch (e) {
if (i === retries - 1) throw e;
await new Promise((r) => setTimeout(r, 1000 * (i + 1)));
}
}
}
Extending the System
Email digest: Instead of (or in addition to) Slack, send a weekly email digest of all pricing changes detected over the past 7 days. Query the pricing_changes table and format with a templating library.
Historical trends: Chart pricing over time by storing each detected change. Use a time-series visualization to show when competitors raised or lowered prices.
AI analysis: After detecting a change, call OpenAI to generate a competitive analysis: "Competitor X raised their Pro tier from $49 to $69/month. They added SSO as a feature. This positions them above us on price but may justify the increase. Recommended action: update battlecard."
Multiple channels: Route alerts to different Slack channels based on which competitor changed — #competitor-a-intel, #competitor-b-intel — to reduce noise.
FAQ
How quickly does KnowledgeSDK detect changes? KnowledgeSDK checks monitored URLs at regular intervals. For most plans, changes are detected within 1-6 hours. This is sufficient for pricing page monitoring where the business impact is measured in days, not hours.
What if a competitor's pricing page uses heavy JavaScript? KnowledgeSDK handles JavaScript rendering automatically. Pricing pages built with React, Vue, or other frameworks are supported without any additional configuration.
Can I monitor login-protected pricing pages? You can pass session cookies when registering the webhook. However, cookies expire and login-protected pages are generally considered proprietary. Stick to public pricing pages.
What's the cost of running this system? Webhook registrations are included in all plans. You pay for scrape/extract calls when KnowledgeSDK detects a change and sends the diff. For 5 pricing pages that change once a month, you're looking at 5-10 scrape calls/month.
Can I use this for non-pricing pages? Absolutely. The same architecture works for monitoring product feature pages, job listings, press releases, or any public web content.
Start monitoring competitor pricing today. Get your KnowledgeSDK API key at knowledgesdk.com/setup.