Business Context
Enterprise teams spend upwards of $40K/yr on platforms like Klue to track competitors, but the data is often stale and requires manual analyst curation.
The core problem — Building an automated system that monitors competitor websites, detects meaningful pricing/strategy changes (while ignoring noise like A/B tests), and alerts founders in real-time.
Architecture
Constraints
Competitor sites frequently block traditional scrapers. Changes can be visual (layout) or structural (DOM). LLMs are highly susceptible to prompt injection from malicious competitor sites.
System Design
We built a distributed scraping pipeline using headless browsers that capture full-page screenshots. These images are fed into Gemini 2.0 Vision to extract structured diffs. A custom scoring engine ranks the business impact of these changes (e.g., pricing update = High, typo fix = Low).
Implementation Details
The system includes a "Lobster Trap" security layer—a middleware that sanitizes all extracted text to prevent competitor sites from injecting malicious instructions into our LLM pipeline. Background jobs run on dedicated worker nodes via queues to ensure reliable scraping at scale.
Tech Stack & Rationale
Next.js & Vercel
Chosen for edge-optimized rendering and scalable serverless API routes for webhook ingestion.
Gemini 2.0 Flash Vision
Allowed us to extract UI diffs and competitive pricing tables directly from raw screenshots, bypassing DOM obfuscation.
PostgreSQL
Used for strictly typed relational storage of competitor telemetry and historical pricing data.
Outcomes
The platform achieves 60-second signal extraction (compared to days for human analysts). It successfully defends against 100% of tested prompt injections and processes thousands of URLs daily.
Need something built to this standard?
Let's talk about your product, your constraints, and whether we're the right team to build it.
Book Technical Discovery