Web Intelligence Crawler

Problem Statement

Manual research on companies, markets, and technical topics does not scale. We needed a system that could crawl sources on a schedule, evaluate relevance, store embeddings for retrieval, and improve queries over time — not a one-off scraper script.

Solution Approach

Web Intelligence Crawler is a full platform with isolated projects, continuous operation, and agent-assisted feedback:

• Crawl layer: Scheduled discovery and parsing across configured sources
• Storage: GCS-backed artifacts plus ChromaDB vector store for semantic search
• RAG: LangChain document processing, chunking, multi-query retrieval, and re-ranking
• Agents: Orchestrated evaluation and feedback loops that adjust keywords and queries from performance data
• Operations: FastAPI REST API, React dashboard, metrics, and alerting

Technologies Used

• Backend: Python, FastAPI, LangChain
• Search: ChromaDB embeddings and RAG query paths
• Frontend: React monitoring dashboard
• Cloud: Google Cloud Storage integration
• Ops: Deployment guides, API docs, and phase-by-phase implementation notes

Key Achievements

• Multi-project architecture with isolated data per research area
• Continuous 24/7 crawling with scheduler and monitoring
• Multi-agent workflow: evaluate content, analyze patterns, update configuration
• Complete documentation set (setup, testing, deployment, API reference)

Impact

• Reference implementation for agentic feedback loops in the NWP engineering handbook
• Reusable pattern for research automation, competitive intelligence, and content orientation
• Demonstrates data + AI delivery beyond static dashboards and ETL templates

Lessons Learned

• Feedback loops need measurable evaluation criteria before agents can improve queries
• Vector stores require clear project boundaries to avoid cross-contamination
• A monitoring dashboard is essential when crawlers run continuously