Skip to content

News Intelligence Pipeline

A constraint-based pipeline for real-time news analysis, built in five files to show clean architecture. It ingests news over HTTP, scores sentiment and surfaces trending topics, and serves an interactive Streamlit and Plotly dashboard, packaged with Docker.

PythonStreamlitPlotlyTextBlobPostgreSQLDocker

Context

A real-time news-analysis pipeline built under a deliberate constraint: exactly five files, to prove that clean architecture does not require a sprawling codebase.

Problem

Continuously ingest news, score sentiment, surface trending topics, and present the results interactively, while keeping the system simple and reliable.

My role

Sole author: the ingestion, the sentiment and trending analysis, the dashboard, and the containerization.

Architecture

Async HTTP ingestion (aiohttp and requests) collects articles; TextBlob scores sentiment and the pipeline derives trending topics; pandas, NumPy, and scikit-learn handle processing; an interactive Streamlit and Plotly dashboard presents the results; and Docker packages it for AWS/Azure-ready deployment.

Key decisions

  • Capped the whole system at five files to force clear module boundaries instead of incidental complexity.
  • Used asynchronous ingestion to keep throughput up while staying a single, simple process.
  • Chose TextBlob and lightweight analysis so the pipeline stays dependency-light and reproducible.

Outcome

A working, containerized pipeline with a live analytics dashboard. Any numbers shown come from collected sample runs, not a production deployment.