Show HN: Large Scale Article Extract of Newspapers 1730s-1960s
Not a chatbot. A semantic search engine that reads 600k newspaper pages using multi-model AI pipelines to turn 230 years of unstructured archives into queryable intelligence.
Why it matters
Developer built an end-to-end AI product (layout detection → OCR → LLM classification → agentic search) that solves a real discovery problem in historical data. Demonstrates practical multi-model orchestration and AWS scaling challenges that enterprise teams face when productizing AI.
The key facts
13 to know600k+ newspaper pages extracted (5TB of content)
Time investment: 3,000 hours over 7 months
Date range: 1730s–1960s (230+ years)
Multi-model pipeline: layout tech + OCR + LLM + vLLM + heuristics
Backend: OpenSearch + Postgres + agentic search API
Source: Chronicling America collection
Core challenge: infinite layout variety, OCR quality, image segmentation
600k+ newspaper pages extracted (5TB of historical content, 1730s-1960s)
Multi-model pipeline: layout detection + OCR + LLM + vLLM + heuristic segmentation
Agentic search tool that writes queries on behalf of users
Built on OpenSearch + Postgres for semantic search
7 months development, ~3,000 hours engineering effort
Source: Chronicling America collection (public domain)
Go to the source
Hacker Newssnewpapers.com
Publisher excerpt: Hello HN, over the past 7 months I've spent nearly 3,000 hours on building SNEWPAPERS, the first historical newpaper archive with full-text extractions, nearly perfect OCR, a vast categorization taxonomy and of course with semantic and agentic search capabilities. Problem: I wanted to search…