End-to-end information retrieval system — crawler, indexer, ranker, and UI — built in Java with no external search frameworks.
Ingests 400+ RSS feeds via an OPML feed list. Respects robots.txt, enforces per-domain rate limits, and validates redirects to block SSRF.
Custom tokenizer with Porter stemming and bigram generation writes a compact binary inverted index. Token and doc metadata are stored in flat text files for fast loading.
Queries score with BM25 for term relevance, multiplied by a PageRank signal built from the inter-blog citation graph. Domain diversification prevents one blog from dominating results.
"More like this" uses dense embeddings and cosine similarity to surface related posts. K-means clustering groups the full corpus into browsable topic areas.