Korean search, RAG retrieval & data extraction — from someone who ran it in production

Korean web, HWP & PDF extraction

Korean public-sector data lives in HWP attachments and PDF tables that generic tooling drops silently. I extract them with tables intact, deliver CSV/JSON/API on a fixed schema, and run the source on a self-healing crawler that recovers when a page changes instead of quietly stopping. API first when one exists. Measured table preservation: 99.4% on 500 attachments from one day of 나라장터 notices (2026-09-08; preferred format, parse errors excluded).

Korean search relevance for Elasticsearch/OpenSearch

For global SaaS and overseas services entering the Korean market: Nori and other CJK tokenizers, user dictionaries, synonym sets, analyzer design for Elasticsearch and OpenSearch. Most "our RAG answers are off" problems are retrieval problems, and most Korean retrieval problems start at the tokenizer. On 183,240 public-tender titles, a corpus-built user dictionary took P@10 from 0.906 to 0.986 — details in the note below.

Hybrid retrieval

BM25 + kNN, fused and evaluated against a frozen golden set of failing queries — before/after measured, not asserted.

Background

Twenty years as a software engineer. At Korea Newswire (2013–2025, principal engineer) I designed and ran the Elasticsearch architecture behind a media-monitoring service: index and component templates, a custom tokenizer, Nori and Mecab customisation, user-dictionary maintenance from new-term extraction, monthly index rollover with snapshot policies, and DB-to-index consistency checks. The ingestion side was a crawling platform generating per-source instances for about 7,000 news sources, ~200K new articles a day, on Python, scrapy and playwright, run on-prem, on Docker Compose and on Kubernetes.

Why Korean RAG fails: start with the tokenizer →

When the table disappears, RAG states a wrong number with confidence →

Miriboa — a bid-document verification service I build and run: requirement extraction and matching over Korean and U.S. federal tenders, with a published benchmark. miriboa.sizlon.io →

Email [email protected] or use the Korean site’s contact form — English is fine.