What It Takes to Label 67,962 Reviews With an LLM
Sentence-level labeling across an 84-aspect ontology, 26 ontology versions, and a model bake-off that kept 95 percent of quality at a quarter of the cost.
"Just run it through an LLM" is the most popular sentence in data work right now. Here is what it actually took to turn 67,962 hotel guest reviews into 285,344 structured, queryable observations for Doha's luxury hotel market.
The ontology is the product
The model reads every sentence of every review and labels it against a hospitality ontology: 16 topics, 84 leaf aspects, each with sentiment, intensity, severity, and confidence. That ontology went through 26 versions and over a thousand rule-set iterations before it held up against real text.
That is the unglamorous truth of LLM systems: the model is a commodity, the schema is the asset. "Breakfast" is not one thing. Variety, quality, service speed, and the omelet station are different aspects with different owners inside a hotel, and a general manager can only act on feedback that lands at that resolution.
Cost engineering is model engineering
We benchmarked frontier models against open-weight alternatives before committing:
- The best model a 16GB GPU could run scored 46 percent on our labeling benchmark. A mid-size model reached 62 percent. A 120B model on CPU hit 83 percent but was unusably slow.
- The configuration we shipped, a frontier model with reasoning disabled, kept roughly 95 percent of top-tier quality at about a quarter of the cost.
- Our first cost estimate was off by 14x. We only found out because we measured before scaling.
In production, a 529-review backlog labels in about 25 minutes for roughly a dollar.
Pipelines fail; design for resumption
The labeling pipeline is parallelized and checkpointed in atomic 50-review chunks. When a run dies, and runs die, it resumes from the last completed chunk instead of re-billing the whole corpus. For long-running LLM jobs, idempotent checkpointing is the difference between an incident and a non-event.
The lesson
LLM projects do not fail at the model. They fail at the ontology, the cost model, and the plumbing. Get those three right and the model choice becomes a line item you revisit quarterly.
The full build, including the competitive-set analytics and the tenanted client portal, is in the case study below.