How to Classify App Reviews With an LLM (Bugs, Requests, Praise)
Classify app reviews with an LLM: a closed taxonomy, a strict JSON schema, real code for bugs, requests and praise, and the pitfalls that wreck accuracy.
The Argus Team
Reply Argus
You classify app reviews with an LLM by handing the model one review at a time, a closed list of categories, and a strict JSON schema it has to fill. You read back a structured verdict (`category`, `sentiment`, `confidence`) instead of prose. Do that across your whole review backlog and 'we have 4,000 reviews and no idea what people actually want' turns into a sortable table: bugs here, feature requests there, praise you can ignore, and the confusing middle flagged for a human.
The naive version is a one-line prompt — 'is this a bug or a feature request?' — and it half-works until it doesn't. The version that survives real reviews is a taxonomy you designed on purpose, a schema the model can't wander out of, and an escape hatch for the reviews that don't fit. Here's all of it: the categories, the prompt, the schema, working code, and the four failure modes that quietly poison your accuracy.
Why not just use keyword rules?
You get surprisingly far with `if 'crash' in review: tag = 'bug'`. Then you meet the reviews. 'The app *crashes* prices, love the deals' is praise. 'Please stop it crashing my old phone' is a feature request wearing a bug's clothes. 'It's not that it crashes, it just feels sluggish' negates the word 'crash' mid-sentence. Keyword rules can't read intent, negation, or sarcasm, and app reviews are made of all three.
An LLM reads the sentence, not the tokens. It knows 'would be 5 stars if it had dark mode' is a feature request from a happy user, that '만들어주세요' ('please make it') belongs in the same bucket as the English asks, and that a 1-star saying 'perfect, exactly what I needed' is mistaken star-tapping or sarcasm — worth a low-confidence flag, not a silent miscategorization. That semantic read is the whole reason to reach for a model.
What categories should you classify into?
The single biggest quality lever is a closed, mutually-exclusive taxonomy you commit to before you write the prompt. Open-ended 'summarize the theme' output is unclusterable — the model will invent 200 near-duplicate labels. Give it a fixed enum and it snaps every review to a bucket you can actually count. A taxonomy that holds up across most consumer apps:
- bug — something is broken: crashes, data loss, a button that does nothing, sync failures. The reviewer expects it to work and it doesn't.
- feature_request — asking for something that doesn't exist yet: dark mode, an export, a language, an integration. Often phrased as 'would be great if…'.
- ux_complaint — it works, but it's confusing, slow, or annoying: bad onboarding, too many taps, an intrusive paywall placement. Not a crash, still friction.
- pricing — anything about cost, subscriptions, billing, or 'too expensive / used to be free'. Route these to product and growth, not engineering.
- praise — positive, no action required. Worth counting (it's your sentiment baseline) but it doesn't create a ticket.
- support_question — a user stuck on something you can answer: 'how do I restore purchases?' These deserve a real reply, fast.
- spam_or_irrelevant — bots, competitor drive-bys, reviews for the wrong app, empty 5-star noise. Filter these before they pollute your themes.
- other — the deliberate escape hatch. If a review fits nothing above, it lands here with low confidence and gets a human glance — not a wrong bucket.
Keep it to roughly six to eight primary categories. Fewer and you lose signal (bugs and performance blur together); more and the model's accuracy per class drops and your inter-label boundaries get fuzzy. Let a review carry one primary category plus optional secondary tags — 'crashes on the pricing screen' is a bug first, `pricing` second, so you don't force a false either/or.
The schema: force structured output
The mistake that eats a weekend is parsing prose. Ask the model to 'reply with the category' and you'll get 'This review appears to be a bug report, though it could also…' and your `json.loads` throws. Every serious LLM provider now exposes a structured-output / JSON-schema mode (also called response-format or tool/function calling) that constrains generation to a schema you define. Use it. The `enum` on `category` is what stops the model from inventing a 200th label.
{
"type": "object",
"properties": {
"category": {
"type": "string",
"enum": ["bug", "feature_request", "ux_complaint",
"pricing", "praise", "support_question",
"spam_or_irrelevant", "other"]
},
"secondary_tags": {
"type": "array",
"items": { "type": "string" }
},
"sentiment": { "type": "string", "enum": ["negative", "neutral", "positive"] },
"actionable": { "type": "boolean" },
"confidence": { "type": "number", "minimum": 0, "maximum": 1 },
"summary": { "type": "string", "description": "One clause, <=12 words" }
},
"required": ["category", "sentiment", "actionable", "confidence"]
}The prompt and the classification call
The system prompt does three jobs: it defines each category in one line (don't assume the model shares your definitions), it forbids inventing categories, and it tells the model to lower `confidence` and choose `other` when unsure rather than guessing confidently. Then you batch — classifying reviews one HTTP call each is slow and expensive, so send 10–20 per request and get back an array. Here it is provider-neutral; swap `call_llm` for your provider's structured-output method.
import json
SYSTEM = """You triage mobile app reviews.
Classify each review into exactly ONE primary category from the enum.
Definitions: bug = something broken; feature_request = asks for a
feature that doesn't exist; ux_complaint = works but confusing/slow;
pricing = cost/billing/subscription; praise = positive, no action;
support_question = user needs help you can answer;
spam_or_irrelevant = bots/wrong-app/noise; other = fits nothing above.
Rules: never invent a category. If a review is ambiguous, sarcastic,
or doesn't fit, choose "other" and set confidence below 0.5.
Set actionable=true only if it implies work for the team."""
def classify_batch(reviews):
numbered = "\n".join(f"[{i}] ({r['stars']}\u2605) {r['body']}"
for i, r in enumerate(reviews))
result = call_llm( # your provider's structured call
system=SYSTEM,
user=f"Classify each review.\n\n{numbered}",
schema={"type": "array", "items": REVIEW_SCHEMA}, # schema above
)
return json.loads(result)
# Feed it real reviews pulled from the store APIs, then bucket:
verdicts = classify_batch(reviews)
bugs = [r for r, v in zip(reviews, verdicts)
if v["category"] == "bug" and v["confidence"] >= 0.6]Where do the `reviews` come from? Both stores expose them over their APIs — the [Google Play reviews API reference](/blog/google-play-reviews-api-reference) and the [App Store Connect reviews API reference](/blog/app-store-connect-reviews-api-reference) cover the auth, the pagination, and the fields you'll feed the classifier. Pull, classify, bucket: that's the loop.
Four pitfalls that quietly wreck accuracy
The classifier will look like it works on your first ten reviews and then drift on the next thousand. These are the failures that don't announce themselves:
- Ignoring confidence. A model forced to pick will always pick, confidently, even when wrong. Read the `confidence` field. Anything below your threshold (start around 0.6) goes to a human queue, not straight into the counts. A pipeline that trusts every label at face value is a pipeline lying to you at scale.
- No 'other' bucket. Without an escape hatch, genuinely novel reviews get crammed into the nearest wrong category and vanish. The `other` class plus a low-confidence flag is how weird-but-important signal survives instead of being averaged away.
- Language blindness. A huge share of reviews aren't in English, and a classifier prompted only in English silently degrades on Portuguese, Korean, or Arabic. Modern LLMs handle 100+ languages natively, but test on non-English samples explicitly, because 'looks fine on my English test set' hides the drop.
- One-shot, one-label tunnel vision. Real reviews are compound: 'love it, but it drains my battery and please add widgets' is praise + bug + feature_request. Force a single label and you lose two of three signals. Primary-plus-secondary-tags captures the compound ones; a hard single-choice schema throws them on the floor.
The model will hallucinate a category if you let it
Without an `enum` constraint, an LLM asked to 'categorize' reviews will happily emit 'user_delight', 'onboarding_friction_v2', and forty other freelance labels: unclusterable, uncountable, different every run. The closed enum in your schema is not optional polish; it's the difference between a dataset you can pivot-table and a pile of synonyms. Constrain generation, then validate the output against the same enum before you write it anywhere.
The payoff of good classification isn't the label — it's what happens next. Once a review is tagged `bug` with high confidence, it can be routed, drafted, and answered in the same pass. Here's a 2-star review classified as a bug (secondary tag `pricing`), then the reply that closes the loop:
Was fine until the update, now it crashes every time I open the checkout to renew my subscription. So I literally can't pay you even though I want to. Pixel 8.
Sorry — that's a checkout crash we introduced on Android 14 in 6.1, and it's exactly the wrong screen to break. The fix is live in 6.1.2 (updating now from the Play Store should clear it), and your subscription hasn't lapsed on our side. If it still crashes after updating, reply here and we'll extend your term and pull the logs.
From labels to a roadmap: clustering the results
Per-review categories are step one. The prize is aggregation — 'we got 340 reviews this month, 61 are bugs, and 44 of those are the same login loop.' That's a second pass: within each category, embed and cluster the near-duplicates, or have the LLM group them into named themes with counts. Now you have a prioritized list, where the login bug outranks the one-off typo report because 44 people hit it, and that list *is* your roadmap.
You don't have to build any of this. [ReplyArgus](/features) turns your incoming App Store and Google Play reviews into an evidence-backed PM roadmap board — the recurring themes surfaced and ranked, each card exportable to Jira, Notion, DevRev, or Google Sheets — and drafts every reply in the reviewer's own language, so sorting and answering happen in one motion. You get the prioritized roadmap without owning the plumbing.
- 1
Step 1 — Pull the reviews
Fetch from the store APIs (or your own store). Normalize to a common shape: id, stars, body, language, date, platform. Reviews you can't read, you can't classify.
- 2
Step 2 — Design a closed taxonomy
Six to eight mutually-exclusive primary categories plus an 'other' escape hatch. Write a one-line definition for each — the model needs your definitions, not its own.
- 3
Step 3 — Force structured output
Define a JSON schema with an enum on category, a sentiment enum, and a confidence number. Use your provider's structured-output mode so the model can't return prose or a freelance label.
- 4
Step 4 — Batch and classify
Send 10–20 reviews per call, get back an array. Route anything under your confidence threshold to a human queue instead of the counts.
- 5
Step 5 — Cluster into themes
Within each category, group near-duplicate reviews by embedding or a second LLM pass, and attach counts. The count is the priority signal.
- 6
Step 6 — Act on the buckets
Bugs to engineering, feature_requests to the roadmap, support_questions and complaints to a reply. Classification is only useful if something moves because of it.
Skip the pipeline: hand structured reviews to an agent
If you're building this to feed an agent, you can skip the plumbing entirely. ReplyArgus exposes your classified, structured reviews over an [MCP connector](/blog/app-reviews-mcp), so you can ask Claude, ChatGPT, or Cursor 'group my unanswered 1-stars by theme and draft replies to the bug ones' and it operates on real store data — no scraping, no schema of your own. That's the whole [agentic-tools](/agentic-tools) surface: the classification is done; you just point an [AI agent for app reviews](/blog/ai-agent-for-app-reviews) at it.
Frequently asked
- How do you classify app reviews with an LLM?
- Send each review to the model with a closed list of categories (bug, feature request, praise, etc.) and a strict JSON schema it must fill, using the provider's structured-output mode. Read back a `category`, `sentiment`, and `confidence` per review, then bucket by category. Batch 10–20 reviews per call for speed and cost.
- What categories should app reviews be classified into?
- A closed, mutually-exclusive taxonomy of roughly six to eight: bug, feature_request, ux_complaint, pricing, praise, support_question, spam_or_irrelevant, plus an 'other' escape hatch for anything that fits nothing else. Allow one primary category plus optional secondary tags so compound reviews aren't forced into a false either/or.
- Is an LLM better than keyword rules for review classification?
- For app reviews, yes. Keyword rules can't handle negation ('doesn't crash'), sarcasm, intent ('would be 5 stars if…'), or non-English text. An LLM reads the sentence rather than matching tokens, so it correctly tags a feature request buried in praise or a bug phrased politely — the cases keyword filters get wrong.
- How do I stop the LLM from inventing its own categories?
- Constrain generation with a JSON schema whose `category` field is an `enum` of your allowed labels, using structured-output mode. That physically prevents freelance labels. Then validate the returned value against the same enum before writing it anywhere, and route low-confidence results to a human queue.
- Does it work for reviews in other languages?
- Modern LLMs classify across 100+ languages natively, so a Korean or Portuguese review lands in the right bucket without translating first. But test on non-English samples explicitly — an English-only prompt and test set can hide real accuracy drops on other languages.
- How much does classifying reviews with an LLM cost?
- It scales with volume and model, but batching 10–20 short reviews per call and asking for a compact schema (not prose) keeps it cheap — most apps classify a full month of reviews for a trivial amount. The bigger cost is engineering time to build, monitor, and re-tune the pipeline, which is why many teams use a tool that ships it.
Classification on its own is a spreadsheet; the value shows up when a bug gets routed, a feature request lands on the roadmap, and a stuck user gets a real answer — fast, because a 2-star review left unanswered for a week is a 2-star review that sticks. Google reported at I/O 2019 that developers who respond to reviews see an average lift of 0.7 stars, and you can't respond well to what you haven't sorted. If you'd rather not build and babysit the classify-cluster-reply pipeline yourself, [start free with ReplyArgus](/signup) — no card. It classifies your App Store and Google Play reviews, clusters the themes onto a roadmap board, and drafts your first reply in minutes, in the reviewer's own language.
Try it
Let Argus draft your next reply.
Watch it answer a real review in your voice. 10-day trial, no card to begin.
Keep reading
App Store Reviews Not Showing? The Full Fix List for Apple and Google Play
App Store or Google Play reviews not showing? Here's every real cause — moderation delay, wrong country storefront, filtering, rating lag, cache — and the fix.
Read moreAre App Store Reviews Fake? How to Spot Them and What to Do
Yes, some App Store and Google Play reviews are fake — but fewer survive than you think. How to spot them, and what to do when they hit your app.
Read more