Feedback Loops and Retraining: Why Products Without Them Plateau in Month 3

The product at day 1 is the same product at day 365 if there's no feedback loop. That's not a metaphor; that's data.

Products with feedback loops compound. Every month, retrieval gets better. Outputs get more accurate. User satisfaction increases. Cost per successful outcome decreases.

Products without feedback loops plateau in month 3 and stay there until they're shut down. Feedback loops are Layer 6 of the six-layer system: the layer that turns a static product into one that learns.

What You're Actually Measuring

Most teams measure the wrong things: accuracy on a benchmark, latency, cost per token. All noise. A benchmark score from model evaluation tells you how the model performs on a fixed test set, not how your product performs with real users. Both matter, but only one of them improves month over month.

The signals that matter:

Signal 1: Did the user accept the output? Thumbs up / thumbs down, explicit selection, no correction needed. If users accept 85% of outputs, your system is working. If they accept 40%, it's not.

Signal 2: Did the user correct the output? "You said X, but the answer is Y." This is pure gold. The user told you exactly where you're wrong. Log it. Analyze it.

Signal 3: Which retrieval result did the user actually use? Your system retrieved 5 documents. Which one did the user click on? That's the signal that your ranking was right (or wrong). Over time, this tells you which ranking signals actually work, and it feeds directly back into your retrieval layer.

Signal 4: How long did the user read the output? If the output is 200 words and the user reads for 3 seconds, they're not reading it. If they read for 30 seconds, they're engaged. Time-on-page is a signal of output quality.

Signal 5: Did the user ask a follow-up question? If the first answer was good, many users just leave. If they have to ask again, your first answer was incomplete. Log it.

These five signals are worth 100x more than "accuracy on a benchmark." They're real user behavior, not laboratory conditions.

What to Log (The Infrastructure)

Day 1: build logging infrastructure.

{
  query_id: uuid,
  timestamp: ISO8601,
  user_id: (hashed),
  query_text: string,
  query_length: int,
  query_embedding: vector,
  retrieval_results: [
    { rank: 1, document_id, score, source },
    { rank: 2, document_id, score, source },
    ...
  ],
  retrieval_latency_ms: int,
  model_selected: string (e.g., "13B" or "70B"),
  model_output: string,
  output_length: int,
  output_latency_ms: int,
  model_cost_cents: float,
  verification_passed: boolean,
  verification_checks: {
    grounded: boolean,
    coherent: boolean,
    complete: boolean,
  },
  user_feedback: {
    thumbs_up: boolean | null,
    correction: string | null,
    follow_up_query: boolean,
  },
  retrieval_clicked: int | null (rank of document user selected),
  time_on_page_seconds: int,
  conversion: boolean (did the user do what they intended?),
}

This is 20 fields. It's not complicated. You need it all. Seriously. Every field.

Why? Because you don't know which signals matter until you have data. Some signals will surprise you. Log everything and analyze later.

The Monthly Analysis (Where Insight Lives)

Week 1-3: ship features, respond to users, maintain the system. Week 4: analysis.

Analysis Step 1: Acceptance Rate

acceptance_rate = thumbs_up / (thumbs_up + thumbs_down + corrections)
goal = 85%

If acceptance drops, why? Analyze:

• By query category: "Tech support questions have 78% acceptance, billing questions have 52%"
• By model selected: "Outputs from the big model have 91% acceptance, small model 67%"
• By retrieval quality: "When retrieval is empty, acceptance is 12%; when full, 89%"

This tells you where to focus.

Analysis Step 2: Correction Patterns

corrections = corrections with explicit text
patterns = group by: what was the user correcting?

Example output:

• "Told me the return policy was 30 days, actually 14" (15 corrections) → retrieval problem or hallucination problem?
• "Didn't mention the exception for digital products" (12 corrections) → retrieval is incomplete
• "Code syntax is wrong; won't run" (8 corrections) → model problem

Fix the top patterns. Ignore the noise.

Analysis Step 3: Retrieval Ranking Quality

clicked_rank = average rank of documents user actually used
goal < 2 (user should click top-2 result)

If clicked_rank is 4.2 instead of 1.5, your ranking is broken. This is the highest-signal feedback you have for retrieval architecture: which documents users select tells you exactly how to re-rank. Why is it broken?

• New ranking signals not working?
• Corpus quality degraded?
• Vector embedding quality dropped?

Debug it.

Analysis Step 4: Latency Impact

acceptance_rate by latency_bucket:
  0-100ms: 87%
  100-500ms: 84%
  500ms-1s: 78%
  > 1s: 62%

If latency > 1s tanks acceptance, you have an SLA problem. If latency doesn't matter, don't optimize for it (save the cost).

Analysis Step 5: Cost vs. Acceptance

for each model / retrieval strategy:
  cost_per_query = sum(retrieval_cost + model_cost + verification_cost)
  acceptance_rate = acceptance for that path
  cost_per_successful_outcome = cost_per_query / acceptance_rate

This is your north star metric. If cost_per_successful_outcome is 3x higher for a low-traffic path, shut it down. If it's better, invest there.

The Feedback Loop (How to Close It)

Monthly analysis → insights → action. Then what?

Action 1: Improve Retrieval Insight: "Billing questions have 52% acceptance; retrieval is returning generic results." Action: Tag documents with "billing" category. Add category-specific ranking signal. Re-rank. Measurement: Repeat next month. Did acceptance improve?

Action 2: Update Prompts Insight: "Corrections often include 'but there's an exception for...' Your prompts don't mention exceptions." Action: Update the system prompt: "Always mention relevant exceptions." This is the fastest lever you have, and prompt iteration costs nothing but a redeploy. Measurement: Do exception-related corrections drop?

Action 3: Update Routing Insight: "Small model outputs for billing questions have 50% acceptance. Big model has 88%." Action: Add billing questions to the "route to big model" list. Feedback is what makes routing decisions empirical instead of guesswork: you move a query class to the bigger model because the acceptance data told you to. Measurement: Did acceptance improve? Did cost increase acceptably?

Action 4: Add Verification Checks Insight: "Code generation correction pattern: syntax errors in 20% of outputs." Action: Add a syntax validation check. Reject outputs with syntax errors and retry. Correction patterns are the best source of new verification rules: every recurring correction is a check you should have been running. Measurement: Did code correction rate drop?

Action 5: Expand Corpus Insight: "Empty retrieval happens on 8% of queries. When it does, acceptance is 12%." Action: Identify the 8% of queries that get empty retrieval. Find content to cover them. Add to corpus. Measurement: Does empty retrieval rate drop?

This is the loop: measure → analyze → act → measure again.

Do You Actually Need to Retrain?

Note what none of those five actions require: a new model. Teams reach for retraining first because it feels like the serious answer, but it's usually the wrong one. Retraining is slow, expensive, and hard to attribute. Prompt changes, routing changes, and corpus expansion are fast, cheap, and measurable.

The decision is a cost-benefit call. If the failure is "the model doesn't know X," fix retrieval or the corpus first. If it's "the model is right but too slow or too expensive," that's an inference optimization question: distill or quantize before you retrain from scratch. Retraining earns its cost only when the failure is a genuine capability gap that no amount of context or routing closes. That's rare in the first year.

Real Velocity From Feedback Loops

Here's what we've seen in production:

Month 1: Baseline. 100% = 82% acceptance, $0.12 cost per outcome Month 2: 3 quick fixes from feedback. 85% acceptance, $0.11 cost per outcome Month 3: Retrieval ranking improved. 87% acceptance, $0.10 cost per outcome Month 4: Routing optimized. 88% acceptance, $0.08 cost per outcome Month 5: Verification added. 91% acceptance, $0.09 cost per outcome (slight increase for quality) Month 6: Corpus expanded + prompt updates. 92% acceptance, $0.08 cost per outcome

6 months of data-driven iteration. 10% improvement in acceptance, 33% drop in cost per outcome.

No new models. No retraining. Just measurement + iteration.

The Checklist (Before You Ship)

• You have logging infrastructure for all 20 fields above (or at least the top 10)
• You're computing acceptance_rate weekly
• You have a monthly analysis process (reserve 4 hours, analyze, generate action items)
• You track: acceptance by category, acceptance by model, acceptance by retrieval quality, cost per outcome
• You have a process to implement feedback: ranking updates, prompt changes, routing changes, verification checks
• You measure the impact of each change (did the metric improve?)
• You're not just logging; you're closing the loop (measure → act → measure again)
• You've communicated to the team: feedback loops are not nice-to-have, they're load-bearing

The Bottom Line

Products with feedback loops are alive. They get better every month. Users notice. Costs drop. Acceptance improves.

Products without feedback loops are dead in the water. Same quality at month 6 as at month 1. No learning. No improvement.

Build the feedback loop on day 1. It takes 6 hours. It pays forever.