Case Study: Support Ticket Triage + Draft Reply Automation

Background

LedgerFlow provides invoicing and payment tools for SMBs. Support volume scaled from 800 to 2,000 tickets/day after a pricing change brought a wave of billing questions. Median first-response time hit four hours. CSAT dipped.

Hiring was slow. They needed automation that made agents faster, not a bot that talked to customers unsupervised.

The problem

Every ticket arrived as unstructured text. Agents manually:

1. Read the ticket and guess intent (billing, bug, onboarding, escalation)
2. Looked up the account in three internal tools
3. Searched the knowledge base for the right macro
4. Personalized the macro and sent

Steps 1–3 took 6–10 minutes per ticket. Macros were stale. Escalations landed in the wrong queue one time in nine.

Architecture

Zendesk webhook (new ticket)
  → Classify intent + urgency (structured JSON)
  → Fetch account context (API)
  → Retrieve relevant KB articles (hybrid search)
  → Draft reply (LLM)
  → Present draft in agent sidebar (Zendesk app)
  → Agent edits / approves / rejects
  → Log diff for prompt refinement

No ticket was sent without agent approval in v1. Two low-risk intents (password reset, invoice lookup) earned auto-send in v2 after review.

Implementation phases

Phase 1 — Classification only (2 weeks)

They trained the team on nothing — instead, they built a 200-ticket labeled set from historical tickets and used GPT-4.1 mini with structured outputs:

{
  "intent": "billing_dispute | bug_report | onboarding | account_access | escalation",
  "urgency": "low | medium | high",
  "confidence": 0.0–1.0,
  "suggested_queue": "tier1 | tier2 | billing_specialist"
}

Tickets with confidence < 0.75 routed to a human triager. Accuracy on the holdout set: 94%. Misroutes: 3.2%.

Phase 2 — Draft replies (4 weeks)

For the top five intents (covering 60% of volume), the pipeline added:

• Account tier, MRR, and last 3 ticket summaries from Postgres
• Top 3 KB articles via hybrid search
• A draft reply in the agent's voice guidelines

Agents saw the draft in a Zendesk sidebar app. One-click insert, edit, send. Acceptance rate started at 41% and climbed to 67% over six weeks of prompt iteration.

Phase 3 — Selective auto-send (week 8+)

Password reset and invoice-link requests hit 97% draft acceptance after review. Leadership approved auto-send for those intents only, with a post-send audit sample of 5%.

Results

Agents reported the biggest win wasn't the draft — it was automatic account context. "I used to have four tabs open before I could write a sentence."

Observability

Every LLM call went through Langfuse with:

• Intent label vs. agent override
• Draft text vs. sent text (diff)
• Time-to-first-response per ticket
• Cost per ticket (~$0.04 average)

Weekly reviews focused on rejected drafts and large diffs — those became prompt changelog entries.

What they'd do differently

1. Label 200 tickets before writing any code — they built the classifier twice because the first label schema was too granular.
2. Instrument agent overrides from day one — the diff logs were the best training signal they had.
3. Set CSAT as a guardrail metric — they almost auto-sent billing disputes in v2; a one-week CSAT dip stopped it.

Takeaway

Support automation isn't about replacing agents — it's about removing the 10 minutes of lookup before the agent can be helpful. Classification + context + draft, with a human approving every customer-facing word, was the pattern that actually shipped in a regulated industry.