The LLM is just an unreliable third-party REST API - Part I

I’ve been trying to get a mental model of agentic AI applications into something that I’m familiar with when thinking about building an application. The first thing that came to mind was web applications. You have your front end, you have your service layer, you have your data store on a high level. The following is an example of building a very crude, very simple CLI that given an array of reported issues or bugs will use an AI agent to go through a discriminated union and classify them in different buckets.

While I was thinking about this, obviously one of the major players in any agentic application is the large language model. And in past projects, I’m sure as it is right now for so many of you, one of the unpredictable and unreliable pieces were third party APIs, therefore, it is not too much of a stretch to say:

The LLM is just an unreliable third-party REST API

That’s the one-sentence mental hook. Every LLM-specific design decision collapses into something a backend programmer has already done, talking to Stripe, Twilio, a flaky internal microservice.

So, treat the LLM like a third-party API you don’t trust and your architecture writes itself.

Here’s what running the CLI on four representative bug reports actually produces, one input per classification bucket (full output in the repo at sample-cli-run.json):

[
  {
    "classification": "actionable_ticket",
    "report_id": "report-0",
    "original_input": "Deleted a project this morning. Now if I paste the old URL into Chrome it still loads our app shell...",
    "title": "Deleted project URL loads app shell with no navigation back to Projects",
    "severity": "medium",
    "category": "project_lifecycle",
    "extraction_confidence": "high",
    "expected_behavior": "Navigating to a deleted project's URL should either redirect the user to the Projects list...",
    "observed_behavior": "The app shell loads on the deleted project URL. The left sidebar shows 'No chats yet'...",
    "steps_to_reproduce": [
      "Create a project and note its URL.",
      "Delete the project via the normal project deletion flow.",
      "..."
    ],
    "engineering_notes": "The 404 error state is being rendered correctly at the data/API level, but the UI route handler...",
    "missing_information": ["Browser version / OS", "..."]
  },
  {
    "classification": "partial_ticket_needs_clarification",
    "report_id": "report-1",
    "original_input": "Contract review gave me a totally wrong answer about the indemnity clause...",
    "title": "AI incorrectly reports unlimited liability when contract clearly caps it at $1M",
    "severity": "high",
    "category": "ai_response",
    "...": "(all actionable_ticket fields, plus:)",
    "clarifying_questions": [
      "Could you share the PDF (or a sanitized version)?",
      "What was the exact question you typed to the AI?",
      "..."
    ],
    "what_unlocks_actionable": [
      "Providing the source PDF or the verbatim indemnity clause text...",
      "..."
    ]
  },
  {
    "classification": "too_vague_request_more_info",
    "report_id": "report-2",
    "original_input": "upload is broken",
    "interpretation": "The user reports that 'upload is broken' but provides no additional context...",
    "clarifying_questions": [
      "What type of file were you trying to upload?",
      "What exactly happens when the upload fails?",
      "..."
    ],
    "suspected_category": "upload"
  },
  {
    "classification": "non_bug_support_question",
    "report_id": "report-3",
    "original_input": "How do I export my project to Word?",
    "suggested_route": "feature_request",
    "suggested_response": "Thanks for reaching out! What you're describing isn't a bug report, but rather a question about available functionality...",
    "reasoning": "The user is asking how to perform an action and notes they can only see PDF in the menu..."
  }
]

Layer-by-layer mapping

The code lives at github.com/KazChe/bug-cli-ai-agent

Where the analogy buys us specific insights

Each of these is a “you already do this with web APIs” reframe of an LLM-specific design move. Gold for the blog:

That last row is the framing that lands hardest: the live eval isn’t a test, it’s monitoring of a vendor.

But as everything, not all mental models are one-to-one. There are places that this thing breaks, and the following is what’s new with building with LLMs…not an exhaustive list, so please don’t cancel me. That’s Part II.

• Non-determinism. Stripe with the same input → same output. LLM → not. That’s why the eval has an agreement threshold, not an equality assertion.

• The schema is what you hope for, not what’s guaranteed. Stripe’s OpenAPI is contractually enforced; Anthropic’s tool input_schema is a strong suggestion the model usually follows. That’s why strict post-call validation is the actual second line of defense, not the first.

• Cost scales with request body, not just request count. Caching the system prompt isn’t a parity move with HTTP caching; it’s a token economics move that has no clean web-API analog.

• Failures are about meaning, not transport. A 500 from Stripe means “couldn’t process.” A weird answer from an LLM means “processed it and produced nonsense.” That’s why your ClassifyError has stages (llm_schema, no_tool_use, final_schema) instead of just HTTP-style status codes.

Part II goes deep on each of the four, with code and an eval run.

Code lives at github.com/KazChe/bug-cli-ai-agent.