Harness Engineering: Why the System Around the Model Matters More Than the Model

The model is not your product. The model is a function call inside your product.

I've spent the last couple of years building AI-powered systems at RapidClaims -- pipelines that process medical records, extract clinical information, and automate coding workflows that used to take humans hours. Before that, I was at Richpanel building customer support infrastructure. Across both, the pattern is the same: the intelligence layer is a small, replaceable piece of a much larger engineering puzzle.

The industry is obsessed with the wrong thing. Every week there's a new benchmark, a new model, a new prompting technique. Fine-tune this, few-shot that. Meanwhile, the systems that actually work in production -- the ones processing thousands of documents a day without falling over -- are defined by everything around the model. The orchestration. The retrieval. The error handling. The observability. The memory architecture.

I call this harness engineering. And it's where the real work lives.

The Demo-to-Production Gap Is an Engineering Gap

Everyone has seen the demo. You throw a document at GPT-4, get a beautiful structured response, and think: we'll ship this in two weeks.

Then reality hits.

The model hallucinates on edge cases. Latency spikes when you hit rate limits. Your pipeline processes one document fine but chokes on a batch of five hundred. The context window fills up halfway through a complex medical record. Retries create duplicate work. There's no way to tell which step failed or why.

None of these are model problems. They're engineering problems. And they require engineering solutions -- the same kind of distributed systems thinking that's been solving hard problems for decades before LLMs existed.

The gap between "works in a notebook" and "works in production" is not bridged by a better model. It's bridged by a better system.

The Model Is a Function Call

Here's how I think about it: in any serious AI system, the model call is one line in a much longer pipeline. Before it, you have data ingestion, parsing, chunking, embedding, retrieval, reranking, context assembly. After it, you have validation, structured extraction, error handling, logging, state management, downstream routing.

The model itself? It takes some text in and produces some text out. That's it. Everything interesting happens around it.

At RapidClaims, our pipelines combine semantic search, metadata filtering, reranking, and structured reasoning across large medical documents. The LLM is the reasoning step. But the retrieval pipeline that feeds it the right context -- that's where accuracy is won or lost. A perfect model with bad retrieval produces garbage. A decent model with excellent retrieval produces results you can trust.

When teams spend months fine-tuning a model to improve accuracy by 2%, they're often ignoring retrieval improvements that could give them 15%. The ROI on engineering the harness almost always beats the ROI on engineering the model.

Orchestration Is Not Optional

Once you move past single-shot inference, you need orchestration. Real AI workflows are multi-step. They branch. They fail partway through. They need to be retried, paused, and resumed.

We migrated from a legacy queue-based system to Temporal for workflow orchestration. The difference was night and day.

With queues, every failure mode was a special case. Message gets processed twice? Write a deduplication layer. Step three fails after step two succeeds? Hope your cleanup logic works. Need to replay a workflow from a specific point? Good luck.

Temporal treats workflows as durable, resumable programs. Each step is an activity with its own retry policy, timeout, and failure handling. If a workflow fails at step four, you fix the bug and replay from step four -- not from the beginning. State is persisted automatically. You get visibility into every running workflow, every pending activity, every failure.

This isn't about Temporal specifically. It's about the principle: AI workflows need first-class orchestration. Not scripts. Not queues with duct tape. Real workflow engines that handle the ugly parts -- retries, timeouts, idempotency, compensation -- so you can focus on the logic.

Retrieval Is Harder Than Generation

This is the thing nobody warns you about when you start building RAG systems: retrieval is the hard part.

Generation is straightforward. You give the model context and a prompt, it generates a response. The quality ceiling is set by the context you provide. Which means everything depends on retrieval.

And retrieval is a deep engineering problem. Chunking strategy alone can make or break your system. Chunk too small and you lose context. Chunk too large and you dilute relevance. Overlap too much and you waste tokens on redundancy. Overlap too little and you miss information that spans chunk boundaries.

Then there's the embedding pipeline. Which model? What dimensions? How do you handle domain-specific terminology that general-purpose embeddings don't capture well? How do you keep embeddings fresh when source documents update?

Semantic search gets you 70% of the way there. The last 30% comes from metadata filtering and reranking. In our medical coding pipelines, pure vector similarity isn't enough. You need to filter by document type, date ranges, patient context. Then you rerank results using a cross-encoder to surface the most relevant chunks. Each layer adds accuracy, but also adds latency and complexity.

Building a retrieval pipeline that's fast, accurate, and maintainable is a full engineering discipline. It's not a prompt engineering problem.

Observability or You're Flying Blind

Here's a question: when your AI pipeline produces a wrong answer, can you trace exactly why?

Can you see which documents were retrieved? What the similarity scores were? How the context was assembled? What the model saw? What it returned? How long each step took?

If you can't, you're guessing. And guessing doesn't scale.

We invested heavily in end-to-end telemetry early on. Every pipeline run produces a distributed trace -- from document ingestion through embedding, retrieval, reranking, inference, and output validation. Every step has latency metrics. Every LLM call logs the input tokens, output tokens, and response time.

This isn't luxury. It's survival. When a customer reports an incorrect code assignment, I can pull up the trace, see that the retrieval step returned irrelevant chunks because the metadata filter was too broad, and fix the root cause in minutes. Without telemetry, that same debugging session takes days of staring at logs and guessing.

Latency monitoring matters too. LLM APIs are not deterministic in their response times. P50 might be 800ms, but P99 might be 8 seconds. If you're not tracking percentiles, you don't know your actual user experience. You know your average, which is a lie.

Backpressure: The Art of Slowing Down

Every LLM API has rate limits. Every embedding service has throughput caps. Every vector database has query concurrency limits. The question is not whether you'll hit them. It's how your system behaves when you do.

The naive approach is retry with exponential backoff. It works for occasional failures. It doesn't work when you're processing a batch of a thousand documents and you hit rate limits on document fifty. Exponential backoff on 950 remaining documents means your pipeline takes hours instead of minutes.

Backpressure-aware orchestration is the answer. Instead of letting every worker blast requests at the API and then individually backing off, you apply pressure upstream. Slow down ingestion. Queue work at a rate the downstream systems can handle. Monitor throughput in real time and adjust.

This is standard distributed systems engineering. It's been solved in databases, message queues, and stream processing for years. But I see AI teams reinventing it badly because they don't realize the problem is the same.

When we moved to backpressure-aware orchestration through Temporal, our throughput actually increased. Counterintuitive, but it makes sense: you waste less time on failed requests and retries when you never exceed the rate limit in the first place.

Memory Is Architecture, Not a Prompt Trick

The default approach to memory in LLM applications is to stuff everything into the prompt. Previous conversation turns, retrieved documents, system instructions, user preferences -- all concatenated into one massive context window.

This works for demos. It fails for production.

Context windows are finite. Even with models supporting 100K+ tokens, you're paying for every token -- in latency and cost. More importantly, relevance degrades as context grows. The model's attention isn't uniform. Important information buried in the middle of a massive context gets less weight than information at the beginning or end.

Structured memory architecture means being intentional about what goes into context and how. Short-term working memory for the current task. Long-term memory in a vector store for retrieval. Metadata-indexed memory for fast lookup by category. Summarized memory for compressing historical context without losing key information.

Each layer has different storage, retrieval, and eviction characteristics. It's a caching problem. And like all caching problems, the strategy matters more than the storage.

The 80/20 That Nobody Talks About

If I had to put a number on it: the model is 20% of a production AI system. The other 80% is the harness.

That 80% includes:

• Orchestration and workflow management
• Data ingestion, parsing, and normalization
• Embedding pipelines and vector storage
• Retrieval, filtering, and reranking
• Context assembly and memory management
• Error handling, retries, and compensation
• Observability, tracing, and monitoring
• Backpressure and concurrency control
• Output validation and structured extraction
• Deployment, versioning, and rollback

Every item on that list is a real engineering problem with established solutions. None of them are solved by switching to a better model.

The irony is that when you build the harness well, swapping the model becomes trivial. It's a configuration change. The system is model-agnostic because the model was never the system -- it was a component.

The Real Competitive Advantage

Teams that treat AI as a model problem build fragile demos. Teams that treat AI as a systems problem build durable products.

The model will get cheaper. It will get faster. It will get smarter. OpenAI, Anthropic, Google -- they're competing on that front so you don't have to. Your job is not to build a better model. Your job is to build a better harness.

The engineering that makes AI work in production is not new. It's distributed systems. It's data engineering. It's observability. It's the same hard, unglamorous work that has always separated systems that work from systems that don't.

The model is the easy part. The harness is the craft.