API Evangelist Conversation with Patrick Kelly on Context Engineering, Code Mode, and Building an Accountability Layer for MCP

I can tell you how we got here. Sideko is an API tooling company I started with my cofounder Elias a couple of years ago. We were building tools to help engineers and product teams turn APIs into nice, shiny, developer- friendly products — taking API specs and automatically creating documentation, SDKs, and mock servers from those specs. We had customers like Prudential using it and a bunch of startups generating SDKs with us. But with AI and the way things are going, the consumer is now much more commonly an agent. There are still humans writing integration code, but it’s rapidly becoming agentic consumption. MCP came out and our customer and sales conversations started switching over to it. Then this code mode technique appeared — you literally give an agent an SDK, expose documentation to it, and let it execute functions. We looked at that and said, this is our bread and butter, let’s just reapply our tech. That’s the shift to Port of Context.

For us the main thing is being able to iterate on our approach to context engineering in a way that makes sense. One way of running our tool: we translate any number of upstream tools and MCP servers down into three — list what’s available, get more details about the SDK, and execute code. We’re turning all of that into three tools so the agent can use them more efficiently. That’s one way of engineering the right context window for an agent. We have other techniques too — a virtual file system technique where we just give it bash tools and execute, so it can go learn about the SDK with grep, look at the types, look at what functions are available, and then execute. The reason it’s tactical is that something as small as changing one sentence in a tool’s description can change behavior, and knowing the result of that one change is actually really hard.

We want to evaluate API usage — how well did the agent call the API, did it use the right parameters, did it do the steps in the right order? Most of the eval frameworks out there are very imprecise. They’ll just take what the LLM says at the end and judge that, hoping it actually made the right calls in the right order. That’s not really what we’re trying to test. So we built something like a bare-bones proxy that’s actually a state-control system in front of the upstream MCP servers. The agent starts the eval from a blank slate, we store state as it modifies things upstream, and when the eval ends we clear state — it deletes everything that was created. Then we can look at the state of the upstream servers right at the end and say, is it what we want, and report back. That’s precise measurement of what actually happened upstream. It took a while to build, and honestly it’s an ongoing search for the right tool for the job.

Primarily we’re dealing with enterprises converting OpenAPI to MCP — that’s the bread-and-butter use case and the fastest way to start playing with this stuff. Because of our time at Sideko we have a lot of tools for improving specs and getting them in good shape, so we use linters with rules specific to MCP generation. For SDK generation we made sure all the examples matched the schemas; in MCP generation the descriptions matter more than the examples. A silly thing we look for: the operation description and the longer description field are often identical — “deletes a pet” and then “deletes a pet.” We lint to make sure that’s not happening and that things becoming tools actually have proper descriptions. We’re more focused on cleaning the mess, or making things work in a messy environment. If a description is far too long, code mode makes it manageable because we’re not shoving the whole thing up front — we let the agent explore dynamically rather than filling up the window.

Exactly. That translation into three tools is one way of engineering the right context window for an agent to use these things, and we have other techniques alongside it. What we really want is to be able to say: if you’re using Anthropic Sonnet, this is the best technique; if you’re using Opus, this one is better. We want to make that whole map for recommending models and which techniques work best, because the techniques are changing so quickly. We want to be the catalog and the guidance — and not just “go read these docs and craft it this way,” but runtime, inline guidance in the moment, so we can pivot and switch on the fly. The goal is to bring all of that together into a much richer, more refined context window that’s going to be successful.

Ideally people are doing both. They’re thinking strategically about their assets — their OpenAPIs, their descriptions — and they have a perfect map. Once they’ve gotten there, it’s time to turn on the agents, and the agents will both have the right definitions in front of them, good descriptions of what’s available, and then the tools they need to effectively manage those descriptions and execution environments. That’s the ideal state. You think strategically about the assets, and then the tactical layer makes the agents actually use them well. We’re looking at having to figure out benchmarking using these assets, and maybe contributing some, to build a framework that can run all of this — because there’s just nothing out there that serves our requirements yet.

I love the idea of rallying behind vacuum and libopenapi. It’s been amazing to work with — it’s incredible how performant it is and how battle-tested it became very quickly. It’s not even an old project and there are already so many technologies dependent on it. Let’s keep promoting it and rally behind it. The number of times I’ve had a spec open on a call with a customer and told them to see if it runs in Swagger — it usually crashes the whole thing — and then I show them the OpenAPI doctor and it actually works there. That’s been so helpful. And ideally we figure out the open-source aspect of what we build alongside the commercial one: get a full set of the biggest APIs out there as a dream eval case, share specs, and build something on top of that common foundation rather than everyone keeping it behind the scenes.