API Evangelist Conversation With Derek Thompson of Ambassador

I’m Derek Thompson. I am both an API engineer by trade, but I also work, uh, Sort of in our space. I worked at a stoplight for quite a while before and through the acquisition by SmartBear. And now I’m over at Ambassador working on our new API platform, Blackbird. Uh, and I focus mainly on our new AI integration. So all the stuff we’re trying to get the LLMs to do for us to help out along the way of, uh, Building new and updating [00:01:00] existing API projects and testing and validating those projects anywhere we can we can find to sort of point these guys to help out a little bit is what we’re trying to do.

Yeah, and I do sort of want to clear up a little bit. We don’t actually do client SDK generation out of black. Our generation and code generation is pointed at the server implementation itself. So your API. So we have sort of like, uh, the first thing we rolled out was just a basic, uh, chat bot that allows you to generate a new service spec. Uh, you know, you can pick your version and go ahead and create your open API specification and then run The second thing we rolled out. Well, with that, we also included some AI tools for, uh, auto fixing existing specifications and the one you’re creating on the fly. So doing basic syntax checking were integrating spectral leaning in now so that you can auto fix any spectral leaning rules and sort of [00:03:00] supply your custom rule sets. And then the other Auto Piece of that, of course, is we wanted to offer an activity where you can update, uh, your existing open A P I spec. And the idea there for us is that that kicks off sort of your workflow. Like the way we want people to be able to use Blackbird is very much in line with sort of the open A P I initiative design first flow. The idea is you come together as a team and you sort of collaborate on what you want to change and you express that first with the A I in an iterative activity. Uh, in terms of changing the underlying specification. Uh, and that’s really like, I think, where we have a big advantage over the sort of standard code gen cases. Uh, where you’re just trying to say, here’s like, say, a git repo. It could could be any style application. It could be anything. Can you make changes to it? Um, we have that specification. It’s almost like a road map to your existing implementation, right? Like, it’s pretty easy to start [00:04:00] correlating what’s in that spec to actually handler code and different things like that. And then the get metadata actually does a really good job if you analyze it of sort of allowing you to do nearest neighbor groupings of, uh, Related code. So you look for the handler code and then what you find from the metadata is it’s related to changes They’re tended to correspond with like some new, uh, database queries over here Some new tests down here some new types or structs over here to handle the uh, Response and request bodies all that stuff and can see that pattern and use that to at least do a first a first pass of generating the initial. Now we run into the limits of these LLMs all the time. I’m not saying it’s generating working code right off the bat. It tends to find the right locations, which is important to me because it gives me a place to start with. I can go in and start polishing the code and actually make it work. But I think that sort of flow and that idea that what’s what’s sort of changing as we introduce [00:05:00] these LLMs isn’t isn’t what I’m doing as an engineer because The important part I’m playing is the intention, right? Like what I want to change, how I want it done. Uh, all the experience I’ve learned over time. In terms of how to implement those changes, all that stuff, that’s the value I’m bringing, I think, more than actually being in an IDE and just like punching out a ton of code, which I do a lot too, and I enjoy doing that, but, uh, I do think we’re going to see a big change to that process being more like directing the LLMs, getting that round of changes in, reviewing that as a human, uh, giving them a second round of directions on fixing and sort of getting a UX experience that’s much more about me Using human language to describe what I want to happen and about the L. L. M. S. Taking that and executing on those.

Yeah, absolutely. We’re trying to just get rid of the sort of like boilerplate code generation part of our existing process, right? Um, and I do think like what we’re going to find and what we are finding already. I think we had sort of talked offline a little bit and mentioned like Where are we at in terms of like, there’s generally this S curve in terms of capabilities, right. And we’re going, we’re obviously been on this uptrend with the LLMs and their underlying. Base model capabilities for a while, [00:08:00] and it’s really quite interesting and challenging to develop developer tooling right now in this sort of shifting sand landscape of, uh, A. I. Because I don’t want to develop tooling that’s going to be obsolete. The next time a model drops, right? Like and you don’t want to do that. So one of the things I sort of seen, I think there’s some strong signal around that. That’s making me a little selfishly, uh, optimistic and happy in terms of continuing to be an engineer at all. And and sort of slinging code for a living. Uh, is that I do think we’re seeing. In a couple of different ways that the core capabilities of the underlying models, at least these transformer based revolution that we’re sort of been been riding for a while, uh, definitely seem to be getting into that realm of diminished returns. Like, I think one of the reasons we’re seeing a longer period of time between the drops of the major models and. It is specifically because like, they’re just not getting [00:09:00] the improvements by, by throwing a ton of extra NVIDIA at it. And they’re not getting the improvements they were hoping to get by throwing a ton of extra synthetic data into the training, into the training sets. I think they get an echo effect and it’s not as helpful as they were hoping it was going to be. And then on the other hand, you see, uh, you know, deep seat come out and you get this idea that, man, you maybe didn’t need that much of that, like horsepower in the initial training anyways. Um, and, and all those things, when I look at them together, tell me like we had this big revolution and we pushed up these base capabilities really quickly, but that might be leveling out. We might not see like. The ability, the improvements, when you’re just talking about like a single iteration, uh, I’ve given you some context, some input context, you pass it into the network and we get some output context. I don’t think we’re, we’re going to continue to see the rapid improvements there that we’ve been seeing. So I think what actually, where you are seeing the improvements today are more like. Like, uh, like, have you used, uh, the cloud [00:10:00] code system at all or any of these? Okay. Yeah. And I think, I think the improvements there and, you know, I had a colleague use it and tell me it looked like, you know, black magic to him when he was watching it make changes in the files in real time and stuff like that. Um, I don’t think it’s black magic. I think they’ve done a really good job of pre analyzing your get repo and doing some embeddings and doing some nearest neighbor, uh, indexing. And I think they’re able to look at what you’ve done and sort of trace through that. But what I’m getting at is I think the improvements we’re going to see over the next few years are going to be more about how can we give these guys the correct context? Like, how can we seed that context and also manage that context? For multiple trips through the model. So as you’re coming out, you’re updating some stateful context in the background. It’s sort of also keeping track of, you know, I’m involved in a multi step task, not a single past task, and here’s how I’m working through it. Uh, but again, we’re not improving the ability of that, that output to be correct. You’re still going to have the hallucinations. You’re still going to have all the stuff we’re seeing today. Um, we’re just going to be producing. With the same sorts of hallucinations at time, but contextually matching more of like what you’ve done in the past, because we’re gonna, we’re going to use your past get history and what you’ve been doing to, uh, to sort of, uh, point the LLMs in the right direction.

That is a hundred percent. And I’m glad you, you mentioned the example, the examples is like. One of the things I wanted to go after right away because I have written so many examples and changed them for my mocks to give me specific responses so I can test different things. Uh, so just having tooling that allows you to rapidly like, you know, one of the things we’re exploring, it’s not in the product right now, it’s, it’s something we’re, we’re still working on is using overlays and generating overlays rapidly to update your example sets to give your mock different. Sort of context, depending on what you’re trying to do or even updating it during a long run of some validation tests so that you can change the responses that are coming back for the specific tests and having an AI generate those examples for me and sort of like work through a big group of them and just make sure they’re, they’re somewhat consistent over time again. I still feel like where we’re at, there’s, there’s still the need for a human to then come in and do a, a visual inspection or run it and, and make some improvements. You don’t often get like just this works right off the bat, uh, from these guys, but it, it saves me a lot of time anyways, especially when you’re talking about generating hundreds of responses and, and different things like that, that you want to use in these mock and testing. Scenarios. It’s really coming in handy. So for me, it’s all about that. It’s like, look at the stuff that’s always been painful for us that we complain about to each other when we’re doing it and and look for opportunities to teach the LLM to do that sort of basic layer of code gen for us. And let me focus on the interesting stuff. Like we’re working on a razzle stuff right now. To me, the interesting stuff is how do you build an engine to execute a razzle documents, right? Like it’s not like How do you, how do you, how do I produce my, my first, like, 20 Erazo documents that I’m going to use for testing? Like, I’ll have the AI do that, and then I’ll fix them up, and then I’ll use them in testing. But I don’t want to write them by hand anymore. That’s, that’s what we want to get rid of, right?

Yeah. And that’s, that’s something I think that we are going to deal with just as a, uh, as an industry. Software development in general, like you can see it all around us, right? Like it’s increasingly hard for college graduates to find jobs. Uh, It’s increasingly hard for juniors to keep jobs because there’s this perception in corporations in corporate America that like, you can just replace these guys now with LLMs, just a couple seniors directing these AI. What else do you need? And it’s like, when your seniors retire, where are you going to get your next round of seniors to do this work? And that’s why I think like, it’s so important, especially when we’re talking in the context of these. These open A. P. I. Initiative documents like [00:16:00] I want to focus our collaboration and work on the documents themselves, right? Like we should be in there understanding as we’re making changes to the open A. P. I. Spec what that means for the implementation and working with new new members of our teams to explain why we’re making these changes, what it means and go over like, uh, you know, have that opportunity to do some some knowledge sharing and the same thing with tests like, you know, it’s not gonna be great if Erazo becomes a specification that’s completely under the hood all the time. That’s just like, AI generates them, AI runs them, and that’s all. You don’t even really know what they are. Because, uh, Erazo You know, it actually is a very interesting way to explain in a broader context what we’re accomplishing with these, with APIs in general, with open API specs in general and the ability to have multiple specs and sort of craft workflows across several services, right? It’s important for us to find ways to share that with new people and onboard them on that. And I, I’m not saying that I have found the solution or I [00:17:00] know the solution to that. I think it’s something we’re going to be dealing with and something we’re all going to have to come together and sort of figure out what it means as, as things shift to bring new people on, to get them up to speed, to pass on that hard one experience. And, you know, you can’t do this job without feeling a little bit like. Most of that hard one experience is just hours and hours often late at night being down in the code, fixing things, figuring things out, being like, Oh, I’ve always seen another engineer make this change. And I’ve always sort of just been like, that’s interesting, but what does it actually mean? And then you go down rabbit holes and you figure all that stuff out. Um, we’re gonna have to figure that out, man. We’re gonna have to figure out what it means to onboard people and bring new new developers in and get them up to speed. Um, but there’s another part of me that also feels like, man, like, The very first program I punched in as a kid into a TI 99 foray was in HEX. I remember quite well because it took me days to punch this thing in. It was a, uh, it was a space invaders clone. I finally got it working [00:18:00] and ran to grab my parents. I was so, so impressed. I played it once and my little brother, who was a toddler at the time, walked in and tripped over the power cord and it was just gone into the ether, man. I didn’t have any medium to save it to or anything like that at the time. My point being, I don’t want to go back to punching Hexen. Like we have constantly sort of added these. Layers of abstraction onto that, that, that core system all the way to the point where like you have languages like Python that are, you know, it’s not human language, but it’s close to human language. It’s an abstraction that like you don’t have to necessarily be a software engineer to use, right? Like you can be a data scientist, you can be someone into statistics and stuff like that and be able to pick it up and, and move pretty quickly. Maybe this is just the next level of that. Like we’re just bringing it up and actually getting increasingly to this level where we can just like express what we want to do in regular human languages and have the AI pick it up again to do that. Now you have to see it with an awful lot of technical [00:19:00] context, you know, with the example with the Razo, that’s, that’s something that has been difficult simply because, you know, the open API specification has been around a long time. The training data has millions of examples. Of these API specs, some of these models, even the top end ones, they’re, they’re cutoff knowledge cutoff is late 23. They don’t know anything about Erazo to get any like sensible results. You actually have to pipe not only the scheme in, but also like, uh, an explanation of, of what, what the different, uh, components do and what they’re for. So it can have any idea of what it’s trying to build. Right. So, yeah.