API Evangelist Conversation with Vincent Biret, Microsoft Graph SDKs Principal Software Developer at Microsoft

Hi Ken, I’m Vincent. Nice to meet you virtually.

Yeah, I’m a principal software developer on the Microsoft Graph Developer Experience team. So Microsoft Graph is one of the main APIs. Uh, Microsoft to do things about calendars, files, emails, and so many other things. And my team is responsible for building the client experience, tools, SDKs, and more. And more recently, we’ve also taken a responsibility for co pilot extensibility. So having the ability to have Microsoft 365 co pilot, call into external APIs and data sources, reason over that, and bring an answer to the user based off that.

So, um, let’s start by acknowledging the fact that Microsoft graph has over 23, 000 operations. [00:02:00] Um, get postpatch put and so on and so forth. Um, over 2000 models, over 5000 endpoints or path items. Um, and so many other things. So it would not be feasible for us to ship SDKs for the whole API surface across, you know, NET, Java, Go, TypeScript and so many other languages if we had to handcraft all the SDKs. Or we would need thousands of People monkey typing away, and that would not be interesting work anyway, right? So to be able to scale through that we are relying heavily on code generation for VSD case So my team has built over the last four years an open source code generator for open API And its name is cute out K. I. O. T. A. And what it does is it takes as an input and open A. P. I. Description processes it and gives you all your models affluent A. P. I. Surface to craft for request [00:03:00] handle sterilization, desterilization, retry handling, compression, redirect all those, you know, nitty gritty details. Eso you can focus on where you add value as a developer, which is building the application, not actually integrating H. T. P. Calls, right? Uh, and so These same, uh, generator, uh, we use to generate v SDKs that we put on the feeds like NuGet, NPM and so on and so forth. But you can also use to create your own SDKs, either as an API consumer or as an API provider. It’s really up to you. Uh, in fact, we have even have GitHub who is using qda to generate various SDKs for their rest, API as well. Um, my team is also because we have so much tooling and a deep presence with open API. My team is also responsible for the open API dot net library, which is the dot net library that parses open API descriptions and serializes open API descriptions. And you can do a number of things. That’s the same library that you’re using. If you’re doing ASP dot net [00:04:00] core APIs and you generate open API descriptions, all of that relies on the same library. And we’re also working very, very hard. The. Open API 3. 1 support and we’re late to the party in this library. So all of our tooling and ecosystem can now benefit from, uh, the improvements of a new open API, um, uh, specification version.

Exactly. Because [00:05:00] if you project on yourself more and more, the applications don’t talk just to one API. If you talk to multiple APIs on the market, right. And then you have one API provider who provides an SDK built a specific way on other API provider, providing an SDK. Built another completely different ways. You have to relearn everything. The third API provider doesn’t have any SDK for your language at all. So the experience, uh, uh, is not consistent at all as a consumer. And then, as an API provider, having to build all those SDKs across so many languages, which you might Or not have the funding for the expertise for and so on and so forth is or very costly time consuming. And this is not where you’re making. You’re adding the most value to put it bluntly. Right? And so we’ve cut out one of the bets we’ve been making is maybe we should turn things around a little bit and Okay. Say, Hey, providers provide the best, cleanest open API description as you can. And there are tools here to help you like [00:06:00] spectral so forth. And, and then consumers, well, don’t bother with SDKs anymore. Use Kyoto or another. generation tool, generate the SDK that you need for just a parts of API that you need, because that’s another thing, right? In graph, we have 23, 000 operations. Our dot line is the case. Something like 20 megs. Maybe you don’t need the whole thing, right? Maybe you just want three operations in their generate just for that. And now you get going and you have a consistent Experience across multiple API provider, and everybody is happy across the board, right? Um, so that’s another bet. We’ve been making, and we’re happy to see more and more people are adopting this strategy as we consume API’s.

Yeah. So for. Probably over a year, 18 months now, uh, padding has, uh, emerged where, [00:08:00] um, one of the ways to make, so, okay, let’s start with the basics. Uh, LLMs are trained on a fixed data set at a fixed point in time. Uh, if you ask just the LLM, a question that’s about information that. You know, appeared after, uh, that point in time, they let them won’t know about that. And, uh, if you need to, um, have access or to reason over data that’s stored in some kind of application or some kind of database, it won’t be able to do it on its own. So one of the strategies to overcome that is called, uh, rag retrieval, augmented generation, I think, and the idea is to say, hey, we’ll. Um, ask a question to the LLM, the LLM will know that it has a couple of extensibility point that it can call out to, they are called functions in the object model of the LLMs usually. So it knows that, Oh, I can go get, I don’t know, emails here. I can go get, uh, calendar events there. I can go do other things, uh, here and there. Right. And of course, most of the times, these capabilities, those [00:09:00] functions are actually API calls. It’s the ability to say, I’m going to call an API with a bunch of parameters, and that’s going to retrieve the data set I’m looking for. And then once I have all the data, I can reason over that and give a more precise or more accurate answer to the user. And better yet, Because APIs are not just about reading. Now you also unlock the potential for VLLM to do actions, to, to, uh, after having reasoned about a few things, um, um, actually change things in the system or the application. A good example of that is, um. Please, LLM, go read all my emails because I cannot keep up with my inbox. And if anybody is asking for, uh, to meet me, uh, go ahead and schedule 30 minutes with them and reply to them automatically with a nice message and something like that. Right? So when you do that, you have the initial question that goes to the LLM. The LLM will understand that, uh, there is an ability to read emails [00:10:00] will go call the APIs to do those things. It’s actually the orchestrator, but I don’t want to get too deep in there. Um, and then once it has all the recent emails will be able to read the content and reason over that identified that one of them is about asking for for you and I to meet, for example, and then we’ll reason again. Go talk to the API that creates, uh, events or meetings and whatnot. Schedule that and everything is done at the end of the day, right? And all of that transit through, uh, API calls. And so Part of my, what my team has been working on is this idea of plugins. So you can say to the LLM, Hey, here are a bunch of things that you can go call and use for your reasoning and to answer the user,

Biggest limitations of, um, LLMs is they have a finite number of tokens, words, or context you can pass to, to the LLM so we can reason about. And so, for example, in the case of Microsoft Graph, it will be impossible to feed it 75 megs of text, which is your description of the API and, and hope for the best to, for it to be able to do anything with that, right. And so that’s one thing where our tooling shines. You have again, because we built so much intelligence into being able to select down to the operation you care about and just generating for that in the SDK world while we took that same capability and brought it to the plug in generation story. [00:12:00] So now you can generate a plug in with attached open API description, which is effectively a subset of the original one about with only what you care about. The other important thing with LLM Is we get confused if we get too many choices and especially if the choices don’t come with enough context or don’t come with enough instructions. A good example of that is let’s assume you have two operations, one which is about creating messages. Creating emails and another one, which is about sending messages, which would create it implicitly. So if you have both operations in context and not enough instructions, uh, the LLM is going to struggle to figure out which one it needs to call to people, even if you say, send an email explicitly, right? Um, And so again, this ability to select and trim down to exactly what you care about will give you faster answers from the LLM, more reliable answers from the LLM. And those are the kind of tooling we’re kind of building, which is currently [00:13:00] in public previews, internal previews and so on and so forth. But yeah, the accuracy of the description LLM to be able to do what you want it to do. Otherwise it’s going to hallucinate and be unreliable and so on and so forth.

That’s. Also, uh, space we’re exploring on our end, uh, you’ve probably heard about that, uh, design language called TypeSpec, uh, which is you can imagine TypeSpec being the TypeScript equivalent to OpenAPI for what TypeScript is [00:14:00] to JavaScript is effectively a higher order design language that eventually transpiles down to an OpenAPI description that you can feed to other tooling and so forth. And so, As we noticed that sometimes some operations like clarity, instructions or other things like that, we’ve been working on extending type specs so we can generate more information in the OpenAPI description to fit later on into the model so there is more context here. I don’t think there is anything out to try yet, but the work is being Done publicly in open source repositories. So it’s easier to track and and play with

Oh, well, I like to work at scale on interesting problems that that that solves. Um, [00:15:00] Issues for the day to day life. Not only for corporations, you know, it’s great to make money and so on and so forth, but also for individuals. Um, I truly believe that the more APIs we enable and integration of those APIs into applications, the simpler our lives all will become. Um, like lately have been, you know, dealing with, um, banks and notaries on one side and also with, uh, health care system on the other side. And all of that is still to this day. You need to call somebody and wait for hours on with a ringing tone and whatever for somebody to pick up and understand what you want to talk about. Right. And when you Post that and say, Hey, well maybe if you had a bunch of APIs that were publicly documented and somebody had a good idea, um, on, on your device to integrate those APIs in an m and so on and so forth, or as an application that talks to between different, uh, providers, uh, that will be much easier. You know, that would be, you know, chatting [00:16:00] with an agent or filling a form somewhere. And I would not have to. Spend so much time on the phone and so on and so forth. And I think that’s going to make everybody’s lives. Um, uh, With a bit less friction, right? You don’t want to spend your time on the phone or anything like that, right? So I can see that APIs and the whole ecosystem around it. And then when you add on top of that, all the LLM stuff will drastically reduce friction for everybody’s life on a day to day function. And I’m working to enable that. So that’s, that’s what gets me going, I guess.