API Evangelist Conversation with Dave Shanley of Princess Beef Heavy Industries

Hey there, who are you? Hey Ken, I’m Dave, Dave Shanley. I also go by the name of Quobics and I’m the founder of Princess Beef Heavy Industries.I do a lot of stuff with OpenAPI and APIs in general.

Heavy Industries. Yes, it’s a, it’s an interesting name. Um, so, uh, you know, I’ll get, I just, I’ll give you the, the short version. Um, I was trying to think of what to call what I was doing and I was coming up with, you know, typical one name. things. I was like, this is very boring and all sounds very common. So, um, whilst thinking for a name, uh, my daughter who was two at the time, uh, she used to watch a television show. And so it’s still, it’s still around. It’s called blazing the monster machines. It’s a kid’s show about monster trucks. And, uh, she used to call it. Princess Beef. There’s no mentions of princesses or beef in any shape or form in the show, but she used to call it Princess Beef. I’m like, where did you get that from? What a great name for a band, Princess Beef. She used to play these toy drums and she called them the honey drum. I don’t know why she called them honey drums, but see, I thought Princess Beef and the honey, honey drums. Now that’s a name for a band. But, you know, I’m too old and, uh, way beyond that part of my life now. So I thought, you know, I’ll just call the company [00:02:00] that princess beef and then the heavy industries part, it’s a harken back to, uh, my nineties hacker roots, uh, used to do a lot of stuff in the hacker scene in the, in the nineties. And, uh, there was a group called loft. Heavy Industries and they kind of disbanded in the early 2000s, but they were one of my inspirations for building developer tools, tooling. Um, and I thought, why not revive that name? Heavy Industries sounds like, you know, a nineties skater brand. So I thought, cool, why not? I

Yeah, it’s a, it’s a great question. Um, so, I’ll take you back to the, the, the fizzy days of 2019. Um, it’s when I [00:05:00] started working on this problem. The reason why I started working on this problem was, I was at VMware, and I was the architect, uh, an architect of, well, I was the architect of APIs at VMware. Um, or part of VMware. Anyway, um, the problem was, was Customers couldn’t locate APIs. They couldn’t use the APIs. They were, uh, you know, written by different teams. There were open API specs. There were wisdom files. There was just a mishmash of swagger in there as well. Uh, half of these. These specs were code generated. No one had validated them. The documentation that was generated from them was atrocious. It was PDFs, you know, 500 megabyte PDFs. It was absolutely appalling. So there was just a mess everywhere. There was no standardization. There was no governance. There was no consistency. There was no, you know, standardized documentation. There wasn’t, there was nothing. Uh, there was no way to know how good. Or bad anything was. So starting with that, that, that problem, I started looking on the internet. So, okay, let’s go just, just instead [00:06:00] of reinventing the wheel here, I’m sure there’s companies out there that have already gone and done this work already. So I’m filling with my APC is falling out for some reason. I don’t know why. Um, And, uh, I came across spectral. It was like, great, my problem is solved. I don’t need to do any more work. And then I started feeding these specifications from, and these were, these were VMware enterprise products. So the, um, uh, you know, they were very large, very large API services, and they were kind of grown over many, many years. So there’s a lot of technical debt there. Anyway, I started feeding these specs into spectral. wouldn’t return like nothing happened. Um, so, okay, that’s not good. It’s not good. It’s not going to, it’s not, this is not going to work. So when I finally like cut the specs down, like chop them down just so I could get spectral to respond. I realized that it. This was never going to work in the long run because Spectral has a couple of design problems, uh, not because it’s a bad tool. It’s an excellent tool. In fact, Vacuum is just, it’s used the design of Spectral, but I built [00:07:00] it in with a different architecture. It’s fully asynchronous, uh, whereas Spectral isn’t. It can’t be because it’s built in JavaScript. So when, if you think about these huge, huge specs and you’ve got lots and lots of rules, it’s running through them one by one, bit by bit, bit by bit. And It just takes for it’s exponentially slower. So, I thought I’m gonna, gonna have to fix this. So, uh, we tried, I actually got funding, went to the CTO of VMware, got funding for it. We tried to build a linter, failed miserably. Had a team of three people, couldn’t make it work. Spent three months on it, just didn’t work. Um, and so then we, we, we flipped. I said, okay, we, we can’t reinvent Spectre. All those guys are geniuses, not gonna do it. Um, let’s go and do documentation. So we started building documentation. That’s, you know, a hot topic. Now there’s a bunch of startups doing really great docs out there, like scalar, uh, great docs. Um, uh, so we, we did this, but we, it was for scale. So, you know, these were 10, 000 paths and, you know, a hundred thousand objects, huge, huge surfaces built a [00:08:00] tool that would read those specs in and actually generate. All the content. So, and it actually worked really well and it was SEO friendly. It was lightning fast. It was, looks good. It was consistent. Um, so the reason why I started to build vacuum was we built a bunch of tooling. Internally, we started to fix the problem. All the developers started to adopt it because it was great. They weren’t force fed it. It’s just like. This solves a real problem for, for us and our use cases. I went back to the CTO. Um, and I said, we should open source this. And I don’t think I sold it very well, to be honest, the conversation went to let’s turn it into a product. We need to open source this stuff because the industry is screaming for high quality tooling that isn’t able to operate this level. Smaller specs, smaller companies, smaller tool sets, smaller specs, you know, special is okay. It’s still slow, but it will do the job. But when you feed, when you’re trying to feed in hundreds of gigantic 50 megabyte specs, it just It can’t do that. It just can’t go. So [00:09:00] anyway, uh, that didn’t work out like the CTOs. I know let’s put it into a product and put it into a business unit. And like, that’s not going to work for me. So I thought the only way for me to do what I want to do and do conflict free would be to exit the company and build it myself. So that’s what I did. I quit VMware and I started building, I started building. Vacuum from, from scratch, basically looking at the special source code, seeing what they’d done, how they’d done it. And what I didn’t want to do was create a whole new standard, you know, like yet another standard, yet another DSL for describing rule sets. So what I wanted to do was make it a drop in replacement as much as possible. So if you’re running Spectral today, you can use vacuum and put your specs in or put your rule sets in. It should just work out of the gate within a certain degree, depending how, and I can come to that in a minute of why there’s some in this. There’s some variances between the two, but mostly it’s, it’s a drop in replacement. Um, and I spent [00:10:00] about. Six months trying to get work, trying to just trying to get the whole thing to work, uh, trying to get it to understand circular references. And the more and more I did, the more and more I realized is that I, this isn’t. I don’t just need a linter, I need an entire compiler to make this work. So, LibOpenAPI is, is like a, it’s a Go library for parsing. It’s a toolkit for OpenAPI. That was born inside Vacuum. It did all the parsing, had all the models, um, and what I did is I extracted it out of Vacuum and then created it. It’s its own standalone library for anyone who wants to use, like, the compiler to, you know, to, to read, parse, understand, do diffing, things like that, and then continue to build. Vacuum, but it took me, it took me a super long time to get it right. And I was on the verge of quitting number of times where I just thought, how can, why is this so hard? And it is hard because you know, the further and further and further you get into open API, the more complicated, the more, uh, cyclo, the [00:11:00] cyclomatic complexity of what you’re building becomes almost infinite, particularly with recursive polymorphic data structures and Jason schema. It’s like, wow. So, um, I just kept going. And what I noticed is when I just put the code out there within a couple of months, you know, it was just me. And then all of a sudden a few people popped up, uh, like Dan Daniel G Taylor, who’s built humor and he’s also built, um, uh, rest dish, uh, fantastic tools for, uh, HTTP tools and a, and a go framework. He reached out cause he was having a similar problem. There was a, there was another library out there called. Kin open API. Um, and, uh, it, it, it’s great library, fantastic library. Uh, but it didn’t support 3. 1 to do that. You needed to rebuild the entire library, the entire thing. It just wouldn’t work. And what I wanted to do with diffing, like semantic diffing. I wouldn’t support it either. So he was look, Daniel G. Telly was looking for some of these, these features. He [00:12:00] reached out to me, uh, we started, you know, built some stuff for him. He helped me get it off the ground because he adopted it with his tooling that gate that gained some traction. People started seeing it. They started adopting the tools as well. And all that goodness from, from that point forward. Uh, so yeah, it was what I want to do is pass and understand open API contracts. I want to build something like the doctor. We’ve got a visual. a visual experience of managing rules, rule sets and seeing the results and, uh, you know, being able to manage those rule sets, but then, uh, grand divisions of what I want actually want to do is. Create a marketplace, right? So you’ve got your rule sets and you want to be able to say, okay, I want to share these and I want to import those. I like those five rules from this company. I like these five rules on a mix and match, create my own shopping cart of governance and save it and then publish that. And, you know, and, you know, create this, this new economy or marketplace or [00:13:00] community or however you want to. Describe it free economy, obviously, um, of, uh, of governance, but yeah, I mean, but in order to do that, so I’m going on a bit here, I apologize. I’ll, I’ll pause on that. You ask questions, but the, the, the, the, the order for me to do, to do all that I needed a compiler and B I needed a, a, a quality engine, uh, you know, linter, so I need to lint it to work just like spectral, but I need to be lightning fast and there’s only a few ways to do that you build it as a compiled tool using something like you do in Java, but. And I was talking about Java, there’s a lot of great Java tools out there, but it’s very heavy, and there’s better tools out there, in my opinion, so I picked Go, because I love Go, but you could have done it in C you could have done it in Rust, anything that’s going to compile down to native code and run as a, as native machine code is what we need, and not be interpreted like what we have with JavaScript and Spectral, so, Build that, get it fast, make it fast, make it really [00:14:00] fast, make it really, really fast. And then, only then can you start to build the rest, the rest of the tool chain, the rest of the stuff. And that’s where wiretap and OpenAPI changes kind of emerged from that, that foundation. Um, because I wanted to do it real time. In fact, uh, there was a team in VMware that built, uh, a web service around Spectral. And it was called the, the rest. The API quality service or something like that. I can’t remember the name of it now. It was a very long time ago, but it was, and actually there’s a number of companies that tried to do the same thing. I’m not going to name names, but they run Spectral as a service because it’s so slow, it can’t be real time. So it’s a submit your job, wait for Spectral to run. You get an email or you get a, you know, a callback or, you know, a webhook or something and we’ll say, hey, we’re done. And then, you know, you get your results and you go and look at it, but it took a huge amount of compute. It was very slow and it’s not real time. You couldn’t use it inside VS Code as a, [00:15:00] you know, as a, as a linter. It just wouldn’t respond quickly enough. So anyway, that’s what most companies have taken that approach and realized that it just doesn’t scale. That’s why Vacuum’s there. Hopefully there won’t be a need for another one. You know, if we get to the point where they’re, you know, I don’t know, you never know what the future holds. And then I’ll probably be doing the same thing again in rust or a zig or, or, I dunno, maybe not C but, but who knows, who knows what the future holds, but hopefully this, this should be able to cope with today’s workloads for the next five to 10 years. Oh, shut up now. Sorry.

Yeah, it’s, it’s, it’s a great, it’s a great point. Actually, the, the JavaScript functions already exist. They’ve been in there for a while now. So you can use them if you want to, um, yeah, you can. Yeah, it’s got its own JavaScript engine. Now it will execute all that JS as you can build like go the go plug in system. It’s pretty horrible. Uh, there’s not much I can do about it. It’s part of the language. It will work, but it’s not ideal. So that, and also means you’ve got to compile stuff. So the reason why I built the JavaScript one is because it’s dynamic. You can just feed in a script and it will run without recompiling anything. You know, no go code required. But, you know, I’m not particularly a fan of that, you know, it works, it can be used, but I’m much more of a fan of function based stuff. So what I like is that you’ve got the core rule set, the core functions, which are things like truthy, defined, you know, alphabet, those are the core functions. And pretty much if you can define adjacent. to look up that it will work for [00:19:00] anything. They will, those core functions do pretty much anything. And I haven’t found the need to add any more like that. You know, they’re touring complete basic programming and Prince principles. No, it doesn’t exist. Does it not exist? Is it greater than equal to less than so those core functions? I found that can complete the majority of jobs when there’s, you know, custom logic, like you, you want to be able to do calculations on an operation or. Parameter or a path or a series of paths, then you need some custom logic, right? So you need to be able to do something that in that approach, what I prefer to do is build functions, like it’s, it’s a specific function and I give it a specific rule that then becomes like an individual unit. You can say to anyone out there, uh, here’s a built in function. Use it, don’t use it. It’s a rule, enable it, don’t enable it. Um, but I always prefer people to stay with the core functions because. They’re fast, it works, it does most, most of it. If you need that custom stuff, build yourself a custom [00:20:00] function. And don’t do it in JavaScript, do it in Go. So it runs, you know, at native speed versus, you know, deserialization, serialization happens with the JavaScript stuff. It will slow it down, won’t be great. Um, but yeah, there, there, honestly there, there shouldn’t be anything that you can’t complete. With those, those core functions. And most people just use those core functions. They use the other rules that I’ve created in there are for, you know, additional, uh, you know, governance compliance, things like that. It’s, uh, um, you know, it’s up to, it’s up to people to use them or not. But, um, when it comes to like expansion, I do want to be able to create, um, A simpler way to put more custom logic in there. Uh, I haven’t done that yet, but without creating code, if there’s a way to create, um, some form of an extension to the DSL that allows you to put some, you know, maybe some rules in additional sub rules. I don’t know. I haven’t quite got there, but to be honest, Ken, I haven’t had that many people ask for [00:21:00] that type of stuff. People seem to be. Really comfortable writing their own custom rules using the core functions. Or, you know, if they actually are part of the ecosystem, actually building their own functions, there’s a number of people in the community that are just building their own code, compiling it and running it, running it with vacuum in there, in their pipelines as, as plugins, essentially. Um, so it depends on how deep you want to go. And I’m casual user, use the core functions, hardcore user. You’re going to be in custom code pretty quick.