API Evangelist Conversation with Danny Preussler on Reopening the SoundCloud API, the Agent That Scraped Its Way to a Token, and Writing Documentation for Machines

Thanks for inviting me. I am Danny, an engineer living in Berlin, Germany. It is funny that I work with APIs now, because my background is much more on the client side — that was most of my career. Back in the day when there were BlackBerries I was writing BlackBerry apps, and then this thing called Android came up, so I jumped to Android and worked for a couple of companies like eBay and Groupon doing that. Then there was Viacom, so a bit of streaming, which brought me in the end to where I am now at SoundCloud. Even there I did Android apps, and they made me manager of the team at some point. At some point I did not like management anymore and said, what else is there? Let me jump to the API team, because I like this — I started talking to developers again. I have been at SoundCloud seven years now, and on this team for about two and a half.

This outdates my own SoundCloud time, so I interviewed a few people here. SoundCloud started in 2007, and in 2008 there was already a public API — very early on. The idea was eat your own dog food, and we literally built SoundCloud in the beginning on the same API that people out there could use, which I think was a very cool approach, and it lasted a very long time. There are nice stories from that period. When I joined, the Android lead of the team had actually been hired because he wrote an Android app with the public API that was better than the SoundCloud one, so they just hired the guy. People built a lot of cool stuff around it, up to businesses — there is someone still at SoundCloud who built a tool called SC Planner, where artists could schedule releases. It was bought by a company called Repost, which was bought by a company called SoundCloud, so it ended up back with us. It must have been a very cool time. There was a lot of freedom back then.

Things shifted, for multiple reasons, and it ended up in what a lot of people out there know — that we kind of closed down our API program. It was meant in the beginning to be very temporary, in 2017, which was also a big layoff year at SoundCloud, but it became a much longer pause than was initially planned. SoundCloud had major label content all of a sudden, which needed to be much more protected. And to be honest, when you looked at the most famous apps out there, they were scrapers and downloaders. Although people tried to build cool stuff, there was always this abuse and scraping problem. The numbers were very interesting — in 2016 we had a hundred thousand new apps registered on this API, and it was a free API, we just gave people access. Imagine managing at that scale. So the program closed for a while. There was a form online and people were always referring to that form, asking when we would reopen, but really you had to find someone at SoundCloud who could create your credentials.

We tried a soft opening. In the beginning you had to write a ticket and we validated your case, and then it got a bit out of hand, so there was a portal on our website where you could fill things out and reach it yourself. Then AI changed everything. Our docs were very outdated, and we started seeing that endpoints we had not mentioned anywhere in our documentation began spiking — people were using them more. There was a deprecated endpoint that had been deprecated for five years, and suddenly why do we have more traffic there? Because all these models are trained on what is out there, on Stack Overflow and co, so people started using things in ways they were not supposed to be used. This made us realize we needed to revamp our documentation — and not for a human reader, but for a machine reading it. Spotify did a great job there too.

We stumbled on the question of how you create an app. We ran some experiments where we basically gave an agent — I will not say which one — the instruction, hey, I want to create a SoundCloud app. It created everything, then asked for my API key, and I said I do not have one. So it literally started scraping our website, got a token from there, and changed everything from our public API over to our internal API. That is a warning sign. It makes it so easy now to do this. The whole idea of a public API is, let us funnel them, we are in control, we can see abuse. But if it is that easy to bypass, we have to make it easier to give you an API token. You cannot write a ticket and get an answer two weeks later. So we said, let us build a script. The first version was just for localhost, then we realized that was not enough, so now you open a web page, sign in, give a code, with as little human interaction as possible. We want to publish an npm package, but it is not there yet — things are moving so fast that every day counts, so we just put the script out there.

Exactly what we also had. All of a sudden the ticket volume went up. People messaged us asking, can you change the redirect — and wait, I myself made it possible to do that yourself, like a year ago. But we found there was some outdated documentation that all the LLMs went into, and it said there that you have to write a ticket. So people started writing tickets before they were even calling anything, when they would have found the right information elsewhere. All this information that is out of sync — these agents run into it, and they will make it visible. We discovered things and thought, oh my god, this has been outdated for so long, and it was still live in our developer portal. I do not know if anyone went into it. Maybe they opened an issue. We had a community that helped each other, which by the way also changed — people got a bit sick of helping someone who is just vibe coding and does not know what they are doing, so that part went down a bit. Which made the documentation more important again.

When we opened the API it was a soft opening, very gradual, and at a certain point you could tell the scrapers had noticed — they registered a lot of apps even with the delay, they found it. That is one reason why for now we put it behind a subscription, so we kind of know who you are, which helped a lot keeping this at bay. Spotify gated it too, but it is still an open API, you just need a subscription with us. Back in the day we had no idea what people were using or what they were actually doing. We do not own the clients, which is why a lot of things broke — we never did versioning, we try to be very backward compatible, because we had no idea who was out there unless they were a partner we could reach. So one of the things before we reopened was that we needed an API gateway. We use Tyk. That gave us control — we can measure which endpoints and which apps are being used, we get authentication and rate limiting, alerts and reports. We can increase and decrease your rate limit. We want to allow this, but we also want to stay in control.

I like these developers who do something cool with it. SoundCloud has this problem of ownership of the public API — once we split it and it is not our own dog food anymore, who owns this, what are we gaining, we are giving it away for free. But one of our biggest consumers is Lyrion, the Logitech music server. There is open source software that runs it and you can have SoundCloud with it, and I love this — a free open source tool that keeps your old hardware alive and connects it to SoundCloud. Of course we also have DJ partners who integrate SoundCloud, and my job is to talk to them. When people think of the SoundCloud API they think about building a music player, and internally people sometimes ask why we would want a copycat of SoundCloud. But actually most people have a huge collection and want to optimize their playlist management, because it is too fiddly in the app or the website. We have churches that upload their preachers every Sunday. It is creators writing the tools they do not have, and now they can vibe code their own workflow tools together. That is where I like to see this go.