The Sub Delta: a music discovery engine I've rebuilt for a decade

The Sub Delta is the project I come back to when I want to try something new with nobody's permission needed. I started it in 2015 to answer one question: given artists I already like, what should I listen to next? It crawls SoundCloud's social graph from those artists, scores what it finds, and gives each user a discovery channel that updates every day.

Because it's mine, it's where I learn. Each rebuild taught me something I later used at work: queues and retries, cost-aware data modeling, monorepos, and now building with AI agents.

SoundCloud has more underground music than anywhere else online and few good ways to find it. Its own recommendations lean toward what's already popular. I wanted something that starts from my taste and looks a few steps further out.

• SoundCloud's API is rate-limited and only partly documented.
• A graph crawl fans out fast. Each artist leads to likes, reposts, follows, and related artists, several levels deep.
• It's a side project, so it has to be cheap to run and hard to break while I'm not watching it.
• One developer, fitting it around a full-time job.

• Channels and springs. A channel is a taste profile, and springs are the seed artists and tracks it starts from. The rest of the system hangs off that model.
• Queue-driven crawl. The scan runs as queue-triggered Azure Functions, one hop per message, so each step can fail and retry on its own. Handlers rethrow on failure, so nothing fails silently.
• Hourly scan slots. Channels are spread across the hours of the day, which keeps pipeline load and SoundCloud's rate limits even.
• Scoring with decay. A track's score combines engagement, tag matches, follower counts, and trend, and persistent scores fade on a half-life so one good day doesn't keep a track on top.
• Graph in Cosmos. The graph moved from a dedicated graph database to partitioned SQL queries in Cosmos DB. It kept the traversal model and cut the database bill.
• A pnpm monorepo. Shared packages for models, services, and pipeline logic feed the Next.js site, the admin app, and the Functions, so a type changes in one place.

• All of it. I designed the data model and the scoring, and built the pipeline, the site, the admin app, and the operations tooling.
• Wrote the SoundCloud packages it runs on and published them as open source.
• Added accounts, public channels with a Discover page, and a taste loop that uses what you like to re-rank your For You feed.
• Set the repo up for AI agents: rule files, a system map, and agent skills that check production health, query the database, and read logs.
• Ran a full engineering audit of the codebase in 2026 and closed its findings over a series of PRs.

The hard part has always been the fan-out. The earliest version tried to crawl the whole graph in one function call, which can't survive a fan-out that size. Putting each hop on a queue made the crawl harder to follow but possible to retry, and every version since has kept that shape.

Cost shapes the design. A dedicated graph database was the natural fit, and it was expensive. Rebuilding the graph as partitioned SQL queries in one shared throughput pool kept the model I wanted for much less money.

• Live at thesubdelta.com, with a free tier and a Pro tier.
• Rebuilt from a C#/.NET prototype into TypeScript, then from a single Next.js app into a monorepo with the pipeline on Azure Functions.
• Three open-source SoundCloud packages came out of it.
• More than 1,000 commits in the current monorepo since 2023, with most recent work done through AI coding agents and the same guardrails I use at work.

• Next.js
• React
• TypeScript
• Azure Functions
• Azure Cosmos DB
• Azure Storage queues and tables
• Redis
• NextAuth
• pnpm workspaces
• esbuild
• Vitest
• Vercel