Algorithms in Music Create New Frontiers of Emotion

I remember the first time I heard algorithms in music.

Growing up in Alaska in the 90’s, our largest local shopping center had a relatively impressive CD collection, and there was a little kiosk where you could bring a CD case over and preview the music. In the pre-mp3, proto-internet era, above and beyond watching MTV, listening to the radio, and hanging with friends, this was one of the only ways an Alaskan teenager could discover current music: browsing covers through vaguely-organized genre collections and listening to 10 seconds at a time on a grubby pair of public headphones. Drawn in by the curious cover depicting a deranged smile on an off-centered face with a matte white background, it was here I discovered The Richard D. James album by Aphex Twin. I was instantly transfixed by the intensely fast and digital-sounding drums on 4, Fingerbib, and Milkman, which feature glitchy, rolling cascades of drum patterns contrasting against these pretty harmonies, that kind of decorate the brutal complexity of the rhythms with a peculiar irony.

At the time, electronic music was just making its way into the popular consciousness by way of acts like The Prodigy, Fatboy Slim, and Chemical Brothers – producer/bands that laundered the electronic sound onto the MTV airwaves by borrowing heavily from alternative rock aesthetics. Aphex Twin rode this wave into the mainstream with “Come to Daddy” – a hard and dark punk-esque anthem of distorted guitars and vocals, erupting with a sound that would later inspire a generation of breakcore producers and industrial rockers alike. But The Richard D. James Album did not pretend to be appealing to 90’s rock fans… it was electronic music. No frontman, no guitars, no obvious drum kits, just unapologetic electronic sound. 

In a 1997 interview with Keyboard Magazine, Aphex Twin described the making of The Richard D. James Album in 1994 as an era where he was writing his own algorithms to program the music and rhythms. Samples would be randomized with loops changing rhythmic structures and keys autonomously. Electronic music was, at the time, heavily characterized by the limitations of hardware samplers and synthesizers: The hypnotizing repetitiveness of trance, house, and acid was largely influenced, if not dictated, by the default design functionality of the instruments that the music was made on. The 16-step loops and the preset bank memory sizes of the time inherently limited the density of ideas and sounds going into a track. Aphex Twin broke from this trend by using trackers and self-programmed software to introduce compositional complexity, in ways that sounded naturally effortless, to areas of electronic music that had been otherwise difficult to edit manually.

I didn’t have words for it yet, but this was the moment I fell in love with algorithms being used to make musical structures… not just as symbolic maths to dictate rules and procedures, but as a wholly unique aesthetic unto its own. Aphex Twin’s music sounded like it was neither relying on the remediation of pop culture musical ideas, nor was it falling within the conventions of loop-based electronic music. Instead, it was music that was built around hidden systems that blossomed into complex patterns and sound collages, like something beamed down from a higher lifeform – organized sound from beyond what a human could play or even conceive of.

There is something otherworldly-beautiful and gloriously alien about music that is composed programmatically outside the realm of how humans have written and performed music since the first drum beat rang out beside a campfire. If music is the language of emotion (as my friend and mentor Stephen Webber explains better than I ever will in his TedX talk The Most Important Thing) how marvelous it is to feel an emotional stirring that appears from beyond the boundaries of what a human being is capable of adequately expressing physically or mechanically… The music might not be human, but the feelings it evokes are.

It’s been over 30 years since The Richard D James Album came out, and in that time, there have been countless examples of music-making algorithms that push the boundaries of musical expression to superhuman levels. Notably, the Breakcore “boom” of 2003-2009 brought rhythm-mashing machines out of IDM art galleries and onto the dance floor, with artists like Venetian Snares, Squarepusher, Xanopticon, Dev/Null, Aaron Spectre, and TechDiff leaning into the brutal novelty of 200bpm+ break-neck speed and complexity with the algorithmic assistance in custom-made Reaktor, Max, hardware patches. It was a race to push jungle and harcore rave conventions to the absolute limits – a dadaesque fetishization of loud and off-color absurdity, which hit a boiling point around the same time that dubstep hit the scene in 2009, and by then perhaps it felt like a relief to return to more simplistic grooves, because suddenly the breakcore sound largely vanished.

And then there are the more quiet examples such as Brian Eno’s Generative Music, an album released as software, which also helped to coin the idea that a piece of music can live outside of time, with programs that are set up to potentially play music forever autonomously and without repeats. While Eno himself characterized Generative music as restricted to “curiosity” in pop culture at the time, Autechre’s Oversteps managed to bring computer-made musical composition into the collective consciousness over college radio waves. On Oversteps, Max patches were used to generate rhythms, harmonies, and melodies differently every time, for both their performances and track productions. An alleged Autechre max patch was passed around while I was in college during this era, and it was nothing short of magic to open it up, hit space bar, and hear an endless Autechre track unfold. 

 

The feeling a classical composer might have as the baton drops and the symbols on paper come to life through sound might not be all too different from how an audio programmer feels when compiling software and hitting play. You spend this time preparing a special arrangement of symbols and instructions, and then feed the program to the orchestra, hoping that the code translates into a sound that is far beyond what any one person could replicate. One could argue that synth patching and audio programming lives under the same umbrella of meta-artistic expression alongside composition itself. 

In conversation with my friend, composer and technologist Gadi Sassoon (also a virtuosic technical instrumentalist), he once told me that he feels like commanding the flow of signals, be it for a synth or an orchestra, is a form of music making that truly frees the ego. Programming as a composer, and composing as a programmer… instead of making the sound directly, one creates the conditions that beautiful flora rise from.

As Brian Eno said, some composers are like gardeners: “Someone who will plant seeds and wait to see what exactly will come up.” Each flowerbed is a little different, even if they share the same seeds. In this sense, generative algorithms perhaps manage to reflect back into music some of the mediated chaos that is inherent to nature. If music is a man-made invention, an aesthetic taming of physics, maybe delegating musical decisions to degrees of randomness can help us reconnect with the untamed nature from which the physics emerged. This is especially evident in the work of Xenakis and Roads which lead to granular synthesis: finitely controlled randomness allows us to simulate nature with more accuracy and ease than programming each raindrop by hand. Once you break sound into algorithmically-assembled fragments, these systems act as schematic concepts, and can be iterated upon endlessly to create modern IDM/glitch masterpieces like Bubblegum by Woulg.

The limitations of what a human could reasonably achieve or even imagine is a formidable bottleneck to overcome in the world of music, but one with rich rewards. Sufficiently advanced music synthesis technology is nothing short of magic, such as the custom concatenative synthesis algorithms used in Rob Clouth’s 2020 debut album Zero Point, where thousands of sound fragments rotate with perfect cohesion to animate the grooves across each track. This record made an enormous impact on the direction of both my aspirations as a researcher, music maker, and software designer, and was also the inspiration for DataMind Audio’s plugin Concatenator.

Our current AI era is an odd thing to live through, in context of everything that’s come before in the generative audio space. Generative algorithms in music went from high-brow raves and rare vinyl to suddenly being everywhere. It went from Brian Eno self-describing it as a mere curiosity, to an all-out culture war controversy, in just a year with the rushed rollout of disruptive AI song generators. 

The difference between AI music and the algorithmically generated music I fell in love with so long ago, is that algorithms have been used up til now to push sound forward, and to push towards expanding the capacity for human emotional expression; not just to hear something, but to feel something that could not have been felt otherwise. AI song generators, on the other hand, tend to converge upon the most probable outcome based on a prompt, which consequentially tends to look backwards into the history of music for remediation. 

There are plenty of ways we can subvert the slop and push AI into making sound that actually pushes sound forward instead of plagiarising the past. If getting an expected outcome from a large model is how big tech would like us to use AI, it’s important to remember that misusing technology is what gave us cultural phenomena like autotune voices, distorted guitar, and clip to zero mastering. 

Here are a few ideas for using AI ethically and in a way that honors the lineage of generative algorithms in audio by pushing sound forward:

• Own the Data: To actively engage with what audio data goes into training a model is a creative act in and of itself, which will have an enormous aesthetic impact on what the model sounds like. Being responsible for what goes in means being responsible for what comes out. This is the foundation of how we train Artist Brains at DataMind Audio: the artist chooses what original sounds go into their small model, and each model sounds distinct and original.
  
  
• Get Creative with the Model Training: To actively engage with the training process presents many creative opportunities for how an AI music model might sound out the other side. For instance, on my last album Data Séance, I trained a small model on my bass sounds, and then used that checkpoint to train on an entirely different collection of bass sounds, which resulted in a model that was crossed between both data sets.

• Means to a beginning, not an end: Imagine you made a model that sounds exactly like you, and then sampled it to make something new. Imagine you and a friend made a model based on both of your music, and prompted the model to envision a collaboration that did not otherwise exist, and then remixed the outcome together. Imagine evoking uncanny and unnatural hallucinations rather than avoiding them. 

Even though artists like Holly Herndon, Reeps One, Gadi Sassoon, patten, Portrait XO, Dadabots and myself have made inroads on these ideas and more, it remains difficult for music producers thus far to get creative with AI in these ways. This could be partially because so much of the public-facing AI tech on the market today have been designed for the widest addressable consumer market, and partially because the GPU and data required to make this stuff is still beyond what most people have access to.

There have been, however, some developments recently that promise to put the power of AI back into the hands of music makers that wish to push the art of digital music making forward: Stability AI has just released Stable Audio 3.0, which comes with a way to train your own LoRAs. This means that soon it will be very easy for everyday music producers to build their own generative AI machines based on their own audio and for their own use. The ceiling just shattered in the walled garden of big tech deciding how models are trained and who gets to use them. The artist decides where your data goes, who gets to use the model and the model’s outputs, and all of it can live on a local device… But even more exciting, finally we might see this tech used more widely for sonic exploration rather than remediated exploitation.

I think that, after hearing the music people have made, 13 year old me who just discovered Aphex Twin would be proud of the way we’ve used AI and the tools we’ve made at DataMind Audio – a song like SAGES by Ólafur Arnalds & Loreen which uses our Combobulator plugin makes all of these efforts running a music tech company worthwhile for me. Because it’s not about what the tools can do… it’s about the frontiers of emotion that the tools give us access to, and how they might open corridors of musical experience that were otherwise inaccessible.

As Ethan Hine put elegantly in AI Slop Predates AI (which was mentioned in Adam Neely’s incredible video essay Suno, AI Music, and the Bad Future): “I love 1980s hip-hop because there’s no way to predict it from projecting trends in 1970s pop… Now that 1980s rap exists, it’s easy to feed it into an AI as training data and get more 1980s rap, but there’s no way that AI could have produced 1980s rap if it was only trained on 1970s pop.” I think this insight illuminates not only how human remediation is still superior to AI remediation, but also towards how musical innovation tends to be situated in the context of a time, place, and community.

What can we do right now to expand the limit of human expression in music from our situated positioning within our current time, place, and community? This is the kind of inquiry that leads both the R&D at DataMind Audio and the advice in my classroom and retreats for making innovative music in the DAW. And if we get this right, maybe the next 13 year old kid who stumbles across a new sound will pick up the baton and help push sound forward for the next generation.

by Ben Cantil
May 27, 2026