Meet Holly Herndon: singer who deepfaked her own voice

Are we reaching a tipping point where we should embrace deepfake technology rather than be disempowered by it? Artists are beginning to experiment and take back control of their digital selves with surprising results.

Have you ever secretly wished that you could sing in your favorite musician's voice? Experimental musician Holly Herndon trained an AI on her voice so that she could sing almost anything in multiple languages. In a recent TED Talk, the artists unveiled Holly+, an AI-powered instrument that will allow anyone to sing using her unique voice.

In a live demo, musician PHER sang into a microphone that amplified his natural voice. Midway through his performance, he switched to a second microphone that made his voice sound identical to Holly's. In addition, when he sang into both microphones, it sounded like a duet. But is there more to this technology than a novelty digital ventriloquism act?

The music video for Kendrick Lamar's "The Heart Part 5" showcased the creative potential for AI technology as the rapper seamlessly morphed into O.J. Simpson and Will Smith. The normalization of swapping faces or bodies in everything from Hollywood movies to Tom Cruise's deepfake videos means it's no longer creepy for many. But seeing somebody inherit the voice of another artist could make you feel uncomfortable about where we are heading.

However, if you take a peek behind the curtain, you will find a neural network bringing the imitated singing voice to life through advances in machine learning. But artistic experimentation will also require artists to carefully navigate copyright laws designed for an analog world.

The legal complexities of digital twins

Holly Herndon refers to her AI vocal clone as her digital twin. But Holly+ could soon develop a life of her own and spark future debates around voice ownership, and we have been here before. In 1988 Bette Midler famously sued and won her case against Ford Motor Company after they paid backing singers to impersonate her in a version of "Do You Want to Dance." A few years later, Tom Waits also won a case against Frito-Lay for performing a vocal impersonation of him doing a Doritos radio commercial.

Herndon believes that DAO (Decentralised Autonomous Organization) stewardship of her voice model rights might be the answer. She believes that, eventually, she will be in total control with the release of tools that support the creative usage of likenesses and provide licensing opportunities that DAO members and all profits would approve would also be shared amongst everyone involved.

Vocal deepfakes are here to stay. A balance needs to be found between protecting artists and encouraging people to experiment with new and exciting technology

Holly Herndon

When the creators of Rogue One digitally recreated Grand Moff Tarkin (Peter Cushing) and Princess Leia (Carrie Fisher), it also started a conversation about what happens to our image rights after we die. On her website, Herndon shares how she is much more comfortable with distributed ownership of the rights to her voice model with a DAO of stewards who are equally invested in maintaining her value and reputation than a soulless music executive.

Herndon's AI deepfake twin also hit the headlines recently for making the brave and bold move of covering Dolly Parton's Jolene. Holly+ nearly pulled it off with impressive results, but something was missing. Upon listening more closely, it quickly became apparent that AI cannot replicate the raw emotion and vulnerability of a human voice delivering one of the most powerful songs of all time. At least not yet.

The AI vocals and accompanying music video feel strangely hypnotic and disturbing in equal measure. However, rather than comparing with the original, it almost succeeds in taking something old and turning it into something new.

The cover uses a vocal model trained on Burden's voice, which means Dolly Parton would receive royalties as the songwriter as she would with anyone covering her material. But could the industry be heading in a new direction where royalties could be shared with the human voice the AI was trained on via NFT minting and new methods like the Zora auction model?

Once we get over the knee-jerk reaction that AI could replace musicians and a dystopian future of deepfakes of leaders spreading dangerous disinformation and users blindly following virtual influencers, there is something to be much more optimistic about. Herndon's Holly+ is a tool that will enhance the creativity of an artist rather than replace it. There is also an argument that the copyright laws from our analog past have held back artists and stifled creativity.

What is creativity anyway? If we look under the hood, everything is a remix. However, experimenting, copying content and combining it with other elements, and transforming it into something completely different is the foundation of the music industry. It is responsible for everything from rock n roll to electronic dance music.

Musicians begging AI not to please don't take their man in 2022 is understandable. If it's in the right hands, it could open up new genres and dramatically transform the entire creative process by bringing together the best of both worlds. But artists must remember that the human spirit, raw emotion, and vulnerability should be at the heart of everything and use technological tools to enhance everything else. Ultimately, video never killed the radio star, and it’s unlikely AI will replace musicians.