Riffusion is a Stable Diffusion machine learning system adapted for musical synthesis
The launch of a new project called "Riffusion" was announced. It develops a variant of the Stable Diffusion machine learning system adapted to generate music instead of images. The music can be synthesized based on a suggested template or a natural language text description.
The music synthesis components are written in Python using the PyTorch framework and are available under the MIT license.
The project is interesting because it continues to use "text-to-image" and "image-to-image" models for music generation, but manipulates spectrograms as if they were images.
In other words, classic Stable Diffusion is not based on photographs and images, but on spectrogram images that reflect the change in the frequency and amplitude of the sound wave over time. Consequently, a spectrogram is also formed at the output, which is then converted into an audio representation.
This is the v1.5 stable diffusion model with no modifications, just fitted on text-paired spectrogram images. Audio processing occurs downstream of the model.
You can generate infinite variations of an ad by varying the seed. All the same web user interfaces and techniques like img2img, inpainting, negative indications, and interpolation work out of the box.
spectrograms
An audio spectrogram is a visual way of representing the frequency content of a sound clip. The x axis represents time and the y axis represents frequency. The color of each pixel gives the amplitude of the audio at the frequency and time given by its row and column.
It is mentioned that the method can also be used to modify existing sound compositions and synthesize sample music, similar to image modification in Stable Diffusion.
For example, the generation can set sample spectrograms to a reference style, combine different styles, smoothly transition from one style to another, or make changes to an existing sound to solve problems such as increasing the volume of individual instruments, changing the rhythm and replacement of instruments.
The STFT is invertible, so the original audio can be reconstructed from a spectrogram. However, the spectrogram images of our model only contain the amplitude of the sine waves and not the phases, because the phases are chaotic and difficult to learn. Instead, we use the Griffin-Lim algorithm to approximate the phase when reconstructing the audio clip.
The patterns are also used to generate long-duration compositions, made up of a series of closely spaced passages that vary slightly over time. The separately generated fragments are combined into a continuous flow by interpolating the model's internal parameters.
The frequency bins in our spectrogram use the Mel scale, which is a perceptual scale of pitches judged by listeners to be the same distance from each other.
Below is a hand drawn image interpreted as a spectrogram and converted to audio. Play it through to get an intuitive feel for how they work. Notice how you can hear the tones of the two curves in the lower half, and how the four vertical lines in the upper half produce rhythms similar to a hi-hat sound.
A Fourier transform (a mathematical transformation used to transform signals between the time domain and the frequency domain) is used to create a spectrogram from sound. When recreating sound from a spectrogram, there is a problem with phase determination (only frequency and amplitude are present in the spectrogram), for which reconstruction the Griffin-Lim approximation algorithm is used.
The interface is implemented in TypeScript and is also distributed under the MIT license . The trained models are released under the permissive Creative ML OpenRAIL-M license for commercial use.
If you want to know more about it, you can check the details at the following link.