Ambisynth

Introduction After some disappointing results on single-class WaveRNN models , I thought I should repeat the same experiment with a different architecture. Everybody keeps asking why Yong was getting such better results . He was using SampleRNN, and the reason I have been avoiding that is because, even though all of the architectures are painfully slow at generating audio, SampleRNN is the only one where I haven't seen any work that suggests that it might eventually be made faster. It is difficult for me to imagine a viable service where people have to wait several hours to generate a few minutes of 8-bit audio; at that point it might make more sense to generate the audio in advance that people can download as needed. If we are going to do that, why don't we just put an internet-connected microphone in the woods and stream high-quality wav files to our servers? Without the ability for a sound-designer to adjust the synthesis parameters, manipulate the model in realtime, blen...

Introduction So far, we have been trying to get better sounds by improving the training process. However, it occurred to me that we could also try to improve the sampling process -- the way that we get audio out of a model that has already been trained. I was thinking about My favourite paper , where he gives a few tips for sampling. The main one makes use of the fact that the model outputs a probability distribution. After a forward pass through the network, we can modify that distribution to make the most probable things even more probable, or vice-versum. In our case, we can raise the whole distribution to some power. If the power is greater than one, the distribution will get stretched up and down, making the mountains higher and the valleys lower (making the output more predictable), and if the exponent is less than one, the distribution will get flattened out (making all options equally likely thereby making the output more noisy and random). The sampling temperature is defi...

Introduction My previous posts all deal with single-class models, meaning that I train a network on *only* cafe sounds, for example, so later that model can generate only cafe sounds. In principal, it should be possible for a single model to be trained on *several* classes, e.g. cafe AND lakeside. Later, the user should be able to specify which type of audio they want the model to generate. The main benefits to this are: If the model is able to learn a common representation of the different classes, it might produce better audio. This was reported in the original wavenet paper , where they say, perhaps surprisingly, that training wavenet on several human voices and selecting the desired one later produces better-sounding results than training on a single human voice. For similar reasons, a single multiclass model might be smaller, in terms of the file size, than several single-class models. With the implementations I am using, a single-class wavenet model trained ...

Introduction After some semi-encouraging and semi-disappointing results with WaveRNN, I wanted to see if I could get better results on single-class models, and I wanted to train more models so I could see if the difficulties are anomalies associated with particular datasets, or what. Improvements After my lakeside model exploded , I thought some regularization might help. So I implemented a few things. Gradient Clipping This is a no-brainer. I clip the gradient norm to ±1. This seems to help models not explode, and speech models seem to train faster. Dropout There evidently isn't much consensus on how dropout should be applied to RNNs. Some papers say to apply it only to the input connections, or only to the output connections. Some say apply it everywhere except the recurrent connections: Reference One Reference Two while other papers say it is best to apply it directly to the recurrent connections: Reference Three . Every paper shows some version of this gr...

Introduction In a previous post I talked about noise profiling . That generated some interest amongst our team members, so I made a web version that is easier to use than the previous command-line version. Links The code for the project is in the Ambisynth Private Repository . I'm not posting a link to the live web-page here because we might try to monetize it later. Features There are several noise profiles available by default. Users may also upload new recordings or corpuses of recordings, and the webpage will analyze them and create a new noise profile. Users can can play back synthesized sounds in real time or download a wave file. The theory of operation is described in more detail in the previous post. Figure 1: screenshot of the noise profile tool Future Work 1) Allow users to control the analysis window size and overlap. 2) Collect several noise profiles for each corpus and either average them, interpolate between them, or collect a distribution from them and...

Introduction Adversarial autoencoders have been used to synthesize images, so I thought they might be employed to synthesize sound as well. One obvious difference is that images are static in time, while sound could go on forever, with each new bit of sound depending on all previous sound. This is a difficult property of sound, and to simplify it, I think autoencoders might initially provide a hybrid approach somewhere between granular synthesis and deep learning. My idea is that an auto encoder might be able to generate short windows (grains) of audio that would behave computationally like images, and that could later be assembled via granular synthesis . There might later be a separate machine learning thing that would tell us in what order to assemble the windows (more on this anon). The code described in this post is in the Ambisynth Private Repository Autoencoders Figure 1: Schematic of an autoencoder. A window of audio samples goes in the top, flows through a bottlen...