How Little Data Do We Need to Train SampleRNN?

-

In my previous experiments, I was training SampleRNN and Wavenet on a 52-minute dataset, and got satisfactory results. However, we wanted to know how the models perform with fewer data. I trained SampleRNN using the Cafe scenes from DCASE 2016.

Reference Samples

For reference, here are a couple of representative samples from the original DCASE dataset, with the reduced quality that I used for training (16-bit, 16kHz).





Example 1: Audio clips taken from the Cafe dataset

Samples

I trained 6 separate SampleRNN models with varying amounts of data, each for 100 000 iterations (about 31 hours for each model). By "iterations", I mean that for all models, I set the minibatch size to 52 audio files, and trained for 100 000 minibatches. This means that for the smallest trial, each audio file was presented to the network 100 000 times, and in the largest trial, which has 6 times as many audio files, each file was presented to the network 1/6 as many times. I generated five 30-second samples from each model; Here are representative samples:

num files minutes audio
52 8:40
104 17:20
156 26:00
208 34:40
260 43:20
312 52:00

Example 2: Audio clips synthesized by RNN with various amounts of training data.

Result

Based on this, it seems like 45 minutes is about the minimum amount of audio we need. This is a single-class model, and it remains to see how much audio we need per class for a multi-class model. It may also be that the models here with less data were trained for too long -- perhaps this experiment could be repeated by holding constant the number of times each file is presented to the network (i.e. fewer iterations for less data).

Additional Observations

The synthesized samples seem to over-represent the amount of high-pitched electronic beeping sounds which are present only very sparsely in the training data.

Future Work

This should be repeated with Wavenet, and later with multi-class models. For now, I will put that on the back burner until I have had some time to experiment with multi-class models more generally.

Comments

  1. UnknownAugust 17, 2018 at 1:56 PM

    Quite surprised by this result, only 45 minutes of audio needed for a single class model, thought it would have been at least double/even more than that amount?
    I am not quite sure what you are deducing here?
    Is it that you believe the system can be "over-trained" and effectively there is a "sweet spot" time wise/sample wise for that class of data?

    ReplyDelete
    Replies
    1. MikeAugust 20, 2018 at 3:46 AM

      These are all good questions. I'd like to know what this sounds like with much more data, but I think 52 minutes is the most I have from a single class right now. I don't think this is a sweet-spot for this specific class (although I don't know for sure), I think this should generalize to other classes.

      Delete
  2. AnonymousAugust 17, 2018 at 2:13 PM

    It is interesting to hear the degradations at each step down from the 43 min case, which raises a question about how to get reliable subjective evaluation of these samples. I'd like to know whether further quality improvement is audible for longer training datasets.
    It's also notable that the occurrence of any unpleasant or annoying noise in the training data seem to become more frequent, especially if they have a distinct pattern, e.g. periodic or repetitive.

    ReplyDelete

Post a Comment

Popular posts from this blog

Ambisonic Rendering in the Story Bubble

-
Image
Figure 1: Sound Sphere in the Center for Vision, Speech and Signal Processing (CVSSP) at the University of Surrey. The person is shown standing in the intended orientation, facing the front of the sphere. Introduction I recently needed some audio content that I could quickly render for playback in the sound sphere shewn in Figure 1. There are a reasonable amount of free ambisonic recordings available online, e.g. Angelo Farina Youtube So I spent some time putting together tools to render ambisonics in the sphere, and here I wanted to share what I learned in case anybody else at CVSSP wants to try this out in the future. Speaker Arrangement The sphere has 24 speakers (plus a subwoofer which I will henceforth ignore), which at the time of writing are arranged as shown in Figure 2 and Figure 3. Figure 2: Speaker layout. The center of the diagram represents the top of the sphere. The top of the diagram represents the front of the sphere. There is one spe
Read more

How I calibrated my contact microphone

-
Image
Introduction I used the Hsu-Nielson method to find the frequency response of my contact microphones. This method involves cracking a mechanical pencil graphite on the microphone transducer. The microphone in question is the one shown above, and described in detail here: http://michaelkrzyzaniak.com/marshmallow.php I specifically used the white, low-gain variant. Experimental Setup and Results Figure 1: The mechanical pencil used in this experiment, with 0.5mm number-2 graphite Figure 2: Three millimeters of graphite were extended from the pencil Figure 3: Audio interface used for experiment (Scarlett 2i4), showing the settings used. Figure 4: Video showing the graphite being cracked. A small amount of putty was used to prevent the tip of the pencil tapping the transducer after the graphite breaks. The graphite is placed on the transducer and constant pressure is applied. When the graphite breaks, the pressure is released in a very short time (shorter than th
Read more

WaveRNN

-
Image
Figure 1: CVSSP's Pet Wyvern. Introduction I've been thinking about how we can make audio synthesis faster. In particular, I would be interested in realtime soundscape synthesis, partially because I think it would be good for the project, and partly because it would be well aligned with my own personal research goals. I found two options: Parallel Wavenet For a while, I was looking at parallel wavenet . This can supposedly generate samples in realtime, and is now used in the Google Assistant. However, I have been unable to find a good, vanilla implementation of it, and the paper is really sparse on details. There are some projects on GitHub where people have started implementing it more than 6 months ago and have not finished, so given the time constraints of this project, implementing it myself doesn't seem feasible. Moreover, the training process is very intricate and involves training a couple of networks separately and then together -- which makes it really har
Read more