10 ways 10 hours of raindrops can help you live to 100

-

Introduction

I was listening to this stuff generated by SampleRNN. It seems almost unreasonably good compared to some of the results I have been getting. The most obvious difference is that that person is using 30 to 50 hours of training data, as compared to my typical 1 hour. To a certain extent I have been avoiding using huge datasets, because in a way Ambisynth makes more sense if we can use small datasets -- i.e. if you have 50 hours of soundscapes, why synthesize more? So for now, this post represents more of a (forbidden) research question: how good can synthesized soundscapes be; let's assume practical considerations will take care of themselves at some point in the future.

Dataset

So I made a 10.5-hour dataset with audio from this amazing website (it's well worth a good listen): https://recordingsofnature.wordpress.com. The recordings were all made with stereo microphones attached to trees in the woods near lakes between 3 and 5 in the morning. I chose several files that were dominated my rain. For posterity, I used these files:
  • file0065_trim_18db.mp3
  • file0100_58-2-15_db10-5.mp3
  • file0104_8-31_10-5db.mp3
  • file0108_trim_10-5db_6-36.mp3
  • file0113_730-940_10-5db.mp3
  • file0153_4-5db.mp3
I prepared them as required by the different Neural Net architectures.

Results

SampleRNN

I trained SampleRNN for 110 307 iterations (about a day) on this data, and sampled it with a temperature of 1. Some of the results are in Example 1.




Example 1: Sounds generated by SampleRNN trained for 110 307 iterations on a 10-hour rain dataset.

I then continued to train for another couple of days, up to 338 640 iterations. The results are not noticeably different, although maybe less birds overall, at least in the few samples I generated.



Example 2: Sounds generated by SampleRNN trained for 338 640 iterations on a 10-hour rain dataset.

WaveRNN

I trained WaveRNN for 120 000; iterations (about 3 days), and generated samples every 6000 iterations. The results were no different than from before, which is to say muddled noise that never really converged on anything in particular.

WaveNet

I trained Wavenet for 150 000 iters on the same dataset and generated several 30-second samples. One of them is in Example 3.


Example 3: Sounds generated by wavenet trained for 150 000 iterations on a 10-hour rain dataset.

This sounds reasonably rain-like, but none of the samples generated anything resembling birdsong, and the sounds all have undesirable silence and clicking. In this case, I presented the data as 6 about 2-hour audio files, so the clicking should not be the result of discontinuities at the file boundaries, as I had suspected before.

Conclusions ans Apotheosis

WaveRNN really sounds good -- these are easily the best results we have gotten in the past 5 months. I want to try to find another large dataset and try to replicate the results. I also still want to *try* to run some listening tests, so at some point I am going to have to cut off training more models and just do it with what we have, but I think this is worth pursuing before doing any tests with human subjects.

Comments

  1. willbDecember 12, 2018 at 3:15 AM

    Sounds good.....

    ReplyDelete
  2. AnonymousDecember 12, 2018 at 7:29 AM

    The SampleRNN results do sound good. So (and I think you know what's coming) why is that?

    ReplyDelete
  3. MikeDecember 12, 2018 at 8:09 AM

    I think it is just a matter of having more data. It probably has something to do with overfitting, but to be honest I'm not really sure how overfitting applies to synthesis. I keep wondering if overfitting might be good here, like if we could learn to synthesize the training data more or less exactly, that should sound pretty plausible. I remember making Markov text models years ago, and training with 4 or 5 novels. The model would end up grabbing one phrase from one novel, the next phrase from another novel where the transition is common to both novels, and so forth. That might be a reasonable way of synthesizing audio. I keep imagining that something similar might happen with overfitting, however, that seems not to be the case here.

    ReplyDelete

Post a Comment

Popular posts from this blog

Ambisonic Rendering in the Story Bubble

-
Image
Figure 1: Sound Sphere in the Center for Vision, Speech and Signal Processing (CVSSP) at the University of Surrey. The person is shown standing in the intended orientation, facing the front of the sphere. Introduction I recently needed some audio content that I could quickly render for playback in the sound sphere shewn in Figure 1. There are a reasonable amount of free ambisonic recordings available online, e.g. Angelo Farina Youtube So I spent some time putting together tools to render ambisonics in the sphere, and here I wanted to share what I learned in case anybody else at CVSSP wants to try this out in the future. Speaker Arrangement The sphere has 24 speakers (plus a subwoofer which I will henceforth ignore), which at the time of writing are arranged as shown in Figure 2 and Figure 3. Figure 2: Speaker layout. The center of the diagram represents the top of the sphere. The top of the diagram represents the front of the sphere. There is one spe
Read more

How I calibrated my contact microphone

-
Image
Introduction I used the Hsu-Nielson method to find the frequency response of my contact microphones. This method involves cracking a mechanical pencil graphite on the microphone transducer. The microphone in question is the one shown above, and described in detail here: http://michaelkrzyzaniak.com/marshmallow.php I specifically used the white, low-gain variant. Experimental Setup and Results Figure 1: The mechanical pencil used in this experiment, with 0.5mm number-2 graphite Figure 2: Three millimeters of graphite were extended from the pencil Figure 3: Audio interface used for experiment (Scarlett 2i4), showing the settings used. Figure 4: Video showing the graphite being cracked. A small amount of putty was used to prevent the tip of the pencil tapping the transducer after the graphite breaks. The graphite is placed on the transducer and constant pressure is applied. When the graphite breaks, the pressure is released in a very short time (shorter than th
Read more

WaveRNN

-
Image
Figure 1: CVSSP's Pet Wyvern. Introduction I've been thinking about how we can make audio synthesis faster. In particular, I would be interested in realtime soundscape synthesis, partially because I think it would be good for the project, and partly because it would be well aligned with my own personal research goals. I found two options: Parallel Wavenet For a while, I was looking at parallel wavenet . This can supposedly generate samples in realtime, and is now used in the Google Assistant. However, I have been unable to find a good, vanilla implementation of it, and the paper is really sparse on details. There are some projects on GitHub where people have started implementing it more than 6 months ago and have not finished, so given the time constraints of this project, implementing it myself doesn't seem feasible. Moreover, the training process is very intricate and involves training a couple of networks separately and then together -- which makes it really har
Read more