Audio Landscape Explorer

Overview

Generates a navigable 3D landscape using an audio file. First, you watch the landscape being generated, then you can walk around the landscape, and hear the part of the sound that made the ground under your feet.

Implementation

Loading: Uses Audio Context to read a sound file (uses filereader API). The program reads frequency data via an Analyser Node (provided by the Audio Context).

Generating: During this stage, the magnitudes of the frequncies are used as weights in determining the height of the current vertex. Data is collected every frame, and accumulates for reading only once every five frames to enhance performance. Once enough data is accumulated, a point is added to the heightmap. This point also takes into concideration the heights of its neighbors, in order to make the terrain look smoother. The camera is locked to the currently caluclating piece of the landscape during this state. The color values of the vertices are determined by their height. The generator also keeps track of the time value in the song and puts them in the same indices in a separate "timemap".

Exploring: Once the audio context reads an end in the sound, the camera falls onto the landscape, and you gain control over it. When the player starts moving, the context starts playing the sound from the time fetched from the time map at the indices obtained from the player's x and z positions. The sounds stops if the player stops, or if the player is not moving to new segments (so that you cant run in place and still play the entire sound).

User Guide

Runs well on Chrome and Firefox. Other browsers not tested.

To start, click the browse button to select a sound file to generate around.

After the sound loads, you will start to see the landscape being created strip by strip as the sound plays.

Once the last strip has finished being generated, you can start walking around the landscape, and it will play back the part of the sound that made the ground where you are walking.

If you stop walking, it stops playing the sound.

Sample output

Landscape of a Calm Instrumental

Landscape of an Electronic Song

Electronic Song during Generation

Video

The Project