Soundweaver

High Level Design

Western music has two main types of keys: major and minor. In its most common form, pieces in a major key are typically brighter in sound. People tend to feel more excitement and happiness when these pieces are played. With pieces in minor keys, people tend to feel more calm or melancholic. The goal of this project was to reflect these emotions using colors on the VGA screen. People tend to feel happiness with green, sadness with blue, and anger with red. Therefore, the goal was to match the emotions of the sound with the color. There were two main parts of this design: the music analysis, which differentiated the keys from each other to find the mood (emotion) of the music, and the animation, which displayed the colors onto the VGA screen using a boid algorithm imitating starling murmurations. Figure 3 shows an HSV 360 degree color wheel. In our program, we use the degree value (starting with 0 degrees pointing straight up in the red section and increasing counter-clockwise) to represent colors. When the program averages or changes colors, it is using the degree value on this color wheel. This wide range of colors is made possible using Bruce Land’s 8-bit color VGA driver, which allows for more color values at the expense of screen resolution.

Figure 3: Color Wheel from 8-bit coloring.

Music Analysis

To analyze the music, a sound input is broken up into intervals. A musical interval is the distance from one note to the next. For example, a major second is a step difference between the original note and the new note. A perfect fifth is three and a half steps away from the original note to the new note. Because the color value on a screen can be broken up into three sub-values for red, green, and blue, the possible intervals are broken up into three categories: major, minor, and dissonant. In simpler terms, the intervals are broken up into three moods: happy, sad, and angry.

Calculating the intervals of a new note takes two steps. The first step involves calculating the frequencies of the new note. This was done using a Fast Fourier Transform analysis based on Professor Hunter Adams’ Cooley-Tukey FFT program. This step takes samples at different frequencies and measures the amplitude of the voltage at each frequency. Second, the program iterates over all the frequencies to find the amplitude peaks to identify the notes played. For example, if a sound has the highest amplitude at a frequency of 440 Hz and that amplitude is comparatively well above that of other frequencies, then the note played was a singular A note.

Every iteration the FFT runs, the three frequencies with the highest amplitudes are identified. An amplitude threshold is then compared to the highest detected amplitude to determine if a new sound was played. If there was a new sound played, the differences in amplitude are then calculated for each of the three amplitudes to see if one, two, or three frequencies (up to a triad) were loud enough to be considered played frequencies, or if they were merely harmonics or outside noise. The algorithm then calculates the interval of the most recent highest frequency with the previous sound’s highest frequency. If two frequencies are played, then the interval between the lowest frequency and highest frequency is additionally calculated. Similarly, for three different frequences were played in the new sound, the intervals between the lowest frequency and the middle frequency and the middle frequency and the highest frequency are additionally calculated.

The intervals are calculated using a logarithmic ratio of frequencies known as cents. This equation is shown below in Figure 4:

Figure 4: Cents equation.

The difference between each interval is 100, allowing the program to easily calculate the intervals between the notes. Using C as an example, below in Figure 5 is the interval and its cents value:

Figure 5: Cents table.

Once the cents value is found, the last step is to match the interval with a color. Though intervals are not individually trapped to an emotion, for the purpose of this project, intervals are broken up into happy (green), sad (blue), and angry (red). The idea when splitting up the intervals is to have a generally balanced color gradient, allowing for a possibility for a large range of colors to appear on the screen. With this in mind, green was set to denote unison, major third, perfect fourth, and perfect fifth intervals. Blue would denote major second, minor third, minor sixth, major sixth, and octave intervals. Finally, red would denote minor second, tritone, minor seventh, and major seventh intervals.

Once the program determines the intervals, a predator boid would be first triggered to represent each interval. In a separate core running the boids algorithm, the predators that are turned on will both affect the color of the surrounding boids and chase them away, creating a splash effect. The second thing the algorithm does is calculate an overall mood. This is an average color based on the past twenty highest detected frequencies.

Animation

The animation on the VGA screen implements a modified version of the Boids algorithm under the hood. Figure 6 below shows an early test version of the program. The music analysis done on a separate core can turn on up to three predators and assign each an affecting color to spread to the boids that are within their chase range. In this version, the predators are always invisible. The predators will move throughout the screen space, and if there are no new notes, the boids will ignore them. However, once the program detects new notes, the predators will turn on, and the boids that are within the predatory range of the predators will turn away. These boids will also have their color set to the average of the affecting colors of the predators that are chasing them. Boids that are not in the predatory range of any predators will have their color shift towards the overall mood by 15 degrees.

Figure 6: Early boids implementation.

To complete the animation, the screen is split into square tiles, as shown in the Figure 7. After the Boids algorithm updates the position of a boid, its position gets mapped to the tile on the screen that contains it. When this happens, the boid’s color values gets added to the tile’s color accumulator value, and the tile’s boid counter increments. After all the boids have their effects on the tiles updated, each tile will calculate its color. Tiles containing no boids will have their color set to the overall mood color. Otherwise, the tile will have the average color of all of its contained boids (or the color accumulator divided by the boid counter). Predators do not directly affect tile color, but instead affect the boids which do affect tile color.

Figure 7: Tiled animation.