Dynamic Word Cloud

The idea was to program a typographic projection, consisting of relevant key words, which is dynamically generated in the course of a speech to provide the audience with a guiding thread while listening. The words displayed in this projection are outlining the essence of whats being said by the speakers. After the conclusion of the speech the final result could also be printed out to offer the participants an individual memento of the talk, discussion or presentation. Prototyping this idea seemed to be an interesting internship side project to work on whenever I had the time.

The first step in realizing this visualization was to take the raw audio output of a speakers microphone and converting the words spoken into text, using speech to text software. The resulting text is separated into individual words. Those words are then sifted through to sort out conjunctions, interjections, pronouns and essentially every other word that isn’t carrying any significant meaning on it’s own or cannot convey meaning due to the lack of context in the randomized visualization it is going to end up in. Usually speakers tend to use words they’d like to emphasize on more frequently. The same thing applies to essential information like the name of the topic for example. This is why the the occurrences of each remaining word are counted in the next step. The resulting occurrence-count is then mapped onto the size and color of the respective word. Finally all the words are randomly placed on the available surface to achieve a cloud-like visualization.

To make a prototype for this word cloud work as intended I had to (re)familiarize with different tools and technologies.

First I needed to find a sufficiently functional speech to text converter that I could run in the browser. After attempting to utilize many different Chrome browser extension, trying to use the built in speech recognition of Windows and among other things installing Amazons Alexa on a Raspberry Pi amounted to pretty much nothing I ultimately ended up setting up a Node.js server running some JavaScript to access Googles Web Speech API.

With this solution I could continuously put in audio via microphone and get a clean string back. Node.js in combination with Socket.IO also allowed me to effortlessly pass on the results to vvvv via OSC.

For most of the programming the existing vvvv nodes where sufficient but I wasn’t able to find nodes to prevent the overlapping of words which would inevitably occur if iI just spread them out randomly. Overlapping greatly diminished the readability so I also had to write a custom node for vvvv in C# to prevent it from happening.