Dynamic Word Cloud

Open brainstorming sessions are a prevalent way to kick start the development of interesting concepts. These sessions are usually intentionally kept unrestricted to yield as many ideas and approaches as possible. Many of the ideas generated that way never leave the sketch book. They might not fit the needs and desires of the particular client perfectly or may just be to ambitious for the given scope. However every once in a while an idea sparks our personal and/or professional interest and curiosity and gets worked on in our spare time nonetheless. The project described in this blog post falls into that category. 
The Dynamic Word Cloud
So while thinking about ways to augment the experience of talks, panel discussions and presentations the concept of a dynamic word cloud came up.
The idea was to program a typographic projection, consisting of relevant key words, which is dynamically generated in the course of a speech to provide the audience with a guiding thread while listening. The words displayed in this projection are outlining the essence of whats being said by the speakers. After the conclusion of the speech the final result could also be printed out to offer the participants an individual memento of the talk, discussion or presentation. Prototyping this idea seemed to be an interesting internship side project to work on whenever I had the time.
How it Works
The first step in realizing this visualization was to take the raw audio output of a speakers microphone and converting the words spoken into text, using speech to text software. The resulting text is separated into individual words. Those words are then sifted through to sort out conjunctions, interjections, pronouns and essentially every other word that isn’t carrying any significant meaning on it’s own or cannot convey meaning due to the lack of context in the randomized visualization it is going to end up in. Usually speakers tend to use words they’d like to emphasize on more frequently. The same thing applies to essential information like the name of the topic for example. This is why the the occurrences of each remaining word are counted in the next step. The resulting occurrence-count is then mapped onto the size and color of the respective word. Finally all the words are randomly placed on the available surface to achieve a cloud-like visualization.
Technical Setup
To make a prototype for this word cloud work as intended I had to (re)familiarize with different tools and technologies.
First I needed to find a sufficiently functional speech to text converter that I could run in the browser. After attempting to utilize many different Chrome browser extension, trying to use the built in speech recognition of Windows and among other things installing Amazons Alexa on a Raspberry Pi amounted to pretty much nothing I ultimately ended up setting up a Node.js server running some JavaScript to access Googles Web Speech API.
With this solution I could continuously put in audio via microphone and get a clean string back. Node.js in combination with Socket.IO also allowed me to effortlessly pass on the results to vvvv via OSC.
Since I was already well acquainted with vvvv I used it to handle all of the sifting, sorting and rendering.
For most of the programming the existing vvvv nodes where sufficient but I wasn’t able to find nodes to prevent the overlapping of words which would inevitably occur if iI just spread them out randomly. Overlapping greatly diminished the readability so I also had to write a custom node for vvvv in C# to prevent it from happening.
Examples
Here are additional examples for word clouds generated from some very different speeches, talks and presentations.
President Donald Trump Speech at CPAC 2017, Friday, February 24, 2017
Apple WWDC Keynote 2017
Elon Musk talks Climate Change and Carbon Tax at the Sorbone 2015
Martin Luther King Jr. “I Have a Dream”, 28 August 1963, Lincoln Memorial, Washington D.C.
With the guidance of my patient colleagues I went through the process of generating an idea and developing it into a functional prototype. This internship project was a valuable experience for me and and the advancement of my humble programming skills along the way certainly didn’t hurt either.

vf

Related Posts

KOMMENTARE