Building a virtual reality music instrument

Work in progress. This article is intended to reflect the work in progress from loose ideas to some kind of playable prototype. Comments, ideas and feedback are welcome at any point. Latest update 7.15 pm gmt+2 Sep. 19–2015

First steps: The elements

6.50pm gmt+2 Aug. 26–2015
My initial plan is to try to define the elements needed for getting something to work. Basically to get an idea about the technicalities before getting too far ahead thinking about possible implementations and specific ideas. Getting deep with the basic concepts and investigating techniques and methods would allow me to understand better what could be done, and then ultimately allow me to work on a clear idea or concept that could work within the framework. The goal is clear; to make playable prototype of a music instrument that can be played in virtual reality. And if it happens that I don’t make it all the way, then at least it would be a good process of learning and playing with the possibilities of this exciting technology.

Another preliminary note. Since I have access to a few basic hardware and software options I will make my case evolve around these rather than pinpointing any product or tool that could do the job. I do not have an Oculus Rift to play with, but I’m confident that also lighter, cheaper and more accessible products and services can take me, us, a long way.

So. the elements:

The viewer. The visual and audible access to the space. Ultimately this would be the Google Carboard with somekind of home made Android app and with earphones, that gives both visual and audible access and full control over the experienced perspective. In the simplest version this could also be just a computer screen with a sound output and a fixed perspective. But, virtual reality will only ever feel real if you perceive yourself as a free agent within the virtual space. Free agency means control; and being in control over what you want to see is a very basic part of this, and for that reason at the very core of the illusion of virtual reality.

The space. A 3D space for coordinating the controller and the perspective. This is also where the sound of an instrument would live, which actually kind of makes is more like a 4D space including sound - or time as you could also define the existence of a sound - as the 4th dimension. For this I expect the Unity3D engine to be a platform that could be used. Also inherently with the Leap Motion setup comes a 3D space as the stage for the controller. I’m not sure if this space can easily be projected to a viewe with perspective though.

The controller. Some way of feeding movement or action into the space and make this cause a sound. The simplest controller with the Google Cardboard/Android and other mask solutions is your head movement. Controlling something basically means interacting with it in a way that makes a change to the object. And by changing the perspective by moving your head, you are also changing the object, in this case the space and sound around you. You can build an instrument from that.
And off course the Leap Motion is a competitor here. I have previously written about some of the limitations in using leap motion as a controller but it seems like the whole scene has matured a lot since i wrote that, and having something else than your head movements is the only real way to go beyond the obvious, the headbanging app. And that’s definately an ambition with this.

Next up: Questions

9.20am gmt+2 Aug. 27–2015
Next step is to get more specific with connecting things and mapping out options. An open question is how to make the controller work inside the viewer, or inside the same space, where you then have basically two controllers; your head and your hands. This needs to take place in the space, preferably without any latency and real time, and then ported through the Cardbord with an Android mobile phone. We’ll let’s see. At least the questions are starting to present themselves.

Also a simple proof of concept with using Unity3D as the platform and getting some kind of sound activation going on in the space designed would be nice to have established soon.

Second steps: Connecting the dots

9.36 pm gmt+2 Aug. 27–2015
I’ve now drawn some models to get an idea of how to connect the elements.

Basically I see the Leap Motion as the controller, Android in Google Cardboard as the viewer and Unity3d as the space.

So, ideally what we want is something like this, Model 1:

..a direct connection between the Leap Motion and Android running an app of some sort with a Unity3d space with the actual sounds and instrument.

But after a bit of initial googling on this model it seems like it might be hard to pull off, compared to a seemingly more complicated, but noneless more often tried route among fellow travellers in VR: Connecting the Leap Motion tracker and the Android through a computer, like this, Model 2:

Either way, the job is now figure out how to get the elements to play together, real time. So more research and googling ahead.

Oh, and by the way today I also came with a name I think is suitable for the project: Extraudionary VR :)

The research

9.58 pm gmt+2 Aug. 27–2015.
Some examples of people and projects connecting the Leap Motion and Android/Cardboard:

And this cool guy building a VR guitar with Leap Motion. Not sure if it’s gonna work with Cardboard.

10.21 pm gmt+2 Aug. 27–2015
Oh. So apparently the Model 1 can be ruled out. The Leap Motion guys are busy stabilizing an alpha SDK for Android, and even though i’m lucky enough to have a Snapdragon 801 (OnePlus One) as required:

“the alpha isn’t ready for public demos or presentations, such as hackathons or work published on the web.” Leap Motion

So on to the next option, the Model 2. I’ll now try figure out a way of connecting everything through Unity3D on my computer, and run it all from there.

The framework

11.05 pm gmt+2 Aug. 28–2015
Ok, so now the plan is to get a framework going in Unity3D, that can connect the Android in the Cardboard and hopefully the Leap Motion too. By now my focus is the first part. After all the Android in the Cardboard would be some kind of controller too, and also i’m getting ideas stretching beyond the mere head-banging, so maybe there is hope yet for something very simple, using your mere being as conductor of music.

Specifically I’ve been thinking of a 360 degree piano, simply playing the next note as you turn around, rising in octaves as you continue passing the off set. But let’s see, if everything connects, I’m confident that we can set up something even more elaborate.

The elements of sound

11.20 pm gmt+2 Aug. 28–2015
A huge, and yet completely unaddressed question at this point is: What is an instrument? What is music? Let’s not be too picky at this stage, and simply state that music is something that occurs when instruments of sounds are in use (or recorded). Then I believe music can basically be broken down to: Is there sound? Yes or no. In terms of computers and physics, a simple vibrating wave form in some kind of frequency, that is either on or off. I should mention that I have a deep belief in music as a form of communication rather than as a product. I beleive music can carry a message and a meaning that cannot be defined by timely definitions of a medium or a product and cannot be justifiably equated with a price tag or specific value in any given situation. It simply is waves of meaning and feeling that you either adopt, rebel against or don’t care about.

So the on/off thinking in terms of sound is the foundation. Off course lot’s on exciting stuff can be applied on top that. But we need to start with something simple.

11.45 pm gmt+2 Aug. 28–2015
Also I’ve been sidetracking a bit and playing with graphics for a potential logo. I know it’s early stage, but sometimes you just gotta go, where you wanna go. And I guess I kind of fear the process of really diging into the understanding of Unity and sort out all the details and functionality (even though it seems like the n00b-friendliest tool ever in the history of digital design). OK, enough words, I’m gonna throw the logo draft in here, and dig in :)

Setting up Unity with Android

0.42 am gmt+2 Aug. 29–2015

  1. Open the Unity Getting Stared with Android development guide.
  2. Get Android Studio up and running.
  3. .. hmm

.. this is over my head. I’m stuck with way too many unknowns and no real experience in setting up a development environment. The fear seem to be justified.

So, I’m changing plans, rolling back a bit and sticking to the ideas level to keep my head clear. And hoping to hook up with some smart and interested people tomorrow who know a bit more about the technical aspects of this. I can probably set up something inside Unity on my own, but for the idea of getting it out of the computer in any kind of preview on the phone, I definitely need some help. It would be awesome to hook up some game developers or Android/Cardboard savvy folks tomorrow and get something up and running. I’ll cross my fingers and see what happens :)

Setting up Unity with Android

12.14 pm gmt+2 Aug. 30–2015

I had a great time yesterday at the maker weekend Framtidsrummet 2015 in Helsingborg. It ended up being mostly talk, sharing inspiration and presenting various awesome stuff from with the Cardboard and Leap Motion. It did not really push this project much further, but it was a good boost of energy and shared inspiration, and a good way to test out some thoughts and ideas on the concept level. The greatest part though was experiencing people’s virgin reactions to stuff like Evolution of Verse from VRSE on Cardboard and Chordion Conducter and Collider with Leap Motion. There is true magic to be discovered here. And so the seach continues.

A music game or a music instrument

2.25 pm gmt+2 Sep. 19–2015

A simple but important distinction needs to be made. That between a game and an instrument. Games can be musical and instruments can be playfull, but in order to keep focus on what we are trying to achieve we need to understand the difference between the two.

A musical instrument is about adding sounds to the world. It’s about subtracting sounds from it and absorbing it in yourself. That in turn is what we call music listening or music experience. A part of this music experience also takes place when playing a music instrument, but as a natural side effect, not as the end goal.

In a music game this is different. Here the music listening experience is a main goal, and playing is supporting that goal.

With a musical instrument we want to have tool of expression, not a tool of impression. A game gives you an impression, but doesn’t let you express yourself beyond the boundaries of the initial setup. A music instruments let’s you express your self, and with that expression you can create an impression for others.

First draft

7.15 pm gmt+2 Sep. 19–2015

I manage to get a first draft ready for a virtual music space with Unity3d. It works in Firefox and Safari. Not in Chrome


One thing is playing an instrument to your self though, but when shared between people music can become something truly special. So finding a way where more people can join in on the experience of playing music or at least join in as an audience, could be a great next step for this.

Some links for further investigation and background



UX driven product designer with 20 years experience in editorial design and digital publications. Let’s work together!

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Jesper Vestergaard

Jesper Vestergaard


UX driven product designer with 20 years experience in editorial design and digital publications. Let’s work together!