Final Project Brainstorming Part 2

After the initial week of brainstorming and rapid prototyping, the ideas we had still seemed a bit vague.  The team asked Federico to create some visuals for the Processing part of the project.  Instead, Federico showed up with 12 rough sketches on paper drawn in pen.  For the class time, which was a workday, we went over each of his sketches, which contained variations of ideas that used balls, Wiimotes, and crowd tracking.  At some point in the discussion, we seemed to converge on a particular idea that was from the sketch.  Everything clicked at the same time and seemed to fall into place.

Final Project Brainstorming Part 1

In ideation, the process of forming new ideas, people use divergent thinking before converging to accept a viable path for implementation.  This post reports the formative and iterative processes we used to generate ideas.   The nature of collaboration, multiple goals of teachers and students, and technological complexities constrained and set a context for accepted ideas we chose to implement.

We had about a week and one half to begin narrowing our search.  In the first session, which was after class, the four of us decided to wander outside to sit on the grass.  In our first discussion, we went around talking about what our concept of play meant.  In the discussion, we talked about throwing things in the air, such as baseballs, or other kinds of balls.  This kind of play differs from games in that there isn’t a goal or winner.

From that description, we thought we would need to track these objects using a camera and blob detection.  I thought putting a wiimote inside of a ball would give some expressive ability.  We immediately set to work on getting a prototype working.  Fu Hao played with an object tracker in OpenCV.  I started using OSCulator to start connecting a wiimote to Ableton Live.

(Click here to see the slides)

This is the set of slides that we presented.  The idea presented here is more fire related than what we ended up using.  It also expects a level of detection that we did not end up using.



Using OSCulator as a MIDI Bridge – OpenCV, Processing, OSCulator, and Ableton Live. Really.

Over the past few nights I have been working on bringing audio content from Processing into Ableton Live.  While, in general, I consider myself to be a proficient programmer, I have spent too many hours looking for ways get Python and other languages to speak MIDI.  My technical approach to connecting and encapsulating software components of the Interactive Performance and Technology  is to create network messages that work as interfaces.  This creates reusable components for future projects.

OSCulator supports communicating with Wiimotes and MIDI using a UDP based message protocol called OSC.  At this point in the project, the lack of complex messages is low priority, making way for hands on and empirical practice (hacking).

Combining thoughts from a discussion form and blog post about sending OSC messages, I made a Python based example and Java/Processing example for sending an OSC message that contains, pitch, velocity, and on/off.  I mapped the camera sensing from Fu Hao via Processing to send MIDI messages to Ableton Live. For the project, this allows us to connect the vibration of the Wiimote to the visuals on the screen to give the performer feedback.  It could also be used for additional sonification, but I’m not sure what should sound like.

Instead, here is a video of a demo of sonification control using this bridge.

What you can’t hear me explain in this video is that horizontal position is mapped to low and high pitches, while tempo is mapped from slow at the bottom to fast at the top.  This demo is surprisingly expressive, which is not really conveyed very well by this video.  If someone want’s more technical details about how this works, leave a comment.  This code will be available soon in at .


Response: Dance-Inspired Technology, Technology-Inspired Dance

Reading Summary

In this reading, Gonzalez, Carroll and Latulipe report on the complexities of engaging in co-design of interactive dance.  Leaning on experiences of co design between computer scientists, choreographers, and dancers in “case studies”, they develop five design principles.

1. Connected Kinetics

The mappings between dancer movements and visualizations should be clear to the audience.  It should be clear that visuals are affected and manipulated by the dancers.

2. Augmented Expression

Visualizations should be an extension of dancer expression.  The technology exists to widen the range of expressions available to dancers and choreographers.

3. Aesthetic Harmony

Balance, ebb and flow, of emphasis between dancers and technology should create an aesthetic complementary to each other, integrating the mediums.

4. Interactive Build

As the choreography progresses in intensity, technology should match and converse with the choreographic build.

5. Integrated Process

Choreographers and Computer Scientists should engage in iterative design early on in the design process, which informs design for all.


The authors use six co-designed performances to contextualize and support the above design principles.  After engaging in similar processes, I agree.  I also empathize in that I feel I understand that complexities can create conflicts.  The details in this paper echo statements said after sketch 1 from dancers, who felt quite constrained when using kinects.  However, these technological devices do create an augmented expression that would otherwise be impossible.  I think the following statement encapsulates the reason that so much thought and effort is needed to achieve well designed interactive performances.

“In order to inform the overall gestalt, thoughtful consideration must be given to all parts.”

Sketch 2

This post is a summary and reflection of the concept, process, performance, and lessons learned for Sketch 2. We were asked to produce a 2 minute that highlights the technique we used. The intention, and result, is to develop a “sketch” performance and technology that “gets the idea across”. In the end, the performance went rather poorly for a number of reasons that may have been avoided if we had more time to iterate and develop the concept, technology, and performance.

Formative sketch from Catherine.

Formative sketch from Catherine.


The concept for this sketch is Alice in Wonderland and exploring the technique of forced perspective. We refer to the book by Lewis Carroll and the 1951 Disney cartoon. The part that we focus on is the part where Alice finds and falls into the rabbit hole. Our idea, which came from Catherine, who is generally interested in illusion, is to create an experience centering around forced perspective. We accomplish this illusion by using interactive media and projection mapping.

There are two phase in the concept. The finding and the falling phase. In the finding phase, we emphasize the forced perspective of a forest where it appears Alice is smaller when closer and larger when farther from the audience. Next, the falling phase shows Alice falling as a variety of alice in wonderland type objects float around her. In the concept items fall move faster or slower, depending on how close the performer is to the audience.

Our concept uses technology that we used in Sketch 1: Processing for visualization, a Kinect for sensing the performers position, and a projector. New technology and systems we use include the hanging canvases we constructed for projection mapping with MadMaper, and a plugin that uses the Syphon protocol for transferring what we rendered in processing to MadMapper.



Our process, given the short time to work on this sketch, was to to discuss the concept, split up tasks, and work on. We developed and iterated the concept, then started working in parallel as possible.

We split up the roles for each member of the team. Catherine constructed the hanging surfaces for projection mapping. She use MadMaper to calibrate, and led hanging them, which we did as a team. Catherine also created all of the image visuals for the Processing code. I was responsible for creating the Processing code for the first phase of the concept, which I explain more later. Megan performed, choreographing a dance, picking music, and a costume. FuHao worked on the Phase 2 falling visualization in processing. Megan found and negotiated music, choreographed a dance, and performed the dance in the performance.

The color theme is reminiscent of the old 50’s cartoon of alice in Wonderland, centering on the Cheshire Cat.  Before switching over to a less saturated/boring real world color scheme that emphasized the transition to a new and colorful world.

Cheshire Cat


Color Scheme

For the finding phase, we use trees that have variable heights for the forced perspective illusion. After Catherine generated some initial trees, I created a generative algorithm that placed the trees in a kind of grid. Here’s a top down view:

Top down of camera and trees that creates parallax effect.

Top down of camera and trees that creates parallax effect.


Even though the trees are two dimensional pictures, they appear like they are in three dimensions. Because the threes are drawn, they appear three dimensional. To accentuate this feeling of depth, I used a parallax technique I first saw used in old Super Nintendo games.

The parallax technique is an extension of artistic principles of distance. In landscapes and other types of art, artists draw objects that are far away as closer together and with less detail. From the perspective of an forward facing observer, the closer an object is to the viewer, the faster it seems to move. So trees that are farther away move more slowly than trees that are close.

One technical issue I experienced in implementing this visualization was that some of the trees seemed to have black boxes around them. This was a simple Z buffer issue. Because I use low level access drawing mechanisms from within the Processing framework, I have to make sure that the objects in order from farthest from camera to the closes to the camera. Processing renders the pictures as I tell it to and is not a 3d engine meant for handling these issues. FuHao had similar problems that I notified him about, which he then fixed.

The idea for this scene is that the Kinect would be able to sense the depth of the dancer. However, as we came closer to the deadline, we realized that because this dance is choreographed, we could slowly change the size of the trees to have the desired effect with less technical complexity.

For the falling scene, phase two, objects move up in the screen and have a random rotation. The objects are again visual components drawn by Catherine. FuHao also create some cat eyes that randomly blink as they fall. All of the objects that fall allude to the story of Alice in Wonderland: clocks with randomly placed hands, the cheshire cat, bottles of potion, and more.

The idea is that a Kinect would sense the depth (distance from audience to the dancer) but modify the speed at which the performer appears to fall.
Catherine constructed two hanging surfaces from sheets of some kind of white polymer material. The wings were cut at angles so that the forced perspective effect would be stronger. Once we had one wing set up and hanging, we testing the projection mapping setup. The scale of the wings created a really immersive feeling to me. Later, as both wings were up, Catherine was able to create a kind of box by using bits of sky from the trees for the top, a spotlight from a white circle.

Getting processing to work with MadMapper was straightforward. We used a Syphon plugin. However, this created an extra programming step where one draws to an image in memory rather than the usual processing routines.


As I briefly pointed out before. I was happy with the way that the visuals turned out in terms of immersion, scale, and the ad hoc spotlight. The Friday before the performance, everything seemed like it was coming together. We finally had a song picked out, the visual components seems close to correct, and the wings were up and looking good. The mac mini seemed slow, but it was working. If you look at the commit log on github, you can see the number of commits on Sunday and Monday is higher than normal.

Through some lapse in communication combined with my own schedule on that Sunday and Monday being very very busy, the last minute integrations didn’t happen. This resulted, lamentably, in including a handful of the graphics Catherine created. She included more trees in SVG format. However, the code for trees expected PNG files (which are rastered). FuHao thought that the performance was scheduled for Wednesday instead of Monday. In addition, the code that FuHao had did not include code compatible with Syphon. So I integrated what was there.

To make things worse, the mac mini we used for the performance went even more slowly than when we tested it before. Because of this increase in lag, the timing of the music was off, which understandably, disoriented Megan. The visible artifacts caused by the lag made everything look jumpy and just bad.

It went pretty poorly compared to how I envisioned it. Anything that could have gone wrong did.  Here a a couple of photos by Glen Vigus in Viz from the performance and a video of the visualization from processing.


Screen Shot 2013-05-01 at 12.16.01 AM


Lessons Learned

Having little time to prepare a performance can create a disaster. A corollary is that the more time one has to prepare a performance, the better it will be. Perhaps these short term sketches amplify what might go wrong. To avoid doing things the wrong way, do them the right way. Get it right fist and then you don’t have to change it later. We should have come to an agreement about the technical specifications and tested on the actual hardware sooner. Having time for the performer to test the complete system also would have helped our performance be better.

Short time frames for projects create a pressure that accentuates strengths and weaknesses. We produced very interesting artifacts in a short amount of time, but the poor integration didn’t show that work in a very good light.  But, perhaps, that is what a sketch is.



Sketch 2 Directions

Sketch 2 is about using projection mapping techniques to fit the theme of Space Harmony.  For this sketch, we are going to show forced perspective, and hopefully create a feeling of falling though a rabbit hole.

The theme come’s from Lewis Carroll’s Alice in Wonderland.  We consider this scene from the 1951 cartoon:


From this and discussions with my team and instructors we decided to:

  • Have a 30 second exploration phase
  • Begin falling for the rest
  • Use floating for the falling dance
  • Have objects to add texture
  • If possible, we are going to show the tunnel/hole as more warped as Alice (Megan) “falls” down the rabbit hole.


Learning from previous experience with the Kinect, we are going to make it control the speed of decent, but  it will be passive.  This will give our dancer as much space as she wants to move around.  When she moves into the sensor range, she can adjust the speed of decent and it will stay where she puts it.



Reflections on Sketch 1: Lessons Learned


This monolithic post reflects on my experience with Sketch 1 for the Interactive Performance and Technology course.  I say monolithic because it been better to post my progress more as I went along.  Perhaps the memoir nature provides better context.

The Concept

DirctIVY senses a dancer’s movements with a Kinect to map the directness and indirectness of a dancer’s movements to closed/angular and open/organic visualization on a projected screen.  The idea of directness and indirectness comes from Laban’s ideas on the space dimension of Laban efforts.  Laban uses three dimensions of relationships of movements: time, weight, and space.  My favorite diagram of this is from Laban’s Effort(Laban, Rudolf, and Lawrence, F. C. Effort. (1947). London: MacDonald and Evans.).


Instead of identifying movements from individual efforts, the concept is to measure and visualize movement higher level and more abstract level.  Identifying dimensions of effort on a scalar level affords greater expression visually than measuring binary representations.

To put this concept into practice, the team and I collaborated to develop, test, and perform what we could with the given time, technical, and contextual constraints.  I have organized this post in terms of the lessons learned in (1) ways of thinking affect communication, (2) collaboration , (3) Processing, working with (4) Kinect and recognition, and reflections on the (5) performance.

Ways of Thinking (1)

Ways of thinking here refers to my own thoughts and internal perspectives as well as the perspectives and of my teammates.

“My Aesthetics”

I didn’t understand that word very well before starting this sketch.  It sounds very etherial compared to the normal diction I hear in the Computer Science discipline.  In Computer Science, I literally get paid and feed my family based on my ability to discretize and organize information from the real world into useful computational models.  It seems like this word references artistic perspective that acts to produce products.  Instead of boxing up a set of styles or attributes, “aesthetics” acknowledges the tacit nature of the knowledge of the artist/designer.

My team is diverse and has diverse perspectives, producing diverse ways of communicating.  While I am a Computer Science student, I am taking the course as Visualization student to expose myself to visualization software and techniques.  I’ve already taken Computer Animation and Computer graphics, so I wanted to learn something new.  I like to communicate with demos that show a proof of concept (you can see an example of this a few posts  back).

Fuhao works in Dr. Chai’s lab.  He used charts to communicate the effectiveness of the direct/indirect recognizer.   Megan, the Dance Science student, would show us dances.  Catherine drew sketches and iterated on concepts.   Alex, the industrial engineer,  used examples and code to communicate.

Collaboration (2)

In addition to in-person contact, we use three systems to collaborate: email, facebook, github.

We also tried to use google docs
Email: Megan used email to send us the music files she worked on. I found email to be useful for sending binary files.

Our Facebook group (which was created the day after the team was assembled) had the most immediate communication responses, but conversations became disorganized quickly. Posts in a Facebook feed are ordered by a combination of recency and user interest. Retrieving discussion threads that were more than a couple of days old required scrolling past the loaded content.

I’m a bit of a evangelist. At the beginning of last Fall, I moved around 70 code repositories of the Interface Ecology Lab from SVN to github. Our team repository is at  Github enables issue tracking, which can integrate goals, milestones, issues, code commits, and team communication.   For example, in issue #1, Fuhao and I figured out how to integrate Processing with UDP communication from a Kinect application.

(3) Processing

Processing is a really convenient way of creating visualizations quickly.  The non javascript version basically runs off of Java with a heavy serving of syntactical sugar.  Unlike “real” java, you don’t have to put functions/methods in a class.  Processing seems fast enough and expressive enough for most visualizations.  Online forums and examples help serve as starting points for learning the framework.

The revision history for the sandbox folder in github shows an evolution in my own process for creating visualizations for this concept.

This example shows me creating 10 LabanLine s at .1 increments of directness.  This helped me, but I eventually moved on to animating a visualization of just one line.  After we collected data on the first friday after the team was assembled, I asked Catherine to show me a filigree.  These can be drawn as spirals.


A little direct.


Once I had a basic implementation for this, I used data we collected as a team that Fuhao prepared an put it into a python simulator.  I also added a way to consume the UDP in Processing with Java sockets.  This let me test how the dancer would be seen by the system.  The example bellow is an early visualization of the dancer.

You see filigrees and the circle around the dancer gets bigger and smaller based on the directness (controlled by keyboard at this point).  In the next version, I have added a visualization that uses a convex hull of the dancer instead of the awkward circle.  I have also added leafs that Catherine drew on the tangents of the filigree.

The convex hull felt much more expressive, however, it changed shapes quite rapidly and felt distracting.  An improved result follows.

It’s hard to tell by the picture, but what is happening now is that the convex hull has been reduced to a width and height maximum.  Also, the filigrees are now fading as they move.

 The box that is generated uses curves with random offsets at each point.  So each line of the box is made curvy by placing extra points on the line, randomly moving them, and drawing all of the points with curve() from processing.  Her, the line goes from A to B.  The straight line represents the path the line would take if it weren’t curvy.  The yellow lines show the path from the points on the straight line to the spot with added randomness.

This is closer to the final product, with more varied colors around the box.

(4) Kinect and Recognition

My previous post shows the idea I had for measuring directness as the predictability of a path.  It looks like this does not work in practice, either because of resolution errors or our own failure to tweak the system.  Instead, Fuhao took an nearest neighbor approach in Matlab using 3 data collections: indirect dancing, direct dancing, and alternating between both.

Nearest neighbor search is a technique used in information retrieval and artificial intelligence to perform classification of data.  In Fuhao’s adaptation, he finds wether the current dancer position is closer to training data from direct or indirect to get an estimation of the directness.

To collect this data, we used a Kinect and asked Megan to dance in directly, indirectly, and alternating every 10 seconds.  Each collection was about 30 seconds.  The raw files are about 1 gig each!  The rendered files that contain the xyz points are 500kb in ASCII and 100kb in binary.

At first, the system we used from the T.A. was limited to a throughput of 10 fps per second.  Peizhao helped get the Kinect sending 30 fps.  While we used a 30 fps system for the in the dress rehearsal, I think we must have accidentally used the 10 fps system in the performance.

(5) Performance

The performance went really well, but I’m not sure that all of the elements really matched each other very well.  There were many factors that affect the overall experience of the piece that I didn’t anticipate, manage, or plan well: lighting, the sensor contraints, and tweaking minor things in general.

Lighting is an issue with interactive performances that use projection behind a dancer because you want to keep the projections screen bright, but also want to light the human performer.  If you simple use a light on from the back, then it washes out the projection screen.  We used a barn door to light Megan, but we didn’t put much thought on it.

The constraints of the sensor limited the dancing space to about a 3 by 9 foot space.  This is a much smaller space than Megan is used to dancing in and it represents less than one third of the overall projection screen width.  To put Megan “in” the box, we have to take into account the perspective of the audience.  Our system works well for someone looking directly from the front center of the audience. Because Megan was situated about 15 feet from the projector, the audience to the left or right of center saw the box too far to the left or right.

I thought about scaling the box visualization to be relative to the space (see figure above), but ran out of time.  The idea would be to increase the amount of expressible space.

Catherine and I did some last minute tweaks before the performance.  Because the system for measuring directness wasn’t integrated in time, I added a smoothing function for the keyboard modified direct/indirect number.   The smoothing function uses a value of wanted directness and moves the actual directness to be 10 percent close each frame.  I also added four vines to the box instead of one and replaced all random functions with Perlin noise instead of uniform random.  We also added some flair effects.  You can see the performance on the video below:

Processing and Kinect – Replaying Recorded Kinect Data for Development

This uses the format from Peizhao.

First, get the code by cloning the repository at . (look up git or github if you need to)

The folder we are interested with here is processing/sandbox/communication.

This folder contains the files:

./data/DirectIndirectRawData.txt   Kinect data from a raw (ascii values) format that we don’t use anymore.
./data/kinect_indirect_slower.bin  Kinect data from binary format.  If you want to know about the format, look at our discussion on github.
./udp/processing_consumer/consume_udp/consume_udp.pde   Example listener and visualizer in processing.
./udp/python_udp_simulator/   Opens files in /data.
./udp/python_udp_simulator/   The file you want to run to generate UDP data.

To run, run /udp/python_udp_simulator/  This starts the UDP traffic.  Then, open consume_udp.pde in processing (I used 2.5>).  You should see this:

The processing file has some comments.  The way that you might use it in a project is to copy this as a file in your sketch.  You could use the variables that get updated below directly:

PVector root_position = new PVector(0,0,0);
Vector<PVector> all_positions = new Vector<PVector>(21);//raw kinect data
Vector<PVector> all_positions_p = new Vector<PVector>(21);//converted to 2d

all_positions_p contains screen coordinates.   You need to make sure you don’t write to all_positions or all_positions_p or you may run into synchronization issues.

Or, you could call a different function from the within function dancerPositionAlteredEvent.

Please comment if you have questions and I will revise this post.