Computer vision and VR – an experiment to bridge the physical with the virtual world between 2 users (2020)

What kind of conversations can emerge between two users – one in the physical and the other in the virtual world? Referencing the movie “Dogville”, I came up with this project drawing on the notion of “taking control” that could either help or exploit the other (at a later more narrative-developed stage), depending on how the players choose to act.

In the example above, I used OpenFrameworks to create a 2d interactive map of Dogville, that would indicate the position of a player (green dot), an NPC (yellow dot) and a spotlight (white dot). User 1, would be able to control that white dot, by actually moving a spotlight around on a paper, that OpenFrameworks is able to track through computer vision and link it to the white dot movement. This spotlight would then be translated as a virtual spotlight in a VR environment that another user is experiencing through OSC (that enabled the communication of data between OpenFrameworks and Unity). So User 2 is actually able to see the spotlight movements of User 1. Then the position of User 2 would also be depicted back into the OpenFrameworks environment (green dot) becoming visible to User 1 too.

This was a test to create a basis for future interesting interactions between those 2 users that would respond to each others actions in those two different environments.