8 bits, 8 players, 8 projectors, and one Nintendo Entertainment System

What do you get if you combine the nerdy ingenuity of ETH Zurich, the creativity of Disney researchers, and a swanky Swiss night club with a 360-degree projection system? The world's only cooperative 8-player, 8-bit Nintendo Entertainment System capable of continuous, panoramic side-scrolling on that aforementioned 360-degree display, of course.

At this point I would recommend that you look at the photo gallery and at least one of the videos below so that you have a mental image of how this all slots together. I will attempt to describe the system, too, but words don't really do it justice.

So, as you probably know, Super Mario Bros is a single-player side-scrolling game for the NES/Famicom. You constantly move forwards, jumping over obstacles and on top of goombas, until you get to the end of the level. That's about it. There's no going back, you can't zoom out to see where you've been, and the max number of players is two: one for Mario, one for Luigi.

In May this year, the Eurographics 2015 conference was jointly hosted by ETH Zurich and the University of Zurich. At around the same time, Disney Research's Bob Sumner had just founded the ETH Game Technology Centre. Sumner had been tasked with coming up with "a cool concept" for the dinner at the end of the conference—and so that's exactly what he and his fellow engineers did.

Back in July, just after I'd taken a look at IBM's Solar Sunflower, I visited Sumner in Zurich. As we sat by Zurich's Limmat river, in between sipping some cold beers and lamenting the costs of living in our respective cities, he explained the core concept and motivation behind the project: "Basically, we observed that the 8-bit era of gaming had a huge collective influence on so many people, but the actual gaming experience was typically an individual one. We wanted to turn this idea upside down, and elevate the NES console experience into a group experience where people can play in a collaborative setting."

How do you turn the NES into a 360-degree gaming machine?

First things first: the team used a real NES with real game cartridges. There was no hacking of the console itself, nor of the game ROMs. Everything that you see in the photos and videos was created with DIY hardware and software.

To connect eight controllers to the NES, they used an Arduino-based multiplexer. Video output from the NES is fed through an upscaler, to get the output up to a solid 576p at 50Hz. Audio output goes directly from the NES to the room's sound system.

Next, the upscaled video is fed into a "tracking PC." This PC is running some custom software that takes incoming video frames and copies them into a new, much wider output buffer. The software looks at the background of each frame as it comes in and tries to line it up with the previous frame, stitching them together into a panorama. This sounds quite simple until you remember that a) the background can move very quickly or slowly, depending on the player, and b) in most side-scrolling games the player can also go backwards.

Once the tracking PC is done working its magic, the video output is sent to a media server, which in turn drives eight projectors (two for each wall). Because each wall has a slightly different aspect ratio, and the media servers stretch the output image slightly to cover the entire space, the tracking PC also has a real-time "GPU algorithm" to correct the distortion. The end result, as you can see throughout this story, is a very sharp rendition of Super Mario Bros, Castlevania, and other NES classics.

How can eight people play Super Mario Bros?

The final piece of the equation is turning original one- and two-player NES games into cooperative eight-player games. The Arduino multiplexer has two modes of operation: it either cycles through each gamepad after a fixed amount of time; or the tracking PC can tell the Arduino to change to a specific gamepad, depending on how far through the level the players are.

So, for example, Super Mario Bros might switch between gamepads every five seconds, or every time Mario moves a certain distance towards the end of the level. With eight players arranged around the octagon, you could arrange it so that each player gains control of Mario when he's on their section of the wall.

The end result, according to Sumner, is pretty damn fun, if a bit manic. Because of the "progress shifting," it's very hard to use some abilities effectively. The Super Leaf in Super Mario Bros 3, for example, which allows Mario to fly, is very hard to use when it might require three consecutive players to work together in unison to achieve lift-off. Or, you might jump towards a goomba to kill it, but then lose control because your five-second window is up.

What if you don't have eight projectors?

After Bob had finished explaining how their crazy NES hack worked, my first question, of course, was whether I could give it a go. Sadly, Bob no longer had access to a venue with a 360-degree projection system—nor did he have a whole day spare to set it up again.

Still, I didn't go home empty-handed. At the Disney Research offices (which are really cool, incidentally) they had whipped up an alternative output system: an Oculus Rift. It was a bit janky, and lacked the eight-player cooperative mode, but it did kind of work. Basically, as you move through the level, the tracking PC draws a continuous panorama—and as the panorama unfolds, you can move your head around to change your viewport.

The main problem was that I could only turn my neck so far before the pathetic kinematics of my humbly humanoid vertebrae started to suggest that permanent paralysis would soon be on the cards. When that happened, I had to quickly whip my head around to the other side, and then continue playing as normal. It got a bit tiring, but was still rather fun.

* * *

For more technical details on the entire setup, ETH Zurich has published an open-access research paper called "Unfolding the 8-bit era." The paper was presented in London last week at the Conference on Visual Media Production. The project was led by Sumner, who worked with a team of PhD students from ETH Zurich and Disney Research.