Videogames are interactive experiences with emotional highs and lows, providing players with thrilling experiences alongside wondrous vistas. The greatest games can leave lifelong impacts on their players long after the controller is put down.

Emulators serve as a convenient way to relive those past experiences and rediscover hidden gems from one's childhood. But what if an emulator could not only recreate those moments, but enhance them by pushing the games you know and love to new heights? At what point do people say that the must-play experience of the game is not on the original console, but on an emulator?

The Nintendo 64 and Playstation era of emulation brought with them the flexibility of 3D engines, meaning that for the first time emulators could really make a significant difference in how a game looked. The biggest enhancement was the ability to increase the resolution games run at, producing clear images on high resolution displays. While polygon counts remained very low, emulators that supported this important feature quickly became extremely popular.

The GameCube and Wii are exponentially more powerful than either of those consoles. What was a nice enhancement a generation before is now one of the most prominent reasons for using an emulator. Dolphin provides the same enhancement to a vast library of games spanning two full consoles, including many past consoles via the Virtual Console. Many users replay their favorite games on Dolphin because of its ability run them at any resolution alone!

A New Enhancement¶

But that wasn't the team's only attempt to revolutionize the way games looked in the emulator. Long ago, it seemed like another enhancement would be just as interesting and perhaps even dwarf upscaling: Anaglyph 3D. This feature allowed users to play games in stereoscopic 3D and see actual depth using red-cyan tinted glasses in the D3D9 backend.

While the Anaglyph 3D feature had the potential to be a great enhancement, it resulted in a distorted 3D effect and terrible performance thanks to a number of technical and design problems. After a lukewarm response to this supposedly big feature, the Anaglyph 3D settings were left out of a UI rewrite in the Dolphin 2.0 era, silently killing the feature.

Help from the big N¶

With the 3D cinematic revolution in full swing, NVIDIA debuted a home 3D Solution called NVIDIA 3D Vision. This technology consisted of a pair of shutter glasses and a proprietary NVIDIA driver that would allow users to take ordinary games and translate them into stereoscopic 3D automatically. The result was far better than Dolphin's Anaglyph 3D feature: the specialized drivers didn't distort the image or halve the framerate, and the active shutter glasses allowed for full color clarity.

Unfortunately the driver relied on heuristics which were often incompatible with Dolphin, so a hack was needed to improve compatibility with 3D Vision. Part of this hack was a quick-and-dirty implementation of exclusive fullscreen in the D3D9 backend. This allowed 3D Vision to recognize Dolphin, and the results could finally live up to expectations. Unfortunately, using NVIDIA 3D Vision with Dolphin was a difficult proposition at best and suffered tons of issues:

1. NVIDIA 3D Vision didn't take off as expected, resulting in a limited user base.
2. Support for only Windows and D3D meant that Mac/Linux users were completely left out.
3. Dolphin had to use a hack to support it, and that hack did not always work.
4. With no influence over the 3D Vision code, Dolphin developers couldn't work around any of the visual glitches created by the feature.

All of that mixed with the relatively high cost and poor support of the product meant that there was very limited testing with 3D Vision on Dolphin. Limited testing meant limited use, less developers to work on issues, and a general disinterest in the feature.

Lost Ambitions¶

With two 3D implementations that failed to take off, Dolphin coasted along with the feature laying mostly dormant within the emulator. Anaglyph 3D was a total failure, and 3D Vision had serious barriers even for those who were interested. Though Anaglyph 3D was quickly dropped, NVIDIA 3D Vision had enough potential (and was easy enough to port) to remain in Dolphin for as long as D3D9 remained.

The run finally ended in 4.0-155 - when D3D9 was dropped for accuracy reasons. 3D Vision was not given a proper send-off, instead left as a casualty to larger changes. The reaction was surprisingly mild: an issue report here, a forum post there, but overall? Virtually nothing.

Months passed with no news about the feature, save for a few users who used the same rough work-arounds as before the hack existed. The once-promising feature was left in the past as the emulator moved forward. There was no point to bringing back the old hack with its problems, so all of the issue reports regarding 3D Vision were closed as the team finally moved on.

Exclusive Fullscreen Support¶

A flicker of light for the fallen feature lit when Armada expressed interest in implementing Exclusive Fullscreen mode for the remaining D3D backend. Exclusive Fullscreen was shown to reduce input latency, make games run smoother, and lowered overhead – things that would improve overall experience for everyone, not just a handful of special setups. This task amounted to a rewrite of Dolphin's window hierarchy, simplifying it so that D3D11 could obtain Exclusive Mode Fullscreen without hacks. Coincidentally, that was exactly the feature that NVIDIA 3D Vision needed to once more be able to detect Dolphin. Of course it didn't work at first, but once Armada realized that Exclusive Fullscreen could restore 3D support, he added it to his list of priorities for Exclusive Fullscreen and worked with testers to revive the old feature.

After many painful weeks of tweaking, configuring, and bug hunting, Exclusive Fullscreen support was merged to Dolphin, marking the return of third-party 3D support! Dolphin could work again with NVIDIA 3D Vision out of the box and Tri-def with a little setup. While the results from these third-party solutions were very nice, they still left out a large chunk of the user base and didn't always work reliably. So not wasting any time Armada set out on another attempt at implementing Stereo 3D natively within Dolphin.

Native Stereo 3D¶

NVIDIA's 3D Vision spoiled everyone with how easily it created good 3D. On the other hand, Dolphin's anaglyph support never impressed anyone. It suffered from distortion, lack of depth, and other issues that made it very difficult to recommend. In order to make Dolphin's new native support better, Armada carefully studied what NVIDIA did right, and what Dolphin did wrong.

Stereoscopic Projection¶

The technique for displaying a 3D image onto a computer display is known as stereoscopic projection. It begins with defining two cameras, one for each eye, which are separated by roughly the same distance as a person's pupils. Since Dolphin shows the image on a screen and not directly on the user's eyes, it needs to create the illusion of objects both beyond and in front of the physical screen. For this to work, a virtual screen has to be defined, and everything is then projected relative to this virtual screen.

The method that was used to generate this effect in the old Anaglyph 3D feature is known as a "toe-in projection", it involves rotating the cameras slightly inwards so that they're both pointed at the same point on the virtual screen. This works fairly well when looking at the center of your screen, but the sides of the screen will suffer from heavy distortion known as the Keystone effect. This distortion happens because in rotating the cameras the virtual screen being looking through has also been rotated. There are now in fact two different virtual screens which only merge into a single screen in the center of the image.

To fix this problem, a different kind of camera that doesn't rotate the virtual screen is needed AND it still has to create the effect of looking at two different perspectives beyond the screen. The solution is "off-axis projection", which skews the image projected into the cameras instead of rotating the cameras themselves. Using this method the two cameras are converged on the same virtual screen despite the fact that they're aimed straight forward.

In a real camera this projection is achieved with a set of lenses, but in a 3D game engine it is implemented as a set of projection matrices. Normally a game that natively supports stereo 3D would define two projection matrices from scratch. However since Dolphin is emulating consoles, it can't simply define its own projection matrices; it has to base it on the single projection matrix given by the game. While Armada was working a few things out on paper, Comex suggested that we could simply apply a shearing transformation on the projection matrix from the game. This would produce two cameras with a skewed image, one for each eye.

Unfortunately sending two matrices to the GPU was inefficient, so Armada looked into generating them on the GPU itself. However, someone had already faced this same problem and overcame it: NVIDIA. Armada noticed that Dolphin's latest attempt at stereoscopy was quite similar to NVIDIA 3D Vision's methodology, and discovered that NVIDIA has generously published a specification of their algorithm. Instead of defining two projection matrices, 3D Vision simply applies the shearing transformation directly to the 3D objects in the game. After confirming this method produced the correct results it was applied to Dolphin with great success.

Stereoscopic Rendering¶

Now that 3D objects could be projected correctly, Dolphin still needed to actually render the game for each eye, which proved to be yet another challenge. Most games which natively support 3D stereo will simply render the game twice every frame (multi-pass rendering), however this poses a significant problem for Dolphin since the Gamecube and Wii were only ever designed to render the game once per frame.

This problem was never tackled in the old Anaglyph 3D feature; all it would do was show the user two different frames at the same time: Every odd frame rendered would be displayed for the left eye and every even frame would be displayed for the right eye. Each eye would in fact be seeing two different points in time, which made fast-paced games extremely difficult to watch. It also effectively halved the visual frame rate, since the rendered frames were being divided over the two eyes. 60 FPS games like F-Zero GX would max out at 30 FPS per eye while 30 FPS games like Wind Waker would max out at an unplayable 15 FPS per eye. A clean and efficient solution was needed in order to handle this problem without compromising the framerate of the game.

To render the game twice within the same frame Dolphin needs two Embedded Framebuffers (EFBs); one for the left and right eye. This is not a problem since Dolphin is in control of the EFB and already manipulates its size for enhancing the internal resolution. With that out of the way, Dolphin just needs a way to duplicate all the geometry, and there are several ways to do this.

One solution was to just duplicate every draw call: for every object that is rendered by the game Dolphin would send it twice to the GPU and render the output to the left and right EFBs, but since Dolphin is already bottlenecked by the amount of draw calls doubling those did not seem like a good idea.

Then ector suggested using a geometry shader to duplicate all 3D objects on the GPU and transform them so that they're rendered from different perspectives. This would give the result of rendering the game twice without any extra work on the CPU! Looking for previous implementations Armada was able to find a few research papers with significant technical limitations, and a tweet from John Carmack that tackled at least one of those limitations.

Those two sources combined was more than enough information to begin implementing this technique in Dolphin. But that doesn't mean there weren't still serious problems standing in the way.

The main problem with this technique had to do with EFB copies. At first the EFB was made twice as wide, and would render the game to the left and right portions of the EFB using viewports as mentioned in the tweet. But what would happen when Dolphin needs to make a copy of this EFB and display it as a texture in-game? Users would see the game twice in one eye!

Immediately degasus suggested that instead of making the EFB twice as wide, we should give the EFB an extra layer and use the relatively obscure technique of multilayered framebuffers.

During the geometry shader the GPU will render the 3D objects for each eye to separate layers of the EFB. This way when the EFB is displayed in-game Dolphin can show only the layer appropriate for each eye. And when it displays the final image on the monitor the layers can be in any configuration required instead of just side-by-side. Another advantage is that Dolphin no longer used the "viewport select" feature that John Carmack mentioned which brings the minimum requirement to OpenGL 3.3 instead of OpenGL 4.3, allowing this feature to support a much wider range of GPUs.

However, all of these methods have one limitation: sometimes Dolphin can't add an extra layer to an EFB copy, just like it can't always increase the internal resolution of such a copy. This happens when an EFB copy has to be directly stored in the system memory allocated by the console instead of the video memory allocated by Dolphin. The memory allocated by the console doesn't take into account that the EFB is now twice as large, so Dolphin can only use one layer at the native resolution which results in the EFB for the left eye being shown in the right eye. Essentially, some games that do post-processing, which often forces Dolphin to use a copy from RAM, are not fully compatible with Stereo 3D.

Despite that limitation, overall compatibility is still staggeringly high. The stereo 3D feature has never worked better and the performance impact is minimal.

It should be noted that Oculus and NVIDIA have introduced a new OpenGL extension which implements single-pass rendering without the need for a geometry shader, allowing GPU vendors to optimize specifically for it in their drivers and GPUs. Optimizations which Dolphin will benefit from.

Dolphin Stereo 3D Support¶

By natively supporting Stereo 3D, Dolphin can now support this feature for a huge range of graphics cards, as well as handle challenges and problems with 3D on its own terms. This means there is a huge possibility for improvements and game-specific modifications to make 3D more enjoyable.

Dolphin now features:

• 3D Stereoscopy in the OpenGL and D3D backends on all platforms with any graphics card that supports at least OpenGL 3.3 or OpenGL ES 3.1 with the Android Extension Pack.
• Side-by-Side, anaglyph, top-and-bottom and 3D Vision output allows Dolphin to support a wide range of monitors and glasses.
• Proper 3D projections brings minimal distortion and better depth than the old 3D solution.
• The 3D behavior can be adjusted to avoid glitches and problems.

There are two main settings should be adjusted when searching for the perfect output:

• Depth: controls the distance between the two cameras; some games are at a different scale and require the cameras to be further apart or closer together for the best relative scale.
• Convergence: controls the distance objects need to be to appear in front of the screen. Usually in a third-person game, one would want objects behind the character to appear in front of the screen, but every game has this character at a different distance. Without adjusting this setting there often won't be any convincing out-of-screen effects.

Usage and Limitations¶

Like any enhancement, 3D support is not guaranteed to work in every title. Games made for the GameCube and Wii had no consideration for a feature like this much like they weren't meant to run in HD. The surprising part is that many games translate virtually perfectly into 3D thanks to how they were designed and coded, and others work fairly well with a few minor adjustments.

That said, 2D games, such as The Legend of Zelda: Four Swords Adventures, should be considered an obvious example of a game that won't see any benefits. Games like these disregard the concept of depth, thus the 3D conversion code has nothing left to do besides move the whole screen in and out.

A far more annoying issue is that 2D elements in 3D games will sometimes be drawn at screen level, meaning they have no depth, even though it would be far better if they did. For most games this is only a minor annoyance, but it can make certain games nearly impossible to play. Fortunately, some titles that suffer from this problem can be assisted through texture hacks. These basic texture hacks can remove the offending graphics and transform a game like Starfox Assault from impossible to play into an absolute beauty in 3D.

Why care about Stereo 3D support?¶

None of the above matters if no one uses the feature. A majority of Dolphin's users lack a stereo 3D setup and may not get why bringing back this feature is worth even a mention. Even this article said it was so unimportant that it was dropped without even a murmur.

Why is bringing back stereoscopic 3D support something to celebrate?

The answer is simple: stereo 3D support has gotten better over the years and so has Dolphin. The old 3D solutions didn't work well enough, Dolphin didn't run nearly as many games, and barely anyone had the hardware on hand to test. Dolphin was in a different place, a place where merely getting games running well was still the main task at hand. Dolphin today is a much more polished emulator, so enhancements like stereo 3D work far better as a result.

Dolphin plays some of the greatest games ever made. People are always looking for new ways to experience their favorite games, and as long as there are people inspired by the games, new enhancements, features, and discoveries will continue to be made. Even the implementation of 3D Support has opened the door for another exciting feature in the near future...

Future: Virtual Reality Support¶

Stereoscopic 3D support implements the most basic need required for Virtual Reality support, so it's the logical next step. While it may come as a surprise, at this point one of the biggest hurdles for official VR support is a licensing issue - the EULA on the Oculus SDK prevents it from being used in any open-source application licensed under the GPL.

The Oculus SDK license is quite draconian. In short, Oculus can void your license to use the Oculus SDK, and with that the entire application, on grounds that:

• The SDK is being used in an application that can "cause health and safety issues".
• The SDK is being used "to interface with unapproved commercial virtual reality hardware".

This is in stark contrast with Dolphin's GPL license which grants everyone the freedom to use the software for any purpose. That means Dolphin cannot make or distribute builds that integrate with the Oculus SDK while still protecting your right to use the Dolphin emulator as you see fit.

Fortunately an early version of OSVR was released recently which allows open-source applications to support many different VR devices with one open-source SDK that is GPL compatible. Using that SDK, Armada implemented a prototype build that could turn any game that is supported by stereo 3D into a full VR experience. That's what makes emulation such a beautiful tool: the games that we remember can be made even greater, even stronger than before with all kinds of enhancements. With virtual reality on the horizon, one can only imagine what other wonders these amazing games could hold.

Dolphin Virtual Reality Technical Preview: Exploring Outset Island