Augmented Reality is more than Virtual Reality
Ghost Objects and Devices
This is introducing an entirely new concept, expanding the possibilities of virtual objects far beyond those known or imagined from real ones:
Just a little while ago, I thought what might happen if we could actively use our eye accommodation and parallax to pick out spots in three dimensional point clouds. Raw tomography data for example, is fairly huge by the sheer number of pixels and therefore hard to rotate in real time. Calculating features and surfaces for a solid modeling is even way more difficult.Looking at rotating raw data pixel clouds, our brain already generates a perfect 3D impression, even with a 2D screen, but many features are just occluded by others even if we render for optimum transparency and actively look at features from all sides.
Just confining the rendering process to the center of view has long been tried, but what if we also use depth accommodation, focus and eye parallax that is, to render the right depth area denser and crisper than others?
This would save a lot of computing power as well, as we only have to render each pixel for a small area of main interest, 1/1000 of the volume for example, and all the rest at far lower resolution.Literature revealed several hints on actively confining rendering efforts to area and depth, but only for solid modeling. Just one paper deals with the manipulation of pixel cloud rendering by eye steering, in this case by certain eye gestures, using a single eye tracker [95]. Nevertheless it was pointed out that two eye trackers and other approaches would be tried next. I hope this work is still proceeding.
But the medical application and the rendering approach are just some out of numbers of others that could use active depth steering. In virtual and augmented reality, it could really add an entirely new world of experiences.
Imagine virtual objects that are like faint vapor when not in use, being in place but only just hinting their existence, and then suddenly become solid when you accommodate to their proper distance. Or several layers of windows on your computer screen (to mention the most old fashioned stuff), not simply being transparent but changing transparency and crispness according to depth accommodation. Other than the transparency of current window managers, just hardly useful, this could really be working at last.
Here we see three windows arranged in space in different distances. Left, the nearest window is accomodated at, and the others automaticalled are made transparent and blurred. Beneath, the hindmost window is selected, and to the right, the middle one. This can of course only give a faint impression what it would look like, as with a real implementation, the 'ghost' windows would have a different distance and stand out easily, by parallax or in case of head movements.
Of course here we are at the limits of conventional displays again, as it would really be hard to accommodate by parallax on a screen that does not deliver the appropriate focus.
With vision goggles, this is all different. Virtual windows here could not only be arranged all around you, but this way also in several layers, enabling not only dozens but many hundreds of them for concurrent use.What if we not only enable looking through walls, but fully automate this by making the wall disappear as soon as we just look towards infinity, changing its humble appearance for a view coming from outside cameras, or for a giant screen cinema event?
Several layers of virtual bookshelves stacked in front of a wall, all kinds of gadgets, devices and utilities shown as vapor or faint outlines when not in use (Vaporware, anyone?), but getting into life if you eye-point at them in three dimensions. A virtual globe for example, in the middle of your living room, five feet tall but not activated by an icon, instead by looking at its own faint preview.
These virtual objects should of course disappear as soon as you are approaching real ones covered by them, just for safety.It is also a really new and unknown realm, as it is not yet obvious how the interaction with such objects would feel:
If a 3D solid dissected this way, giving the impression of some super microscopic or ultra narrow depth-of-field view, would interact with our visual system in a convenient way, if the reactions of the object would make our accommodation go astray, if we could easily learn to willfully accommodate towards 'nothing', beyond or before things that we really see, to activate the others still hidden, these are very complex questions that currently no one can answer.
Maybe a properly designed system would work well, while a poorly designed one would cause headaches and eye strain. Nevertheless the enrichment of the virtual experience by these things will be overwhelming, and any research in this field will be necessary and important, given alone the many fancy applications we could imagine.
There are many possible varieties of the basic technology as well: We could use a refractometer assembly that measures eye focus though the lens, instead of dual eye trackers measuring parallax (squint).
This would even work with one eye alone. In an eye operated mobile phone for example, with a monocular see-through display, eye steering of the device would become way easier, as no icons or controls at the edge of the visual field would be necessary. Transparent vapor icons and buttons could be used instead, and activating them by eye pointing would be very secure, as it would only work when accommodation would fit as well.Eye pointing onto links in a virtual, transparent web page would be depending on proper accommodation as well, i.e. if we look towards the far it wouldn't react. The page would also automatically be transparent at far accommodation and reappear at near accommodation, half transparent then for example, so using complex virtual interfaces in the real world would be a lot easier even if the device would not support any real world placement or registering of these objects (binding them to real world locations, as real things are).
Eye pointing of this kind could perfectly be combined with finger pointing, the device would easily and unambiguously know when you hit a key or grab a handle.
In the end, 3D eye pointing will make many interfacing tasks a lot easier. The little extra effort of a refractometer (an IR camera looking at your retina through the lens) for example, can enable a monocular goggle phone almost as powerful and convenient as a full fledged virtual vision system, without the much larger effort of motion tracking and location registering.
Many other applications will be possible. A 3D chess game for example, or a 3D Quake game with transparent walls, or of course many serious applications as well, like the 3D image of an engine seen inside the real object and actively revealing any of its depth layers one at a time, selected just by the eye accomodation of a service technician, this way being enabled to really see all of its inner detail in absolute perfection.
Rolf R. Hainich, June 3, 2008
home more notes order
Copyright © 2006-2011 Rolf R. Hainich; all materials on this website are copyrighted.
Disclaimer: All proprietary names and product names mentioned are trademarks or registered trademarks of their respective owners. We do not imply that any of the technologies or ideas described or mentioned herein are free of patent or other rights of ourselves or others. We do also not take any responsibility or guarantee for the correctness or legal status of any information in this book or this website or any documents or links mentioned herein and do not encourage or recommend any use of it. You may use the information presented herein at your own risk and responsibility only. To the best of our knowledge and belief no trademark or copyright infringement exists in these materials. In the fiction part of the book, the sketches, and anything printed in special typefaces, names, companies, cities, and countries are used fictitiously for the purpose of illustrating examples, and any resemblance to actual persons, living or dead, organizations, business establishments, events, or locales is entirely coincidental. If you have any questions or objections, please contact us immediately. "We" in all above terms comprises the publisher as well as the author. If you intend to use any of the ideas mentioned in the book or this website, please do your own research and patent research and contact the author.
