Augmented Reality is more than Virtual Reality
Architectural aspects
of Augmented Reality glasses
While the 'holy grail' of augmented reality technology may be the creation of display glasses with fully integrated computing, as powerful as a current desktop PC, this will still have to wait a few years until power consumption and weight become convenient.The most likely current solution are 'dumb' display glasses constituting the visual equivalent of a headphone, and being connected to a personal information device (PID), usually a smartphone. While such display glasses may still have to be more complicated than headphones, as we want to include orientation cameras, eye tracker etc, the according computation should nevertheless be outsourced to the PID.
This device could receive the data from one or more media players and other networked devices, and also orientation camera and sensor data from the glasses, and deliver the glasses a readily rendered display video integrating all active sources.
The following illustration shows this for virtual TVs or computer displays. With High End display glasses (for more on this click here), this would provide a real alternative to large HD displays, combining low cost, low power consumption, high resolution and perfect 3D rendering (overcoming almost all limitations of physical screens), and this with glasses not heavier (potentially even lighter) than current shutter glasses.
Sample augmented reality scenario, including three virtual screens
In our example, we have decorated walls with dark rectangles, freeing space for the insertion of virtual display screens. This way, we need no masking capabilities within the display glasses..
The individual rectangles may be enhanced with block codes (in infrared colors, e.g.), or with a relatively dark image, a dark furniture surface, or a simple rollup curtain. Using natural images for orientation is state of the art meanwhile, and the image could be subtracted from the virtual video in order to make it disappear. The different layouts of the rectangles serves for their identification and for assigning them different video streams. The video streams can be provided by several - real - source devices (TV receivers, computers, media players) and are, e.g., transmitted to the user's hardware via WLAN. It may even be possible to omit the PID in this case: the orientation camera could also send its raw images to the source devices and leave them the computing of orientation and coordinate transforms.
A crucial point in our example is a sufficiently good registration of the virtual images to the real world. For distant objects, we can fortunately assume a collimated display model, translating the glasses' display coordinates directly into angles relative to the display (
,
in the illustration). For a complete explanation click here.
What we get is a non-changing, linear relation between the image of a position camera fixed to the glasses and the projected image of the display glasses' displays. Once the camera has located the corners of one of the rectangles, this gives us the exact positions where to place the corners of a video display window in the glasses displays. All that remains is a simple projective transform of camera pixel to display pixel coordinates.
This works well for larger distances to the displays (the usual 2..3m from a TV should behave like infinity). For closer distances, it can be adapted to a certain degree, by applying relatively simple corrections.
The architectural possibilities in this example are many, and screens are but one - quite simple - example for virtual devices. An example quite similar to the TV screens shown are real appliances with virtual enhancements, like the washing machine with a virtual control panel from “The End of Hardware” (right image): the entire set of controls is replaced by a little wireless chip, and the virtual controls displayed are operated by finger pointing, recognized by the orientation camera(s).
A wide variety of other virtual or partly virtual devices are indeed very similar to our virtual displays, only their image sources may differ, and also the rendering methods and transmission formats used (e.g., vector graphics or game engine based scene descriptions).
These ideas have first been presented to a larger audience at the BiTS2012 conference.
Reducing power consumption and and weight
Low power consumption is an important aspect for the construction of convenient display glasses, as it allows for cable-free operation using small rechargeable batteries. The accumulators used int current mobile phones or digital cameras often weigh 20g or less, and are allowing for continuous operation enduring many hours, even though considerable computing tasks have to be performed and mobile phones also need considerable power for RF transmission. Given the fact that the main functional elements of display glasses can be reduced to thin chips and thin plastic elements for the optics (mirrors), the remainder of the construction will be extremely light. This means low power consumption is most crucial.
In this context, special attention has to be given to technical questions of video compression and transmission, and the distribution of processing tasks between different devices. Orientation cameras require only VGA resolution, because the location of edges once found in an image can be computed to small fractions of the pixel size. Transmitting their raw images to an external processor can, e.g., be done in MJPEG format, avoiding latency times and incurring low computing requirements and power drain.For transmitting HD video, the first choice currently is h.264. This can result in large delay times because of features like forward- and backward interframe optimizations, and also needs lots of processing power. It is however possible to modify interframe optimizing algorithms to forward operation only, still sending a 'difference to the last frame' instead of a full frame, but avoiding additional latency, except for the necessary computation time. The transmission bitrate is very variable in this case (bursts occurring when full frames are transmitted), but this is not really a problem considering current WLAN technology.
Another possibility is the external computing of orientation data and of rendered perspectives, including inlays.
This short assessment already shows that many architectural decisions have to be made for such applications, and also many new standards have to be defined. The definition of common standards will indeed be crucial for the success of this technology. Any brand of display glasses will have to 'play' with any kind of media or virtual device sources. Only if we can use the same pair of display glasses for all applications offered, these glasses will be desirable to have and to wear.
home news order
Copyright © 2012 Rolf R. Hainich; all materials on this website are copyrighted.
Disclaimer: All proprietary names and product names mentioned are trademarks or registered trademarks of their respective owners. We do not imply that any of the technologies or ideas described or mentioned herein are free of patent or other rights of ourselves or others. We do also not take any responsibility or guarantee for the correctness or legal status of any information in this book or this website or any documents or links mentioned herein and do not encourage or recommend any use of it. You may use the information presented herein at your own risk and responsibility only. To the best of our knowledge and belief no trademark or copyright infringement exists in these materials. In the fiction part of the book, the sketches, and anything printed in special typefaces, names, companies, cities, and countries are used fictitiously for the purpose of illustrating examples, and any resemblance to actual persons, living or dead, organizations, business establishments, events, or locales is entirely coincidental. If you have any questions or objections, please contact us immediately. "We" in all above terms comprises the publisher as well as the author. If you intend to use any of the ideas mentioned in the book or this website, please do your own research and patent research and contact the author.