Augmented Reality is more than Virtual Reality


Mobile phones will soon have hundreds of gigabytes of storage and replace a camcorder, a PC, anything. But how to use all this?
Traveling with 2 pieces of luggage, looking for my connecting flight, I just imagine fumbling around with this thing, trying to see any informations on its dwarfish display. Taking a pen, trying to stab little blots on the display, doesn't help either. This technology, getting as near to the personal communicator as anything so far, will surely hit the wall in the near future, simply because there is no acceptable display, and rollout displays or projectors won't do. Long ago I realized this problem, and now technology is ready to solve it. We need to eliminate the screen in favor of a near-eye projector, glasses with a tiny add-on that could finally weigh less than 20g.
Forget those ancient virtual reality goggles. Forget anything hampering direct sight.
It can be done way better.

Virtual objects, virtual devices will surround us, everywhere. They will soon replace most of today's user interface hardware, screens, keypads, entire installations, and they will do a lot more. Nevertheless, these virtual things won't clutter up our view at all, because other than with some classical approaches to augmented reality, here they will be seamlessly integrated and fixed to the real environment, rather than sticking in front of our eyes.

We will have to envision entirely new applications and usage habits. It's a new world to explore.
No more desktops mapped to a computer screen, but operating systems mapped to the real world. Available anywhere. Boosting productivity beyond any expectations.
When I started my activities back in 1993
[1],[2], generating steady, high quality images merged into natural sight was already resolved, and the only major obstacles were computing power and small high resolution displays.
Ten years later, these obstacles were gone, but very little had happened in all the other fields. Somewhere, programs and joint projects were launched whose aims looked ten years old. Basic parts of augmented reality technology still deliver themes for theses all over the world (fortunately at least this happens), but it's usually a matter of chance if somebody, maybe from institutions not generically involved in the thematic at all, picks up this or that problem because of necessity. Very few researchers around the world are dedicatedly concerned with the thematic, and a few small or medium enterprises are developing in this field. The funding of this research is still microscopic compared to that of large screen display technology, for just one example.
In the beginning, only Japanese companies built camcorders, i.e. had the full spectrum of technologies at hand for tiny display glasses. Meanwhile, mobile phone technology delivers the best starting point and it is available in many countries. This is just good luck, not the farsightedness of managements.

Only a very small number of books cover augmented reality, and even fewer really deliver a strategic view. Some of the most important issues, including visual orientation and mask displays for example, are hardly treated or not even at all.

So I literally had no choice but to write this book. It has two main objectives:
- showing as many applications as possible, because this already induces a lot of thought and inspirations about realization;
- demonstrating the feasibility, i.e. naming the technological approaches as well as calculating the results achievable. Delivering the recipes. Otherwise, the usual skeptics could just too easily do anything away as mere speculations.
It should be readable to the (interested) public, but it can't be entirely popular science, as it has to treat several issues in a way that professionals can profit. Yet this isn't such a contradiction, as professionals from different disciplines also usually don't know a lot of each other's field.

Many augmented reality projects now deal with screens all over the place, projected items, lots of additional installations. In the context considered here, this would be sort of installing phone booths everywhere, rather than using a mobile phone. It may be the right thing for certain applications, but generally shouldn't we try to reduce hardware and costs, not to multiply them? Recent studies say copper will get rare even in spite of recycling. So will other resources. It will therefore be wise anyway to push forward technologies that reduce large appliances, office space, wiring.

We will see that the actual IT hardware of the ultimate perceptional interface could fit in a sugar cube and take less than 10 milliwatts of power. Micro and nano technologies will be very important here. Some tasks will turn out easier than anticipated. We don't need tactile feedback all the time: nobody wants gloves. We don't need gestures: too complicated. Pressing virtual keys is so simple to implement, you wouldn't believe it, but even this won't be the end. We'll finally operate anything by just looking at it. It will be like magic.

Current near eye displays won't do. We need something that dynamically adapts to eye motion, incorrectly sitting glasses, focus and aperture effects, geometry changes. Also of course something that does not impair direct sight at all, has very high resolution, a very large field of view, and a mask display to make virtual objects non transparent.

It can be built, and for the basic functions somebody will already have to reinvent the wheel, as many of the problems have long been solved in some military projects. Yet the entire technology still doesn't have a really supporting structure, no powerful industrial basis. In the academic field, mainly the ISMAR conference developed into an important platform during the last years. Yet it is not done with research papers covering this or that mathematical or programming problem on still inadequate interface hardware. It is not done with fancy demonstrations or utterly expensive industrial installations on the same basis. The central topic of this book - in spite of its title - will have to be hardware, the new, better one, that is, and these issues still are missing most of the time.
What is necessary in this field is an entire, ambitious, multi threaded development program that involves, or better, is initiated by major industrial companies, has adequate funding and a sufficient time scale, and aims for nothing less than perfection.

The device that we are aiming at will require:

  • Eye tracker
  • Mask display
  • Position cameras
  • Dynamic image generator
  • Dynamic fit and focus adaptation
  • No straps, no screws, no belts, no clamps !

IIn current products, hardly anything of this is realized at all. It's not only technology that we have to deal with. First of all, applications have to be envisioned, fantasy has to be stimulated, the entire way of thinking has to be fully comprehended.
I remember when I had my first programming course back in the 70's. Nobody could afford a private computer back then, but I had heard a lot about it. I knew digital circuits, but I wondered about all those silly articles describing 'thinking' machines. This couldn't work, so what? After writing my first lines of code, I knew. This thing wasn't meant to think. Within minutes I figured out how to build one. It was all simple and clear.
The important issue was not how to do it, but first to know what it should do. Just as with all innovations:

The most difficult and important step is not to find the answers, but to ask the right questions. So it is here. We know there is something that could do amazing tings. We have to do a lot of thinking about what it could realistically do and what not. Once the goals are defined, we will be able to build it. Hence, this book is meant to ponder the technology as well as its applications. The entire scope. I went as far as to write many pages of fiction, in order to make this more colorful. Only by exercising our fantasy and delving into the scenario will we be able to find new applications, and there are many. Utterly important here it is to sort out the impossible and the unacceptable, or this would really be nothing but fiction.

Just "can't be done" is not an option. I've seen quite some examples where would-be experts said something wouldn't work, and a short time later it had been accomplished.
Applications dictate the construction. I try to define the necessities and the technologies that could do it. You'll see that it will finally be working. We will be able to build those ultra light plastic glasses that don't even look as if they got any super technology in them. We will have to face all those problems arising from the everyday use of virtual objects. And a lot more.

Displays and especially optics have been widely neglected. Maybe nobody realized the potential of a really good solution? So I'll address this extensively, including new ideas like holographic optics and image generators that may change things entirely.
The main chapters in this book I started to write in 2004 and 2005 (even 1993 in some detail, as nearly everything in the earlier papers is still current and true).
First is an introduction, outlining the technological and economical basics and a lot of applications. Then a piece of fiction, that in my opinion will say more than any of these brittle tech ideas and that is, as said, crucial for the comprehension of the entire thematic.
The applications list is also continued in part 4, with emphasis on virtual media. There I'll also address themes like 3D and surround cinema and holographic and light field encoding.
Part 3 goes into technical detail, exploring possible solutions. Even this is meant to be understandable to the general public as well as to experts from a wide variety of fields. This 3rd edition contains a lot more optics basics that I collected in an appendix to get a more readable main part.
This may also help anyone interested in actual construction work in figuring out what to do and how. Sensitize technologists and managers to new possibilities. Motivate people to become involved in this new development, or at least to keep it in mind with their everyday decisions.

This book is not primarily an overview of current research nor a summary. It is an investigation of technology and its possibilities, and in the course of it we will develop many entirely new ideas as well.
The objective is to fill all the gaps, at least in theory, that are parting us from the ultimate vision, in order to prove that it can be done, and to show how it will probably be accomplished.

As soon as a first - even a simple - implementation succeeds, drawing competition and accelerating the development, it will start an avalanche. An eye operated mobile phone integrated into a pair of glasses, for example. A killer application.

foreword            3rd edition

notes          content






Copyright © 2006-2011 Rolf R. Hainich; all materials on this website are copyrighted.
Disclaimer: All proprietary names and product names mentioned are trademarks or registered trademarks of their respective owners. We do not imply that any of the technologies or ideas described or mentioned herein are free of patent or other rights of ourselves or others. We do also not take any responsibility or guarantee for the correctness or legal status of any information in this book or this website or any documents or links mentioned herein and do not encourage or recommend any use of it. You may use the information presented herein at your own risk and responsibility only. To the best of our knowledge and belief no trademark or copyright infringement exists in these materials. In the fiction part of the book, the sketches, and anything printed in special typefaces, names, companies, cities, and countries are used fictitiously for the purpose of illustrating examples, and any resemblance to actual persons, living or dead, organizations, business establishments, events, or locales is entirely coincidental. If you have any questions or objections, please contact us immediately. "We" in all above terms comprises the publisher as well as the author. If you intend to use any of the ideas mentioned in the book or this website, please do your own research and patent research and contact the author.