Tag Archives: hololens

Hands On with Remote Assist, a compelling HoloLens application, at Microsoft Build

May 9, 2018 Tim Anderson

One of the announcements at Microsoft’s Build event in Seattle was Remote Assist, an application for the HoloLens augmented reality headset. Today I got an opportunity to try this out, and was impressed.

The company has set up a hotel room as a factory with some big machines that whir and hum and a scary power supply cabinet. I took the part of the engineer when a fault stops the machines and the factory grinds to a halt. Unfortunately I knew nothing about how to fix it.

The solution was to put on the HoloLens headset and make a call to someone better informed. The concept is pretty simple. The remote technician sees what you see, so can easily inspect the screens as well as the physical state of the machinery. Along with the video call, HoloLens and Remote Assist lets the remote technician add annotations to the real world, pinpointing the exact button, cable or belt that needs attention. They can also stick images or documents next to the machinery to show you what to do.

The fake factory floor was a little contrived, but nevertheless effective in showing how this approach can make it easier to offer support. The alternative in this kind of case is a site visit, which with specialised equipment might well involve a flight and several days delay. The cost is not only the travel expenses, but also the possible extended downtime while machinery is idle.

If you have ever tried to guide someone at the other end of a telephone through fixing a physical problem, you will know how tricky it can be. Send me a photo. No, that’s not quite the right spot. Send me another photo. What happens if you change that setting? No, not that setting! Sorry, now you have two things to fix …

Remote Assist makes this much easier and I can see the potential. HoloLens may seem expensive for a casual purchase, but could soon pay for itself in this context.

There is no date yet for availability, but the technology will enter limited preview soon.

HoloLens: a developer hands-on

May 2, 2015 onlyconnect 1 Comment

I attended the “Holographic Academy” during Microsoft’s Build conference in San Francisco. It was aimed at developers, and we got a hands-on experience of coding a simple HoloLens app and viewing the results. We were forbidden from taking pictures so you will have to make do with my words; this also means I do not have to show myself wearing a bulky headset and staring at things you cannot see.

First, a word about HoloLens itself. The gadget is a headset that augments the real world with a 3D “projected” image. It is not really a hologram, otherwise everyone would see it, but it is a virtual hologram created by combining what you see with digital images.

The effect is uncanny, since the image you see appears to stay in one place. You can walk around it, seeing it from different angles, close up or far away, just as you could with a real image.

That said, there were a couple of issues with the experience. One is that if you went too close to a projected image, it disappeared. From memory, the minimum distance was about 18 inches. Second, the viewport where you see the augmented reality was fairly small and you could easily see around it. This is detrimental to the illusion, and sometimes made it a struggle to see as much of your hologram as you might want.

I asked about both issues and got the same response, essentially “no comment’’. This is prototype hardware, so anything could change. However, according to another journalist who attended a hands-on demo in January, the viewport has gotten smaller, suggesting that Microsoft is compromising in its effort to make the technology into a commercially viable product.

Another odd thing about the demo was that after every step, we were encouraged to whoop and cheer. There was a Microsoft “mentor” for every pair of journalists, and it seemed to me that the mentors were doing most of the whooping and cheering. It is obvious that this is a big investment for the company and I am guessing that this kind of forced enthusiasm is an effort to ensure a positive iimpression.

Lest you think I am too sceptical, let me add that the technology is genuinely amazing, with obvious potential both for gaming and business use.

The developer story

The development process involves Unity, Visual Studio, and of course the HoloLens device itself. The workflow is like this. You create an interactive 3D scene in Unity and build it, whereupon it becomes a Visual Studio project. You open the project in Visual Studio, and deploy it to HoloLens (connected over USB), just as you would to a smartphone. Once deployed, you disconnect the HoloLens and wear it in order to experience the scene you have created. Unity supports scripting in C#, running on Mono, which makes the development platform easy and familiar for Windows developers.

Our first “Holo World” project displayed a hologram at a fixed position determined by where you are when the app first runs. Next, we added the ability to move the hologram, selecting it with a wagging finger gesture, shifting our gaze to some other spot, and placing it with another wagging finger gesture. Note that for this to work, HoloLens needs to map the real world, and we tried turning on wire framing so you could see the triangles which show where HoloLens is detecting objects.

We also added a selection cursor, an image that looks like a red bagel (you can design your own cursor and import it into Unity). Other embellishments were the ability to select a sphere and make it fall to the floor and roll around, voice control to drop a sphere and then reset it back to the starting point, and then “spatial audio” that appears to emit from the hologram.

All of this was accomplished with a few lines of C# imported as scripts into Unity. The development was all guided so we did not have to think for ourselves, though I did add a custom voice command so I could say “abracadabra” instead of “reset scene”; this worked perfectly first time.

For the last experiment, we added a virtual underworld. When the sphere dropped, it exploded making a virtual pit in the floor, through which you could see a virtual world with red birds flapping around. It was also possible to enter this world, by positioning the hologram above your head and dropping a sphere from there.

HoloLens has three core inputs: gaze (where you are looking), gesture (like the finger wag) and voice. Of these, gaze and voice worked really well in our hands on, but gesture was more difficult and sometimes took several tries to get right.

At the end of the session, I had no doubt about the value of the technology. The development process looks easily accessible to developers who have the right 3D design skills, and Unity seems ideally suited for the project.

The main doubts are about how close HoloLens is to being a viable commercial product, at least in the mass market. The headset is bulky, the viewport too small, and there were some other little issues like lag between the HoloLens detection of physical objects and their actual position, if they were moving, as with a person walking around.

Watch this space though; it is going to be most interesting.

microsoft, software development, windows

Windows 10 and HoloLens: quick thoughts and questions following the January reveal

January 21, 2015 onlyconnect 3 Comments

Microsoft is revealing its Windows 10 plans in stages, presumably in part to build up expectation and get feedback, and in part because some pieces are ready to show before others.

Today in Redmond Microsoft shared a number of new features. In quick summary:

Windows 10 will be a free upgrade for all Windows 7 and 8.x users, at least for the first year.

Comment: this is necessary since the refusal of Microsoft’s user base to upgrade from Windows 7 is a strategic roadblock. For example, Windows 7 users cannot use Store apps, reducing the market for those apps. It is more important to persuade users to upgrade than to get upgrade revenue. Windows 10, of course, will have to be compelling as well as free for this initiative to work, as well as providing a smooth upgrade process (never a trivial task).

Windows to evolve to become a service Executive VP Terry Myerson says this in this post:

Once a Windows device is upgraded to Windows 10, we will continue to keep it current for the supported lifetime of the device – at no additional charge. With Windows 10, the experience will evolve and get even better over time. We’ll deliver new features when they’re ready, not waiting for the next major release. We think of Windows as a Service – in fact, one could reasonably think of Windows in the next couple of years as one of the largest Internet services on the planet.

And just like any Internet service, the idea of asking “What version are you on?” will cease to make sense – which is great news for our Windows developers.

Comment: What does this mean exactly, beyond what we already have via Windows Update? What does Myerson mean by “the supported lifetime of the device”? What are the implications for the typical three-year Windows release cycle? I hope to discover more detail soon, though when I enquired whether there will be, for example, a “Windows 11” I was told, “We aren’t commenting beyond what’s stated in post that you reference.”

Project Spartan (a code name) is a new browser developed as a universal app – this means an app built for the Windows Runtime (“Metro”) environment, though in Windows 10 these also run in a window on the desktop, blurring the sharp distinction you see in Windows 8. Project Spartan features, according to Microsoft’s Joe Belfiore, a new rendering engine along with features includes the ability to annotate web pages with keyboard or touch/stylus, and the ability to save pages for reading offline. There will also be “enterprise mode compatibility for existing web apps”, which means that old IE will live on.

Comment: Creating a new browser is a bold step though it may be as much for marketing reasons as anything else, since IE has a tarnished reputation. The advantages of the new rendering engine, and the way compatibility will be handled, are not yet clear. Another point of interest is compatibility issues caused not only by the new engine, but also by running in sandboxed universal app environment. Looking forward to more detail on this.

Windows 10 across PC, tablet and mobile: the OS will have the same name on all three, universal apps (like a new mobile Office) will run on all three, and there are new efforts to synchronize content. For example, notifications will sync across phone and PC/Tablet.

Comment: Sounds good, but there are a few downsides. One is that Windows Phone is tied to the same release cycle as full Windows, which is rather slow. Currently Windows Phone is falling back as it waits for Windows 10 in respect of both operating system upgrade and also the universal app version of Office – which is already available for iOS and Android. CEO Satya Nadella said today that there will be new “flagship” Windows phone devices, which is good news for what is currently a neglected platform, but it will be hard for the platform to thrive if it is constantly waiting for the next big Windows update. Update: if “Windows as a service” means no more monolithic upgrades but constant incremental improvement, perhaps this will not be the case. Watch this space.

Cortana coming to Windows PC and tablet: we saw Microsoft’s digital assistant, powered by Bing search, demonstrated on full Windows.

Comment: Cortana is impressive and fun, but I am not sure how much the feature enhances the platform. On the phone I do not use it much; the problem is that speaking to your phone “what meetings to I have today” and getting a spoken response is a great demo, but in practice it is easier to glance at the calendar, especially as voice control only works in quiet scenarios. The other aspect of Cortana is the personalisation it brings to things like web search or reminders; more data about our preferences and activities can bring some magic. This is Google Now territory, and while Microsoft’s approach to privacy may be preferable, Google will be hard to match in respect of the amount of data it can draw upon.

DirectX 12: Microsoft showed a demo of its latest DirectX graphics API, claiming up to 50% better performance and up to 50% less power consumption.

Comment: this is solid good news. If games run best on Windows 10 a significant enthusiast community will want to upgrade right away. Further, DirectX is not just for games.

Xbox One integration: Microsoft showed how Xbox Live team or competitive games can work across Xbox One and PC, and how games can be streamed from XboxOne so that the console becomes a kind of games server for your Windows 10 tablets and PCs. Xbox One will also run universal apps.

Comment: Better integration between Windows devices and Xbox is long overdue and can help to promote both. Xbox One though has a bit of a Windows 7 problem of its own, with Xbox 360 remaining popular simply because of the huge numbers of games that have not been ported. If only Microsoft could introduce backwards compatibility …

Surface Hub: this is a giant 84”, 4K display wall-hanging PC which you can use as an interactive whiteboard for meetings and so on. It seems to be the next innovation from the Perceptive Pixel folk who also developed the table-top Surface device.

Comment: Looks cool, but it will be expensive. May help to encourage businesses to keep faith with the Windows client.

Microsoft HoloLens: this was the big reveal, a secret project that, we were told, has been developed in the basement of the Microsoft Visitor Center on its Redmond campus.

HoloLens is a headset which enables 3D augmented reality: projected images are seen like holographic images in the space around you, and you can interact by gesture detected by cameras and motion sensors in the headset. Look carefully at the following image:

In this example, the demonstrator is assembling a quad copter using a palette of 3D components in Holo Studio, an application which uses the technology. However, note that you only see the quad copter through the HoloLens headset, the image from which in this case is merged with a view of the demonstrator herself using a custom camera:

If you had been in the room, you would see the quad copter only on the screen, not in the room itself. Therefore I suspect this is more accurately described as augmented reality than holography, though the scene does look holographic if you are wearing the headset.

In a final flourish, Microsoft a 3D printed version of the quad copter which duly flew up and down; I am sure the motor and so on was NOT 3D printed, but it made a lovely demo.

Apparently NASA loves the technology and will be using it with Mars Rover in July in a project called OnSight – read the NASA release.

Bringing it down to earth, Microsoft also stated that all universal apps will have access to the HoloLens APIs.

Comment: This looks amazing and must have potential for all sorts of scenarios: architects, planners, marketing, games and more. The tough question I suppose is how much it has to do with Windows 10 as experienced by most users.

In closing

Microsoft surprised us today and deserves kudos for that. Nobody can accuse the company of lack of innovation; then again, Windows 8 and the original Surface were innovative too, and proved to be a disaster. I do not think Windows 10 will be a disaster; we have already seen in the preview how it is an easier transition for Windows 7 users.

A key thing to note from a developer and technical perspective is that universal apps are right at the centre of the Windows 10 story. That is a good thing in many respects, since we get Store deployment, sandbox security, and a degree of compatibility across phone, PC, tablet and Xbox One. But is the Store app / Universal app platform mature enough to deliver a good experience for both developers and users, bearing in mind that in Windows 8.x it is really not good enough?

Look to Microsoft Build at the end of April, which Myerson said is the culmination of the Windows 10 reveal, to answer that question.

Tim Anderson's IT Writing