Friday, August 13, 2004

Why no pen-centric UIs?

Microsoft has released an updated version of the TabletPC OS in the latest Windows XP SP2 (it is downloadable now if you want to try it out). The major change you can see for the TabletPC is the improvement of the TIP (Text Input Panel). This is better, but hardly a radical improvement to the UI.

At a recent TabletPC workshop put on by Microsoft and the University of Washington, many researchers complained about Microsoft refusing to release a pen-centric UI for the TabletPC. Instead they have relied on a UI that is simply incremental over the existing desktop UI. The pen is little more than a mouse pointer in the existing GUI. The TIP is just one such example of a hack that is necessary to let the pen input text into existing applications without the apps having to know anything about it.

The reason Microsoft has given for this decision to many of us privately in the past is that they actually did develop a purely pen-centric UI and it tested quite well. The problem was that people either said or exhibited problems when moving between that novel UI and their existing GUI apps, which they would still have to do in the future. This feels like a bad compromise or a great research problem.

How do you allow someone to learn an entirely new UI that takes advantage of the unique input characteristics of the TabletPC platform, yet still allow them to easily use their existing GUI metaphor when it is more appropriate or necessary? Can you make this new UI easy to learn and allow it to become more pen-centric as the user gets more expertise? Think about how Marking Menus start out as a simple pie menu (a fairly traditional GUI interaction -- except for the circular part) and then moves to a gestural interface as the user become more expert with it. You could push this research even further if you consider multimodal UIs that use pen, gesture, and speech rather than just pens.

What should Microsoft do? What is the interesting research here for those of us who want to push harder? What do you think?


Richard C. Davis said...

How can Microsoft promote a new pen-based user interface, yet still allow the use of the GUI metaphor? I think the question is moot until Microsoft provides a compelling reason to use its pen UI. What is the “killer app” for the TabletPC? The app has existed in all of our dreams for years, and yet it still has not been realized. It’s an electronic notebook that extends the capabilities of existing notebooks in four ways. First it allows notes to be edited in some simple ways, because we all occasionally run out of space where we need it. Second, it allows scraps of information from the outside world, such as pictures, scanned documents, or nearby people’s notes to be put on the page. Third, it allows notes to be archived and retrieved easily and reliably. And finally, it allows searching through the notes for recognized text.

I thought this killer app might have been a pipe dream until I saw the “Journal” application that shipped with the TabletPC. Its UI was something of a GUI/Pen UI bastard, and it didn’t work perfectly in either situation. The tool buttons, for example, were clumsy to use with the pen, and when using a mouse and keyboard, there wasn’t a good way to enter typed text. It had bigger problems, such as sluggish performance with large collections of notes, but none of these overshadowed its strengths. Journal provides the best DRAWING experience of any ink application I have ever used, and its search capabilities are surprisingly good. Since the other problems looked easy to solve, I had hopes that Journal would evolve into the killer app I had been waiting for.

But when will it come? Microsoft seems to be investing all of its energy in OneNote, and this is becoming more tailored to keyboard/mouse input than pen input. I have not heard of any new versions planned for Journal. Why on earth not? Perhaps Microsoft believes that people are content to find a level surface and pull out a laptop computer whenever they want to write something down. I think they are very mistaken, but perhaps they will not be convinced until another company comes along and produces the application that they should have. The beauty of it all is that this application could run under Windows, regardless of what other applications are installed.

Once this electronic notebook begins to gain acceptance, then it is time to talk about how to transition between PenUIs and GUIs. I think that there’s a lot of interesting work to be done in beautification of sketched documents and incorporation of beautified sketches into “Office” documents. But that work can’t really begin until people start taking electronic notes on a large scale.

In any case, I think Microsoft should get off its butt and finish what it started.

Yang Li said...

What is the right territory for pen-centric UIs?
Why no pen-centric UIs? So what is the motivation to have Pen-centric UIs? I think it is to enhance the activities in our daily life that have not or have not been well supported by GUIs, rather than just to make existing GUIs pen-compliant. GUIs have dominated HCI for decades which successfully support quite many tasks, e.g., its killer app of Word Processing. This leads to another question - what are the killer applications for pen-based UI? In anther word, what are pen-centric UIs good at and are also meaningful to our lives? I think the tools for note-taking and design (both informal and formal) can be the killer apps for pen-centric UIs.

Evolution: Embed pen-centric UIs into GUIs or the opposite?
We are always able to find the trace of the future or the previous UIs in the current UIs. Back to the time of Command-Line based UIs, GUIs used to be provided as some apps that can be started from a command console. And now, in the time of GUIs, the command console, as a legacy of Command-Line UIs, is enabled as an app. Maybe pen-centric UIs have to go through this process. Now we are only able to see some pen interactions and apps in GUIs. And in the future, one naive idea is that GUIs probably would be encapsulated into “a piece of paper” in pen-centric UIs (as a command console is encapsulated a window in GUIs). So how to efficiently present the semantic equivalence of these two kinds UIs to users during this alteration?

Revolution: A completely new interaction paradigm?
A fundamental research problem that has not been solved is what the paradigm or the model of pen-centric UIs is. We all agree Pen-Paper should be the metaphor. However, it is far ambiguous away from WIMP for GUIs. Multi-modal UIs with pen and voice could be an important part in this paradigm. A nice analog here is mouse and keyboard for GUIs.

I am interested in both pen-based interaction techniques and paradigms. In spite of these unsolved research issues, I think it is normal that big companies always intend to take cautious steps for new things. So they always leave a great space for researchers. :)

James A. Landay said...

Wow, I really must have been spacing when I read that... I worked at Go! There are many interesting things in the Go model that we still don't see in most mainstream OSs (e.g., embedded applications -- it was very cool). They were simply too early (expensive, slow hardware). Run the GO OS on today's tablet PC and it would rock and be way better than Windows XP.