Thursday, March 19, 2009

Time to claim success on electronic sketching of UIs?

In Las Vegas this week (March 18th), Microsoft demoed Microsoft Expression Blend 3 with SketchFlow. SketchFlow is a new tool that appears to be a commercial strength version of our previous research tools in this space:
  • SILK -- for sketching, storyboarding, and prototyping GUIs
  • Denim -- for sketching, storyboarding, and prototyping Web sites (w/ lots more in terms of functionality and testing than was possible in SILK)
  • K-Sketch - informal prototyping of animations

Much of the functionality of these three research tools has been embedded in a full strength web development system (Expression Blend 3). The image above shows the UI, but doesn't really capture it (you need to watch the video below).

Watch this video of the demo (go to about 2/3 of the way in If you go too far in and they are already sketching, just back up a bit. The video is quite long.):
(click on Day 1 Keynote)

More info on it here:

The talks use the term "informal" all over the place. Clearly our "informal user interfaces" work has had impact on industry. I know this often takes many years (we first showed SILK in 1995!). But I want to thank Brad Myers (my PhD advisor) and all of the students, staff, and postdocs that have worked on this project (especially Jimmy, Jason, Mark, Richard, and Yang). You should all be quite happy to see this come to fruition. It sometimes takes many years to see impact and many other researchers never see it.

What do you guys think? Are we done? What don't they do?

PS Watch the Buxton intro at the beginning to see a lot of the motivation. I wonder if they will claim to never have seen our work and were instead motivated by Buxton's recent book?


toomim said...

Rad to see tool support for early-stage design! Storyboards+actions instead of code!

I think it's cool how they took these ideas, but repurposed them for a traditional GUI builder. Big innovation without a total paradigm shift!

There's support for informal representations... but you specify all widgets with the mouse and keyboard. No pens, no electronic ink! No recognition, no AI errors! You start with a structured mockup and click a theme to make it LOOK informal. Then you go hifi by changing the theme.

No separate ink-drawing tool. No fully-pen interface. No ink recognition. No data export to a gui builder. No pie menus. No paradigm shift! Just storyboards and informal early stage design! Cool!

Love the storyboard widget... beautiful design details.

Keith Butler said...

It wouldn't make sense for Expression Blend to cite Buxton's book to justify informal prototyping. The use of the term "sketching" is really different than that in Bill's book. In the early chapters and in my conversations with him it was clear that sketches are not supposed to be immature versions of the design at all. Instead, designers should use sketching as a way to gain understanding and insight of the problem their design is supposed to solve- It's much more like Jacobson's use of domain modeling, than like lo-fi prototyping.

James A. Landay said...

Others have previously done this with mouse-specified widgets (e.g., Jonathan Meyer's EtchaPad -- see -- and also Denim 2.0 had widgets that you could "stamp" out rather than draw). SILK also let you go "hi-fi" from the underlying sketch representation.

I think the pen is still the right way to go. The recognition has gotten WAY better since SILK and note that Denim requires almost no recognition. The Tablet PC is the part that hasn't taken off, but I think we are very close now.

We always knew we could build this with standard widgets in a GUI (and in fact many participants requested it), but that wasn't interesting from a research perspective. If we were making product, I'm sure we would have gone in that direction until the Tablet PC market matured.

James A. Landay said...

Buxton opened the entire session presenting the new version of Experssion. He used his slides from his standard book talk and the product team referred back to him as they talked about the features. I agree that Bill uses a different definition of "sketching", but in my opinion he is simply making a broader use of the term that still encompasses sketching on paper or screen. I think he advocates using sketching as a way of exploring the design space. That is what "low-fi" prototyping also affords, so there really isn't a disconnect here IMHO.

I'm not complaining about this. I think it is a great triumph for the ideas, wherever you might think they came from. I just wish Bill would recognize the work (there is not a single reference to it in his entire book and in his talk he shows his "friend's" sketch-based storyboarding tool -- which obviously came much later than SILK or Denim).

James A. Landay said...


Funny thing you brought up AI & sketch recognition, as the entire idea of Denim was to reduce the amount of that needed in the tool and I think it does a fairly good job of it.