Pencils and Pixels:

An Exploration of Physical and Digital Sketch

Techniques for Tangible Interaction Design

Solomon Bisker
Methodologies of Visualization (Design Drawing I)
Fall 2009, Carnegie Mellon University
All of the works in this project were created using a hybrid of physical pencil, pen and marker sketching techniques learned in this class, and digi-
tal design tool techniques learned during my career as an Interaction Designer. The objective of my project was to explore how blending physical
and digital sketching techniques could boost my repertoire of sketching skills as I pursue my Masters degree in Tangible Interaction Design. My
hypothesis in starting this project was that sketching tangible interactions poses unique challenges that neither hand-sketching nor digital mock-
ups alone are currently suited to tackle. This felt like it would be particularly true in interactions that have environmental context (“mobile” or built
environment scenarios) and interactions that involve posing and positioning of the the human body (touch screens, hand-held objects, and the
like.) To that end, I came up with simple scenarios of tangible interactions that I felt might benefit strongly from these hybrid techniques. Then, for
each scenario, I attempted to use both hand-sketching and digital manipulation of sketches to play to the strengths and weaknesses of both medi-
ums in developing final concept sketches for each scenario.

Before we dive into these interaction scenarios, there are few general tips worth noting that are useful to anyone creating or digitizing hand-drawn
sketches for later digital manipulation. By reading these general tips upfront, the reader will hopefully be able to follow my individual techniques at
home without making the same mistakes I did.

If possible, use a high-quality color scanner
for your work. The scanned image on the left
was from a professional color scanner. The
colors in the image are much richer (and truer
to the original sketch) than those captured by
the “consumer” color scanner in the image on
the right.
If you intend to scan your drawings
in black and white to eliminate the
paper color, make sure no marker
bleeds through from the other side
of the page. In this case, a light
marker was translated to pitch
black, rendering the lines under
that marker unusable.
Drawings of actual interfaces will not stretch or skew cleanly to match the ac-
tual dimensions of your screen. This means that to get an interface drawing to
drop into a photo clearly, you will want to measure out the proper sspect ratio
of the device and translate that directly to the paper before sketching (perhaps
practicing to try to achieve dimensions close to that ratio from memory, if you
are looking to digitize quick skteches from brainstorming sessions.)
 I began by capturing the signage form off of a bathroom sign, Here, I was working through how shoulder height
and then applying that form to various people I encountered. and angle influences attitude.

Digital designs often play to the

strengths of digital design tools. One
such strength is the ability to make
pixel-perfect lines and shapes, which
can be used to emphasize simplicity
of form. This is seen in public sig-
nage, where “bathroom man” figures
are recognizable and perfect in form. I experimented with sitting poses of students, but decided that having to then stylize
And yet, the bathroom man invari- the student’s chairs would detract from their poses.
ably feels almost *too* perfect. This
reflects the inability of digital tools to
let one sketch the subtilty of a person
in a pose.
Here, I’ve used sketching to capture
a “pose” of a “bathroom man” doing
a real action (in this case, one of my
professors giving a speech). Then,
digital tools add precision without los-
ing the humanity of the pose.
As my professors stood before my classes, I realized that the signage form left much room for capturing a pose’s
spirit in arm movement and head position.
I recreated the angles and positioning of the body parts in Illustrator as a series of rectangles with rounded ends,
eventually adding a skewed rectangle as a torso that would capture shoulder position.

Once the shape began to take form, I began manually correcting the proportions on my sketch by adjusting shape sizes. It was crucial working with vectors in Illustra-
tor (as opposed to bitmaps in Photoshop) to maintain the smoothness in the shapes as these scaling tweaks were made.

I found myself getting more and more nitpicky with the form as the shape entered a“uncanny valley” of idealized form.
I ultimately wound up referencing a real bathroom sign to bring it to an idealized state.

I attempted to recreate the purity of the “bathroom man” by overlaying basic

circles, lines and other shapes on top of my raw sketch. In this way, I was able
to recapture my original angles, both of the head tilt and of the arm gesticula-
tion. Once I brough the shapes into the right poses to capture the sketch,
the shapes had to be resized in order to better capture the idealized propor-
tions of a signage figure. In this sense, while my sketch might have captured
the *pose* more faithfully, I then needed technology to help me tighten those
overly thick thighs, pull up that torso and otherwise make my person more
“idealized.” The end result is “the ideal man,” albeit in an oddly human pose.
The biggest failing I saw was one of perspective - most sign people are drawn
head-on, and it was difficult for me to capture the pose while maintaining that
head-on feel. All digital work done in Adobe Illustrator.
 I started by trying to capture the shape of the hand,”clenching” the hand to emulate
holding a leash and working on thumb grip.

I then took my improved hand drawings and refined then in the context of projection, explor-
ing how a person would “project” their pet.

Interaction design increasingly needs to incorpate the context of physical space into
the design of digital products. It is often not the interface at all that has to be envi-
sioned, but rather the motions the body goes through - and how the digital aspects of
product relates to that physical motion - that needs to be sketched and storyboarded.
A colleague’s research on the impact of pico-projectors on the virtual pet movement
inspired me to sketch out a vision for walking a virtual dog. In this sketch, my physical
hand drawings had to capture the tactile feeling of “leading” a dog, while the digital
alterations to the sketch needed to approximate the virtuality and hyper-realism of a
“projected pet” in a realistic manner.
Finally, I explored how to frame the context itself - realizing that people would want to project the pet at a
certain size, and in proper relation to their walking.
After finding a dog in an appropriate pose on Flickr, the Photoshop “Magic Wand” tool was used to remove much of the
grass from the scene (with hand-erasing removing the rest.)

I discovered that the picture of the dog was captured at too low of a resolution to fit nicely with the resolution of the drawing. Since the drawing was a stylized image, I
lowered the resolution of the drawing so the dog would appear “realistic” in the context of the scene.

A white blur was added to the scene where the

projector would eminate light out of the user’s grip.

I decided to focus on projecting

a “real” pet, using Flickr to find
royalty-free pictures of dogs in
approximate poses appropriate
for the story. I then used Adobe
The final edited dog picture was made slightly transparent, to re- Photoshop to digitally edit the
flect potential weakness in the projector’s strength. dog out of its original photo and
to scale it appropriately to fit
into the user’s projection space
both size-wise and perspective-
wise. Finally, tweaks was added
to emulate the pico-projector’s
actual operation.
 I started by exploring the human form, trying to faithfully sketch a person facing away from
This envisionment was done with similar motivation to the pi- me, with his arm raised as if to touch the touchscreen.
co-projector sketches, except that I now wanted to see how
my existing 2D interaction design tools (OmniGraffle, Visio,
etc) could be used in sketches of a real world interaction.
Rather than a “user’s eye view”, a straight on third person
perspective was used in my sketch such that the 2D inter-
face wouldn’t need to be skewed to match the scene. My
hope was that the sketch would let me see whether a human
body’s proportions would be compatible with a touchscreen
interface for finding rideshares I had created for city use.

This process took some time - it turns out I had trouble imagining just how the back of ones
body looks, in terms of shape, angle and proportion.

Eventually, my sketches converged on an acceptable body form, and I explored the use of
marker to add clothing which would emphasize that form.
The part of the body figured that intersected with the scene was copied and pasted onto a new Photo-
shop file in order to render the background transparent.

The 2D screen mockup was then dropped on where the person had previously been. The screen was
stretched slightly to fit the drwaing, as the aspect ratio was inconsistent between my UI and the drawing.
Again, Adobe Photoshop was the appropriate tool
- but this time, it was needed to digitaly modify the
sketch itself. Specifically, the portion of the per-
son intersecting the screen was digitally removed,
with areas of the sketch without body parts being
rendered transparent. This allowed me to paste my
Finally, the top of the person was reattached to his body on top of the screen, screenshot to scale directly into the scene.
as if to reconstruct the designed scene by layers.

The sketch showed me that my interface was sufficient for an urban touch
screen context, but arguably did not take enough advantage of the screenspace
I had been provided. My figure’s arm seems to have a higher precision of point-
ing than I would have expected at this scale and distance from the screen. As a
result, I suspect the entire interface could have either packed in more informa-
tion or been placed into a smaller form factor without sacrificing usability.
Most of my issues in sketching this interface came from sketching the outline of the screen.
An incorrect shape would look wrong immediately in the context of a photographed photo.

Once the shape began to take a proper form, I began to play with drawing the interface itself in a “sketchy” manner to focus on issues of layout and
size. I did this by blurring out system elements and smaller interface text.

I tried to improve the boundaries of the display screen by focusing on keeping the lines parallel, but I found that made
me pay less attention to giving the screen a proper width-height ratio.

My last envisionment tried to take an opposite approach to my typical interface sketching technique - drop-
ping a hand-drawn interface into a digital photo or other context (as opposed to using Visio to mock it up and
dropping that into the photo.) I thought perhaps this might be useful to test out interfaces very, very early in a
sketching process, where it is unclear whether or not it’s worth the time to even mock up an idea in Visio. In
this example, I tried to create a fake mobile Treemap news interface for placing onto a photo of my holding an
iPhone screen.
Photoshop’s Magic Lasso tool was used to select and copy the interface sketch so it could be pasted into the photograph. Similar to the dog photo earlier,
our photo is too high resolution for our sketch to fit the scene - so I scaled down the photo resolution to match the drawing.

Then, the image was cleaned up with the Eraser tool before Free Transform was used to rotate the sketch to match the angle of the
screen in the user’s palm. An additional Free Transform on the photo itself was then used to scale it to the size of the drawing.

Above: An attempt at shrinking the photo to match the drawn

The first attempt (the upper of the two final results shown at right) looked too
squished, so I tried again, this time using the original photo and applying a second
Free Transform to the drawing to match the device screen’s actual aspect ratio.
screen’s aspect ratio.
Below: An attempt opposite of that above - trying to stretch the
interface drawing itself to match the photograph. In this case,
stretching the interface works better than shrinking the picture,
but neither looks ideal.

I created this rendering in Adobe Photoshop of my interface projected onto a

photo to see how the interface might feel in the palm of my hand. This seems to
quickly let me see how interface elements might relate to, say, the shape of my
hand and my thumb. It seemed clear from this composite sketch that buttons on
a Treemap interface would be clickable on this scale, and this might have encour-
aged me to explore the concept for a design further in Visio. However, I realized
quickly that for hand-drawn interfaces, it becomes that much more important to
draw the interface aspect ratio correctly. Visio-created interfaces - especially
those done in vector - might stretch to fit your drawings, but your drawn interfaces
will not stretch as cleanly to match your photographs.