Programming Languages for

Compressing Graphics
Morgan McGuire
Shriram Krishnamurthi
John F. Hughes

Brown University
{ morgan | sk | jfh}@cs.brown.edu
Encoding Images as Programs
Describing Images (1)
setPixel(0,0,BLUE);
setPixel(1,0,BLUE);
setPixel(2,0,BLUE);

setPixel(50,0,WHITE);

setPixel(100,0,RED);
setPixel(101,0,RED);

The Cost of Bandwidth
Major cost of doing business on the web
Yahoo!: 65,000,000 pages/day
Valve: 1M /software patch
Image compression is a one-shot activity
Low vs. high bandwidth users
Multiresolution
Describing Images (2)
repeat 50 times setNextPixel(BLUE);
repeat 50 times setNextPixel(WHITE);
repeat 50 times setNextPixel(RED);
nextRow();
repeat 50 times setNextPixel(BLUE);

American Flag
Doesnt compress as well as the French flag in
the repeat n times language.
Adding More Primitives
drawRectangle(BLUE, );
drawRectangle(RED, );
drawRectangle(WHITE, );
drawStar(WHITE, );

Is this enough?
Describing Complex Images
Describing Complex Images
JPEG: language consists
of frequency domain
instructions
GIF: language consists
of setPixel and
dictionary lookup
Preferred format
depends on the image
Observations
Images are programs
Even within one language, many possible
descriptions produce similar images
Lossy compression
Description length depends on language
and image complexity
Best compression when the language
matches the image
The Obvious Compression Scheme
Compress the image in several formats
TGA, GIF, JPG, SVG, SWF
Choose the best
Add a byte to the front of the file specifying
the compression language
Problems with the Obvious Scheme
None of the formats may be particularly good
for our image
Even JPEG tops out around 50:1, has serious
artifacts
Lacking ideal features like:
Multiresolution
Time/Space tradeoff
Introduction of new formats requires new
browser plug-ins
Describing Complex Images
How can we do better than
the obvious approach for
images like this?
Using an Expressive Language
(compose
(vertical-gradient BLUE WHITE)
(polygon DARK-BLUE )
(polygon BLACK )
(* (polygon )
(blur tree-texture)
YELLOW))
Using an Expressive Language
What if we design a really expressive language for
representing images?
Because the data is a program the decompressor is
part of the data
Each image gets its own custom format
High compression
We have control over multiresolution, perceptual
artifacts
Package for the web as a plug-in
We only need to upgrade the plug-in when the language itself
changes
Whats wrong with this plan?
Example
setPixel(0,0,WHITE);
setPixel(1,0,WHITE);
setPixel(2,0,WHITE);

(blur (+
(rotate (banana))
(distort (* RED
(circle))) )
?
2 Mb 2 kb
setPixel Language
Expressive Language
The Encoding Problem
?
Pierre cant code!
The Encoding Problem
setPixel(0,0,WHITE);
setPixel(1,0,WHITE);
setPixel(2,0,WHITE);

(blur (+
(rotate (banana))
(distort (* RED
(circle))) )
2 Mb 2 kb
Photoshop menu
EMACS
Where is the Save as Code Menu?
It is easy to convert from an image to
programs in the GIF/JPG language
More expressive language = harder
conversion
How much harder?
Much Harder!
Converting the image into a program
that produces it is a search problem
The search space is the space of all
possible programs
This is an infinitely large space
Tempering Expressiveness
Good compression languages are
ones where:
Expressive power is large
Searching is easy

How do we make searching easy?
Steerable Search Techniques
Genetic Algorithms
Inject domain information through fitness function
Metropolis Search
Inject domain information through transition
probabilities
Simulated Annealing
Inject domain information through gradient
estimation
Perceptual Fitness Function
Tweaking this is the domain-experts job
Perfect fitness function not necessary (or
possible!)
|
|
.
|
+ +
+

\
|
=


x x x x
x x x x
g i g b i b
g i g e i e
n
g i m
15 ) ( ) ( 35
10 ) ( ) ( 40
100
1
) , (
e (i ) = edge filter, b (i ) = convolve with Gaussian,
|i
x
| = color magnitude
Designing the Language
Desirable language properties for
compression
Expresses many images compactly
There are many programs for which
another, shorter program exists that
produces visually similar output
Desirable search properties
Mutations safety
All programs terminate
The Evolver Language
Scalar := real between 0 and 1
Vector := Scalar x Scalar x Scalar
Matrix := Vector*
Value := Matrix | Scalar | Vector
Operator := Add | Collage | Blur | Noise |
Call := Operator x Expr*
Expr := Call | Scalar | Vector
The Evolver Language
Automatic coercion between Matrix,
Vector, Scalar
Every operator has the same domain
and range
Primitives include stock images and
textures
No looping constructs
No functions!
: Not the Ultimate Compressor!
: Not the Ultimate Compressor!
Copying code is sometimes good
Multiple instances of a pattern in an image
often differ slightly
Hard to evolve both definition and
multiple applications
Added Benefits of Search
Artist can save the image immediately
Webmaster uses Evolver toolkit to search for
an equivalent program
It is easy to find a large program quickly
Webmaster lets Evolver continue to run
Upload new, smaller encodings as they are
found
More time = less space = less cost
Multiple constraints: size, time, artifacts,
decompression time
Results
Proof of Concept
TGA 1:1 (128x128)
Proof of Concept
JPG 24:1 (64x64)
Proof of Concept
Evolver 50:1, infinite detail resolution
Proof of Concept
TGA Evolver JPG
Lake Matheson
Original
Lake Matheson
Original Compressed 50:1
Gradient
Original Compressed
Aspens
Original
Aspens
Original Compressed 54:1
Maples
Original
Maples Code
Collage(HueShift(HueShift(Min(RockImage(), Collage(Rotate(0.22693111, {0.4339271, -0.060890462, -0.14983156}),
{0.9689341, -0.31166452, 1.0}, EnvironmentMap(Interpolate(Derivative({0.5260445, -0.9943271, -0.83629435}),
FishImage(), 0.22693111), Collage(VSplit(VGradient(), Interpolate(Min(RockImage(), Collage(0.22693111, {0.90638816, -
0.3161332, 1.0}, EnvironmentMap({0.3538252, -0.11179591, 0.76402247}, {0.3538252, -0.11179591, 0.76402247}))), -
0.75621074, Derivative(HueShift(LowColorNoise(), {-0.60136193, -0.9961748, 0.956824})))), {-0.6025814, -0.5151359, -
0.2444776}, MiniBlur(Blur({0.48381335, 0.37744927, 0.18049468})))))), LeafImage()), LeafImage()),
Rotate(Rotate(Interpolate(Max(Rotate(LeafImage(), Cosine({0.13357106, -0.48899084, 0.46273336})), LeafImage()),
{0.26036343, -0.2474052, 0.3318561}, Add(Rotate(Interpolate(Max(LowNoise(), LeafImage()), {0.26036343, -0.2474052,
0.3318561}, Add(Rotate(Interpolate({0.26036343, -0.2474052, 0.3318561}, {0.26036343, -0.2474052, 0.3318561},
Max(Rotate(Blur(Noise()), FrequencyStars()), BitAnd(VSplit(LeafImage(), {-0.3782095, 0.06973941, 0.7708523}),
Collage(Blur({0.19214037, 0.7060751, 0.9632803}), {-0.94875985, 0.9535051, 0.9628181}, {-0.94875985, 0.9535051,
0.9628181})))), FrequencyStars()), Zoom({0.84155905, 0.44450688, -0.6368634}, Interpolate(0.97325927, {0.43938103,
0.8003519, -0.8865588}, FrequencyStars())))), FrequencyStars()), Zoom({1.0, 0.35874906, -0.42753658}, {1.0,
0.35874906, -0.42753658}))), FrequencyStars()), {0.30287892, -0.7879979, 0.756324}),
Distort(Distort(HueShift(Distort(Distort(Distort(HueShift(Distort(Rotate(Distort(Distort(HueShift(Distort(Rotate(Distort(Disto
rt(Blur(Distort(Distort(RockImage(), ArcTangent(ExpandRange(ColorNoise()))),
ArcTangent(ExpandRange(ColorNoise())))), ArcTangent(ExpandRange(ColorNoise()))), ArcTangent(ColorNoise())),
RockImage()), ArcTangent(ColorNoise())), RockImage()), ColorNoise()), Blur(Add({0.214469, -0.05106278, -0.8334819},
Blur(Blur({-0.44914088, 0.86714524, -0.038012877}))))), Distort(Distort(Blur(Distort(Distort(-0.32682085, SunriseImage()),
ArcTangent(0.84263784))), LeafImage()), ArcTangent(ColorNoise()))), ArcTangent(ColorNoise())),
ExpandRange(RockImage())), 0.315652), ColorNoise()), Blur(ExpandRange(Add({0.44753784, 0.15750253, -0.9017423},
{0.214469, -0.05106278, -0.8334819})))), ExpandRange(RockImage())), ColorNoise()), Blur(Rotate({0.10588625,
0.2359776, -0.20337643}, {0.2809281, -0.97692156, -0.49766022}))))
Maples
Original Compressed 56:1
Multi-resolution
JPEG 14:1 Evolver 56:1
Multi-resolution
JPEG 14:1 Evolver 56:1
Related work
Searching for programs
Massalins Superoptimizer
Frigo & Johnsons FFTW
Palsberg, Lucier & Mamillapalli
Karl Sims
Programmatic image compression
Fractal compression
MPEG-7
Steerable search techniques in graphics
MLT
Radiosity
Conclusions
It is tractable to search the space of all
programs!
The visually similar criterion makes
computer graphics an interesting domain
Keep using JPEG for now
Future directions
Improve image quality/compression
How can the design of searchable
languages be formalized?
How do expressive constructs affect the
search problem?
Other interesting domains: animation,
audio compression, image search, robot
controllers
Questions
http://www.cs.brown.edu/people/morgan/evolver