MethodsinTextureSynthesisUsingImageSamples

VicenteOrdonezRoman
ComputerScienceDepartment
StonyBrookUniversity,SUNY
StonyBrook,NY11790
vicente.ordonez@gmail.com

Abstract

Usingtextureshasaddedasignificantperceivedrealismtocomputergeneratedimages.Sinceproducing
visual realism is one of the major goals of computer graphics, several approaches have been explored
throughouttheyearsforgeneratingtextures.Oneofthemiscreatingtexturesgivenimagesamples.The
problem can be more specifically formulated as finding the method that produces a texture map of
arbitrary size as output that has the same appearance as a given input image sample. This technical
reportdescribesseveraltechniquesproposedinthecomputergraphicsliteraturefromthelasttenyears
forobtainingsuchtextures.

1. Introduction

Several approaches for generating textures have been explored in the computer graphics literature.
Proceduraltexturegenerationconcentratesontryingtocreatesuchtexturesusingfunctionsorcomputer
algorithms. Such procedures were studied since the early days of computer graphics. It is easy to devise
functionsforgeneratingtextureswithhighlystructuredpatterns,forinstance,generatingthetextureofa
chessboardcanbetriviallyimplementedusingsuchapproach.Someproceduralmethodsalsoprovedtobe
useful for generating some types of non structured textures. One of such techniques that proved to be
usefulforgeneratingrealistictexturesofseveralnaturalphenomenaisthePerlinnoisetechnique[1].In
contrast,imagebasedtexturegenerationprocedurestakeanimageasinputandusecomputeralgorithms
forgeneratingatexturemapofanarbitrarysizethatresemblesinappearancetheinputimagesample,see
Figure1.Thesemethods,inadditionofhavingproventobeusefulforbothstructuredandnonstructured
patterns,haveallowedgeneratingtexturesofsemistructuredpatternsthatwouldhavebeendifficultcases
forproceduralmethods.Theselastsetofmethodsalsotendtobevisuallymorerealisticthanprocedural
techniques since they better capture the details and stochastic processes of the real world. Xu et al. [2]
present an interesting comparison between procedural and image based texture synthesis using a brick
texture. Other more recent efforts in texture synthesis that are not covered in this technical report are
thosebasedon3Dtexturesandvolumetrictextures.

This technical report presents methods used in several papers that used relevant techniques on texture
synthesisusingimagesamples.Thenaiveapproachforusingtexturesfromimagesistakingtheinputimage

and repeating it sequentially. This simple method leads to visible artifacts in the seams between
consecutiverepetitionsoftheimage,soseveralmethodshavebeenproposedtogenerateanarbitrarysize
texturefromaninputimage.Thefirstsetofmethodsforconstructingsuchtexturesexplorestheideaof
conceivingtheimagesampleasastatisticaldistributionthatneedstobereplicatedatalargerscale.This
maybeachievedbytryingtomatchtheglobalstatisticalpropertiesofthetargettexturebasedontheinput
texture or trying to match local statistical properties. This section presents the work of Wei & Levoy [3].
This report also presents the work on texture synthesis of Ashikhmin [4] which is also using pixel level
texture generation and additionally presents a technique for allowing user interaction in the texture
generationprocess.

Figure1.(a)Inputimagesample(b)Outputtextureofarbitrarysize.
(CourtesyofWei&Levoy[3])

Anothersetofmethodsforgeneratingtexturesisbasedontheideaofpreservingthelocalappearanceof
the input image in the output texture by directly taking patch samples from the input image and then
blendingthemtogetherinavisuallyappealingmanner.Severalworkshavebeenpublishedaddressingthis
problemandthisreportwillanalyzethreeofthem.ThefirstoneistheChaosMosaictechniqueintroduced
byXuetal.[2],whichisbasedonthegeneralideaoftakingrandompatchesfromtheoriginaltextureat
different scales and then trying to blend them together by blurring the edges of the patches in order to
reduceartifacts.ThesecondpaperistheworkofEfros&Freeman[5],wherethegeneralideaistoselect
patchesfromtheinputimagethatagreebysomemeasurewithitssurroundingpatchesinthetargetimage
andthenreducingtheedgeartifactsbyusing minimumerrorcutsbetween the patches. The thirdpaper
using this technique is the one introduced by Kwatra et al. [6] where the patches are chosen using the
graphcutstechnique.

2. PixelBasedTextureSynthesis

Thebasicframeworkforthistypeofmethodsisdefiningthemodelthatwillbetterrepresenttheprocess
that produced the given input image and then simulate this process to generate the output texture of
arbitrarysizeonepixelatatime.ThissectionpresentsthemethodusedbyWei&Levoy[3].Thesimplicity
oftheirsingleresolutionmethodfordecidinganewpixelontheoutputimageexemplifiesatypicalpixel
based texture synthesis approach and their attempts to improve performance show the penalty on

efficiency incurred by their proposed basic single resolution method and the efficiency issues of several
similarpixelbasedapproaches.

The basic algorithm considers as input the given image sample and a noise texture of arbitrary user
specifiedsize.Itdecidestheintensityofanoutputpixelastheintensityofthebestmatchingpixelinthe
inputtexturebasedonthesimilarityoftheirneighborhoods.Itonlyconsiderstheneighborhoodofpixels
thathavealreadybeenassignedanintensityvalueintheoutputtexture.Ithandlesthecasewhenthereare
no pixels assigned so far by using the intensities in the input noise texture. This method is clearly
computationally expensive. It performs an exhaustive search in the input texture for every pixel in the
outputtextureandbothitsperformanceandoutputlargelydependsontheneighborhoodsize,seeFigure
2.Thesingleresolutionalgorithmdescribedaboveisextendedtoamultiresolutionschemeinordertoget
betterperformance.

Theideaofthemultiresolutionalgorithmistoapplythetechniqueinmultipleresolutionsstartingfromthe
coarselevelsinthepyramid.Thisideatakesadvantageonthefactthatlargepatternstructuresintheinput
image can be represented with fewer pixels at the lower resolutions and so we can just transfer this
informationfromthoselevelstothefinerresolutionsinthepyramiditeratively.Thismethodprovedtogive
similar results using smaller pixel neighborhoods to those of the single resolution algorithm using larger
pixel neighborhoods. Nevertheless, the exhaustive search step of the algorithm is still a problem. As
explainednext,theypresentafurtherimprovementregardingthisadditionalissue.

Figure2.TextureoutputsfortheWei&Levoyalgorithmusingseveralneighborhoodsizes.(a)5x5
neighborhood(b)7x7neighborhood(c)9x9neighborhood.(CourtesyofWei&Levoy[3]).

ThepaperusesamethodknownasTreeStructuredVectorQuantization(TSVQ)forimprovingthesearchof
pixels with similar neighborhoods in the input image. This method applied to this problem consists of
representingeachpixelneighborhoodasavectorandthencreatingabinarytreestructureofcodewords
using the following scheme: It uses the centroid of the input vectors as the root for the tree and then
dividesallthevectorsintwochildgroupsusinglocallyoptimalcodewordsforeach.Thisprocessisrepeated
for every new level in the tree hierarchy until matching some halting criteria. Using this structure to
performsearchesdoesnotleadtotheoptimalcodewordbutitisusuallyagoodapproximation.Therefore

the results will not be exactly the same results as the results using exhaustive search and will vary
dependingonthesizeofthecodeword.

ThequalityofresultsofthispapercompareswiththoseofEfros&Leung[7],maindifferencesarethathere
thepixelneighborhoodisfixed.Theypointoutthathavingafixedpixelneighborhoodallowsimprovements
intheefficiencyofsearchliketheonesuggestedusingtheTSVQtechnique.

2.1ATextureSynthesisMethodforNaturalTextures

ThissectionpresentstheworkofAshikhmin[4]onsynthesizingnaturaltexturesallowinguserinteraction
aswellashisimprovementoverpreviousmethodsfortheparticularcaseofsometypesofnaturaltextures.
Ashikhmin builds upon the previously discussed single resolution method of Wei & Levoy and also uses
some ideas from the chaos mosaic technique [2] discussed later in section 3.1. So this technique even
when generating one pixel at a time is also using some ideas of the patch based methods for texture
synthesisdiscussedinsection3.

Figure3.SampleoutputofthealgorithmforsynthesizingnaturaltexturespresentedbyAshikhmin.The
imageontherighthighlightstheboundariesbetweentheirregularshapedpatchestakenfromtheinput
imageonperpixelsteps.(CourtesyofAshikhmin[4])

AshikhmindiscussesabouttheproblemsofusingtheEuclideandistanceasameasureofvisualsimilarity,
sincethisoperationmightnotmodelaccuratelyvisualperception.Thealgorithmhepresentsisimplicitly
copyingirregularshapedpatchesfromthesampleimageandplacingthemintheoutputtexture,seeFigure
3. But the method is still perpixel and indeed is a modification of the Wei & Levoy algorithm presented
before. The algorithm avoids the exhaustive search performed by the other algorithm by using the fact
thatifonepixelatposition(x,y)wasalreadyplacedatsomepositionintheoutputtexturethenwecan
inferthenextpixelintheoutputtexturebyusingtheneighboringpixeloftherecordedpixelatposition(x,
y)intheinputimage.So,foreverypixelintheoutputtexture,insteadofdoinganexhaustivesearchinthe
inputimagesampleforpixelsthatagreeonsomepixelneighborhood,itlooksforanewpixelinaspecial
arraythatisusedtorecordpositionsintheinputimagethatwerepreviouslyplacedintheoutputtexture.
Itisnotalwaysthecasethattherewillbeonlyonematch,insuchcasetheonethatmatcheswithmostof

the pixels in the neighborhood is taken. There are a couple of issues that need to be resolved given this
simple description of this algorithm of texture synthesis. The first one is that it is necessary to remove
already recorded pixels in the special array. The second one is that when the algorithm is reaching the
bottomoftheoutputtextureinrasterscanordertheremightbesomeartifacts,furtherdiscussionabout
whythis isnotaproblemforthe particularcases thisalgorithmisintended toworkcan befoundin the
originalpaper.

Additionallythispaperpresentsamethodforallowinguserstocontroltheoutputtexturemapproperties.
Theideaistotakeausergeneratedsketchofthedesiredoutputtextureandthentosynthesizethetexture
while keeping the target appearance of the user generated input. In order to achieve this, the original
algorithmismodifiedforthispurpose.

3. PatchBasedTextureSynthesis

Thissetofmethodsconstructtheoutputtexturemapbytakingimagepatchsamplesfromtheinputimage
andthenstitchingthemtogethersothattheylooklikeacontinuoustexture.Thistechnicalreportexplores
severalmethodsthatusedifferentwaysforselectingthepatchesfromtheinputtextureimageandseveral
techniquesforstitchingthemtogethersothattheirappearanceseemscontinuous.

3. 1Chaosmosaic

The chaos mosaic technique proposed by Xu et al. [2] relies on using randomness and a deterministic
stochastic function to model the stochastic nature of the given sample image in the process of patch
selectionandplacement.

In order to generate a chaos mosaic texture the first step is to construct an initial output texture by
repeating the input image sample and therefore constructing a tiled texture. Then a random patch is
selectedfromeachofthosetiles.Theselectedrandompatchshouldbeofsizelessthanthesizeofeachtile
itself and the paper suggests a size between 0.5 and 0.75 times each tile, where each tile has the same
dimensions as the input image sample. They also mention that more than one random patch might be
selectedfromeachtilebuttheyusedonlyonerandompatchpertileintheirexperiments.Thenextstepis
placing those random patches on top of the tiled texture. The positions of the random patches are
determinedbyaniterativesystembasedonthecatmapofArnold[8].Usingthismodelandassumingthe
outputtileisofsizembym,thenthepositionswheretherandompatchwillbeplacedaregivenby:

Afterexecutionofthislaststep,thetransitionsbetweenpatchesinthetiledtexturefromthepreviousstep
are now covered by the patches placed on top of the output texture. The problem is now the edge
transitionbetweentherandomlyplacedpatchesinthisstepandthebackgroundtiledtexture.Thisissueis
addressedbyfilteringthepixelsneartheedgesoftherandomlyplacedpatchesusingasmoothblending
filter.


ThispaperalsosuggestsanencodingoftheChaosMosaictextureforfast3dgraphicsandtexturedelivery
throughtheinternet.Theysuggestthatusingthistechniqueitissufficienttotransfertheimagesampleand
asetofprecalculatedparametersoftheChaosMosaictextureinordertogenerateitfastontheendpoint
ofthetransmissionandthereforesavingbandwidthresourcesandhavingabetterlatency.

3. 2ImageQuiltingAlgorithm

TheImageQuiltingalgorithmproposedbyEfros&Freeman[3]introducesthenotionofminimumerrorcut
boundariesbetweentilestosolvetheweaknessoftheblendingmethoddescribedinChaosMosaic.Italso
presents another algorithm for selecting and placing the patch samples based on Euclidean distances
betweenneighboringregions.

Thefirstpartofthealgorithmstartsbychoosinganinitialrandompatchfromtheinputtextureandplacing
thispatchin thetopleft cornerofthe output texture.Tochoosethepatch thatgoestotherightofthis
initialpatchitsearchesonallpossiblelocationsintheinputtextureforapatchthatmatchestheleftpatch
for some overlapping region between them. The similarity measure used here is the Euclidean distance
betweenallthepixelsintheoverlappingregion.Thesizeoftheoverlappingregionisuserdefinedandby
defaultischosenas1/6ofthepatchsize.Thisprocessisrepeatedinrasterscanorderforeverypossible
patchlocationintheoutputtexture.Ingeneraltheoverlappingregionthatwillbeconsideredisthetopleft
overlappingregion,exceptforthefirstrowandfirstcolumnwhichwillonlyusetheleftandtopoverlapping
regionsrespectively.ThisprocessisindeedsimilartotheexhaustivesearchperformedinWei&Levoy[3],
withthedifferencethatthissearchisnotperpixelbutinaperpatchbasis.Itisalsoclear,althoughina
lowerextent,thatthesameefficiencyissuesoftheexhaustivesearcharisehere.

Figure4.Firststepischoosingtheconsecutivepatchesbasedontheoverlapconstraintandthen
calculatingtheminimumerrorboundarycutbetweenconsecutivepatches.(CourtesyofEfros&
Freeman[5])

The next step of the algorithm is calculating the minimum error boundary cut between each patch, see
Figure4.Thismayalsobeinterpretedinsomeextentasselectingarbitraryshapedpatchesfromtheinput
texture so that the pattern structures that are part of the input texture are preserved. Using the patch
selection process described before does not eliminate the tiled effect between adjacent patches in the
outputtexture.TheChaosMosaictexturemethodaddressesthisproblembysmoothingthepatchedges,
butthissolutionleadstoblurredtextureswhiletheminimumerrorboundarycutswilltrytopreserveall
thefrequencyinformationonbothpatches.Theminimumerrorboundarycutiscalculatedasfollows:First
calculate the error surface between two given patches in the overlapping region. Then using this region,
calculate the cumulative error for all possible paths. For instance, for calculating a minimum vertical cut

betweentwohorizontalconsecutivepatchesthecumulativeerrorforallpathswouldbe:

min
,

,
, ,
,

The authors report an execution time between 15 seconds to several minutes depending on the input
textureandtheoverlappingsize.Asstatedbeforethebottleneckistheexhaustivesearchforselectingthe
nextpatch.Efros&Freemandonotaddresstheefficiencyissuebuttheysuggestanoptimizationforthis
problemusingtheresultsofLiangetal.[9],whichreportedlyperformsinrealtime.Theyalsopresentan
interestingapplicationoftheimagequiltingideaextensionfortransferringthetextureofaninputimageto
atargetimagewhilepreservingtheshapeandilluminationcuesofthetargetimage.

3.3GraphcutTextures

The Graphcut Textures proposed by Kwatra et al. [4] represents the image or portions ofthe image as a
weightedgraph,whereeachpixelisanodeinthegraphandtheedgeweightorcostbetweentwoadjacent
nodesisgivenbysomemeasureoferrorbetweenadjacentpixels.Theyinitiallyproposeameasureusinga
simple difference between pixel intensities and then refine this cost function by adding local frequency
information into account. In its most simple form, the graph cut method can be used to find the cut
between two adjacent patches instead of using the minimum error cut boundary proposed by Efros &
Freeman.Theysuggestthisapproachasaninitialintuitionforusinggraphcuts.Theideaistomodelthe
overlapping surface between two adjacent patches as a graph, giving infinite weight to the nodes at the
borders of the overlap region. Then the problem of determining which pixels to use from each of the
patches is resolved by using the minimumcut or maxflow problem for the graph. They suggest the
solutionstothisproblemprovidedby[10]and[11].

Figure5.ComparisonoftheImageQuiltingalgorithmandtheGraphcuttechniqueresultsforagivenimage
sample.(CourtesyofKwatraetal.[6])

The graph cuts technique is still more powerful than just being a replacement for the dynamic
programmingapproachforchoosingthecutbetweentwopatchesproposedbyEfros&Freeman.Itallows
placing a new patch in any arbitrary location in the output texture and also takes into account previous
graphcutcalculationsinordertomakenewcuts.Calculatingthegraphcutsforanarbitrarilyplacedpatch
intoanimageissimilartothegraphcutprocessdescribedbefore,oneofthemainmodificationsisthatthe
patchiscompletelyenclosedinsidethealreadysynthesizedpixelsandthereforethegraphisboundedinall
directionsbyinfiniteweightededgesthatconnectwiththerestofalreadysynthesizedpixels.

Apendingdiscussionisaboutthewaytheychoosethenewpatchlocationforthencalculatingthegraph
cuts. They propose three ways for selecting the patches from the input image sample: The first one is
selecting the entire image as a patch and placing it into a random location, and then the graphcut
algorithmwillcuttheappropriateportionofthisimagetodothestitching.Thesecondwayisplacingthe
patchinoneofthelocationsthatminimizestheerrorbetweenthepatchandthealreadysynthesizedpixels
intheoutputtexturebysometolerancerange.Thispatchselectionissimilartothepatchselectionmethod
used in Image Quilting but here the error is computed along the entire patch and not only in the
overlappingregion.Thethirdmethodistakingasubpatchregionfromtheoutputtextureandthenlooking
back in the input texture for a close match chosen randomly from a set of candidate regions using a
probabilistic function. Those two last methods still have the problem of exhaustive search similar to the
onediscussedbeforewhenpresentingtheImageQuiltingalgorithmandtheWei&Levoyalgorithm.Here
theauthorsproposeandimplementaFastFourierTransformbasedaccelerationtechniqueforperforming
thissearch.

Thepaperalsodescribesseveralsuggestionsandenhancementstothealgorithmandhoweachofthose
will potentially get better results for some cases. More importantly they demonstrate their algorithm
usefulnessforsynthesizingvideotexturesbytakingintoaccountthetemporalinformationintheproblem.

Conclusions

Therehavebeenoutstandingadvancementsintermsofquality,efficiencyandgeneralityofthealgorithms
forsynthesizingtexturesfromimagesamples.Theinitialattemptsmainlyfocusedontryingtogeneratethe
outputtexturepixelbypixelusingsomeregionstatisticscalculatedfromtheinputimage.Thesemethods
already recognized some efficiency issues and limitations, so several methods for solving them were
presented.TherewereotherspecializedmethodsliketheonepresentedbyAshikhmin,whichtriedtoget
betterqualitywhilelosingsomegenerality.Thenextsetofattemptsforsolvingtheproblemusedtheidea
oftakingpatchesfromtheoriginalimageandthentryingtostitchthemtogether.Severalsolutionswere
presentedinthisarea,withdifferentmethodsforselectingthepatchesfromtheinputimagesampleand
differentmethodsforstitchingthepatchestogethersothattherearenotvisibleartifactsbetweenthem.
Another remarkable point about all these methods is that most of them offered some sort of aside
aggregated value by pointing out some additional applications. For instance Efros & Freeman proposed
using their method for transferring textures from one image to another, Ashikhmin suggested using his
method for user guided texture synthesis, Kwatra et al. proposed his method for video textures and in
generalthe useofgraphcutshasbeenappliedtootherproblemsaswell.Soevenwhentheproblem of
texturesynthesisitselfisimportant,themethodsusedforsolvingthisproblemmightpotentiallybeused
forsolvingotherrelatedproblemsbothincomputergraphicsandimageprocessing.

REFERENCES

[1] Perlin,K.AnImageSynthesizer.ProceedingsofSIGGRAPH1985.InComputerGraphics(1985),vol.19.
ACMSIGGRAPH,pp.287296.

[2] Y. Xu, B. Guo, and H.Y. Shum. Chaos mosaic: Fast and memory efficient texture synthesis. Technical
ReportMSRTR200032,MicrosoftResearch,April2000.

[3] LiYi Wei and Marc Levoy. Fast texture synthesis using treestructured vector quantization. In
SIGGRAPH00,pp.479488.

[4] M.Ashikhmin.Synthesizingnaturaltextures.InSymposiumonInteractive3DGraphics2001.

[5]Efros,A.A.,andFreeman,W.T.2001.Imagequiltingfortexturesynthesisandtransfer.Proceedingsof
SIGGRAPH2001,pp.341346.

[6] Kwatra, V., Schodl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video
synthesisusinggraphcuts.ACMTransactionsonGraphics,SIGGRAPH2003,pp.277286.

[7] A.A.EfrosandT.K.Leung.Texturesynthesisbynonparametricsampling.InInternationalConference
onComputerVision,pages10331038,Corfu,Greece,September1999.

[8]V.I.ArnoldandA.Avez.ErgodicProblemsofClassicalMechanics.Benjamin,1968.

[9] L. Liang, C. Liu, Y.Xu, B. Guo, and H.Y. Shum. Realtime texture synthesis by patchbased sampling.
TechnicalReportMSRTR200140,MicrosoftResearch,March2001.

[10]Ford,L.,Fulkerson,D.1962.FlowsinNetworks.PrincetonUniversityPress.

[11] Sedgewick, R. 2001. Algorithms in C, Part 5: Graph Algorithms. AddisonWesley, Reading,

Massachusetts.