Bright AClikeLuaDerivative

TerryMoore tmm@mcci.com MCCICorporation

7/15/2008 1

Whoweare
Systemengineeringcompany,specialistsin USBtechnology Ninetypeople HeadquarteredinIthaca,NY;sitesinAustin,Tokyo,Taipei, SeoulandEurope Focusedoncellphoneindustry
Over500millioncellphonesthatuseMCCItechnology TwoofthetopfourcellphoneOEMs TwoofthetopfourcellphonePlatformVendors

Additionalmarketsinsettopboxes,carnavi systems

7/15/2008

FocusofPresentation
WhatdidwelearnaboutLuabasedonthe changeswemade? HowareweusingourreskinnedLua

7/15/2008

MCCIsProblemSpace
Ourcustomersarehugeengineeringteams
manyproducts,shippedinhighvolume yearsofprepworkforonemonthsproduction! veryriskaverse

Oursoftwarehastobeintegratedintotheir developmentenvironments
eachenvironmentisdifferent eachenvironmentevolvesunpredictablyand asynchronously

Wehavetomaintaineconomyofscaleanddeliverbug fixesacrossallthedifferentconsumers Weuseautomationintensively

7/15/2008 4

WhatAutomationtoUse?
Java,Perl,Python,etc.,arewellaccepted
Forbrevity,letssayLDJforlanguagedejour

IftheyrenotusingLDJ,alltheseLDJsarevery heavyweight;thisgeneratesresistancetousing ourautomation IftheyAREusingLDJ,theyllhavetheirown version,anditwontingeneralbethesameas theversionwereusingortheversionanyother customerisusing MostLDJshaveenormouslibrarieswhichaddto thecomplexity

7/15/2008 5

WhynotLua?
Luaisaresearchlanguage,targetingembeddedscripting
needstoevolve backwardcompatibilityislessimportantthanexploringnew waysofsayingthing LuaasastandalonelanguageissecondarytoLuaasan embeddedlanguage

MCCIneededalanguagethatwouldemphasizebackwards compatibilityandstandalonetoolapplications
backwardcompatibilityiscritical Brightusedasastandalonelanguageisaprimaryusecase Lua3.2to4.0madeusrealizethatinordertouseLua technology,weneededadegreeofindependence

7/15/2008

Whatdidwechangeandwhy?
WelikedLuaalot wehopedforitsgeneraladoption andwewantedto stayoutoftheway Wechangedsyntax somethingalmostlikeLuaseemedworsethan somethingquitedifferent
weswitchedtoClikesyntaxforsomewhatcynicalreasons

Wechangedsemanticstomeettheneedofaproductionenvironment
Zerooriginindexing Undefinedvalues Nolocalesensitivity Empahsis onscriptportabilityoverfunctionality

Wechangedthecommandlinewrapperprograms(bright.exeand brightc.exe)tobemoreliketheUnixequivalenttools WechangedtheexternallyvisiblenamesofalltheCAPInamespace entitiessoasnottocollidewithLua. Weaddedsomethingsweliked

7/15/2008

Threekindsofchanges
Trivial nothinginterestingaboutthem Small somewhatinteresting,butnotamajor changetotheflavorofthelanguage Large majorchangestotheflavorofthe language Curiously,theeffortinvolvedwasinverseto thescaleofthechange

7/15/2008

TrivialChanges
Clikesyntax
Thiswastrivial,inthesensethatitwasasimple exerciseinthelexer andparser Moredetailsastowhatwedidlater iftherestime

Createdmanpages Wroteareferencemanual(adaptingliberally fromtheLuareferencemanual) With Clikesyntaxwegotbitwiseoperators enormouslyconvenient

Ofcourse,have toconverttoLONGfirst
7/15/2008 9

SmallChanges
Indexoriginzeroisaverysmallchange,conceptually
YoucanwriteLuaorBrightwithoutknowingtheorigin,ifyoure careful
function GetOrigin() for i,v in {1} do return i; end; end; _ORIGIN = GetOrigin();

Thenenumerationofanarraycanbewrittenas,e.g.,(inLua)
for i=_ORIGIN,#t-_ORIGIN do .

Zerooriginmakesstrsub()lessconvenienttouse,however, astheresnopleasant zerooriginmappingunlessyouuse 2asyourstartpointfornegativeindexing

7/15/2008 10

SmallChanges
.<id>notationdistinguishesreflexiveuseof stringsfromnormalstrings Changestowrapperexecutablesforstand aloneuse
Addcoptionforsymmetrywithsh c Allow#!prefixincompiledscripts Allowmultichunkcompilation(andteachcompilerto producethe#!prefix) Addfallbackmain()invocationinthebright.exe wrapper
7/15/2008 11

LargeChanges
AddingUndefined,andmakingNULLavalidkey anddatumfortables
codechangeswererelativelyminor,onedayswork flavoroflanguagechangedsubstantially IfNULLisavalidkey,thenNULLcannotbeusedasthe distinguishedendvaluewheniteratingovertables IfNULLisvaliddatum,thenpresence/absencetesting requiresextralinguisticfeatures

TheVMswerecompatible uptothispoint(sigh).
7/15/2008 12

Whatwelearned
MakingamoreClikelanguagesubstantiallyreducedresistanceto adoptioninMCCIscommunity Changingtozerooriginreducederrorsforprogrammersswitchingback andforthfromBrighttoC Theundefinedvaluemakesprogramsfailearlyontypos asdesired
Worksverywellforglobalandlocaltypos Returningundefinedformissingtableentriessimilarlymakesprograms morerobust Productivityandreliabilitywentupnoticably &immediately

ChangingtablestohaveNULL(nil)asafirstclassvalueisveryconvenient
butitreallychangestheimplementationandstylesubstantially

BitwiseoperatorsareEXTREMELYconvenient(eveniflua_Number isa double)

Luashouldaddthese

HavingaClikesyntaxallowsforsomecleverhackswhenchecking/using complex#includefiles
13

7/15/2008

WhynameitBright?
Itssortofapun
LuainChineseis. Ifyouaddsuntomoon,( +)yougetthe character,ming2,meaningbright. Mingwasalreadytaken,hence

7/15/2008

14

HowdoweuseBright?
AsaCrossplatformProgrammingLanguage RapidPrototyping Shellscripting
weuseitlikeawk

EmbeddedScripting CHeaderFileCrunching

7/15/2008

15

CrossplatformProgrammingLanguage
documentationgeneration sourcereleasegeneration automaticdependencygenerationforour buildsystem Theminorchangesmadetolua andluac were veryhelpful

7/15/2008

16

RapidPrototyping
Problem:remotecustomerwithbrokenhardwareandonly aTektronixscope Solution:builtatooltorecoverUSBhighleveldatafrom onlyadifferentialtraceofthedatalines
differentialtosingleendedconversion phaselockloopforclockanddatarecovery NRZItonormaldata CRCcalculation Tokenrecognition Totaleffort(sinceitwasbuiltstepbystep):about4hours.This wouldtakeaweekinC.

Forlowlevelhardwareoperations,thebitwiseoperatorsof Brightareextremelyuseful
7/15/2008 17

EmbeddedScripting
MCCIscrossplatformversionofNetBSDmake(1) supportsscriptinginBright.
extremelyconvenient becauseitremovesdependencyon externalcomputation toolsforcomplexmakeoperations allowsustohaveonemakefile thatworksanywhere,forany target

MCCIsusbrc toolcompilesUSBinitializationcodefrom highleveldescriptions weuseBrightforscripting informationabouthardwarelimitations AllofMCCIsUSBtestapplicationsuseBrightasthetest scriptinglanguage MCCIsversionofusbview usesBrighttolearnhowto decodedeviceclassdescriptions
7/15/2008 18

CHeaderFileCrunching
ItseasytogenerateaBrightprogramfroma wellformedheaderfile Thismakesiteasytodocertainkindsoftests onheaderfiles,andtouseCdefinitionsin Brightscripts Weusethis,forexample,foranassemblerfor aspecialpurposekernelVMmcciport.sys.

7/15/2008

19

FutureDirections
Completemodulesystem somewhatdifferentthan Lua,asthegoalistoeliminatefirstorderglobals 64 bitintegers try explicitexceptionhandling
usingcall() forthisisclumsy nothingaselaborateasC++isintended

Optionalstrongertyping
internallyimplementedversionofourCreateClass facility (again,forproductivity)

Stealfeatures fromLua5.1(#operator,iterators) Makethelexer availabledirectly

7/15/2008 20

SupplementalSlides

7/15/2008

21

Whatdidwechangeandwhy?
WelikedLuaalot wehopedforitsgeneraladoption andwewantedto stayoutoftheway Wechangedsyntax somethingalmostlikeLuaseemedworsethan somethingquitedifferent
weswitchedtoClikesyntaxforsomewhatcynicalreasons

Wechangedsemanticstomeettheneedofaproductionenvironment
Zerooriginindexing Undefinedvalues Nolocalesensitivity

Wechangedthecommandlinewrapperprograms(bright.exeand brightc.exe)tobemoreliketheUnixequivalenttools WechangedtheexternallyvisiblenamesofalltheCAPInamespace entitiessoasnottocollidewithLua. Weaddedsomethingsweliked

7/15/2008

22

ChangestoWrapperExecutables
Lua4swrappersweretoosimplisticfor productionuse
Mostimportant:changedbrightc (luac)tocombine multipleinputfilesintoasingleoutputfile
compiledscriptelaboratesbytecodeforeachfileinturn

Changedbright.exe(lua.exe)toinvokeglobalfunction main(ARGV)
onlyiftheglobalchunkdoesntreturnanexplicitvalue onlyifmain()isdefined

Allowed#!asfirstlineofcompiled(.bro)scripts Minorchangestocommandlineoptions
7/15/2008 23

WhatLuathingsaremissing?
Newfeaturesaddedin5.0and5.1
Upvaluesarenotgeneral,andusetheLuaV4syntax Nothreads Nestable longstringconstants Booleanvaluesupportwasaddeddifferently;noboolean type The# operator(goodidea,that) Thenewmodulesupport Weaktables Libraryimprovements

Automaticconversionbetweenstringsandnumbers Localesensitivityforprogramtext
aprogramhasthesamemeaning,nomatterthelocaleineffect atparsetime
7/15/2008 24

Newsemantics
Anewtypewasadded:undefined,withasingle distinguishedvalue,(alsocalledundefined).Allvariables initiallyhavevalueundefined.
Anyattempttoevaluateanundefinedvalueresultsinanerror.

Tablesemanticsareextended
nil (bright:NULL)isavalidtableindex,andavalidtablevalue Ifanindexvalueisnotinanarray,theresultistheundefined value Newexpressionsyntax:<v1>in <v2>allowsaneasywayto checkwhether<v1>isakeyinthetableexpression<v2> Entriesmustberemovedusingtdelete(t, k) t[k]= NULLnolongerremovesindexk.

7/15/2008

25

WhatCthingsdidweadd?
Language
AllbinaryandternaryfunctionsfromC:
bitwise&,|,^,<<,>> weforcenumberstointeger,dothe bitwisemath,thenreturntofloatformat. ISOe ? v1 : v2 andgcc e ?: v

The<iso646.h>alternatetokens ThealternatetokenspellingsfromISOC(writing<%for {,andsoforth. for(;;) {} anddo {} while ()

Extras
TRUE,FALSE,NULL arereservedwords,andpredefined.

AllthereservedwordsfromC++arealsoreservedwordsin Bright
7/15/2008 26

WhatCsyntaxdidwechange?
Commaisusedformultipleassignment,notmultipleexpression evaluation
x, y = f(), g() isthreeexpressionsinC:evaluatex;assignf() toy,andevaluateg(). x, y = f(), g() istwoexpressionsinLuaandinBright:evaluate f(),evaluateg(),thenassignrespectiveresultstoxandy.

Exponentiationisuseful;wekeptit(butuse**insteadofLua ^. Concatenationisexpressedusing..ratherthanmoreClike juxtaposition.(Butthetokenizer willcatenate literalstringsif theyrewrittensidebyside.) Doubleandsinglequotesbothdelimitstrings 'a' isthesameas "a",not0x41. FunctionsaredefinedasinLuaorAwk:function f() { } Nocompiletimetypes
7/15/2008 27

Dotnotation
The.<id>syntaxgeneratesthestring<id>,butexpresses theintentionthattheprogrammerisprovidingthenameofakeyin atable
v = (.n in ThisTable) ? ThisTable.n : 0;

IthinkIstolethisfromatomnotation inanolderLisp?

Perhapsabetterexample:
if (! (.Lib in globals()) Lib = dofile("mcci-v1.bro");

Makesreflexivity somewhatmoreexplicit byconvention,ifyou write .foo,youmeanfoo asanidentifier insomekindofreflexive context,whereas"foo" isastringforsomekindofexternal comparison
canslightlysimplifytheproblemofrenamingtableindices,ifused consistently:asearchfor.foowillfindmorecorrectinstancesthana searchforfoo.
7/15/2008 28

Builtin LibraryAdditions
Becauseoftheglobalnamespaceissue,we decidedtoprefixallbrightadditionswith bright_.
bright_diropen(),bright_dirread(),bright_dirclose() equivalenttothefamiliarUnixroutines bright_stat(),bright_stat_decodemode() portable versionofstat() bright_shortpathname() returnsthe[system dependent]shortversionofapathname date()wasextendedinasimilarwaytosomeof
7/15/2008 29

WhatLua5workdidweduplicate?
Weaddedseparateenvironmenttablesfor eachfunction(butdiditdifferently,andmore conservatively,i.e.basedontheLua4 mechanisms)
thiswasdoneinanticipationofBrightmodules, whichsofarhavenotbeenfullyimplemented

Miscellaneous:break,hexconstants, modulo(%,definedexactlyasinLua5,and probablyforthesamereasons)

7/15/2008 30

Brightstandardlibrary
Inadditiontothenormalbuiltinlibraries,MCCIhasastandard libraryofBrightfacilities,writteninBright. Normally(butnotnecessarily)referencedascontentsoftableLib Interestingwork
Lib.Disclose(),isakintounpack() fromLua5.1 namedbyanalogy withAPL. Lib.GetFlags()isastandardcommandlineparsingpackage Lib.Basename(),Lib.Dirname()areOSindependentfilenameparsers Lib.CreateClass()createsabstractclasseswithstrongertypechecking Lib.CreateStructureClass()createsabstractclasseswithspecificbinary representations(forinteroperatingwithothersystemcomponents) Lib.VectorToString()isliketable.concat()fromLua5.1

7/15/2008

31

Example Lib.CreateClass
cID = Lib.CreateClass( .ID, { { .string, { .number, }); .sName }, .Id }, cTARENTRY = Lib.CreateClass( .TARENTRY, { { .generic, .Parent }, { .number, .HeaderPos }, { .string, .sPathName }, { .string, .name }, { .number, .mode }, { .number, .size }, { .number, .mtime }, { .ID, .uid }, { .ID, .gid }, { .number, .type }, { .string, .linkname }, { .string, .prefix } });

cTARFILE = Lib.CreateClass( .TARFILE, { { .generic, .File }, { .generic, .CurrentEntry }, });

7/15/2008

32

DesignDecisionsthatWorked
Addingadefaultcalltomain()inthebright.exewrapper makeslargeprogramslookmuchnicertoCprogrammers AddingUndefinedgreatlysimplifiesdebugging NULL asatablevalue;TRUE andFALSE assynonymsfor 1 andNULL. Weallowedlocaldeclarationsinfor(;;),muchasinISO C99,whichwasverynice:
for (local i = 0; i < Max; i=i+1) { f(i); }

ismorereadable(toourCprogrammers)than
for i=0,Max-1 do { f(i); }

Both,ofcourse,arepermitted.(Thelatterissomewhat faster.)
7/15/2008 33

Drawbacks(whatwemissed)
Thelibraryroutinenamesshouldhavebeen mappedmorecloselyontotheirCequivalents. Weshouldhavedonemoreworkonmodularity, orbackportedtheLua5work. Ourprogrammersmisscompoundassignment (+=,etc)andswitch() strsub()ssemanticsarenotwelladaptedfor zeroorigin. ItwouldhavebeennicetohavetheBool type
7/15/2008 34

Thanks
ChrisYokum ofMCCIdidalotoflibrarywork, andwasourfirstenthusiasticinternaluser TheLuaprojecthasbeenincredibly understandingaboutoursomewhatheretical approach

7/15/2008

35