Solutions For Automating IT Job Scheduling: Greg Shields

Solutions for Automating IT Job Scheduling
Greg Shields
sponsored by
SolutionsforAutomatingITJobScheduling
GregShields
IntroductiontoRealtimePublishers
by Don Jones, Series Editor
Forseveralyearsnow,Realtimehasproduceddozensanddozensofhighqualitybooks thatjusthappentobedeliveredinelectronicformatatnocosttoyou,thereader.Weve madethisuniquepublishingmodelworkthroughthegeneroussupportandcooperationof oursponsors,whoagreetobeareachbooksproductionexpensesforthebenefitofour readers. Althoughwevealwaysofferedourpublicationstoyouforfree,dontthinkforamoment thatqualityisanythinglessthanourtoppriority.Myjobistomakesurethatourbooksare asgoodasandinmostcasesbetterthananyprintedbookthatwouldcostyou$40or more.Ourelectronicpublishingmodeloffersseveraladvantagesoverprintedbooks:You receivechaptersliterallyasfastasourauthorsproducethem(hencetherealtimeaspect ofourmodel),andwecanupdatechapterstoreflectthelatestchangesintechnology. Iwanttopointoutthatourbooksarebynomeanspaidadvertisementsorwhitepapers. Wereanindependentpublishingcompany,andanimportantaspectofmyjobistomake surethatourauthorsarefreetovoicetheirexpertiseandopinionswithoutreservationor restriction.Wemaintaincompleteeditorialcontrolofourpublications,andImproudthat weveproducedsomanyqualitybooksoverthepastyears. Iwanttoextendaninvitationtovisitusathttp://nexus.realtimepublishers.com,especially ifyouvereceivedthispublicationfromafriendorcolleague.Wehaveawidevarietyof additionalbooksonarangeoftopics,andyouresuretofindsomethingthatsofinterestto youanditwontcostyouathing.WehopeyoullcontinuetocometoRealtimeforyour educationalneedsfarintothefuture. Untilthen,enjoy. DonJones
GregShields
IntroductiontoRealtimePublishers.................................................................................................................i Chapter1:DoINeedJobScheduling?TenQuestionstoAskYourself..............................................1 ThePainoftheNonHomogeneousITEnvironment...........................................................................2 ConstructinganUnfriendlyApplication....................................................................................................4 DefiningITJobScheduling..............................................................................................................................7 TenQuestionstoAskYourself....................................................................................................................10 1.HowMuchTimeHaveYouWastedinCompletingTasksManually?...............................10 2:WhatIstheCostofanErrorIncurredDuringaManualTask?...........................................10 3:WhereCanYouGoforaHeadsUpDisplayofITActivities?................................................11 4:HowDoYouManageCrossSystemCommunications?..........................................................12 5:AreYouConcernedAboutIdleTimeDuringaTask?..............................................................13 6:HowManyTasksExistinYourEnvironmentthatYouDontKnowAbout?.................13 7:CanYouMonitorITTasksAcrossEveryTeam,Platform,andApplication?................14 8:BuildVersusBuy:IsaHomegrownSchedulerGoodEnough?...........................................14 . 9:WhatAreYourStepsforLinkingtheResultsofOneTaskwiththeActionsof Another?..........................................................................................................................................................15 10:HowDoYouHandleErrorsinaCustomCodedScript?......................................................15 DoYouNeedJobScheduling?......................................................................................................................15 Chapter2:SevenUseCasesforAutomatingITJobScheduling.........................................................16 SevenStoriesinITJobScheduling............................................................................................................17 Story#1:FiveAdministratorsFiveScheduledTasksLibraries...............................................17 Story#2:ConsolidatingTasksfromEveryApplication...................................................................18 Story#3:TheScriptthatWasntaJob.....................................................................................................20 Story#4:WhenGatheringDataIsMorethanGatheringData......................................................22 Story#5:ControllingDataTransferasaWorkflow..........................................................................24 Story#6:ComplexJobsandtheNeedforTriggers............................................................................26 Story#7:SecuringtheDataCenterfromtheEaseofScriptExecution....................................28
ii
GregShields
JobSchedulingsStoriesAreitsValueAdd............................................................................................30 Chapter3:WhatMakesanITWorkflow?ATechnicalDeconstruction.........................................31 AWorkflowIsanITActivity,Quantified................................................................................................31 Capturable......................................................................................................................................................32 . MonitorableandMeasurable.................................................................................................................33 . RepeatableandReusable.........................................................................................................................34 Securable.........................................................................................................................................................35 AGroundUpITWorkflowConstruction...............................................................................................36 . JobLibrariesandtheValueofTriggers..................................................................................................44 AnITWorkflowTellstheComputerWhattoDo............................................................................45 Chapter4:ImplementingEnterpriseJobScheduling:ARequirementsChecklist....................46 CreatingaRequirementsSpecforJobScheduling.............................................................................46 Requirement#1:TheSolutionShallIntegratewithAllPlatformsandApplicationsthat ComprisetheBusinessSystem..............................................................................................................48 Requirement#2:TheSolutionShallExposeAllPropertiesandMethodsUsedbyThose Systems............................................................................................................................................................48 Requirement#3:TheSolutionShallIncludeSupportforScriptLanguage Independence................................................................................................................................................49 Requirement#4:TheSolutionShallSupportQueuesforJob/PlanPrioritizationand PerformanceManagement......................................................................................................................49 Requirement#5:TheSolutionShallSupportFileBasedTriggers........................................50 Requirement#6:TheSolutionShallSupportMessageBasedTriggers..............................50 Requirement#7:TheSolutionShallSupportEventBasedTriggers....................................51 Requirement#8:TheSolutionShallSupportTimeBasedTriggers.....................................51 Requirement#9:TheSolutionShallSupportVariablesandDynamicDataAcrossJobs andPlans.........................................................................................................................................................51 Requirement#10:TheSolutionShallSupportIntraandInterWorkflow Communication............................................................................................................................................52 Requirement#11:TheSolutionShallSupportJob/PlanExecutionviaaBusiness Calendar...........................................................................................................................................................52
iii
GregShields
Requirement#12:TheSolutionShallSupportandReportonJobObjectDependencies .............................................................................................................................................................................53 Requirement#13:TheSolutionShallIncludeClient,Browser,andMobileDeviceUser Interfaces.........................................................................................................................................................54 Requirement#14:TheSolutionShallSupportNesting,Chaining,andLoadBalancingof JobsandPlans...............................................................................................................................................54 Requirement#15:TheSolutionShallUseanObjectOrientedManagementInterface55 Requirement#16:TheSolutionShallUseaCentralizedJobandScriptLibrarythat IncludesCommonlyUsedBuiltInItems..........................................................................................56 Requirement#17:TheSolutionShallConsolidateErrorandOutputMessagesfrom ExecutedJobs................................................................................................................................................57 Requirement#18:TheSolutionShallSupportaCentralizedSecurityModelAcrossJobs andPlans.........................................................................................................................................................58 Requirement#19:TheSolutionShallIncludeaCentralizedChangeManagementModel .............................................................................................................................................................................58 Requirement#20:TheSolutionShallIncludeaCentralizedAuditDatabasethat IncludesMetricsandAlerting................................................................................................................59 So,DoYouNeedaJobSchedulingSolution?.........................................................................................60
iv
GregShields
Copyright Statement
2011 Realtime Publishers. All rights reserved. This site contains materials that have been created, developed, or commissioned by, and published with the permission of, Realtime Publishers (the Materials) and this site and any such Materials are protected by international copyright and trademark laws. THE MATERIALS ARE PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. The Materials are subject to change without notice and do not represent a commitment on the part of Realtime Publishers its web site sponsors. In no event shall Realtime Publishers or its web site sponsors be held liable for technical or editorial errors or omissions contained in the Materials, including without limitation, for any direct, indirect, incidental, special, exemplary or consequential damages whatsoever resulting from the use of any information contained in the Materials. The Materials (including but not limited to the text, images, audio, and/or video) may not be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any way, in whole or in part, except that one copy may be downloaded for your personal, noncommercial use on a single computer. In connection with such use, you may not modify or obscure any copyright or other proprietary notice. The Materials may contain trademarks, services marks and logos that are the property of third parties. You are not permitted to use these trademarks, services marks or logos without prior written consent of such third parties. Realtime Publishers and the Realtime Publishers logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. If you have any questions about these terms, or if you would like information about licensing materials from Realtime Publishers, please contact us via e-mail at info@realtimepublishers.com.
GregShields
[EditorsNote:ThisbookwasdownloadedfromRealtimeNexusTheDigitalLibraryforIT Professionals.AllleadingtechnologybooksfromRealtimePublisherscanbefoundat http://nexus.realtimepublishers.com.]
Chapter1:DoINeedJobScheduling?Ten QuestionstoAskYourself
Istillrememberthatdaythebosswalkedintomyoffice.AndIstillrememberherfateful yetenticingwordsthattalkedmeintotacklingtheproject,TheProjectThatWouldChange Everything.YouretheScriptingGuy.Youllmakeitwork,shesaid.Iagreed,andtherestis history. Youreprobablyfamiliarwiththisstory:BigprojectsinITstartthatway.Bigprojectsthat mayseemgreatonpaperbutthengetvery,verycomplexastheyredeveloped.Bigprojects whosemultipleapplicationsacrossmultipleplatformsrequiresubstantialintegration effort. Butthisstoryisntonlyaboutthatonebigproject.Itsalsoaboutalltheotherlittle automationsscriptsforWindowsandOracleandActiveDirectory(AD),SQLpackages, Linuxcronjobs,andsoonthatcreepintoeveryITenvironmentovertime.Those automationsabsolutelysolveabusinessimmediateneeds.Butwithoutcentral management,theyalsocomewithacost.Thatcostarisesasscriptsage,technology changes,andthescriptownersrelocateorleavethecompany. Backthen,IwasknownasTheScriptingGuy.Ifyouneededaquickdatatransformation,or ascheduledmovementoffilesfromonesystemtoanother,Iwasyourgotoperson.Ihad developedacommandofthemajorscriptinglanguages,alongwithalltheothernecessary addonsoneneededtobeTheGuy.Overthecourseofseveralyears,myintegration prowesshadgrowntoincludeplatformandapplicationspecifictechnologiessuchasWMI, ADSI,SQL,someOracle,andevenabitofLinuxandIBMAIX. IputthatknowledgetowhatIthoughtwasgooduse.Overtheyears,Ihadgottentothe pointwheremuchofmydailyresponsibilitywasautomatedsortof.Sometimesmylittle automationsbroke.Sometimestheywereaccidentallydeletedorotherwisewipedout throughtheregularchangethathappensinanydatacenter.Sometimestheirneedwent away,oraserversconfigurationwasupdated,andasaresult,Ihadtogofindthemonce againandrememberwhattheywereintendedtodo. Theresultwasanotverywelloiledmachine;onethatcreatedevenbiggerproblemsthe dayImovedontoanewemployer.Yousee,Ihadlittleautomationsscatteredaroundthe companyserverswithmynameonthem.LastIheard,theyrestillfindingthemyearslater, usuallyaftersomeprocessbreaksthatnobodyeverknewexisted. Youcanprobablyguesswhatweweremissing.WeneededenterpriseITjobscheduling. ThatswhyImwritingthisbook,toexplainwhatthisapproachisandhelpyourealizeyou couldprobablyuseitaswell.Throughoutthisbook,Iintendtoreturnbacktothatbig projectsstoryalongwithafewoftheotherlittleonestoshowyouwhy.
GregShields
ThePainoftheNonHomogeneousITEnvironment
ThisbookwouldntexistifeveryITtechnologyseamlesslytalkedwitheachother, transferringdata,events,andinstructionsacrossplatformsandapplicationswithease.If everytechnologycouldperfectlyscheduleitsactivitieswithitselfandothers,youwouldnt bereadingthesepages. Butyouare,andconsequentlythisguideindeedexists. ItexistsbecauseITjobschedulingisataskthateveryenterpriseneeds,asdomanysmall andmidsizeorganizationsaswell.Jobsinthissenserepresentthoselittlepackagesof automationIdiscussedearlier.Someworkwithasinglesystem.Othersintegratethe servicesofmultiplesystemstocreateakindofmixedworkloadthatproducesaresult yourbusinessneeds. Considerafewofthosejobsyoureprobablyalreadycreatingtoday: Reportsaboutactivitiesonanemailsystemneedtobecollectedanddistributedto securitygroups CustomerfilestransferredinboundtoaLinuxFTPserverneedtobeingestedupon receiptintoaSQLdatabase ProvisioningADusersrequiresseveraladditionalstepsacrosssystemssuchas Linux,emailapplications,andOracledatabases NewrecordsinaMicrosoftSQLorOracledatabasetriggeractionstooccurinoneor moremiddlewaresystem
Yourjobsmightbelesscomplex,workingwithonlyasinglesystemorapplication.Orthey mightbeexceedinglyso,requiringtheparticipationofmultipleapplicationsacross differentplatformsindifferentpartsoftheworld.Atissuehereisntnecessarilyhow complexyourjobsare.Thesimpleoneshavemanyofthesamerequirementsand necessitateasmuchduediligenceasthecomplexones.Rather,theissuehasmoretodo withtheworkflowthatsurroundsthosejobs,andthesolutionsyouimplementtomanage, monitor,audit,andotherwisekeeptabsoneveryactivityatonce. ItalsohastodowiththeverydifferentlanguagesandtechniquesthateachITtechnology usesandrequires.Thosedifferencesrepresentabigheadacheinsidetodays heterogeneousdatacenters.YourIToperatingenvironmentsurelyhasWindowssystems. ButitprobablyalsohasLinux,Oracle,HPUX,Solaris,andothers.Youprobablyneedto transferXMLdocuments,DOCXfiles,andXLSXspreadsheetsovermultipleprotocolslike SMB,FTP,andSSH.Evenyourmonitoringcomesinmanyflavors:SNMPforswitchesand routers,WMIforWindowssystems,andallthevariousUNIXandLinuxwidgetsforkeeping tabsontheiractivities.
GregShields
Youcanimaginejustafewoftheheadachestheseradicaldifferencesinapplicationsand platformscreate: Everyoperatingsystem(OS)speaksitsownlanguage Everyapplicationspeaksitsownlanguage EveryapplicationandOSusesitsownsecuritymodelthatdoesntnecessarily integratewellwitheachother Transferringdataorschedulingactionswithinandbetweeneachisdifficultand sometimesimpossiblewithnativetoolsalone Mostimportantly,centrallymanagingeveryjobeverywherejustisntpossible
SolvingthesefiveproblemsistheprimarymissionofanITjobschedulingsolution.Froma centrallocation(seeFigure1.1),anITjobschedulingsolutioncreatesaplatformonwhich torunallyourlittleautomationpackagesthatmightotherwisebespreadacross technologies.Usingacentralizedsolution,adatabasejob,aUNIX/Linuxjob,andanFTPjob areallpartsofthesamemanagementframework.Allbegintheirexecutionfromthesame place,andallaremanagedandmonitoredfromthatsinglelocation.
Figure1.1:AnITjobschedulingsolutioncentralizesjobsofalltypes,acrossall platformsandapplications. Asyoucanprobablyguess,suchasolutionhastobeexceptionallycomprehensive.A solutionthatworksforyourcompanynotonlyneedstosupportyourmanagement, monitoring,andworkflowneeds.Itmustbemorethanjustamorepowerfulversionofthe WindowsTaskManager.ItalsoneedstosupporttheintegrationsintoeveryOS,platform, andtechnologythatyourbusinessprocessesincorporate.Thatswhyfindingasolutionfor automatingITjobschedulingcanbesuchachallengingactivity.
GregShields
Tohelpyouout,youcanconsiderthisguidetobeakindofautomationideafactory.Its fourchapterswillpresentyouwithquestionstoaskyourself,helpingyouframeyourneed forajobschedulingsolution.Itdeliversasetofrealworldusecasesforseeingscheduling inaction.ItdeconstructsanITjobsothatyoucanpeerinsideitsinternalmachineryand understandthepowerofacentralizedsolution.Anditwillconcludewithachecklistof requirementsyoushouldconsiderwhenseekingthesoftwarethatcreatesyoursolution. Illbeyourguide,andthroughoutthisprocessIllshareafewofmyownstoriestobring somerealworldexperienceintothiscomplextopic. Note Inthisbook,youllhearmeusethetermjobscheduling.Anothercommonly usedtermforjobschedulingisworkloadautomation.Forthepurposesofthis book,youcanassumethatthetwoareinterchangeable.
ConstructinganUnfriendlyApplication
Now,backtomystoryfromlongago.EveryOSandapplicationcomesequippedwith multiplewaystoperformitscorefunctions.Youalreadyknowthis.AnOSincludesoneor morescriptinglanguagestoenactchangeandreaddata.Everymoderndatabasehasits ownschedulingandautomationfunctions,enablingthecreationofpackagesforinserting andselectingdata.EvenmiddlewaretechnologiesandapplicationshavetheirownAPIs, whichcanbeinterfacedeitherinsideoroutsidetheapplication. Buttheinternallanguagesandautomationsthatcomewithaproductarerarelyequipped tohandleactionsoutsidethatproduct.EvertrytouseanXMLdocumenttoinstructaSQL ServertoupdateanOracledatabaserowsothatanSAPapplicationcanprovisionaprocess toanAIXmainframe?Whew!ThatsprettyclosetothesituationIexperiencedasIstarted onTheProjectThatWouldChangeEverything. Letsstartwithalittlebackground.Whythatprojectwasneededisreallyunimportant,as iswhatweweredoingwithitsdata.Whatisimportant,however,aretheinterconnections betweeneachofitsdisparateelements.Multipleapplicationsrunningatopmorethanone OS,integratingwithdifferentdatabases,andrequiringdatafrombothinsideandoutside theorganizationwasjustthestart. Togetgoing,Iattemptedtodiagramitscomponents,creatingsomethingclosetowhatyou seeinFigure1.2.Atahighlevel,thissystemwasconstructedtoaggregateasetofdata fromoutsideourorganizationwithanothersetontheinside.Ourproblemwasthemany differentlocationswherethatdataneededtogo.
GregShields
External Data Source
SAP Server
Oracle Database
FTP Server
IIS Server
SQL Database
Active Directory Server
User
Email Server
UNIX Mainframe
Figure1.2:Myunfriendlyapplication. LetmebreakdownthemassofarrowsyouseeinFigure1.2.Theflowofdatainthissystem startedviaanFTPfromanexternaldatasource.Thatdata,alongwithallitsmetadata, neededtobestoredinasingle,centralizedSQLdatabase.There,permissionsfrom WindowsADwouldbeappliedtovariouspartsofthedataset.Somedatawasappropriate forcertainusers,withotherdatarestrictedtoonlyacertainfew.Informationinsidethe FTPdatastreamwouldidentifywhoshouldhaveaccesstowhat. UserscouldinteractwiththatdatathroughaMicrosoftIISserverrunningahomegrown Webapplication.ThatWebapplicationusedXMLtotransferdatatoandfromtheSQL database.CertaintypesofdataalsoneededtobeaddedtoourcompanySAPsystem runningatopOracle,requiringdatatransformationsanddeliverybetweenthosetwo systems. Occasionally,portionsofthatdatawouldneedtobeingestedintoaUNIXmainframefor furtherprocessing.There,itwouldbeconsolidatedwithdatafromotherlocationsfor greateruseelsewhereinthecompany.Anemailserverwouldensureuserswerenotified aboutupdates,newdatasets,andothersystemwidenotifications.
GregShields
Thatsalotofarrows,andeachofthosearrowsrepresentanintegrationthatneedstobe laidintoplaceinorderfortheentiresystemtofunction.Eacharrowalsorepresentsan activitythatneedstohappenataparticularmomentintime.Dataheadingtowardsthe Oracledatabaseobviouslycouldntbescheduledtogothereuntilitwasactuallyreceivedat theSQLServersystem.Usersshouldntbenotifiedunlesssomethingimportanttothem wasactuallyprocessed.Justtheschedulingsurroundingeacharrowsintegrationwasa complextaskuntoitself. Doesthislooklikeoneormoreofthesystemsthatarecurrentlyinyourdatacenter?If youredoingmuchwithdatatransformationandmovement,youmighthavethesame schedulingheadachesyourself.Thatswhytherearefourcriticalpointsthatareimportant torecognize: Youcanfindinterconnectedsystemslikethiseverywhere.Incompaniesacrossthe globe,ITsystemsareconstructedinwaysjustlikethis,usingtechnologiesjustlike these.Soalthoughthisprojectwouldindeedchangeeverythingformycompany, otherselsewherearealreadydealingwithslightlydifferentbutmostlythesame integrationsforotherreasons. Constructingandmanagingthissystemisnotnecessarilyanactivitylimitedtojust developers.Imnotadeveloper.Imasystemsadministratorwhojusthappenedto beconsideredTheScriptingGuy.Thus,inthemindsofmysuperiors,Imadethe perfectcandidateforleadingthisprojecttocompletion.Thesameholdstruewith theworldsothersystemsaswell.ITadministratorsaretaskedwithconstructing systemsthataremadeupofmanymovingparts.Gettingthemalltotalkwitheach otherandscheduletheiractivitiesisonlythefirststepinthatbattle. Itisentirelypossibletobuildthisusingscriptingandapplicationspecificautomations, butdoingsoisareallybadidea.MystrewnaboutSQLSSISpackagesandVBScripts andcronjobsneverpreparedmefortheadministrativeoverheadofarealsystem ratherthansomejumbleofunconnectedactions.Successfullyschedulingthe activitiesacrosseverypieceoftheinfrastructureisntataskthatworkswellunless itscentralized.Evenworse,managingandmaintainingthoseautomationsbecomes arisktothesystemsoperationastheygrowinnumberandgetmorecomplex. Theschedulingofactivitiesinasystemlikethisrequiresmorethanjusttheclockon thewall.Thinkaboutthedataflowbetweentheseindividualcomponents,andthe eventsthatneedtotriggerotherevents.Simpleactionsinsidesimplesystemsmight workjustfinewithatimeanddatebasedschedule.Isit3:00AM?Startthebackups! Butsystemswithgreatercomplexityandmoredependenciesrequiremore powerfulmeansofdeterminingwhentodosomething.Thosedecisionscanbemade basedonreceivingapieceofdata,seeingachange,readingalogfile,oranyofthe myriadotherstatechangesthatoccurinanyITsystem.AgoodITjobscheduling systemwillgiveyouplentyofconditionsthatyoucancustomizeforidentifying whenanactionshouldoccur.
GregShields
ItisthecombinationofthesefourrealizationsthathelpedmeunderstandthatIneededto stepoutsidemyapplicationspecificmindset.IthelpedmerealizeIneededtolookto solutionsthatscheduleactivitiesacrosseveryplatformandeveryapplication.Thatswhen IstartedlookingintoenterpriseITjobschedulingsolutions.
DefiningITJobScheduling
LetsnowtakeastepbackfromthestorylineandthinkforaminuteaboutwhatITjob schedulingshouldbe.Ivealreadysuggestedthatajobrepresentssomesortof automationthatoccurswithinanITsystem.Butletsgettechnicalwiththatdefinition.I submitthatanITjobrepresentsanactiontobeexecuted.AnITjobmightberunninga batchfileorscriptfile.Itmightberunningashellcommand.Itcouldalsobetheexecution ofadatabasejobortransformation.Essentially,anythingthatenactsachangeonasystem iswrappedintothisobjectwellcallajob. Usinganobjectorientedapproach,itmakessensetoconsolidateindividualactionsinto separatejobs.Thissingleactionperjobapproachensuresthatjobsarereusable elsewhereandforotherpurposes.ItmeansthatIcancreateajobcalledConnecttoOracle DatabaseandusethatjobanytimeIneedtomakeanOracledatabaseconnection anywhere. Nowifeachjobaccomplishesonething,thismeansthatIcanstringtogethermultiplejobs tofullycompletesomekindofaction.IllcallthatstringanITplan.Aplanrepresentsa seriesofrelatedjobsthatcanbeexecutedwiththeintendedgoalofcarryingoutsome change.Figure1.3showsagraphicalrepresentationofhowthismightwork.
Figure1.3:Multiplejobsareconnectedtocreateaplan. InFigure1.3,youcanseehowthreedifferentjobsareconnectedtocreatetheplan.Job27 connectstoanOracledatabase.ItpassesitsresulttoJob19,whichthenextractsasetof datafromthatdatabase.Onceextracted,thedataneedstobesentsomewhere.Job42 completesthattask,asitFTPsthedatatoalocationsomewhere. Theresobviouslyanarttocreatinggoodjobs.ThatsatopicthatIlldiscussingreater detailinChapter3,butIneedtointroducesomeofthebasicshere.Agoodjob,forexample, mightnotnecessarilyhaveanyspecificdataorhardinformationthatsstoredinsidethe job.RatherthanaconnectionstringtoaspecificserverforJob27,amuchbetterapproach wouldbetousesomekindofvariableinstead.
GregShields
Developersusethetechietermparameterizationtorepresentthisgeneralizingofjob objectsandthesubsequentapplicationofvariablesattheirexecution.Figure1.4shows howaparameterizedplancanlinkthreegenericjobs.Atthepointthisplanisrun,those jobsarefedthevariableinformationtheyneedtoconnecttotherightdatabase,extractthe rightdata,andeventuallypassitontothecorrectFTPsite.
Figure1.4:Feedingparameterstojobsinaplan. Byparameterizingtheplaninthisway,Inowgetreusabilityoftheplaninadditiontoall theindividualjobsthatmakeupthatplan.ShouldIdowntheroadneedtoattachtoa differentdatabasesomewhere,pulloffadifferentsetofdata,andsendittosomeotherFTP site,Icanaccomplishthisbyreusingtheplanandmodifyingitsvariableinformation.Thats reusabilityontopofreusability! Theresobviouslyquiteabitmoretothiswholeconceptofworkingwithjobsandplans.Ill spendmoretimeinChapter3helpingyouunderstandthevariouscharacteristicsthatcan beassignedtoajobandaplanaswellasotherobjectsatypicalITjobschedulingsolution willuse. Butthereisonecharacteristicthatmeritsattentionbeforemovingon.Thatcharacteristicis thescheduleitself,whichneedstobeappliedtotheobjecttotellitwhentorun.I mentionedearlierthatschedulingforlargesystemslikeTheProjectrequiresakindof flexibilityyoujustcantgetbylookingattheclockonthewall.Rather,thekindsofjobsthat projectneedstendtobemorerelatedtoactionsorstatechangesthatoccurwithinthe system.
GregShields
LetsassumethatFigure1.4sPlan7relatestosomedatatransferthatneedstohappen insideTheProject.Inthiscase,letsassumethatthedatatransferoccursbetweenitsSQL ServerandUNIXmainframe.Figure1.5showsagraphicalrepresentationofhowthismight beapplied.There,youcanseehowthreedifferentschedulescouldpotentiallybeattached tothenewlycreatedplan: Invoketheplanat3:00PM Invoketheplanwhenasetquantityofnewdataarrives Invoketheplanwhenprocessorutilizationislessthan30%
Figure1.5:Applyingascheduletoaplan. Anyofthesethreeschedulescanbeappropriate,dependingontheneedsofthesystemand itscomponents.Forexample,thefirstschedulemightbeappropriateifadailydatadumpis allthatsnecessary.Inthatcase,adate/timecentricschedulemightbeallthatsnecessary tocompletetheaction.Verysimple. Thesecondandthirdtaskshighlightsomeofthemorepowerfulschedulingoptionsthat couldalsodrivetheinvocationoftheplan.Inthefirst,theplanisexecutednotbasedon anytimeofday.Rather,itexecuteswhenasetquantityofnewdatahasbeenaddedtothe database.Thiscouldbeasmartsolutionifyouwantthesetwodatabasestostayroughlyin syncwitheachother.Itisreallypowerfulwhenyouconsiderhowdifficultthatkindof schedulingwouldbetocreateifyouwereusingjustthenativeSQLorUNIXtoolsalone. Thatthirdscheduleisparticularlyinteresting,becauseitcouldbeusedaloneorin combinationwiththesecond.Thatthirdscheduleinstructstheplantorunonlyifthe serverisntterriblybusy.Usingitincombinationwiththesecondallowsyoutomaintaina levelofsynchronizationwhilestillthrottlingtheuseoftheserver.Agoodjobscheduling solutionwillincludeawiderangeofconditionsthatyoucanapplytoplanstodirectwhen theyshouldkickoff.
GregShields
TenQuestionstoAskYourself
Again,IlldivedeeperintothisdeconstructionofanITworkflowscomponentsinChapter 3.Butbeforeyoucantrulyappreciatethepowerofthismodularapproach,therearelikely afewquestionsthatyoureprobablyaskingyourself.Ifyourenot,letmehelpyououtwith alistoftengoodquestionsaboutyourownenvironmentthatyoushouldprobablyask yourself.Youranswerstothesetenquestionswilldeterminewhetheryoullwanttoturn thenextchapters.Ifyoureexperiencingzeroheadacheswiththetoolsyouhavetodayfor schedulingyourITactivities,youwontneedtherestofthisbook. Everyoneelsewill.
1.HowMuchTimeHaveYouWastedinCompletingTasksManually?
Itwasntmanyyearsagothatoneofmyjobswasinkeepingasetofserversupdated. Monthlyupdateswerederigueurwithsomeonevenshorterschedules.Eachcamewitha veryshorttimewindowwhentheycouldandshouldbeapplied.Thebigproblemresulted fromthefactthattheseupdatestypicallyrequiredaserverreboottogetthemapplied. Atthattime,ourrebootwindowwasintheweehoursofthemorning,manyhourspastthe usual8to5workday.Forme,stickingaroundonceamonthtocompletetheseupdates representedahardshiponselfandfamily.ThatswhyIcreatedmyownautomationthat wrappedaroundtheseupdatesinstallation.Formysolution,whenupdatesweredropped intoaparticularlocation,theywereappliedatthenextwindow.Mymobiledevicenotified meshouldanyproblemsoccur. Fromthatparton,addingupdatestoserversmeantsimplyaddingthemtotheright locationandmakingsuremymobiledevicewasnearthebedside.Yes,sometimestheyd experienceaproblem,butthosecouldbefixedthrougharemotecontrolsession.Successful monthscouldgobywithoutlossofsleeporimportantfamilytime. AlthoughyourITjobschedulingneedsmightnotnecessarilygodownthepathofsystem updateinstallation,thistime(andmoney)savingsbecomesanimportantparable. Computersaredesignedbynaturetobeautomationmachines.Thus,itstandstoreason thatanymanualactivityshouldhaveanautomationfriendlyadjunct.Itisthatadjunctthat canbeapartofyourgreaterschedulingsolution. Canyouaffordtopaytheriskofinappropriateexecution,forgetfulness,orusererrorin yourcriticalactivities?Ifnot,creatingflexibleandreusableworkflowsviaanITjob schedulingsolutionshouldpayforitselfinaveryshortperiodoftime.
2:WhatIstheCostofanErrorIncurredDuringaManualTask?
Thatfirstquestionintroducesthepossibilityofthreekindsofriskinanymanualsystem. Firstistheriskofinappropriateexecution.Anytaskthatrequiresmanualinterventionalso introducesthenotionthatitcouldbeexecutedataninappropriatetime.Or,more dangerously,suchataskcouldbereparameterizedtosenddatatothewronglocationor executeitinaninappropriateway.Thereisarecognizablecostassociatedwiththisrisk.
10
GregShields
Irememberasituationwhereascriptwascreatedthatwouldapplyasetofdatatoa specificserveruponexecution.Thatscripttookasparametersalistofserverstosendthe data.Oneday,ajunioradministratoraccidentallyinvokedthescriptwiththe*wildcard inplaceofalistofservers.Asaresult,datawasdistributedtoeveryserverallacrossthe company.Thatsingleinvocationcostthecompanysignificantlytocleanupthemess. Forgetfulnessandusererrorarebothadditionalrisksthatcanbeaddressedthroughajob schedulingsolution.Insuchasolution,jobsandplansarerunwithintheconfinesofthe systemanditssecuritymodel.Dangerousjobscanbespecificallyrestrictedagainstcertain individualsorexecutionmodels.Centralizingyourjobexecutionsecurityunderasingle modelprotectstheenvironmentagainstallthreeofthesecostlymanualerrors.
3:WhereCanYouGoforaHeadsUpDisplayofITActivities?
Youprobablyhavemonitoringinplacetowatchservers.Youveprobablygotsimilar monitoringfornetworkcomponents,perhapsevenaspartofthesameproduct.Butdoes yourdatacenteralsoleverageaunifiedheadsupdisplayformonitoringjobsalongwith theirexecutionsuccess?Afailureinajobcancausethesamekindsofoutagesandservice lossesasafailureinthenetworkoritsservers. Ifyourbusinesssystemsinterconnectthroughmultipleschedulingutilitiesacrossmultiple productsandplatforms,thereusuallyisntawaytocentralizeallthoseactivitiesunderone paneofglass.WhatyouneedisthesamekindofmonitoringforITjobsthatyouvealready gotinplaceforyourothercomponents. AsyoucanseeinFigure1.6smockup,youcangetthatbyusingacentralizedapproach. There,everyactionacrosseverysystemandapplicationiscentralizedintoasinglescreen. Determiningwhichjobsransuccessfullyisaccomplishedbylookinginoneplace.
Figure1.6:Dailyactivityunderonepaneofglass.
11
GregShields
4:HowDoYouManageCrossSystemCommunications?
Yourdatacenterenvironmentalreadyhasmultipleschedulingenginesinplacetoday. Nearlyeverymajorbusinessservicetechnologycomeswithitsownmechanismfor schedulingitsactivities.Infact,thosemechanismsarelikelyalreadyperformingasetof dutiesforyourservices. Yettheproblem,asyoucanseeinFigure1.7,hastodowiththelanguageseachofthese platformspecificandapplicationspecificschedulingtoolsspeaks.SQL,forexample,comes equippedwithawiderangeoftoolsformanipulatingSQLdataandSQLServersystems;but howricharethosetoolswhendataneedstoexitaSQLServerandenduponaUNIX mainframe?
SAP Scheduling Engine
SAP Server
Oracle Database
SQL Scheduling Engine Oracle Scheduling Engine
SQL Database
UNIX Scheduling Engine
UNIX Mainframe
Figure1.7:Multipleschedulingengines. Often,thenativetoolsarentsufficient,forcinganexternalsolutiontobridgethegap.That solutioncanbeintheformofindividuallittleautomationslikethescriptsthischapter startedwith.Ortheycanbewrappedunderneaththebannerofaholisticjobscheduling solution.Chapter4willdiscussthecapabilitiesyoullwanttolookforinthebestfit solution.
12
GregShields
5:AreYouConcernedAboutIdleTimeDuringaTask?
Consideringtheanswertoquestion4,someplatformandapplicationspecificscheduling toolsindeedincludelimitedcrossplatformsupport.Theirschedulingcapabilitiesmaybe abletofirejobsbasedonactionsorstatechanges. However,onestatechangethatisparticularlydifficulttomeasureacrossplatformsiswhen taskstaketoomuchtime.Taskidlinginastatebasedschedulingsystemcancausethe entireworkbookofplanstocometoahaltifnotproperlycompensatedfor.Essentially,this idletimerepresentswhenpartofwhenataskdoesnotcomplete,leavingthenextone waiting.
SQL Server Send Data
Oracle Process Data
SAP Present Data
Figure1.8:Unmanagedtaskidlingcankillanonautomatedworkflow. Idlingneednotnecessarilybeaproblemwithinapieceofcodeorscript.Itcanbesimply thewaitingthatisnaturalinsometypesofonsystemactivities.Forexample,notknowing whenapersonwillsubmitafileornotknowingwhenapieceofdataisreadyforthenext stepinitsprocessing. Theseidlestatesarenotoriouslydifficulttoplanforusingtimebasedschedulingalone. Withtimebasedscheduling,yourjobsarebuiltwithnointelligenceaboutchangesthat occurwithinasystem.Rather,theysimplyrunanactionatsomesetpointintime.Yourjob schedulingsolutionmustincludethelogicnecessarytoaddthatintelligence.Asyoulllearn inlaterchapters,thatintelligencecanoccurthrougheventbasedschedulingortrigger basedscheduling.Ineitherofthesecases,anonsystemeventortriggerrecognizeswhena changehasbeenmadeandinitiatesthenextstepinprocessing.
6:HowManyTasksExistinYourEnvironmentthatYouDontKnowAbout?
Ifyouhaventyetstandardizedonanenterprisejobschedulingsolution,canyouhonestly sayhowmanytasksareoperatingeverywhereinyourdatacenter?IusedtothinkIknew whereallofthemwereinthatformerjobofmine.ButthenIleft,andtookwithmethesum totalofthatknowledge.AsImentionedatthebeginningofthischapter,thoselittle automationsarestillbeingfoundyearslateroftenafteroneofthembreaksandcauses downtime.Moreimportantly,thoseareonlymyscripts.Therewereothersinthatcompany aswellwithscriptsoftheirownthatprobablyeventuallygotlost. Acentralizedjobschedulingsolutioncreatesasinglepointofcontrolforautomation.It enablesauditorsandITteamstoknowwherechangesarebeingsourcedfrom.Itis essentiallyasinglepointofcontrol,whichmakesauditors,securityofficers,andthe troubleshootingadministratorsveryhappy.
13
GregShields
7:CanYouMonitorITTasksAcrossEveryTeam,Platform,andApplication?
Ifyoucant,youhowcanyoucorrelateissuesacrossthoseteams,platforms,and applications?Ifyoucant,troubleshootingbecomesagameoffingerpointingandproving whynot. Iwasoncetoldastoryaboutacompanyinrealneedofanenterprisewidejobscheduling solution.TheirbusinesssystemwasmuchlikeTheProjectinthatitinvolvedmultiple technologiesacrosssomeverydifferentplatforms.LikeTheProject,managingthatsystem felltoasomewhatdistributedgroupofindividuals.SQLServerwasmanagedbytheSQL Serverteam.SAPwasadministeredbySAPadministrators.EventheADhaditsowngroup ofpeopleresponsibleforitsdailycareandfeeding. Theprobleminthiscompanywasnotnecessarilyitsapplicationspecificschedulingtools. Itwasinitspeople.Thosewidelydistributedpeoplefearedthecentralizationthatajob schedulingsolutionbrings.Thatfearinpartwasduetotheusualtechnologistsfearof centralization,butitwasalsoaresultoftheassumptionthatacentralizedtoolwouldmean recreatingSQL,SAP,AD,andotherjobsonanewandcompletelydifferentsystemoutside theirdirectcontrol. Aneffectiveenterprisejobschedulingsolutionshouldntrequirethecompleterecreation ofexistingjobswithineachplatformandapplication.Recallthatajobitselfrepresentsthe changethatistobemade,theindividualscriptorpackagethatmustbeexecuted.Ajob schedulingsolutionrepresentsthewrapperaroundthatinvokedaction. Thisstoryendsasyoudexpect,afteraverysmallbutverymajorprobleminone subsystemimpactedthesystemasawhole.Fullyunabletotrackdownthatminorjobwith amajorimpact,thecompanydiscoveredwhycentralizingisagoodidea.
8:BuildVersusBuy:IsaHomegrownSchedulerGoodEnough?
IoncebuiltmyownschedulingsysteminthenowancientscriptinglanguageofVBScript. VBScriptisstillinusemanyplaces,andithasalonghistory.Butitsknownfornothaving superiorbuiltinmethodsforschedulingactivities.Thatsaid,itsschedulerworkedfinefor thetaskIassignedtoit.ButthenexttimeIneededascheduler,Ifoundmyselfreinventing thewheel.EvenwiththelimitedcodemodularizationVBScriptcanpresentintoascript,my schedulersreusabilitywasverylimited. Imaginehavingtoreplicatethatschedulingacrossmultipleapplicationsandplatforms usingdifferentlanguagesandevenusingdifferentapproaches,bothobjectorientedand structured.Homegrownschedulersareindeedanacceptablewayofhandlingthetriggering needsofindividualscriptsandpackages;however,aglobalschedulerthatworksacrossall jobsandplansobviouslycreatesasuperiorframeworkforjobexecution.
14
GregShields
Moreimportantly,thehumanresourcesthatarenecessarytokeepahomegrownjob schedulercanbemuchgreaterthantheyseematfirstblush.Thoseresourcesneedtokeep aneyeonthelogicalcodeitselfalongsidethejobsthattheschedulerattemptstorun.In manycases,youllfindthattheextracostsassociatedwithcreatingyourownscheduler includethefactthatthisprojectwilltakeawayfromtimethatcouldbebetterservedby workingonothermorevalueorientedprojects.
9:WhatAreYourStepsforLinkingtheResultsofOneTaskwiththeActionsof Another?
Nowhereisthatvalueaddmorepertinentthanwhentwojobsrelyoneachother.Thiskind ofjobconstructionhappensallthetimewithindistributedsystems.Init,thefirsttaskina stringcompleteswithasetofdata.Thatdataisneededbythenexttaskinthestring.This sharingofinformationcanbehandledthroughfiledropboxesorrichermechanismslike MicrosoftMessageQueueordatabasetriggers. Likewiththeproblemofmultiplelanguages,thesequeuingsolutionstendtowardsbeing veryplatformcentric.Itbecomesverydifficultusingadatabasetriggertoinvokeanaction inAD,forexample.Acentralizedjobschedulingsolutionwithrichsupportforapplications willbecomethecentralpointofcontrolforallcrosstaskactionlinking.
10:HowDoYouHandleErrorsinaCustomCodedScript?
Lastisthehandlingoferrormessagesincustomcodedscripts,aprocessthatitself consumesavastquantityofscriptdevelopmenttime.Errorsarenotoriouslydifficultto trackdown,andbecomeevenmorechallengingwhenscriptsneedtospanplatformsand applications.Errorhandlingrequiresspecialskillsintrappingvariablesanddetermining theirintendedandactualvalues.Alloftheseactivitiesgrowevenmoredifficultwhen scriptsarerunautomaticallyasopposedtointeractivelybecauseerrormessagesinmany casescannotbecaptured.Chapter3willgointogreaterdetailontheerrorhandling functionalityofagoodjobscheduler.Fornow,recognizethatahomegrownscriptwithout errorhandlingisripefortroubleshootingheadachesdowntheroad.
DoYouNeedJobScheduling?
Sodoyouhavegoodanswerstothesequestions?Doyoufeelthatyourexistingscheduling toolsbringyouzeroheadaches?Ifyes,thenthanksforreading.Ifno,thenitislikelythat yournextthoughtswillbetowardthetypesofITchallengesthatanenterprisejob schedulingsolutioncansupport.Withthebasicsdiscussedandtheinitialquestions answered,thenextchapterintroducessevenrealworldusecasesforautomatingITjob scheduling.Itshouldbeaninterestingreadbecausethetypesofusecasesoutlinedinthat chapterareprobablyprettyclosetothoseyouvealreadygotundermanagementtoday.
15
GregShields
Chapter2:SevenUseCasesforAutomating ITJobScheduling
Findmethebusinessthatrunsatopasingleapplicationasingleinstance!andIllshow youabusinessthatdoesntneedITjobscheduling.Everyoneelseprobablydoes. Infact,mostdatacentershavefarmore.Youraveragemidsizedatacenterruns applicationsforhandlingitsdatabases,alongwithmiddlewaresystemsforprocessingthe data.Thatdatacenterrequiresserversandprotocolsforstagingofdatainandoutofthe organization.Applicationsrunatopclient/serveroperatingsystems(OSs)andmainframes, servers,andperhapsevenafewdesktops.Alloftheseelementsneedtocommunicatewith eachother,manydontsharethesameOS,andallsufferunderthemanagement complexitiesbroughtaboutbyproductspecifictoolsets. TodaysITtechnologiesarefantasticinthebusinessprocessestheyautomate,butrareare twothatseamlesslytalkwitheachother.RarerstillistheITproductthatissuperiorallby itselfincreatingandschedulingworkflowsthatmeetbusinessrequirements.Neededto integrateactivitiesamongdisparatetechnologiesisacentralsolutionthatcaninteractwith eachatonce. AnITjobschedulingsolutionisthatRosettaStonebetweendifferentplatforms,OSs,and applications.Itisintendedtobethedatacenterssolutionforconvertingrawtechnology intobusinessprocesses.Inthisbook,Ihopetoshowyouhowtoincorporatesucha solutionintoyourownbusiness. YouvealreadyexperiencedatasteofhowanITjobschedulingsolutionmightwork. Chapter1wasconstructedtohelpyourecognizethatjobschedulingisaserviceyourIT organizationprobablyneeds.Thatsaid,Chapter1sdiscussionintentionallystayedata highlevel.Youhaventyetexploreddeeplythefeaturesandcapabilitiessuchasolution mightbring. Youwontgetthatdeepdiveinthischaptereither.ThatsbecauseIvefoundthatthebest explanationofITjobschedulingrequiresfirstalookattheproblemsitintendstosolve. Onceyouunderstandwhereitfits,youllthenappreciatethelogicbehinditsbehaviors.Itis myhopethatbytheconclusionofChapter1youbegannoddingyourhead,affirmingthat thispurportedsolutionissomethingyourdatacenterdesperatelyneeds.
16
GregShields
SevenStoriesinITJobScheduling
Mytasknowistofurtherenlightenyouwithaseriesofideastohelpyoufindthatbestfit. Theseideaswilltakeplaceintheformofasevenusecases;essentially,sevenlittlestories aboutissuesthathavebeenresolvedormadeeasierthroughtheincorporationofajob schedulingsolution.Thesestoriesthemselveswillbemostlyfictitiousbutarebasedonreal eventsandrealproblems.Illusefauxnamestokeepthenarrativeinteresting. Theresanimportantpointhere.Evenifsomeportionofthesestoriesismadeup,you shouldfindthattheproblemsandsolutionsineacharentfarfromthoseyoure experiencing.
Story#1:FiveAdministratorsFiveScheduledTasksLibraries
Thefirstofthesestorieshasnothingtodowithacustomerfacingsolution.Neitherisit directlyrelatedtoalineofbusinessapplication.Rather,thefirstofthesestoriesstarts simple.ItexplainstheadministrativesituationatCompanyA,amaturecompanywitha procedurallyimmatureITorganization.Lackingmanycentralizedprocesses,operating withmarginallyeffectivechangeandconfigurationcontrol,andmanagedbyfivedifferent administrators,CompanyAsdatacenterisamishmashoffiefdomsandtechnologysilos. Problemis,thesefiefdomsneedtocommunicatewitheachother,eveniftheirmanagingIT administratorswont. John,Bob,Jane,Sara,andJimarethosefiveITadministrators(seeFigure2.1).Eachis responsibleforsomeportionofthedatacenterinfrastructure,witheachhavingsome overlapofresponsibilities.Toaccomplishadministration,theyvecreatedscripts,tasks, andpackagesthatkeeptheindividualbusinessworkflowsrunning.Thoseautomations indeedenactchangeonserversandgetdatamovedfromsystemtosystembutwithno interconnectionofintelligence.
Figure2.1:Aninterconnectionofautomations.
17
GregShields
Figure2.1explainsthisproblemingraphicalform.Init,youcanseethatindividual automationsaresourcedwithoutconsideringtheircontext.IfJohncreatesajob,his informationcannotbebasedoninstrumentationthatisgainedthroughanothercreatedby Sara.Asaresult,thereisnowaytoorchestratetheactivitiesbetweeneachindividual,no waytoscheduleactivitiessothattheydonotconflict,andnowaytobaseinformationor schedulingfromoneoffoftheresultsofanother. Amuchbettersolutionisinaggregatingthesefivepeoplesautomationsintoasingleand centralizedsolution.Throughthatsinglesolution,eachadministratorsjobscanbeseenby theothers.Thejobsofeachpersoncanalsobealignedwiththeneedsoftheothersto ensureresourcesarentoversubscribed.Additionally,becausejobsarecollocatedina singlelocation,informationandinstrumentationfromanyautomationcanbeusedtodrive otherautomationsorfeedintotheirfuturescheduling.
John Bob
IT Job Scheduling Solution
Jane Sara
Figure2.2:SourcingautomationsthroughanITjobschedulingsolution. Inshort,evenifyourautomationsareadministrativeinnature,anITjobscheduling solutioncanbringsubstantialbenefit.
Jim
Story#2:ConsolidatingTasksfromEveryApplication
YetanITjobschedulingsolutionisntsolelyaboutitsactors.Infact,inmanyways,the actorscanbeoneofthatsolutionsleastimportantimpacts.AnITjobschedulingsolution reallyhasmoredowiththedatainadatacenter.Thatswhythesecondstoryinthis chapterdealswiththedifferentapplicationsthatareusedbyCompanyB.
18
GregShields
DifferentfromtheITadministrationexampletoldinthepreviousstory,CompanyBsstory centersaroundtheirlineofbusiness(LOB)application.ThatLOBapplicationiscomprised ofseveralcomponents,eachofwhichisrepresentedinFigure2.3.Transactionsamong thesesystemsoccurthroughacarefullychoreographedsetoftasks,jobs,packages,and workflows.Asyoucanprobablyimagine,thesysteminaggregatecrossesWindowsand UNIXboundaries,andincludesmultipledatabasemanagementsystemsandevenabitof middleware.Itistheclassicbusinessservice.
Figure2.3:Individualschedulersforeachindividualapplication. AlloftheseindividualcomponentsenablethefunctionalityoftheLOBapplication.Butall alsoleveragetheirownbuiltintoolsetsforschedulingactivities:TheSQLserverrunsits SSISpackages,theLinuxSAPserverrunsitsowntasksandcronjobs,andeventhe Informaticaserverenactschangethroughitsworkflows. Youarecorrectinassumingthatthisoneschedulerpercomponentconfigurationcan indeedworkformanysystems.DataandactionsthatoccurinsideInformaticacanbebased onwallclocktimeorotherschedulecharacteristics.TheSQLdatabasecanrunitsSSIS packagesbasedonitsownsettings,andsoon.However,liketheactorsinthefirststory, thisenvironmentislikelytoexperienceproblemsasindividualsystemactivitiesconflict withthoseonothersystems. Contrastthissituationtothesuperiorityindesignonegetsthroughjobconsolidation.In Figure2.4,theindividualtaskschedulersinthatsameLOBapplicationhavebeenreplaced byasingleandcentralizedITjobschedulingsolution.Thisispossiblebecause,asI mentionedinChapter1,aprimarybenefitofsuchasolutionisitsabilitytospeakthe languageofeveryapplicationinthebusinessservice.
19
GregShields
At Job, Scheduled Task Cron Job, SAP Task SSIS Package Oracle Job Informatica Workflow
Figure2.4:Consolidatingtasksacrossapplications. Withthatcentralizationofdataandactionscomesanenhancementtojobscheduling, basedonresultsordatainotherjobs.Thischapterssixthstorywillexplainingreater detailhowtriggeringcapabilitydramaticallyimprovesserviceperformance;butknowhere thatcentralizationofschedulingbringstobeargreaterinstrumentationaboutthehealthof jobsacrosstheentireapplicationinfrastructure.
Story#3:TheScriptthatWasntaJob
JohnisanOracleDBAwhohasbeenwithCompanyCsincetheverybeginning.Inhisroleas databaseadministrator,hebuiltthecompanysbusinesssysteminfrastructurenearlyfrom thegroundup.Asaresult,heunderstandsthosesystemsinsideandout:Hehastunedthe systemovertimetoimproveitsperformanceandweedoutnonoptimizations.Hesbuilt numerousscriptsandotherautomationsthatgatherdata,translateit,processitbetween applicationcomponents,andpresentreportsforreviewbystakeholders.Thatsystemis criticaltothecompany.Itprovidesimportantrevenuedataforitssalesteamsand executives.ItalsomeansalottoJohn. Thenonedaythecompanygrew.Substantially.Overnight.Acquiringacompletelynewline ofbusiness,CompanyCsuddenlyfounditsinternalITsystemsinsufficienttohandlethe newrealityofworkanditsassociateddataneeds. Johnwasapproachedshortlyafterthemergerbysomeveryimportantpeopleinthenow largercompany.Thosepeoplerecognizedhisstrengthsincreatingandmanagingthe originalrevenuesystemthatbroughtmuchvaluetothesmallercompany.Theywanted anothersystem,justlikethefirstone,butthistimeforsellingsprocketsinsteadofcogs.
20
GregShields
Graciouslyacceptingtheoffertoimprovethecompanyandfortifyhisresume,John immediatelyrealizedthatsimplyreplicatingtheoriginalsystemwouldnotbeatrivialtask. Althoughhisscriptsabsolutelydideverythingrequestedofthemintheoriginalsystem, theywerealsohardcodedintothatoriginalsystem.Itsdatabasearchitecturewasdesigned todealwithcogs.Histransformationswerecogbasedinnature.Eventheservernames andscriptnameswerehardcodedintoeachindividualscript,task,andpackage.Worse yet,therewerehundredsoftinyautomationsspreadeverywhere. Translatingevenasimpledatabasejobfromcogstosprockets,likewhatyousee graphicallyinFigure2.5,wouldtakemonthsofdetectivework,recoding,andregression testing.Johnwasinforagreatdealofwork,andtheresultmightnotbeasseamlessly valuableashisoriginalsystem.
Figure2.5:Replicatinganautomationtoacompletelynewsystem. HadJohnsscripts,tasks,andpackagesinsteadbeenobjectswithinacentraljob managementsystem(seeFigure2.6),thisnewcompanyneedmightnotbefraughtwithso muchrisk.
Original System
Additional System
Figure2.6:Objectsremainobjectsastheyretranslatedintoanewsystem.
21
GregShields
RecallChapter1sconversationabouthowgoodITjobsarethosethatarecodedfor reusability.ITplansarethenconstructedoutofindividualjobobjectstoenactchange.In goodjobs,variablesareusedtoabstractthingslikeservernamesandscriptnames sprocketsandcogs,ifyouwillsothatentireplanscanberebaselinedtonewsystems withaminimumofdetectivework,recoding,andregressiontesting.AnITjobscheduling solutiontakestheriskoutofbusinessexpansion,givingITtheflexibilitytoaugment servicesasthebusinessneeds.
Story#4:WhenGatheringDataIsMorethanGatheringData
CompanyDscurrentsituationisaproductofitsownsuccess.Startingasasmall organizationwithasinglemission,theirfocushaschangedandevolvedovertheyearsas linesofbusinesscomeandgo.Indeed,evenwholebusinesseshavebeengraftedonand laterspunoffasthewindsshiftinCompanyDsindustry. Asaresult,CompanyDsuffersundermanyoftheproblemsyouwouldassociatewithany classicenterprisecompany.Ithasthousandsofapplicationsundermanagement,someof whichareusedbyonlyaveryfewpeople.Somehomegrownsolutionsonlyremainbecause theywerecodedyearsagotosolveaspecificproblemforaspecificneedthathasnot changed. Beingacompanythatismorelikethesummationoflotsoflittlecompanies,thesebusiness applicationsarenowherenearhomogeneous.Onebudgetingapplicationmightstoreits datainSQL,anotherinOracle,athirdinsomeobscuredatabaselanguagespokenonlyby ITprofessionalslongpastretirement.Tyingtheseapplicationsandtheirdatatogetherisa bigjobwithbigconsequencestotheorganization. ManyenterpriseorganizationsthatrelyondisparatedatabasesleverageBusiness Intelligence(BI)solutionslikeCrystalReports.TheseBIsolutionsaggregateinformation acrossthedifferentarchitectures.UsingBItoolslikeCrystalReports,datainanOracle formatcanbecomparedandcalculatedagainstdatainaSQLformat,andsoon.Thesetools comeequippedwithrichintegrations,enablingthemtointerconnectnearlyalldatabase formatsallatonce(seeFigure2.7).CompanyDusesCrystalReportstogatherbudgetary dataacrossbusinessunitsandindividualprojectteams.
22
GregShields
Figure2.7:ConnectingsolutionslikeCrystalReportstomultipledatabases. YetFigure2.7doesntfullyshowtherealityofhowCompanyDsdataisgenerated.Before thatdataeverbecomessomethingtangiblethatcanbeingestedintoaBIsolution,itstarts itslifeinsideanynumberofdownlevelsystems.Figure2.8showsjustafewofthose underlyingsystems,allofwhichintegratetocreatethekindofdataaBIsolutiondesiresto manipulate.
Crystal Reports
Oracle
SQL
Other Database
External Company
Figure2.8:UnderlyingjobsmakeBusinessIntelligencedatausable.
E-Commerce Server
23
GregShields
NoticeinFigure2.8howaportionofthefirstbusinessunitsinformationcomesfroma partnercompanyexternaltothedatacenter.Thesecondandthirdbusinessunitshave projectsincombinationthatrequireorchestrationandsynchronizationbetween databases.Thesecondbusinessunithasfurtherintegrationsintoanecommerceserverin ordertogatherafullpictureofbudgetlevels. BIsolutionscanindeedpresentamoreunifiedviewofdataacrossdifferentplatforms,but theydonotprovideamechanismtounifytransactionsbetweendownlevelsystems.For CompanyD,creatingthatunifiedworkflowlieswithintherealmofanenterpriseITjob schedulingsystem.Ifgatheringdataismorecomplicatedthansimplygatheringdata,ajob schedulingsystemtiestogethertheentiresystemtoaccomplishwhatyoureallyneed.
Story#5:ControllingDataTransferasaWorkflow
CompanyEhadabigproblemnotlongagowhentheybeganextendingcustomerservices ontotheInternet.Theyquicklyfoundoutthatyoucanindeedinteractwithcustomers there,butcreatingaholisticsystemthatgetscustomerdatainallitsvarioustypesintothe handsoftherightpersonisntaseasyasitlooks. WhatCompanyEfoundoutisthatdealingwithcustomersovertheInternetautomatically createsalotofdata.Thatdataarrivesinvariousformats,witheachformatrequiringa differentmechanismforhandling. YoucanseeagraphicalrepresentationofCompanyEsstoryinFigure2.9.Internet customersthatdesiredservicesinteractedprimarilythroughaWebserver.Insidethat Webserverwascontainedtherequisitelogictoinformcustomersaboutproducts,and interactwiththemaspurchasesaremade.InterestingaboutCompanyEisthattheir systeminvolvedtwowaycommunicationwithcustomersnotonlyviatheWebsiteand emailbutalsointhetransferofdatafiles.
24
GregShields
Figure2.9:Differentdataformatsrequiredifferentdatahandling. Datafiles,asyouknow,aremuchdifferentthanXMLfilesinWebtransactionsoremails backandforththroughanemailserver.Theyrelarger,theycancomeinmanydifferent formatswhichcreateparticularissueswhenyoureworkingwithunstructuredcustomers, andthemanagementtoolstoworkwiththemdontnecessarilyintegratewellintoother formatsandworkflows. CompanyEneededamultipartmechanismtosolvetheirformattingworkflowproblem. Theyneededtorecognizewhenanorderwasplaced,generateanFTPURLfortheuserto uploaddata,movethatuploadeddatafromalowsecurityFTPservertohighsecurity databaseserver,andfinallynotifytheuserwhenthetransactionwascomplete.Addingto thecomplexity,thoseexactsamestepswererequiredinreverseatthetimetheorderwas fulfilled.
25
GregShields
Youcanimaginetheprotocolsandfileformatsatplayhere:XML,SMTP,FTPandSFTP, alongwithalittleSSHandSOAPtotiethepiecestogether,justlikeyouseeinFigure2.10. ComplexneedslikeCompanyEsrequiredatatransferhandlingthatcansupportthe recognitionthatfileshavebeendownloaded.Suchhandlingcaneithermonitorforafiles presenceoruseeventormessagebasednotification.AnITjobschedulingsolutionwraps filetransferlogicintothelargerworkflow,enablingXMLtotriggerSSH,tofireoffSMTP, andfinallytoinvokeSOAPatthepointtheapplicationrequires.
Figure2.10:Multipleprotocolsatplay,eachwithitsownmanagement. Forcomplexfiletransferneeds,thosethatmustexistwithinabusinessworkflow,ITjob schedulingsolutionssolvetheproblemwithoutresortingtolowleveldevelopment.
Story#6:ComplexJobsandtheNeedforTriggers
ThischapterssixthstorybringsmebacktotheoneIstartedinChapter1.ThereI explainedsomeoftheintroductorypiecesinTheProjectThatWouldChangeEverything.I alsoexplainedhowanITjobschedulingsystemwasveryquicklyidentifiedastheonly classofsolutionthatcouldenablethekindoffunctionalityourcomplexprojectneeded.Let metellyoualittlemoreinthatstory.
26
GregShields
Ourdeterminationhappenedshortlyafterwhiteboardingthevariouscomponentswe knewtheprojectneeded.Figure2.11showsareenactmentofthatwhiteboardingactivity, withonlyafewofitsmyriadinterconnectingmagicmarkerlinesinplace.Myteamknew thatwhenexternalcompanydataarrivedontheFTPserver,transferringthatdatatothe SQLservermustoccurinasclosetorealtimeaspossible.
Figure2.11:WhiteboardingthetriggersforTheProject. FulfillingthatrequirementwithtraditionalFTPalonecreatedafantasticallyproblematic design.TheideaofconstructinganalwaysonFTPsessionthatconstantlysweptfornew datawasaridiculousnotion.Anditwasntagoodideaforsecurity.Neededwassomekind ofagent(or,betteryet,anagentlesssolution)thatwouldsimplyknowwhendataarrived. ThenitcouldprovisionthatdatafromtheFTPserversdatastoragetotheSQLdatabase. Butthatwasntouronlychallenge.Atthesametime,ourSQLandOracledatabasesneeded toremaininstrongsynchronization.ChangestospecificvaluesinSQLmustreplicateto Oracle,alsoinasrealtimeascomputationallypossible.SynchronizingOraclewithSQL meantalsosynchronizingmetadatawithSAP.
27
GregShields
Evenasingleoneofthesenearrealtimerequirementscanbechallengingforadeveloper tobuild.Lackingdevelopers,adevelopmentbudget,andthedesiretocompletethisproject usingofftheshelfcomponents,wedemandedasolutionthatwouldaccomplishthetask withoutrevertingtolowlevelcoding. Whatweneededweretriggers. TriggersaretherealjuiceinanITjobschedulingsolution.Thekindsandcapabilitiesof triggersajobschedulingsolutionsupportsmakesthedeterminationbetweenonethats enterprisereadyandonethatsnotmuchmorethantheWindowsTaskScheduler. TheProjectrequiredawiderangeofthesetriggerstogetthejobdone:OurprojectsFTP toSQLintegrationrequiredafilebasedtrigger,kickingoffajoborplanwhenafile appearedattheFTPserver.MessagebasedtriggerswerealsonecessaryfortheSQLto Oracleintegration,enablingthetwoapplicationstonotifyeachotherabout synchronizationactivities.EventtriggerswerenecessaryintheOracletoSAPintegration, allowingOracletocreateeventsaboutchangestoitsstateandalertingSAPtomake associatedchangesbasedoneventcharacteristics.ThosesameeventtriggersgaveActive Directory(AD)thedatatoquicklytagpermissionsintothedata.Finally,timebasedtriggers kickedoffoccasionaldatatransfersbetweentheSQLdatabaseandtheUNIXmainframe. Wildly,simpletriggersalonewerentsufficient.Allbyitselfeventhebesttriggercouldnt fulfillthemultiserverandmultiactionrealtimerequirementoursystemdemanded.We alsoneededtheabilitytotetherorchainindividualtriggerstogether.Bychaining triggers,wecouldspeedtheprocess,getdatawhereneeded,andensurethesystem remainedinconvergence.Illtalkmoreaboutchainingtriggersinthenextchapter. PaycarefulattentiontothetriggeringcapabilityofyourselectedITjobschedulingsolution. Theverybestwillcomewiththerichestsuiteoftriggeringabilities.
Story#7:SecuringtheDataCenterfromtheEaseofScriptExecution
CompanyFwasamidsizecompanywithamidsizeITorganization.Responsibleforallthe taskstypicallyassociatedwithIT,itsITteamgotalongwellandgenerallyprovidedgood servicetothecompanyatlarge. ButeventhemostwellmeaningofITorganizationsoccasionallymakesmistakes,and sometimesthosemistakesarelargeinimpact.Thatsthesituationthatoccurredoneday aftertwoadministratorsbegansharingtheirquiverofscripts,tasks,andpackages. Youalreadyknowthatoneoftheprimarybenefitsofmovingtoscriptbasedautomationis consistencyinreuse.Bypackagingaseriesoftasksintoascript,thatscriptcanbeexecuted overandoverwithaknownresult.Parameterizingthosesamescriptsmakesthemeven morevaluabletoIToperations.Oncecreated,thesamescriptcanbeusedoverandover againacrossarangeofdifferentneeds.Yetreusabilitycansometimesbeariskaswell. Whenapersoncansimplydoubleclickascripttoactivateit,thechancepresentsitselfthat thatdoubleclickmighthappeninappropriately.
28
GregShields
ThatsexactlywhatwasintendedthedaySaraborrowedoneofJanesscriptsforusein anothersystem.Janesscriptswerebrilliantlydesigned,smartlyparameterized,andwell documented.Builtrightintoeachscriptwasallthenecessaryinformationanother administratorwouldneedtoreusethescriptelsewhere.ForSara,Janesscriptperfectly solvedherproblemathand. Thecentralproblem,however,wasthatSarawasntreallyauthorizedtorunJanesscript. Infact,thewellmeaningSarawasntsupposedtobeworkingonthesystematall.When sheexecutedJanesscript,itbroughtthesystemdownunexpectedly.CompanyFlearneda valuablelessonthatdayintheopennessofsimplescripts. ThatswhyshortlythereafterCompanyFinvestedinanITjobschedulingsolutionfor aggregatingtheirautomationsintoaunifiedstore.Unifyingautomationswithina restrictedrunframeworkenabledCompanyFtoapplyprivilegestotheirscripts.Because theychoseabestinclassITjobschedulingsolution,theywereabletonotonlyapply privilegesonthescriptsthemselvesbutalsothejobs,plans,andevenvariablesassociated withthosescripts.HavingcorrectpermissionsinplacereducestheriskthataSarawill inappropriatelyexecuteascript.But,moreimportantly,italsoreducestheriskthataSara typeworkerwillinappropriatelyattachthewrongvariablestotherightscript,ortheright variablestothewrongscript.
Figure2.12:Applyingsecurityatvariouslevelsinscriptexecution. ConsolidatingITautomationsintoajobschedulingframeworkprovidesvisualizationof scriptswithintheenterprise.Itaddssecuritytowhatmightotherwisebehighlydangerous textfiles.Itcreatesalocationwheretheirsuccessfulexecutioncanbeprovento administratorsaswellasauditors.Anditcreatesanauditableenvironmentofapproved executionthatprotectsthedatacenter.
29
GregShields
JobSchedulingsStoriesAreitsValueAdd
ThesesevenstoriesaretoldtohelpyouunderstandthevalueaddtoanITjobscheduling solution.Inthem,youvelearnedhowITjobschedulingworksforadministrationaswellas complextasksthatmightotherwiseberelegatedtolowleveldevelopers.Youve discoveredhowtriggersandfilemanipulationsareasimportantasdatabasetasksand middlewareactions.Youvealsolearnedhowjobschedulingcreatesthatframeworkfor approvedexecutionthatyourauditorsand,indeed,yourentirebusinesstruly appreciate. YetasImentionedasthischapterbegan,Istillhaventdugdeeplyintotheinnerworkings ofjobsthemselves.Nowthatyouvegainedanappreciationforwherejobscheduling benefitsthedatacenter,letsspendsometimediscussingatechnicaldeconstructionofan ITworkflow.Attheconclusionofthenextchapter,youllgainanevengreaterappreciation forhowthesejobobjectsandtheirplansfitperfectlyintotheneedsofyourbusiness services.
30
GregShields
Chapter3:WhatMakesanITWorkflow?A TechnicalDeconstruction
Computersareusefulbecausetheyllperformanactivityoverandoverwithoutfail.Theartis intellingthemexactlywhattodo. IhopethatreadingChapter2wasasenjoyableaswritingit.AlthoughIlladmitItooka littleliterarylicenseintellingitsstories,IdidsotohighlighttheusecaseswhereITjob schedulingmakesperfectsense.Coordinatingadministratoractivities,consolidatingtasks, generalizingworkflows,gatheringdata,orchestratingitstransfer,triggering,andsecurity areallimportantfacetsofregulardatacenteradministration.Yettoooftenthesefacetsare administeredusingapproachesthatdontscale,introducethepotentialforerror,orcant belinkedwithotheractivities.TheultimatedesireineachofChapter2sstorieswasthe creationofworkflow.Thatworkflowabsolutelyinvolvedeachstorysactors;but,more importantly,itinvolvedtheappropriatehandlingofthoseactorsdata. Inmanyways,workflows,jobs,andplansrepresentdifferentfacetsofthesamedesire: Tellingacomputerwhattodo.Youcanconsiderthemthelogicalrepresentationsofthe littleautomationpackagesIreferencedinthefirsttwochapters.AlthoughIspentmuchof thosechaptersexplainingwhytheyregoodforyourdatacenterandhowtheyllbenefit yourdistributedapplications,Ihaventyetshownyouwhattheymightlooklike. Thatswhatyoullseeinthischapter.Init,youllgetanunderstandingofhowaworkflow quantifiesanITactivity.YoullalsowalkthroughasetofmockupsfromamodelITjob schedulingsolution.Thosemockupsandthestorythatgoeswiththemisintendedto solidifyyourunderstandingofhowanITjobschedulingsolutionmightlookoncedeployed. Butfornow,letsstayatahighlevelforjustabitmore.Indoingso,Iwanttoexplainhow workflowsbringquantificationtoITactivities.
AWorkflowIsanITActivity,Quantified
Aworkflowhasbeendescribedasasequenceofconnectedsteps.Moreimportantly,a workflowrepresentsanabstractionofrealwork.Itisamodelthatdefineshowdatagets processed,whereitgoes,andwhatactionsneedtobeaccomplishedwiththatdata throughoutitslifecycle. Youcanfindworkflowseverywhereinbusiness,andnotallaretechnicalinnature.Think aboutthelasttimeyoutookadayofffromwork.Youknowthattakingthatdayoffrequires firstsubmittingarequest.Thatrequestrequiresapproval.Onceapproved,younotify teammatesandcoworkersofyourimpendingunavailability.Intheworldofpaidtimeoff, youcantjustmissadaywithoutfollowingthatprocess.
31
GregShields
Andyetsometimespeopledojustmissdays.Perhapstheywereverysick,orgotstuckon thesideoftheroadfarfromcellphoneservice.Inanyofthesecases,theworkflowbreaks downbecausetheprocessisntfollowed.Whatresultsisconfusionaboutthepersons whereabouts,andextraeffortinfiguringoutwhattheywereresponsibleforaccomplishing duringtheirabsence. YoucancomparethispeopleworkflowtothedataworkflowsinanITsystem.Datain anITsystemneedstobehandledappropriately.Actionsonitmustbescheduledwith precision.Datamustbetransferredbetweensystemsinatimelymanner.Failurestatesin processingneedtobeunderstoodandhandled.Theresultinanysituationisasystem wheredataandactionscanbeplannedon. Tothatend,letsexplorefurthertheITplanfirstintroducedbackinChapter1.Figure3.1 givesyouareproductionofthegraphicyousawbackinthatchapter.There,youcansee howthreejobshavebeengatheredtogethertocreatePlan7SendDataSomewhere.
Figure3.1:AnexampleITplan. Iwontexplainagainwhatthisplanintendstodo;theactivitiesshouldbeselfexplanatory. MoreimportantistherecognitionthatthisexampleshowshowanITworkflowquantifies anactivityalongasetofaxes:capturability,monitorabilityandmeasurability,repeatability andreusability,andfinallysecurity.Letsexploreeach.
Capturable
Ifindmyselfoftenrepeatingthestatement,Alwaysremember,computersare deterministic!Giventhesameinputandprocessinginstructions,theywillalwaysproduce thesameresult.Yetevenwiththisassertion,whydotheysometimesnotproducetheresult werelookingfor? Thatproblemoftencentersonhowwelltheestablishedworkflowcapturesthe environmentspotentialstates.Awelldesignedworkflow(andthesolutionusedtocreate it)musthavetheabilitytocaptureasystemsstatesandsubsequentlydosomethingbased onwhatitsees.
32
GregShields
Irecentlyheardastorythatperfectlyhighlightsthisneedforcapturability.Inthatstory,a companyrannumerousmissioncriticaldatabasesacrossmorethanonedatabase platform.Mostofthesedatabaseswerepartofhomegrownapplicationsthatthecompany hadcreatedovertime. BackingupthesedatabaseswasaregularchorefortheITdepartment.Althoughthe companysbackupsolutioncouldindeedcompletebackupswithlittleadministratorinput, theconfigurationofmanydatabasesrequiredmanualstepsforbackupstocomplete correctly.Duetosimplehumanerror,thosemanualstepssometimeswerentcompleted correctly.Withmorethan25databasestomanipulate,thathumanerrorbecamethe biggestriskinthesystem.Fixingtheproblemwasaccomplishedbyimplementinga solutionthatcouldcapturethemanualportionsoftheactivityintoanITjob. SuchcaptureisonlypossiblewhenanITjobschedulingsolutionisrichlyinstrumented. Thatsolutionmustincludethenecessaryvisionintobackupsolutions,databasesolutions, andevencustomcodebases.Visionintoeverysystemcomponentmeansknowingwhenthe taskneedsaccomplishing.
MonitorableandMeasurable
Youcantcapturesomethingunlessyoucanmonitorandmeasureit.Justasimportantas visibilityintoasystemisvisibilityintotheworkflowsurroundingthatsystem.Aneffective ITjobschedulingsolutionmustbeabletoinstrumentitsownactivitiessothatthejobitself canrecoverfromanyfailurestates. ThisisofparticularimportancebecausemostITjobsdontoperateinteractively.Once created,tested,andsetintoproduction,atypicalITjobisexpectedtoaccomplishitstasks withoutfurtherassistance.Thisautonomymeansthatwelldesignedjobsmustinclude monitoringandmeasurementcomponentstoknowwhendataoractionsaredifferentfrom expectedvalues. ItseasiesttounderstandthisrequirementbylookingatthesimpleITplaninFigure3.1. Suchaworkflowisonlyusefulwhenitsactivitiesaremeasurable.Moreimportant, measurementofaplanslogicmustoccuratmultiplepointsthroughouttheplans execution.Figure3.2showshowthisbuiltinvalidationcanbetaggedtoeachphaseofthe plansexecution.Init,youseehowthehandoffbetweenJob27andJob19requires measuringthesuccessofthefirstjob.IfJob27cannotsuccessfullyconnecttothedatabase, thencontinuingtheplanwillbeunsuccessfulatbestanddamagingatworst.Youdont wantbaddatabeingeventuallysentviaFTPtoaremotelocation.
33
GregShields
Figure3.2:Validationlogicensuresmeasurability. SimilarmeasurementsmustoccurinthehandoffbetweenJob19andJob42andagainat plancompletion.AsuccessfulITjobschedulingsolutionwillcreatetheworkbenchwhere validationlogiclikethatshowninFigure3.2canbetaggedthroughoutanITplan.This logicshouldnotimpacttheexecutionofindividualjobs,norisitnecessarilypartof whatevercoderunsbeneaththejobobject.Effectivesolutionsimplementvalidationlogic insuchawaytobetransparenttotheexecutionofthejobitself.
RepeatableandReusable
Transparencyofmeasurementalongwithparameterizationofjobobjectscombineto createarepeatableandreusablesolution.Youcanimaginethatcreatingawell instrumentedITplanlikeFigure3.2isgoingtotakesomeeffort.Onceexpended,thateffort gainsextravaluewhenitcanbereusedelsewhere. ReusabilitycomesintoplaynotonlywithineachITjobbutalsowithineachplan.Recallmy assertionbackinChapter1thatanITjobisanactiontobeexecuted.Thisdefinition meansthattheboundaryofanITjobmustremainwiththeexecutionofanaction.Figure 3.3showsagraphicalrepresentationofanIntegratedJobsLibrary.Inthatlibrary,isa collectionofpreviouslycreatedjobs:Job17updatesadatabaserow,Job27opensa connectiontoadatabase,andsoon.
34
GregShields
Figure3.3:ReusingITjobsinaplan;reusingITplansinaworkflow. Eachofthosediscretejobscanbeassignedtoaworkflowforthepurposesof accomplishingsometask.Theycanalsobestrungtogetherininfinitecombinationsto createamorepowerfulITplan.YoucanseeanexampleofthisinFigure3.3.Noticehow Job27representsthebeginningstepofPlan15;italsorepresentsamiddlestepforPlan 22. Oncecreated,bothjobsandplansresideinanITjobschedulingsolutionsIntegratedJobs Library.Fromthere,createdjobscanbereusedrepeatedlyassimilartasksarerequired.In Figure3.3,twonewdatabasesrequiresynchronization.Sinceaplanhasalreadybeen createdtoaccomplishthistask,reusingthatplanelsewherecanbeassimpleasadragand drop.Afterdraggingtocreateanewinstanceoftheplan,theonlyremainingactivities involvepopulatingthatplanwithnewservercharacteristics.
Securable
Chapter2introducedthenotionofjobsecurity.Intheseventhstory,youreadhow individualjobsandentireplanscanbeassignedsecuritycontrolstopreventmisuse.That levelofsecurityisindeedanimportantpartofanyITsystem;however,Chapter2only begantheconversation. ConsiderthesituationwhereanITplanupdatesdatainadatabase.Correctlyconstructing thisITplanrequiresparameterizingtheplantoeliminatespecificrowvaluesoritemsof datatoupdate.However,parameterizingtheplaninthiswayintroducesthepossibility thatsomeonecouldaccidentally(ormaliciously)reassigntheplanandupdatethewrong data.
35
GregShields
ThisriskhighlightswhydeeplevelsecurityisfundamentallyimportanttoanITjob schedulingsolution.Youwantcontrolsinplacetoprotectsomeonefrominvokingaplan inappropriately.Butyoualsowantcontrolsinplacetoprotectcertaininstancesofor triggersforthatplantobeexecuted.EachplatformandapplicationtiedintoyourITjob schedulingsolutionhasitsownsecuritymodel,asdoesthejobschedulingsolutionitself. Mappingthesetwolayerstogetheriswhatenablesajobschedulingsolutionto,for example,applyActiveDirectory(AD)securityprinciplestosomeapplicationwithanon Windowssecuritymodel.DoingsoenablesyoutoleanonyourexistingADinfrastructure forthepurposeofassigningrightsandprivilegesinotherplatformsandapplications. Figure3.4showshowsuchanextendedaccesscontrollist(ACL)mightlook,withtriggers, triggercharacteristics,andeveninstancesofsuchaplanbeingindividuallysecurable.
Figure3.4:Applyingdeepsecuritytoajoborplan.
AGroundUpITWorkflowConstruction
Atitscore,anITworkflowisstillapieceofcode.Somekindsofcodeasolutionsvendor willcreateandincludewithinajobschedulingsolution.Theserepresentthebuiltinjob objectsinyoursolutionsIntegratedJobsLibrary.Othercodemustbecustomcreatedby theadministratorswhousethatsolution.Novendorcancreateobjectsforeverysituation, sosometimesyoullbeauthoringyourown.Notwithstandingwhocreatesthecode,atthe endoftheday,itisthatcodethatneedstobescheduledforexecution. Withthisinmind,letswalkthroughanextendedexampleofconstructingaworkflowout ofindividualparts.YoucanassumeinthisexamplethatanITjobschedulingsolutionhas beenimplementedandwillbeusedtoauthortheworkflow.
36
GregShields
AdiagramofthatworkflowisshowninFigure3.5.Init,eachblockrepresentsanactivityto bescheduled.Itsstorygoeslikethis:Datainasystemneedstobemonitoredforchanges. Aschangesoccur,anITplanmustbeinvokedtogatherthechanges,runscriptsagainstthe data,andmoveitaroundthroughfilecopyandFTPtransfers.Whilealltheseprocesses occur,individualjobswithintheworkflowmusttriggereachotherforexecutionaswellas monitorforserviceavailability.

Query Web Services
Select Oracle Data
Process Script A
Monitor Service
File Copy Data
FTP Data
Monitor for File Presence
Process Script A
Figure3.5:Anexampleworkflow. Youshouldimmediatelynoticethatschedulingisanimportantcomponentofthis workflow.Thatschedulingisntaccomplishedthroughsomeclockonthewallapproach.It isinsteadbasedonmonitoringthestatespresentwithinthesystem(presenceoffiles,WMI queries,logfilechanges,andsoon),andfiringsubsequentactionsbasedonchangesin thosestates.ThisintraworkflowtriggeringisthefoundationofITjobscheduling.Withoutit, schedulingjobsislittlemorethanafunctionoftimeanddate.Aworkflowlikethisrequires amuchfasterresponse,onethatmovesfromsteptostepbasedontheresultsofthejust completedstep.Youonlygetthatthroughtriggering.
37
GregShields
ExplainingFigure3.5sworkflowbeginsatitssecondstepwiththecreationofanOracle PL/SQLjobobject.ThisjobobjectisnecessarytorunaqueryagainsttheworkflowsOracle database.Thisobjectanditsunderlyingquerystringshouldalreadybeacomponentof yourITjobschedulingsolution.Asaresult,creatingthatjobprobablystartsbyclickingand draggingarepresentativeSQLblock(anexampleofwhichisshowninFigure3.6)froma paletteofoptionsintotheplandesignersworkspace.
Figure3.6:OraclePL/SQLobject. Onceaddedtotheworkspace,specificsaboutthisjobobjectsusewillthenbeaddedinto theSQLblockspropertiesscreen.InFigure3.6,youseehowaSELECTstatementiscreated toconnecttoanOracledatabaseandgatherdata.Youshouldalsonoticehowavariable ($DATA_SOURCE)isusedinthiscasetomaintainthereusabilityofthejobobject. ConstructingthatOracleobjectisonlythefirststep.Bydefinition,thereisnologicinitto definewhenitshouldbeinvoked.Accomplishingthisrequirescreatingoneormore conditionalstatements.Inthiscase,theworkflowdesirestoqueryaWebservicetosee whendatahaschanged.Whenithas,theOracleSELECTstatementisinvoked.Figure3.7 showsanexamplescreenwheresuchaWebservicesbindingmightbecreated.This bindingidentifiesthemethodsthattheWebserviceexposes,andisthefirststepincreating thenecessaryconditionallogic.
38
GregShields
Figure3.7:Webservicesconnection. OurexamplenowincludesconditionallogicformonitoringtheWebservicefordata changes.ItalsoincludesconnectionlogicforgatheringdatafromtheOracledatabase.The nextstepintheworkflowrequiresprocessingthatdatathroughtheuseofascriptblock. SuchascriptblockmightbeenteredintoanITjobschedulingsolutionusingawizard similartoFigure3.8.
Figure3.8:Ascriptingjob.
39
GregShields
Inthismockup,ajobobjectiscreatedtoboundascript.Scriptingjobsareexceptionally malleableinthattheycancontainanycodethatisunderstoodbytheITjobscheduling solutionandtargetapplication.InthecaseofFigure3.9,thecodeisVBScript,althoughany supportedcodecouldbeused. Thescriptscodeisenteredintothescriptblock,alongwithotherparameterslikethose seeninFigure3.9:Thoseparametersareassociatedwiththecodeitself,completionstatus, scriptextensions,andsoon.Oncecreated,thescriptbecomesajobobjectjustlikethe othersinthisworkflow. Note Asyoucanimagine,usingcustomcodeintroducesthepossibilityforerror intoanyITplan.YourITjobschedulingsolutionwillincludescripting guidelines,butitshouldalsoincludeinstrumentationtovalidatescript variablesandhandleandalertonerrorsastheyoccur.
Figure3.9:Filecopyjob. Thenextstepinconstructingtheworkflowistwofold.Figure3.5sbranchingpattern illustratestheneedtotransferthescriptsresultstotwolocationsusingtwodifferent mechanisms.Thefirst,seeninFigure3.9,mightbethroughafilecopyjobobject. SuchanobjectislikelytobeabuiltinobjectwithinanITjobschedulingsolutions IntegratedJobsLibrary.Thus,addingthatjobobjecttotheplanmayrequirelittlemore thandraggingitintotheworkspacejustlikewiththeSQLobject.Onceadded,parameters associatedwiththefiletransferarethenaddedalongwithactionsshouldafailureoccur. Noteagainherehowavariableisusedinthefilecopyobjectsparameterstomaintain reusability. Filecopyjobstypicallyperformfiletransfersbetweensimilaroperatingsystems(OSs), suchasMicrosoftWindows.ButgettingdataoffaWindowssystemandontoaLinuxor UNIXsystemrequiresbridgingprotocols.ThatswhyFTPjobsexist.Figure3.10showshow anFTPjobobjectmightlookbeingdraggedintotheworkspace.InFigure3.10,anFTP (technically,anSFTP)jobhasbeencreated.AddedasparameterstothatjobaretheFTP commandsrequiredtotransferthedataaswellasservernamesandcredentials.
40
GregShields
Figure3.10:FTPjob. Note Securingthesecredentialsisalsoimportanttosecurity.Noregulated businessoritsauditorswilllookkindlyonstoringauthenticationcredentials withinanFTPcommandstring.Thus,aneffectiveITjobschedulingsolution shouldprovideasecuredcredentialsstoreforsuchjobs.Thatstore maintainscredentialsecuritywhileallowingtheirreuseacrossmultipleFTP jobs. ImentionedearlierthatmonitoringandmeasurementwerekeycomponentsofgoodIT plancreation.Ifyourenotmonitoringyourenvironment,youwontbepreparedfor unexpectedstates.Onewaytodothatmonitoringcanbethroughatrigger.Ishowa portionofsuchatriggerinFigure3.11.
Figure3.11:WMIbasedtrigger. ThistriggerisusedtofacilitatetheMonitorServiceelementintheworkflow.Forit,a MicrosoftWMIqueryverifiesthestateofaservice(inthiscasetheTlntSrvorTelnet service).Notshowninthefigure,butanimportantpartofthejobcreation,istheactionthe triggerwillaccomplishwhenitdiscoversastoppedservice.Assumingthissample workflowrequiresuseoftheservicebeingmonitored,theactionassociatedwithFigure 3.11willbetorestartthatserviceifitisdown.
41
GregShields
ThisexampleisimportantbecauseithighlightsthekindsofstatecorrectingactionsanIT jobschedulingsolutioncanautomaticallyperform.Ifyourworkflowrequiresspecific serversandtheirservices(ordaemons)tobeoperational,buildingthosecorrective measuresdirectlyintotheworkflowgoesfarintoensuringthecontinuedoperationofthe distributedbusinesssystem. Oursampleworkflowneedstoprocesstwoscriptstomanipulateitsdata.Thefirstyousaw inFigure3.8.Iwontshowyouasimilarviewofthesecondscript.Instead,Illshowyoua constraintthatmightbeapplied(seeFigure3.12).Suchaconstraintcandefinewhenthat scriptneedstobeexecuted.
Figure3.12:Fileconstraint. Recallthatintraworkflowschedulingneedstobemorethanjusttimebased.Timebased schedulersarebynatureinsufficientbecausetheycanonlyprocessdataatprescribed timesoftheday.Doingsocreatesinappropriatedelayforworkflowprocessing.Whatyou reallywantisstepsinaworkflowtofireonceasuccessfulresultfrompreviousstepsis verified. Youcouldachievethisbyrunningtheworkflowlinebyline.However,doingsodoesnt necessarilybasetheexecutionoffollowingstepsoffresultsfromprevioussteps.Thats whyFigure3.12sfileconstraintisuseful.ConstraininganITjobsexecutiontooccuronly whenafileispresentallowsthatjobtokickoffonlyatthemostappropriatetime. Ourexampleworkflowneedstoprocessitssecondscriptafterafileiscopied.Onecan assumethenthatthecopiedfilewillbepresentonthetargetsystem.Thus,addingafile constrainttoajobobjectmeansrunningthejobonlywhenthefileispresentandthe previousstepiscomplete.
42
GregShields
Althoughnotnecessarilyrelatedtothisexample,apairofadditionalconstraintsisworth exploring.ThefirstcanbeseeninFigure3.13whereajobconstrainthasbeenplacedona job.Forthoseplanswhereyousimplywantonejobtofollowanotherafteritssuccessful completion,jobconstraintscanensurethatpathisfollowed.Importanttorecognizehereis that,asconfigured,whateverjobfollowstheoneinFigure3.13willonlybeginifthe previousjobissuccessful.YourITjobschedulingsolutionshouldincludemultipleoptions fordefiningwhenjobsinaplanareallowedtobegin.
Figure3.13:Jobconstraint. Theotherhalfofthisequationisintellingwhichjobtotriggerafterasuccessful completion.YoucanseeanexampleofthisinFigure3.14.Here,ajob(notidentifiedinthe figure)canbeinstructedtotriggeruponthesuccessofthepreviousjob.Using combinationsofconstraintsandtriggersensuresthatfollowingstepsintheworkflowonly executewhenthestateofthesystemisappropriate.
Figure3.14:Completiontrigger. Althoughtimeofdayschedulingisofcomparativelyminoruse,itisstillusefulfromtimeto time.Figure3.15showsanexampleschedulerthatcanbeusedforidentifyingwhenjobs shouldinitiate.Agoodschedulerwillincludenotonlydateandtimebasedtriggersbut alsoschedulingsupportforcomplexschedulingneeds.
43
GregShields
Figure3.15:Timebasedschedule.
JobLibrariesandtheValueofTriggers
WhateverITjobschedulingsolutionyouchooseneedstoarrivewithasuiteofpotential triggersthatdefinewhenjobsarefired.Thesetriggersperformmultiplefunctions.They enableactionstobefiredbasedonknownstatesratherthanrequiringperiodicwakethe scriptupandverifybatchjobs.Theyprovideamechanismtosimplifyeventhandlingon externalsystems,aprocessthatcanbeverycomplexwhenhandledwithinajobobject itself.Theyalsocreatethepotentialfornewtypesofactions,enactingchangebasedon statesthatwouldotherwisebedifficulttomonitorwithinascript. Considerthefollowingpossibletriggersasastartingpointfordefiningwhenyoumight wantactionsfiredinyourdatacenter.Thislistgetsyougoing.Illexpandonitinthenext chapter,whereIdeliverashoppinglistofcapabilitiesyoushouldlookforinasolution: WMIEventSpecifications(usingWQLsyntax) Fileevent(acrossmultipleplatforms) Emailevent(acrossmultipleemailsystems) MicrosoftMessageQueueevent WebServicesevent Startupevent SQL,Oracle,andotherdatabaseevent Virtualenvironmentevent
44
GregShields
Last,althoughthecoreofanyITjobschedulingsolutionisindeedthecodethatenacts changesonsystems,thelastthingyouwanttodoisbegincreatingscriptsifprecreated objectsarealreadyavailable.ThischapterhasdiscussedhowanIntegratedJobsLibrary createsapaletteofpotentialactionsthatyoucanaddtoyourworkspace.Figure3.16 showsarepresentativesampleofwhatonemightlooklike.Paycarefulattentiontothe actionsthatareavailablerightoutoftheboxinyourchosensolution.Youmayfindthat leaningonyourvendorforcreating,testing,andvalidatingobjectsgreatlyreducesyour effortandriskoffailure.
Figure3.16:IntegratedJobLibrary.
AnITWorkflowTellstheComputerWhattoDo
Tellingcomputerswhattodoisindeedanart,onethatsboundedinthescienceoflogic. PurchasingandimplementinganITjobschedulingsolutiononlynetsyouanemptypalette withinwhichyoucancreateyourownautomations.Thatemptypalettedoes,however, comewithsubstantialcapabilitiesforcreatingthoseinstructions.Thischapterhas attemptedtoshowyouwaysinwhichthatmightoccur. Theresstillonemorestorylefttotell.Thatstorydealswithhighlightingthecapabilities thatyoumightwantinsettingupthatpalette.Thatsthetopicforthefinalchapter.Init,Ill shareashoppinglistofcapabilitiesthatyoumightlookforinanITjobschedulingsolution. Someofthosefeatureswillprobablymakesense,whileothersmightsurpriseyou.
45
GregShields
Chapter4:ImplementingEnterpriseJob Scheduling:ARequirementsChecklist
PurchasingandimplementinganITjobschedulingsolutionnetsyouonlyanemptypalette withinwhichyoucancreateyourownautomations.Fillingthatpalettetomeettheneedsof yourenvironmentisthenextstep. Youmightrememberthisideaastheclosingthoughtofthepreviouschapter.Ithighlights animportantrealizationtokeepinmindasyoureconsideringanITjobscheduling solution:Onceyouveselected,purchased,downloaded,andincorporatedintoyour infrastructureanITjobschedulingsolution,whatdoyouhave?Withmanysolutions,not much.Onceinstalled,somesolutionsexposewhatamountstoanemptyframeworkinside whichyoulladdyourownjobs,plans,andschedules. AnITjobschedulingsolutionis,attheendoftheday,onlywhatyoumakeofit.Rightoutof thebox,afreshlyinstalledsolutionwontimmediatelybeginautomatingyourbusiness systems.Creatingallthoselittleautomationsisataskthatsleftuptoyouandyour imagination. ThatswhyfindingtherightITjobschedulingsolutionissofundamentallycriticaltothis process.Therightsolutionwillincludethenecessaryintegrationstoplugintoyourdata centerinfrastructure.Therightsolutioncomesequippedwitharichsetoftriggersthat bringinfiniteflexibilityindeterminingwhenjobsareinitiated.Andtherightsolutionhelps youaccomplishthoseautomationseasily,carefully,andwithallthenecessarytoolsinplace toorchestrateentireteamsofindividuals.Integrations,triggers,andadministrationthese shouldrepresentyourthreeareasoffocusinfindingthesolutionthatworksforyou.
CreatingaRequirementsSpecforJobScheduling
Justthreethings,eh?Thatseasytosaywhenyourejusttheauthorofsomebookon solutionsforautomatingITjobscheduling.Therealworldsimplyisntascutanddried.The realityisthatbusinessestodayrequirejustificationandoftenformaljustificationin ordertoconvertatoolthatsdesiredintoatoolthatactuallygetspurchased. Oftentimes,ITprofessionalsknowviagutinstinctthattheyneedsomethingtosolvetheir currentproblem.Theyoftenevenhaveavaguenotionofwhatthatsomethingprobably lookslike.Thedifficultpartformanyisintranslatingtheirinstinctintoasetof requirementsthatlayoutexactlywhattheyneed.
46
GregShields
ThatswhyIvededicatedthisfinalchaptertoassistingyoutheITprofessionalincreatinga formalrequirementsspecification.Illoutlineasetofrequirementsthatareremarkably similartothoseusedtofindthesolutionformyproject,TheProjectThatWouldChange Everything. Yourememberthatproject,firstintroducedinChapter1?Itsarchitectureisreprintedas Figure4.1.TheProjectThatWouldChangeEverything,asyoucansee,incorporatedarange oftechnologiesalongwithassociatedtriggersformovingdataaroundwhileprocessingit atthespeeditsbusinessrequired.Findingasinglesourcesolutiontoaccomplishallofthis wasntaneasytask.Thus,locatingthesolutionthatworkedforusneededasetofformal requirements.
Figure4.1:TheProjectThatWouldChangeEverything. Inthenextsections,Illlayoutthemostimportantofthoseinformalrequirements language.Foreachrequirement,Illaddabitofextracommentarytoitsstoryand,where possible,showyouamockupofwhatapotentialfulfillingsolutionmightlooklike.Youre welcomeandencouragedtoreusetheserequirementsalongwiththeirjustificationsin yourownspecificationforfindingtheproductyouultimatelyneed. Oh,andyourewelcome.ConsidertheseyourrequirementsforfindinganITjobscheduling solutionthatllworkforyourneeds.
47
GregShields
Requirement#1:TheSolutionShallIntegratewithAllPlatformsandApplicationsthat ComprisetheBusinessSystem
Nottobelaborthepoint,butanyITjobschedulingsolutionyouselectmustworkwith everytechnologyifitstobeuseful.Thatmeanssupportforyourdatabases,alongwith theirqueryandmanagementlanguages.Itmeansintegratingwithyourapplicationseither directlyorthroughexposedWebservices.Itrequiresdirectandindirectintegrationwith allformsoffiletransferbecausedatathatsprocessedalmostalwaysneedstobemoved somewhereelseatsomepoint.Finally,itmustbeabletohandledatatransformation, convertingdatabetweenformatsasitisprocessedorrelocated. Withasketchoftheintegrationpointsthatcompriseyourbusinesssystems,compareits listofproductsandtechnologieswiththosesupportedbytheITjobschedulingsystems youreconsidering.Thosethatdontsupporteverytechnologyshouldbeimmediately removedfromyourcandidatelist.
Requirement#2:TheSolutionShallExposeAllPropertiesandMethodsUsedbyThose Systems
Platforms,applications,andtechnologiesareonlythefirstlevelofintegrationanITjob schedulingsolutionrequires.Inadditiontogeneralsupportforanapplication,sucha solutionmustbeabletodigintothatapplicationsactivitiesandbehaviorsifitistoprocess andmovearounddata. Equallyasimportantasthesupportofthosepropertiesandmethodsistheirexposure withintheITjobschedulingsolution.Noteverybusinesssystemiswelldocumented,and noteveryproperty,method,oractionhasawellknownreasonforbeing.Thus,asolution thatcaninterrogateitsintegrationsforwhatsavailablebecomescriticallyimportant. Figure4.2showshowthismightlookforaWebService,whereamockupITjobscheduling solutionexposesalistofpotentialactionsanddata(inthiscase,propertiesandmethods) withasingleclick.
Figure4.2:ExposingthepropertiesandmethodsofaWebservicesobject.
48
GregShields
Requirement#3:TheSolutionShallIncludeSupportforScriptLanguageIndependence
Orchestratingactivitiesacrossplatforms,applications,andtechnologiesisonlyuseful whentheITjobschedulingsolutioncandosoacrosstheentirefieldofscriptinglanguages. Scriptlanguageindependencereferstotherequirementthatanyappropriatescripting languagecanbeusedwithinanyjobobjectandagainstanyapplicableplatform, application,ortechnology. Figure4.3showshowthismightbeimplementedinasamplesolution.Here,thejobobject itselfdoesnotplaceconstraintsonthetypeofscriptlaunchedwithinthepropertiesofthe job.Inthisfigure,anyscriptcanbeinsertedintotheJobPropertieslocation.Thatsame script,irrespectiveofitslanguage,canbefurtherconstrainedviaparameters,completion status,andotherfactorsincludingpreandpostexecutionsteps.Thisflexibilityis necessarybecauseyoullbeconnectingyourschedulingsolutiontomanytypesof technologies,anyofwhichmayrequireaspecificlanguageforinteraction.
Figure4.3:Scriptsofanylanguagearecomponentsofeachjoborplan.
Requirement#4:TheSolutionShallSupportQueuesforJob/PlanPrioritizationand PerformanceManagement
QueuesinanITjobschedulingsolutionrepresentamechanismtomanageandprioritize jobandplanactivities.AfullyfunctioningITjobschedulingsolutionwillleveragemultiple queuesofdifferingprioritiesinordertopreserveperformanceacrossboththescheduling systemandthoseitconnectswith.Workingwithaseriesofprioritizedqueuesalsoenables akindoffailoverwhentheresourcesneededbyjobobjectsareforsomereasonnot available.Inthisscenario,jobobjectsinonequeuecanbefailedovertosubsequentqueues forprocessing.Theresultisabetterassurancethatjobswillsucceedwhenthesystem experiencesresourceoutagesorothertransientproblems.
49
GregShields
YoucanseeamockupofhowthismightlookinFigure4.4.Here,ajobisconfiguredto executewithinaspecificsubmissionqueue.Thatqueueisgivenapriorityalongwithother parametersthatcontrolitsperformance.Runningjobsinthismannerensuresthatthey executebasedonprioritiesthataredrivenbybusinessrules.
Figure4.4:Individualjobsorplansareassignedtoqueues.
Requirement#5:TheSolutionShallSupportFileBasedTriggers
TheresanideainthesixthstoryofChapter2thatwarrantsrevisiting:Triggersarethe realjuiceinanITjobschedulingsolution.Thekindsandcapabilitiesoftriggersajob schedulingsolutionsupportsmakesthedeterminationbetweenonethatsenterpriseready andonethatsnotmuchmorethantheWindowsTaskScheduler. Itisindeedtheflexibilityofthosetriggers(alongwiththeirassociatedconstraints)that separatesthebestinclassITjobschedulingsolutionsfromthoseyouwontwant. Requirements5,6,7,and8alldealwiththeneedfordifferenttypesoftriggersthatfire basedonstatechangesorotherbehaviorsontargetsystems. Afilebasedtriggerinitiatesjobexecutionbasedonthepresenceorcharacteristicsofafile onasystem.Thesetriggersareparticularlyusefulforidentifyingwhenafileiscreated, thenfiringthejobsnextstepbasedonthatfilecreation.Theycandothesamewhenfiles aremodified,deleted,oranyotheractionassociatedwiththatpieceofdata.Filetriggers becomeimportantforeliminatinglagindistributedsystemsbecausetheyinitiate processingstepsimmediatelyasdataexperiencesachange.
Requirement#6:TheSolutionShallSupportMessageBasedTriggers
MessagingsystemssuchasCORBA,JavaMessagingSystem,andMicrosoftMessagingQueue amongothersarealowlevelsolutionfororchestratingactivitiesacrossapplicationsand platforms.Theircentralizedapproachtosignalingacrosssystemcomponentscreatesan easyframeworkfordevelopers.TheycanbesimilarlyeasyforITjobschedulingsolutions toworkwith.
50
GregShields
Interrogationandintegrationenablesmessagebasedtriggerstocoordinatetheactivities betweenlowlevelsystemsandtheiraccompanyingshrinkwrappedsolutions.Message basedtriggersaresimilartofilebasedtriggersinthattheyimprovejobexecution performancebyexecutingactionsatexactlythemomenttheyreneeded.Yourchosen solutionshouldtieintothemessagingsystemsthatareusedbyyourbusinesssystems, enablingyoutoextendthereachoftheirsignalinginandamongdistributedsystems.
Requirement#7:TheSolutionShallSupportEventBasedTriggers
Likemessagingsystems,eventsarearichsourceofinformationaboutonsystem behaviors.WithvirtuallyeveryapplicationreportingitsstatethroughOSandother onboardeventsystems,anITjobschedulingsolutionwitheventbasedtriggersgainsthe abilitytoorchestrateapplicationeventswithotheractivities. Mostimportanthereistheabilitytocustomizeandtailoreventsinsidethebusiness system.Eventscanbefiredbasedontheactivitieswithinasystem,somonitoringfortheir creationallowsanITjobschedulingsolutiontoimmediatelyinvokeresultingactions elsewhere.
Requirement#8:TheSolutionShallSupportTimeBasedTriggers
Althoughmostofthisbookhasbeendedicatedtohighlightingwhytimebasedtriggers arentgoodenoughformostbusinesssystems,therestillcomesthetimewhenajobmust befiredbasedonwallclocktime.Mostimportanttorecognizehereisthatdateandtime basedschedulingcanbedonewell(whennotdonewell,itcanbeasignificantlimiter).An ITjobschedulingsolutionthatdoesnotincludesupportformultipleschedules,irregular schedules,andhighlycustomscheduleswontbeenoughforyourneeds,particularlyin todaysglobalworkplacewherejobsthatspantimezonesmaybecommon.Seekoutthose thatprovidehighlevelsofcustomizationfordateandtimebasedschedules.
Requirement#9:TheSolutionShallSupportVariablesandDynamicDataAcrossJobs andPlans
Chapter1introducedthenotionofparameterizationwhenitcomestoITjobsandplans. Thisactivityessentiallyabstractseverypieceofdataintovariablesthatcanbeused anywhere.VariablesandothertypesofdynamicdataarecriticaltoreusabilityinanITjob schedulingsolution.Yourchosensolutionmustincludethosethatsupportvariablesboth withinjobsandplansaswellasacrossthem. Oftentimesreusabilityofvariablesandotherdynamicdataacrossjobobjectsisreferredto ascreatingprofilesofdata.Thoseprofilesprovideaneasywaytoreferencedatano matterwhereitbecomesneeded.Figure4.5showshowsuchvariablescanbeinstantiated withinajobobject.There,$ID,@ExecutionUser,and@ExecutionMachinevariableshave beencreatedforlateruse.
51
GregShields
Figure4.5:Variablesarecreatedforspecificuseoracrossalljobsandplans.
Requirement#10:TheSolutionShallSupportIntraandInterWorkflow Communication
Onceinstalled,youllbequicklycreatinglotsofindividualjobsandplansforautomating yourenvironment.AsyoulearnedinChapter3,thosejobsandplansarethediscrete actionsthatultimatelyconnecttocreateaworkflow.AneffectiveITjobschedulingsolution willenablethereuseofvariablesbothwithinandacrossworkflowssothatverylarge automationscanbemuchmoreeasilylaidintoplace. Youveseeninthepreviouschapteranexampleofhowcommunicationwithinandacross workflowsisuseful.Itsnotionofexchangingdataisimportanttosimplifyworkflowsand achieveparallelismofjobprocessingforimprovedperformance.
Requirement#11:TheSolutionShallSupportJob/PlanExecutionviaaBusiness Calendar
Youmightthinkthatasolutionwhoseprimarygoalisjobexecutionperformancewouldnt needtoconsiderthebusinesscalendar.Onthecontrary,itisimportanttorecognizethe impactofjobsonactualsystemperformance.Youdontwanttorunparticularlyresource intensivejobsagainstproductionsystemsduringperiodsofheavyuse.Justtheactof runningthosejobscanhaveanegativeimpactonthesystemasawhole. Determiningthatexactperiodoflowuseisntoftenaneasytask.Businesssystems, particularlythosethatserviceusersacrossmultipletimezones,mayexperience unexpectedhoursofhighandlowutilization.Thecomplexitiesofglobalutilizationdrive theneedforschedulingbasedonabusinesscalendar.Figure4.6showsonerepresentation ofhowthatbusinesscalendarimplementationmightlook.Usingsuchacalendar,the executionofentireseriesofjobscanbevisuallyidentifiedandscheduledtoprevent resourceoverutilization.
52
GregShields
Figure4.6:Schedulingofjobsviaabusinesscalendar. Businesscalendarsarentonlyforresourcepreservation.Theycanalsobeusedtoschedule jobactivitiesbasedonwhendatagetscreated,mirroringthejobexecutiontothebusiness rulesthatdriveitsdata.Forexample,ifyouknowthatasetofendofdaydatawillbe availableatthecloseofeachday,usingabusinesscalendarcanorchestratethecollection ofthatdataacrosstimezonesandinaccordancewithotherbusinessrules.Byfollowinga businesscalendar,itbecomespossibletoalignthetechnicalactivitiesonthesystemwith thepersonnelactivitiesintherealworld.
Requirement#12:TheSolutionShallSupportandReportonJobObjectDependencies
Workflowsthatarecomprisedofnumerousjobobjectswillgrowunwieldyovertime.This happensasindividualitemsgetevermoreinterwoventhroughoutthegreatersystem. Reuseofjobobjectsatthesametimecreatesawebofinterdependenciesbetweenthose veryobjects,whichthenrequiresmanagement.Lackingvisualizationondependencies, administratorscantooeasilymanipulateajobobjectoritemwithoutrealizingits downstreameffects. Figure4.7showsasamplereportwhereonejobobjectsdependenciesarelisted,along withtheirlabel,name,andpath.Thissimplereportisapowerfultoolinacomplexsystem wherejobobjectsfindthemselvesreusedacrosssystems.
Figure4.7:Areportonthedependentobjectsofanobject.
53
GregShields
Requirement#13:TheSolutionShallIncludeClient,Browser,andMobileDeviceUser Interfaces
YourITteamswillfindthemselvesgrowingquicklyreliantonyourITjobscheduling solutionformoreoftheirdailyactivities.Thatsbecauseanyactionorbehaviorthatcanbe characterizedintoascriptorotherjobcanbetriviallyimplementedintoyourITjob schedulingsolution. Fortheseandotherreasons,thatsamesolutionmustsupportmultipleinterfacesfor managementandadministration.Obviously,aclientbasedsolutionwillprovidetherichest interfaceformanipulatingjobsandtheircharacteristics;however,noteveryadministrator isalwaysinalocationwherethatclientGUIisaccessible.Webbrowserormobiledevice interfacesbecomevaluabletoolswhenyoureinthedatacenterandnowhereneararich client.Theybecomeevenmorevaluable(particularlyinthecaseofmobiledevicesupport) whencriticaljobsmightalertinthemiddleofthenight.Chooseasolutionthatincludes numerousinterfaceoptions,andyoullthankyourselfdowntheline.
Requirement#14:TheSolutionShallSupportNesting,Chaining,andLoadBalancingof JobsandPlans
AlargeportionofChapter3wasdedicatedtodeconstructingtheelementsinatypicalIT workflow.Eachofthoseindividualitemscanbeencapsulatedintoajoborplan.Each performssomeaction,andinterconnectingthemincomplexwayssuchasnestingand chainingiswhatmakesjobschedulingsoextensibleacrosstherangeofbusinessservices. Figure4.8showswhataviewmightlooklikeinanITjobschedulingsolution.ThereJobA andJobBaregraphicallyconnectedtoshowhowresultsfromJobAareusedinthe executionofJobB.Althoughsimpleinthisexample,thechainingofinputandoutput representsoneofthecorevaluepropositionsofanITjobschedulingsolution.Its orchestrationofactivitiesacrossalljobsandallsystemcomponentsiswhatenablesthis chainingtooccur.
Figure4.8:Twojobs,theexecutionofwhichischainedtogether.
54
GregShields
Thesameholdstruewithjobnesting,wheretheexecutionofonejoboccurswithin another.Jobnestingfurthersthereusabilityofjobsbyenablinganindividualjobto performanactionwithintheconfinesofanother.Inputandresultingoutputcanbeused betweenjobs. Theresathirdactivitythatsanimportantpartofthisrequirement.Consolidatingchaining andnestingintoanoverarchingsystemhighlightsthepowergainedthroughjobload balancing.AlreadydiscussedtosomeextentasafunctionofRequirement#11sbusiness calendar,jobloadbalancingenablesanadministratortoenactchangeacrossdozensor hundredsofsystemcomponentsatonce.Aneffectivesolutionwillenablethatactionto occurwithoutthefearthatrunningamassivelyparalleljobwillimpacttheplatformor application,orthegreatersystemasawhole.
Requirement#15:TheSolutionShallUseanObjectOrientedManagementInterface
Onecanreallyonlygettothisnirvanaofcompleteautomationifitcanbeproperly visualized.Infact,oneofthegreatestlimitationsofmanyapplicationsandplatformsliesin theirlackofvisualizationtools.Yousimplycantseetheiractivitiesastheyfire.TheITjob schedulingsolutionyouwantwillexposeaworkspaceintowhichjobobjectscanbelaid out,interconnected,andwatchedastheyexecute. Thisvisualapproachtojobcreationbecomesparticularlyimportantasthescopeand complexityofplansincreases.Asyoucanimagine,itsnotthatdifficulttoconnecttwojobs togetherlikewhatyousawbackinFigure4.8.Yetthesituationchangesdramaticallywhen greaternumbersoftasksrequireorchestration,allwiththeirownexecutiontriggersand constrains. AnITjobschedulingsolutionsworkspacedesignerfunctionalitygrowsmoreimportantas complexityincreases.Figure4.9showswhatisstillarelativelysimpleplan;thistime comprisedoffiveseparatejobs.Connectingthesejobsarethetriggers(markedasCT)and constraints(markedasJC)thatcombinetodeterminewhenthenextsetofactionsistobe executed.Inthisexample,threejobsmustcoordinatetheiractivitiespriortothefourthone executing.Onlyafterthatfourthjobexecutescanthefifthandfinaljobcomplete.
55
GregShields
Figure4.9:Acollectionofjobsinaplanwithassociatedtriggersandconstraints. Lookforaneasytouseworkspacedesignertoolinyourchosensolution.Lackingonethat presentsvisualizationsinaneasytounderstandformatmakesyourworkmoredifficult andintroducesthechanceoffailureorerrorinplancreation.
Requirement#16:TheSolutionShallUseaCentralizedJobandScriptLibrarythat IncludesCommonlyUsedBuiltInItems
ScriptsareobviouslythebackboneofanyITjobschedulingsolution,butmanycommon actionsinanITenvironmentarerepetitiveand/oreasilycapturedintoareusableobject. ThischapterinfactbeganwiththeassertionthatafreshlyinstalledITjobscheduling solutioncreatesanemptyframeworkthatyoureresponsibleforfillingwithautomation. Therealityis,dependingonthesolutionchosen,thatframeworkmaybeautomatically populatedwithcommonactionsthatcanbeimmediatelyusable. Asyoucanimagine,havingacollectionofjobsreadilyathandcansignificantlyspeedthe creationofnewautomations.Needtoemailadocument?JustdragtheEmailjobstepinto yourworkspacedesigner.Figure4.10showsamockupofhowsuchajobstepseditor mightlook.There,youcanseearangeofcommonactivitiesthatspanthebreadthofdata centerplatformsandapplications.
56
GregShields
Figure4.10:Aneditorthatincludescommonlyusedjobs. Althoughthesejobstepsalonewontbespecifictoyourneeds,aneffectivesolutionwill includewaysofincorporatingvariablesandotherdynamicdatatocustomizethejobsteps fortheneedsoftheautomationunderconstruction.Moreimportantly,thesejobstepsare pregeneratedandpretestedfromthevendor,whichreducestheriskofscriptingerror andthelevelofeffortintesting.
Requirement#17:TheSolutionShallConsolidateErrorandOutputMessagesfrom ExecutedJobs
Easilyoneofthemostdifficultactivitiesincreatingautomationsisinrecognizingtheir output,whetherthatbethedatayourelookingfororanerrormessage.Mostautomations arenotrunininteractivemode.Instead,theyrerunasbackgroundprocessesthatwork withplatformsandapplicationswithoutexposingtheiractivitiestotheloggedinuser. Thus,theresultingdataanderrormessagesfromthesescriptsarenteasilycapturedusing simplenativetools. AneffectiveITjobschedulingsolutionwilloftenexecuteitsscriptswithinitsownruntime environment,orwithinonewhereoutputanderrormessagescanbecaptured.Executing scriptsandotherobjectsinthiswayenablestheITjobschedulingsolutiontoreturnthis informationtoanadministratorsconsoleforreview.Knowingoutputmessagesfrom executedscriptsassistsgreatlyinthegenerationofthosescripts,easingtheirdevelopment processandreducingtheriskoferror.LookforanITjobschedulingsolutionthatsupports scriptexecutionreportingthatincludesoutputdataaswellasruntimeerrormessages.
57
GregShields
Requirement#18:TheSolutionShallSupportaCentralizedSecurityModelAcrossJobs andPlans
Chapter2sfinalstoryrelayedthepainfulsituationwhereascriptgetsmisused.Script misuse,accidentaluse,ormalicioususeareallcommonrisksinanydatacenter environmentwheremultipleindividualsworktogether.ThatswhyaneffectiveITjob schedulingsolutionwillincludeapermissionsstructurethatcanlockdownjobs,plans,and evenvariablestospecificusersand/oruses. Havingacentralizedsecuritymodelsignificantlyreducestheriskthatascriptwith significantimpactcannotbeaccidentallyormaliciouslyrunagainstdatacenterequipment. Italsoprovidesapointofcontrolforchangemanagementadministratorsandauditorsto monitor.Datacentersthatoperateunderheavyregulationorsecuritycontrolswillgreatly benefitfromcentralizingthepermissionsstructureforscriptexecutionintoasingle solution.
Requirement#19:TheSolutionShallIncludeaCentralizedChangeManagement Model
Securityisnttheonlymissioncriticalrequirementinasolutionthatcouldpotentially makemassivechangesacrosshundredsofsystemsatonce.Nolessimportantaretheneeds forchangecontrolandrevisionhistoryofanyautomationsthathavebeenintroducedinto thesystem. Youvesurelyexperiencedthesituationwheresomethinggotchanged.Whetherthat changeistoasettingonaserveroralineinascript,figuringoutexactlywhatgot changedandwhochangeditinthisscenarioisachallengingtaskthatisntoften successful.Whenchangesaremadethatinappropriatelyalterdata,findingtheexactlineor characteratfaultaddsevenmoredifficulty. ThatswhyanITjobschedulingsolutionthatyoullwanttousewillstorerevisionsof scriptsandotherautomationsforreview.Anexcellentsolutionwillprovideamechanism foryoutoanalyzetheindividualchangesbetweenrevisions,aswellasnotewhichuser madethechange.Figure4.11showsanexamplescreenwhere10revisionsofascripthave beenlogged.There,eachversioncanbeviewedtoidentifywhatgotchanged.
Figure4.11:Revisionhistory.
58
GregShields
Requirement#20:TheSolutionShallIncludeaCentralizedAuditDatabasethat IncludesMetricsandAlerting
Thefinalrequirementheretieseachofthelastfewintoacentralizeddatabaseforauditing, monitoring,andalertingpurposes.Ithasbeensaidrepeatedlyinthischapterthat(in additiontoenhancingjobschedulingitself)aprimaryreasonforimplementinganITjob schedulingsolutionisforcentralizationofjobexecution.Bydefault,thiscentralization automaticallycreatesasinglelocationwhereallactionstoyourbusinesssystemscanbe loggedandmonitored. Administratorandevenuseralertingrepresentusefuladditionstothefeaturesetofsucha solution.RememberthatanyITjobschedulingsolutionsitsinthecenterofyourbusiness service,orchestratingthecommunicationandprocessingofdatabetweendisparate components.Fromthislocation,itisuniquelypositionedtowatchforandalerton behaviorsindata.Thosebehaviorscanbethingsofinteresttoadministrators;or,more often,theyareofinteresttotheusersthemselves.Creatingalertsacrossalltheusual alertingapproachessuchasemail,messaging,instantmessaging,andevenmoremodern techniquessuchassocialmediaoutletsprovideawaytonotifyuserswhenconditionsof interestoccur.Figure4.12showsasimpleexampleofanemailalertthatcanbeinitiated basedoneitheratriggerorotherpreconfiguredcondition.
Figure4.12:Anexamplealert.
59
GregShields
So,DoYouNeedaJobSchedulingSolution?
So,thereyouhaveit20highlevelrequirementsforquantifyingthetypesofcapabilities youneedoutofanITjobschedulingsolution.These20requirementshighlightthemost criticalpiecesthatanydistributedbusinesssystemanditsadministratorswillneedto improvejobexecutionperformancewhilemaintainingconsistencyofworkflows. And,atthesametime,thatsmystory.IntheendTheProjectThatWouldChangeEverything waseventuallyimplementedsuccessfully.Ittooktimetocreatethenecessaryautomations, Illadmit.Buttheworkflowassistancegainedthroughtheuseofacentralizedsystem ensuredthatallourchangeswerelogged,monitored,andcarefullycategorized.Intheend, giventhesameprojectandscopeofwork,Iddoitagainintheverysameway.
DownloadAdditionalBooksfromRealtimeNexus!
RealtimeNexusTheDigitalLibraryprovidesworldclassexpertresourcesthatIT professionalsdependontolearnaboutthenewesttechnologies.Ifyoufoundthisbookto beinformative,weencourageyoutodownloadmoreofourindustryleadingtechnology booksandvideoguidesatRealtimeNexus.Pleasevisit http://nexus.realtimepublishers.com.
60

Solutions For Automating IT Job Scheduling: Greg Shields

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Solutions For Automating IT Job Scheduling: Greg Shields

Uploaded by

Copyright:

Available Formats

Solutions for Automating IT Job Scheduling

[EditorsNote:ThisbookwasdownloadedfromRealtimeNexusTheDigitalLibraryforIT Professionals.AllleadingtechnologybooksfromRealtimePublisherscanbefoundat http://nexus.realtimepublishers.com.]

External Data Source

Active Directory Server

ItisthecombinationofthesefourrealizationsthathelpedmeunderstandthatIneededto stepoutsidemyapplicationspecificmindset.IthelpedmerealizeIneededtolookto solutionsthatscheduleactivitiesacrosseveryplatformandeveryapplication.Thatswhen IstartedlookingintoenterpriseITjobschedulingsolutions.

SQL Scheduling Engine Oracle Scheduling Engine

UNIX Scheduling Engine

SQL Server Send Data

Oracle Process Data

SAP Present Data

IT Job Scheduling Solution

Figure2.2:SourcingautomationsthroughanITjobschedulingsolution. Inshort,evenifyourautomationsareadministrativeinnature,anITjobscheduling solutioncanbringsubstantialbenefit.

IT Job Scheduling Solution

Figure2.5:Replicatinganautomationtoacompletelynewsystem. HadJohnsscripts,tasks,andpackagesinsteadbeenobjectswithinacentraljob managementsystem(seeFigure2.6),thisnewcompanyneedmightnotbefraughtwithso muchrisk.

IT Job Scheduling Solution

IT Job Scheduling Solution

Figure2.10:Multipleprotocolsatplay,eachwithitsownmanagement. Forcomplexfiletransferneeds,thosethatmustexistwithinabusinessworkflow,ITjob schedulingsolutionssolvetheproblemwithoutresortingtolowleveldevelopment.

Select Oracle Data

File Copy Data

Monitor for File Presence

YoucanseeamockupofhowthismightlookinFigure4.4.Here,ajobisconfiguredto executewithinaspecificsubmissionqueue.Thatqueueisgivenapriorityalongwithother parametersthatcontrolitsperformance.Runningjobsinthismannerensuresthatthey executebasedonprioritiesthataredrivenbybusinessrules.

Figure4.9:Acollectionofjobsinaplanwithassociatedtriggersandconstraints. Lookforaneasytouseworkspacedesignertoolinyourchosensolution.Lackingonethat presentsvisualizationsinaneasytounderstandformatmakesyourworkmoredifficult andintroducesthechanceoffailureorerrorinplancreation.

You might also like