CloudBasedMonitoringofIoTDevices

779Views
February20,2015
NoComments
Future
voxxed

ByJohanVos

Aperformantandscalablecloudbasedmonitoringsystemfor
collectingdatafromembeddeddevices
Thenumberofembeddeddevicesthatareconnectedtotheinternetthe
InternetofThings(IoT)isgrowingeveryday.Manyofthesedevicesneed
aninternetconnectiontoretrievedatafromandsenddatatoavarietyof
backendservices.Buttheinternetconnectionoftenisalsorequiredinorderto
managethedevicesandthesoftwaretheyarerunning.
Inorderforadevicetobemanageable,itshouldatleastbepossibleto
monitorit.Anoperator,orthepersonwhoisresponsibleforthemaintenance

Originallypublishedinthe
Jan/Feb2015issueof
JavaMagazine.Subscribe
today

ofthedevice,wantstoknowthecurrentstateofadevice,itshistory,its
currentoperations,andsoon.Withthegrowingnumberofconnecteddevices,
monitoringthesedevicesbecomesahugebutimportantchallenge.Inthis
article,wewillexploreausecasewherecloudbasedmonitoringofconnecteddevicesallowstheoperation
ofalargenumberofkiosksystems.

TheUseCase

CultuurNetVlaanderenisagovernmentorganizationinBelgium.At
therequestoftheFlemishauthorities,thisorganizationtriesto
increasepublicenthusiasmforculture.In2012,CultuurNetlaunched
theUitpasproject,asystembasedonanearfieldcommunication
(NFC)cardthatallowsitsuserstoearnpointswhenattending
culturaleventsandtoexchangepointsforrewards.Userscanscan
theircardsatkiosksystems(seeFigure1),whicharebuiltarounda
RaspberryPiconnectedtoanNFCreaderandanLCDscreen.
Akiosksystemisalsocalledacheckindevice(CID),becausethe
originalgoalwastocheckinauserforaculturalactivity.
Initially,Uitpaswasrolledoutinthreecitiesonly,withabout30CIDs
spreadaroundculturallocationsinthosecities,forexample,near
theaters,cinemas,museums,andsoon.ThoseCIDswereoperated
byCultuurNetonbehalfofthethreecities.
ThesuccessofUitpasinthethreecitiestriggeredaninterestby
morecities,somoreCIDswereputintoproduction,alongwithagrowingnumberofoperationalentities.
Eachcityisnowexpectedtomanageitsowndevicesandoperations.
Clearly,managementcantbedonemanuallyanymore.Also,theneedfordistributedmanagement(each
cityshouldbeabletomonitorandfixitsowndevices)leadstooperationalchallenges.Inordertoaddress
thesechallenges,CultuurNetisnowusingacloudbasedmonitoringsystemthatmonitorsallregistered
CIDsandprovidesaccesstodifferentoperators.Eachoperatortypically,anITmanageremployedbythe
citycanmonitorandmanageonlytheCIDsforthatmanagerscity.
ThesolutionispoweredendtoendbyJava.TheCIDsrunJavaFXonaRaspberryPi.Theysendtheir
monitoringinformationtotheIoTMonitorCloudthecentralpieceofthearchitecture,whichrunsaJava
EE7implementation(GlassFish).Theclientmonitoringapplicationis,again,aJavaFXapplication.Figure
2showsthissetup.

ThedifferentcomponentsalluseJava,butindifferentenvironments.TheIoTMonitorCloudcollectsand
storesallinformationfromallCIDs.ThiscomponentusesJavaEE7APIsandrunsinGlassFish
instancesintheAmazonEC2cloud.ItusesAmazonDynamoDBforstoragecapabilitiesandtheopen
sourceelasticsearchengineforsearchcapabilities.

CIDstoCloud
TheCIDssendmonitoringdataaboutfunctionalevents,aswellasloggingdata,totheIoTMonitorCloud.
BecauseloggingdataiseasilyobtainedinaJavaapplication,thisfunctionalitycanbeusedinany
application.TheCIDapplicationinternallyusestheloggershowninListing1.
Listing1
1

publicstaticfinalLoggerLOGGER=Logger.getLogger("be.uitpas.pi"

Allsystemrelatedevents(forexample,acardbeingscanned,anetworkrequestbeingmade,anetwork
requesttimingout)areloggedusingstandardJavaloggingcommands,forexample:
1

LOGGER.log(Level.SEVERE,"Couldnotreadcard.");

AspecificMonitorHandler,whichisanextensionofLogHandler,iscreatedandaddedtothecreated
logger,asshowninListing2.Bydoingso,themonitorHandlerwillbenotifiedaboutallentriesthatare
loggedontheLOGGERinstance.
Listing2
1
2
3

finalMonitorHandlermonitorHandler=newMonitorHandler();
LOGGER.addHandler(monitorHandler);
monitorHandler.setLevel(Level.ALL);

publicstaticfinalLoggerLOGGER=Logger.getLogger(be.uitpas.pi)

TheMonitorHandleritselfisdefinedasshowninListing3.Wheneversomethingisloggedinthe
applicationbycallingLOGGER.log(),thepublishmethodontheMonitorHandleriscalled,andthelog
messageisprovidedwithadditionalinformationsuchasatimestamp,threadinformation,themethod,and
soon.
Listing3
1
2
3
4
5
6

publicfinalclassMonitorHandlerextendsjava.util.logging.StreamHandler{
@Override
publicvoidpublish(finaljava.util.logging.LogRecordrecord){
...
}
}

TheMonitorHandlerwillsendtheloginformationtotheIoTMonitorCloudusingDataFX,whichisa
JavaFXbasedframeworkthatbringsenterprisefunctionalitytoJavaFX.Oneofitscomponentsisthe
DataSourcescomponent,whichfacilitatesRESTbasedcommunicationwithbackendsystems.DataFX
respectstheJavaFXthreadingmodelanddoesitsworkonabackgroundthread,usingtheJavaFX
Applicationthreadtoreportbacktotheapplication.
ThecodesnippetinListing4showshowDataFXexecutesaRESTrequestfromanembeddeddeviceto
theIoTMonitorCloud,whichprovidesanendpointatthefictiveaddresshttp://iotmonitor.cloud.Theactual
requestcontainsmoreformparameters,butforreadabilitythoseareomitted.
Listing4
1
2
3
4
5
6
7
8
9
10
11
12
13
14

RestSourcerestSource=RestSourceBuilder.create()
.host(http://iotmonitor.cloud)
.path("pimonitor")
.path("rest/monitor/log")
.timeout(5000)
.formParam("cid",myCidIdentifier)
.formParam("message",myMessage)
.formParam("date",String.valueOf(record.getMillis()))
.build();
ObjectDataProvider<String>odp=ObjectDataProviderBuilder.create()
.dataReader(restSource).resultProperty(answer)
.build();
odp.setExecutor(executorService);
finalWorker<String>retrieve=odp.retrieve();

Ascanbeseenfromthecode,aRestSourceiscreatedformakinganHTTPconnectiontoanendpoint,
therebyprovidingtypicalinformationaboutthepath,theparameters,andconnectionsettingssuchasthe
timeoutvalue.TheRestSourceisthenpassedtoanObjectDataProvider,whichmakestherequestand
putstheresultintoapropertynamedanswer.AnExecutorServiceinstanceisthenpassedtotheDataFX
ObjectDataProvider,becauseitcanbeexpectedthatonsomeoccasions,lotsoflogging
informationoutsidethescopeofthisarticle.Seehttp://datafx.ioformoreinformation.
Oneofthemostcommonissueswithconnecteddevicesisthelossofconnectivity.Clearly,sendingalog
messageaboutlostconnectivityisnotgoingtowork.Inthiscase,theMonitorHandlerwillstorealllog
messagestoafilesystem.Oncetheconnectionisrestored,themessageswillbesenttotheIoTMonitor

Cloud.

JavaFXMonitorClientApplication
TheJavaFXMonitorclientapplicationconnectstotheIoTMonitorCloudandvisualizesthemonitoring
informationthattheoperatorisallowedtosee.TheIoTMonitorCloudmaintainsalistofregistered
operators,andassociatesthemwithoneormorecardsystems.AcardsystemisagroupofCIDsthat
belongtogether,typicallyallCIDsinaspecificcity.

Authenticationis,therefore,acrucialcomponentintheIoTMonitorCloudsetup.Figure3showsthelogin
screenoftheJavaFXMonitorclient.Oncetheoperatorisauthenticated,themonitoringconsoleisshown
(seeFigure4).Thetopleftcorneroftheconsoleshowsthelistofcardsystemstheoperatorhasaccess
to,andtheCIDsregisteredwiththosecardsystemsareshownbelow.Themainpartoftheconsoleshows
themonitoringdataoriginatingfromthedifferentCIDsthattheoperatorismanaging.

Typically,thereisahugeamountofdatatobevisualized.Therefore,theapplicationcontainsanumberof
selectorsandfiltersthatallowtheoperatortoselectaparticularpartofthedata,forexample:
Atimeselector,forshowingonlyinformationwithinaspecifictimerange.
Alevelselector,forselectingmessagesofaspecificlevelorarangeoflevels.
ACIDselector,forselectingoneormoreCIDsforwhichtoinspectlogs.ClickingaspecificCIDin
thelistontheleftselectsthespecifiedCIDandshowsitsmonitoringdatainthelist.Ifnospecific
CIDhasbeenselected,datafromallCIDsisshown.
Whenanalyzingdata,operatorstypicallymodifytheselectorsfrequently.Ifallchangestoselectorswould
leadtocallstothebackend,theapplicationwouldnotbeveryresponsive.However,JavaFXin
combinationwithJava8featuresallowsoperatorstomanipulatethedatasetontheclient.Whenthe
applicationisstarted,somedataisretrieved.Ifmoredataisneeded,itwillbeloadedintotheapplication,
anddatathatisnolongerneededcanbegarbagecollected.Thisallowstheapplicationtotakeadvantage
ofthememoryandprocessingcapabilitiesoftheclientsystemitisrunningon.Typically,theJavaFX
Monitorapplicationhasalotmoredataentriesinitsmemorythanitvisualizesinthelist.
Thelistcomponentinthemainpartoftheconsoleisdefinedasfollows:
1

privateListView<Event>eventsListView

TheitemsthataretoberenderedinthisListViewarekeptinaJavaFXSortedListnamedsortedEvents.
ThislistisassociatedwiththeListViewasfollows:
1

eventsListView.setItems(sortedEvents);

AlldataformonitoringitemsthatareretrievedfromtheIoTMonitorCloudarecapturedinan
ObservableListoftypeEventthatisnamedevents:
1

ObservableList<Event>events;

NeweventsareloadedfromthebackendusingDataFXwithcodethatissimilartothecodeinListing4,or
theyaresentfromtheIoTMonitorCloudtotheJavaFXMonitorapplicationoveraWebSocket,whichwill
bediscussedshortly.Inbothcases,thefactthateventsarestoredinanObservableListallowstheListView
thatrenderstheeventstotakealldataintoaccountincludingeventsthatareaddedaftertheListView
hasbeencreated.
NotethattheraweventsarestoredinanObservableListnamedevents,whereastheeventsthatare
passedtotheListView
arestoredinaSortedListnamedsortedEvents.Thereisarelationshipbetweenthosetwolists.First,the
rawlistofeventsisfilteredtocontainonlythoseeventsthatmatchthedifferentselectors.Thisisdone
usingtheJava8StreamAPI,whichprovidesfilteringfunctionality.Wehaveanintermediatefieldnamed
filteredEventsoftypeFilteredListthatisinitiallydefinedasshowninListing5.
Listing5
1

FilteredList<Event>filteredEvents=events.filtered(etrue

TheFilteredListclasswasaddedinJavaFX8,anditprovidesanextremelypowerfulapproachfor
maintainingalistofelementsbasedonanoriginallist,buttakingintoaccountfiltercriteriathatare
specifiedasapredicate.Fromthisinitialization,itisclearthatinitially,allraweventsareconsideredtobe
includedinthefilteredEventslist.Wheneveraselectorchanges,thefilteredpredicateischanged:
1

filteredEvents.setPredicate(validEvent())

validEvent()returnsapredicatethatcheckswhetheragivencandidateeventshouldbecontainedinthe
filteredList.
Inourcase,theboundaryconditionsimposedbythethreeselectorsmentionedpreviouslymustbe
satisfiedbeforeacandidateeventisaccepted.ThisisshownintheimplementationofthevalidEvent()
methodinListing6.
Listing6
1
2
3
4
5

privatePredicate<Event>validEvent(){
returnselectedCid()
.and(isShowLevel())
.and(isInTimeRange());
}

Thereturnstatementinthismethodcontainsthreemethodsthateachreturnsapredicatebyitself.Wewill

examineonlythecodefortheisShowLevel()predicate(seeListing7)however,itshouldbeclearthatthe
otherfunctionsareconstructedinthesamewaybuttheyareslightlymorecomplex.Thispredicatewill
checkthatforagivenevente,theloglevelisatleastequaltothelevelselectedbytheselector.
Listing7
1
2
3

privatePredicate<Event>isShowLevel(){
returnee.getLogLevel().intValue()>=selectedLevel.intValue();
}

ThefilteredEventscouldnowbeshownintheListView,butthereisnoguaranteeontheorderinwhichthe
eventswouldbeshown.ThiscanbefixedbyhavingtheEventclassimplementthejava.util.Comparable
interface,andimplementitinsuchawaythatthemostrecenteventsarealwaysshownfirst.The
SortedListclasswasaddedinJavaFX8,anditallowsinstancesofaListViewtobesortedaccordingto
theirComparator,whichiswhatweuse:
1

SortedList<Event>sortedEvents=filteredEvents.sorted();

ItisthisSortedList,whichextendsObservableList,thatispassedtotheListView.Thus,byusingsome
Java8functionality,weareabletouseasimpleapproachtorenderonlyrelevantentriesinasortedway,
basedonalargesetofraw,unsorteddata.

WebSocket
ThetimeselectorshowninFigure4showsatablabeledLive.Thisallowsrealtimemonitoring.
Realtimemonitoringofdevicesisoftenveryimportant,becauseitcanpreventissuesfrombeing
escalatedintobigproblems.Italsohelpsoperatorstodetectproblemsratherthanrequiringuserstocall
anoperatortoreportaproblem.Therefore,theIoTMonitorCloudiscapableofsendingdataentrieson
thefly,assoonastheyarriveinthesystem.
BoththeIoTMonitorCloudaswellastheJavaFXMonitorapplicationleverageJSR356,theJavaAPIfor
WebSocket.AssoonasauserlogsinsuccessfullytotheJavaFXMonitorapplication,theapplication
establishesaWebSocketconnectiontotheIoTMonitorCloud.TheDataFXframeworkwediscussed
beforecontainsaWebSocketcomponentaswell.ThiscomponentleveragestheclientportionsofJSR
356andopensaWebSocketconnectiontotheIoTMonitorCloud.BecausetheIoTMonitorCloudis
runningontopofGlassFish,whichisthereferenceimplementationofJavaEE7,itcontainsan
implementationofJSR356outofthebox.Hence,itisveryeasytoregisteraWebSocketendpointinthe
IoTMonitorCloudthatwillbroadcastallnewincomingeventstotheclientsthatareentitledtoreadthis
information.

Conclusion
Inthisarticle,wetalkedabouttheimportanceofbeingabletomonitorIoTdevicesinthefield.Whena
largenumberofdevicesisbeingused,andwhenthosedevicesgeneratelotsofdata,acloudbased
solutionispreferredforcollectingthedata,especiallyiftherearedifferentgroupsofuserswithdifferent
setsofpermissions.TheJavaEE7APIsallowyoutodevelopacloudbasedmonitoringsystemthatis

performantandscalablewhileaddressingtheneedsofclients(forexample,byprovidingsupportfor
WebSocket).
ThevisualizationofthemonitoringdatacanbeachievedusingJavaFX.WesawthattheJava8APIshelp
infilteringandsortingrelevantdatabasedonanumberofcriteria.ThisexemplifiesJavaeverywhere:The
embeddeddevicesarerunningOracleJavaSEEmbedded,theIoTMonitorCloudisrunningGlassFish4.1
instances,andthemonitoringclientisaselfcontainedapplicationcreatedwithJavaFX8.
SendtoKindle