Big Data and Cloud Challenges

from IoT

Ricardo Jimenez-Peris
Univ. Politecnica de Madrid
Cooperation Agenda
Opportunities
3 Conclusions

IoT Big Data

Challenges What are the specific
challenges for BigData
from IoT

Big Data
What is Big Data
 Big Data
is not Always Well Understood
Enterprises are now quite
It looks like confused due to the large
a python number of alternatives

It looks
like a rope

It looks like a dagger It looks like a pillar

 3Vs of Big Data
V
Volume
Data Large data size (TBs to PBs) V V
Volume
Velocity
High speed of data V
flow (CEP), data change
(OLTP) and data processing V V
Data Data (OLAP, analytics)
Velocity Variety
Variety V
Diverse data (structured/
unstructured), diverse data V V
models and query languages,
diverse data sources
 What is Different in IoT
Regarding Big Data?
• Big data is characterized by having a particular
challenge in one or more of the 3 Vs: Volume,
Velocity and Variety.
• IoT presents challenges in combinations of
them.
• Most challenging IoT applications impact both
Velocity & Volume and sometimes also
Variety.
 IoT Big Data Requirements
• How are IoT applications characterized with respect data
needs?
• Let us look at different large families of applications:
– Smart-*
• Smart-City.
• Smart-Grid.
• Smart-Transport.
– Connected Things:
• Real-time monitoring of what “things” are doing, their operating
conditions, their issues, their load, their configuration, etc.
• Predictive maintenance: fix before “things” break.
• Optimization: configuration, interaction with humans, energy º
efficiency, most suitable “things” for different uses.
• Analytics to find out the current shortcomings to design next versions
of “things”.
 IoT & Big Data
• One of the most prominent features of IoT is
its real-time or near real-time communication
of information about the “connected things”.
• For challenging IoT applications, the difficulty
lies in doing this at scale (e.g. from 10s of
thousands to 10s of millions and above).
• Smart-* scenarios are in many cases also
characterized due to the Variety of data
sources and data to be stored and processed.
 IoT & Big Data
• Another important feature of the vision of IoT is that
by observing the behavior of “many things” it will be
possible to gain important insights, optimize processes,
etc.
• This vision boils down to solve multiple challenges:
a) to store all the events (Velocity & Volume Challenge);
b) to run analytical queries over the stored events; (Velocity
& Volume Challenge)
c) to perform analytics (data mining and machine learning)
over the data to gain insights (Velocity & Volume &
Variety Challenge);
 IoT & Big Data
• Another angle in the vision of IoT is the ability of
performing Real-Time Analytics.
– How to detect and react in real-time to opportunities
and threats to my business;
• This requirement presents multiple challenges:
– How to process streaming events on the fly (Velocity
challenge).
– How to store streaming events in the operational
database (Velocity challenge).
– How to correlate streaming events with my stored
data in the operational database (Velocity & Volume
challenge).
 IoT Overarching Challenges
• What does really matter for IoT application developers?
• Having a platform to develop IoT applications as easy and
fast as possible.
• What it means for Big Data?
• Delivering platforms able to:
– Correlate streams of data with stored data (Scalable CEP over
scalable SQL/NoSQL).
– Update operational databases with streams of data (Scalable
CEP and scalable OLTP).
– Perform analytical queries over operational databases (Scalable
OLAP+OLTP).
– Support the Variety in different data models and querying each
data (queries and updates over polyglot persistent data).
 First Steps: CoherentPaaS
• Ultra-Scalable transactions.
• Full-ACID NoSQL (Graph DBs, Document oriented DBs, Key-value
DSs), Complex Event Processing (CEP) and SQL (transactional and
analytical queries OLTP/OLAP)
• Transactions across data stores.
• SQL Queries across data stores.
Innovation
• Creating a startup,
LeanXcale to
commercialize this
innovation.
• LeanXcale delivers
an integral big data
management
platform providing
OLTP & OLAP, SQL
& NoSQL
integration.
 Potential Cooperation Areas

Scalable Data
Scalable Management IoT, M2M
Cloud Data Cloud
Management Platforms
IoT, M2M
Cloud
 Big Data: TBs to  Internet of
PBs of data Things
 OLTP & OLAP Potential Cooperation:  Machine-to-
 Big Data Scalable Cloud Big Data Machine
 SQL, NoSQL &  Platforms and
Platforms for
CEP applications on:
 Millions events/s IoT applications IoT & Mobility
 Millions txn/s
 Querying TBs of
data in seconds.
 Contact

Ricardo Jiménez-Peris
e-mail:
CONTACT DETAILS rjiimenez@fi.upm.es
WWW:
http://cumulonimbo.eu