The Event Fabric

bringing IT together What is

µ
Apache Kafka &
What is
Apache Kafka & Why is it Important?
Why is it
Important?

What is Apache Kafka & Why is it Important? | UKOUG Tech17 1

 Heck, I do not even know these people
and they may not know me [personally]
– just my pearls of wisdom. And if they
are late to the party, they can also
check out the historic archives of my
eloquence
Without fretting about the numbers of
readers involved and whether they are
in the same time zone as me and online
when I publish my messages – and
which device they use

It would be so nice if I could

publish my ideas and actions,
accessible near instantly for
everyone who is interested
 Heck, I do not even know these people
and they may not know me [personally]
– just my pearls of wisdom. And if they
are late to the party, they can also
check out the historic archives of my
eloquence
Without fretting about the numbers of
readers involved and whether they are
in the same timezone as me and online
when I publish my messages – and
which device they use

It would be so nice if I could

publish my ideas and actions,
accessible near instantly for
everyone who is interested
• Decoupled communication
• 0, 1 or many followers
• Scalable number of messages (and parties)
• Reliable (mostly available, few messages lost)
• Full history
• Open: cross device, cross location
• Not Sub-second, near real-time fast
• Rate limited (#messages/minute)
• Size limited (140-280 characters)
• Format limited (text)
• Not for private interactions
• Not (really) for programmatic use
 DVX_ORDERS

Oracle Database

ORDERS

Oracle
Database

5
 DVX_ORDERS

Oracle Database

ORDERS

Oracle
Database

6
 µ Oracle Application
DVX_ORDERS
Container Cloud
Oracle Database

Oracle DBaaS Cloud

ORDERS

Oracle
Locally running
Node application
µ Database

What is Apache Kafka and why is it important? 7

 What does the Twitter for System Driven Event Interaction
look like?

• Decoupled communication – organized per topic

• 0, 1 or many Consumers per Topic
• Scalable number of messages (and parties)
• Reliable (distributed)
• Full history
• Open: libraries in many technologie & REST APIs
What is Apache Kafka and why is it important? 8
 µ Oracle Application
DVX_ORDERS
Container Cloud
Oracle Database

Oracle DBaaS Cloud

Oracle Event Hub

ORDERS

Oracle
Locally running
Node application
µ Database

9
 What does the Twitter for System Driven Event Interaction
look like?

• Decoupled communication – organized per topic • Near real-time fast

• 0, 1 or many Consumers per Topic • No Rate Limit
• Scalable number of messages (and parties) • No enforced size limit
• Reliable (distributed) • Anything goes (it’s all byte[])
• Full history • On premises or in cloud, private or trusted
• Open: libraries in many technologie & REST APIs • Very much for programmatic use
What is Apache Kafka and why is it important? 10
Events Consumers

Producers

Robust, Scalable, Fast,

History Retention
Containerized/Cloud-
enabled
Open
Messaging as we know it

• JMS, Oracle Advanced Queuing, IBM MQ, MS MQ, RabbitMQ, MQTT,

XMPP, WebSockets, Oracle Coherence, …
• Challenges
• Costs
• Scalability (size and speed)
• (lack of) Distribution (and therefore availability)
• Complexity of infrastructure
• Message delivery guarantees
• Lack of technology openness
• Deal with temporarily offline consumers
• Retain history
Introducing Apache Kafka

• ..- 2010 – creation at Linkedin

• Message Bus | Event Broker
• High volume, low latency, highly reliable, cross technology
• Scalable, distributed, strict message ordering, ….
• 2011/2012 – open source under the Apache Incubator/ Top Project
• Kafka is used by many large corporations:
• Walmart, Cisco, Netflix, PayPal, LinkedIn, eBay, Spotify, Uber, Sift
Science, Zalando, The New York Times, Airbnb, Coursera, ING Bank,…
• And embraced by many software vendors & cloud providers
• Client libraries available for Node, Java, C/C++, Python, Ruby, PHP, Go,
Rust, .NET, Perl, Scala DSL, Clojure, Swift and more
 Consumers
tcp
Producers
tcp
 Consumers

Producers

Topic
KAFKA TERMINOLOGY
Message

• Topic Key
Value
• Message Time

• == ByteArray Producer Consumer

• Broker Broker
• Producer Topic

• Consumer
 Key
Value Consumers
Time
Producers

Broker
Topic
CONSUMING

• Messages are available to consumers only when they have been committed
• Kafka does not push
• Unlike JMS
• Read does not destroy
• Unlike JMS Topic
• (some) History available
• Offline consumers can catch up
• Consumers can re-consume from the past
• Delivery Guarantees
• Ordering maintained
• At-least-once (per consumer) by default; at-most-once and exactly-once can be
implemented
 Key
Value Consumers
Time
Producers

Broker
Topic
 Consumers
tcp
Producers
tcp
Broker
Topic
WHAT’S SO SPECIAL?

• Durable
• Scalable
• High volume
• High speed
• Available
• Distributed
• Open
• Quick start
• Free (no license costs)
• “Self Fulfilling Prophecy” (positive feedback loop)
CONFLUENT == ENTERPRISE KAFKA

• Freemium model
• Support
• Training
• Confluent Cloud
• Platform and Tools
<CTRL F5>
 F5
F5 Application
F5
CTRLF5
Server
CTRLF5
CTRLF5
 F5
F5 Application
F5
CTRLF5
Server
CTRLF5
CTRLF5
 F5
F5 Application
F5
CTRLF5
Server
CTRLF5
CTRLF5
FAST DATA AND ACTIVE UI

• Handle influx
• Publish findings instantaneously
• Update UI & notify end user immediately
• Analyze in real time

• Decoupled components
• No data loss when a component is temporarily down
• Scalable with volume of events and of number of clients
THE CASE AT HAND

Tweets on
#ukoug17
#ukoug_tech17
#ukoug_apps17
#ukoug_jde17

Show live
tweet feed
for Allow users
conferences like tweets
to
–and show Show live
Client live list of tweet
liked tweets
aggregates
Client per
conference Show a live
list of top 3
Client liked tweets
Client per
conference
DEMO - REAL TIME, CROSS CLOUD, CROSS
TECHNOLOGY PUSH

Tweets on
#ukoug17
you #ukoug_tech17 Client
#ukoug_apps17
#ukoug_jde17
Client

Client
Client
THE CASE AT HAND – STEP ONE
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17

Tweets
Topic
Show live
tweet feed
for
Client conferences

Client

Client
Client
THE CASE AT HAND – STEP ONE AND TWO
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17

Tweets
Topic
Show live
tweet feed
for
Client conferences

Client

Client
Client
KAFKA CONSUMER IN NODE
GET EVENTS PUSHED INTO APPLICATION
THE CASE AT HAND
SERVER SENT EVENTS FOR PUSH BACK
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17

Tweets
Topic

Show live tweet

feed for
conferences
Client

Client
Server Sent
Event
Client
Client
SERVER SENT EVENT – SERVER SIDE

Client

Client
Server Sent
Event

Client

Client
SERVER SENT EVENT – CLIENT SIDE

Client

Client
Server Sent
Event

Client

Client
LIVE TWEET STREAM

Server Sent
Event
THE CASE AT HAND
TWEET LIKES – CLIENT TO SERVER TO ALL CLIENTS
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17

Tweets
Topic
Show live
Allow users
tweet feed
for to like tweets
–and show
conferences
live list of
Client liked tweets

Client
SS
Client E
Client
THE CASE AT HAND
WEB SOCKETS – FOR BI DIRECTIONAL PUSH
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17

Tweets
Topic
Show live
Allow users
tweet feed
for to like tweets
–and show
conferences
live list of
Client liked tweets

Client WebSockets

SSE
Client
Client
TWEET LIKES BROADCASTING

WebSockets

WebSockets
THE CASE AT HAND
STREAMING ANALYSIS OF TWEET EVENTS
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17

Tweets
Show live Topic
Allow users
tweet feed
Show live
to like tweets
for
–and showtweet
conferencesaggregates
live list of
liked tweetsper
Client conference

Client WebSockets

SSE
Client
Client
THE CASE AT HAND - STREAMING ANALYSIS OF TWEETS
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17

Tweets tweetAnalytics
Show live Topic Topic
Allow users
tweet feed
Show live
to like tweets
for
–and showtweet
conferencesaggregates
live list of
liked tweetsper
Streaming
Tweets
µ
Client conference Aggregation

Client WebSockets

Client SSE
Client
KAFKA STREAMS

• Real Time Event [Stream] Processing integrated into Kafka

• Aggregations & Top-N
• Time Windows
• Continuous Queries
• Latest State (event sourcing)
• Turn Stream (of changes) into Table
(of most recent or current state)
• Part of the state can be quite old
• A Kafka Streams client will have state
in memory
• Always to be recreated from topic partition
log files
• Note: Kafka Streams is relatively new
• Only support for Java clients
KAFKA STREAMS
Topic
Filter

Aggregate

Topic
Join

Map
(Xform)

Topic
Publish
 EXAMPLE OF KAFKA STREAMS
Topic Set Conference as key
groupBy

Sum/Avg/Top3 by key
TweetMessage (==conference)
Conference Aggregate
Text and possibly per time
Author window
Hashtag Topic
Latest Conference
Join
Details

Round aggregate Map

to nearest 100 (Xform)

As JSON
Publish
Topic: CountTweetsPerConference
KAFKA STREAMS –
RUNNING COUNT TWEETS PER CONFERENCE
STREAMING TWEET ANALYTICS
PUSHED TO CLIENTS

Server Sent
Event
THE CASE AT HAND - STREAMING ANALYSIS
OF TWEET LIKES
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17
#ukoug_jde17

Show live Tweets tweetAnalytics

Allow users
tweet feed Topic Topic
Show live
to like tweets
for
–and showtweet
Show a live
conferencesaggregates
live list of list of top 3
liked tweetsperliked tweets Streaming µ
conferenceper Tweets
Client Aggregation
conference

Client WebSockets

Client SSE
Client
KSQL FOR DECLARATIVE STREAM ANALYTICS
THROUGH CONTINUOUS QUERIES

create table tweetAnalytics as

select conference
, count(*) create stream retweets
from tweetsTopic as
group by conference select *
from tweetsTopic
where text like 'RT%'
VISUALIZING KSQL VS KAFKA STREAMS
THE CASE AT HAND - STREAMING ANALYSIS
OF TWEET LIKES
Tweets on
#ukoug17 #ukoug_tech17
#ukoug_apps17 tweetLike
#ukoug_jde17
Topic
Show live Tweets tweetAnalytics
Allow users Top3TweetLikes
tweet feed Topic Topic
PerConference
Show live
to like tweets
for
–and showtweet
Show a live
conferencesaggregates
live list of list of top 3
liked tweetsperliked tweets Streaming µ
conferenceper Tweets
Client Aggregation
conference

Client WebSockets Likes µ

Aggregation

Client SSE
Client
WEBSOCKETS – SERVER SIDE
RUNNING TOP 3 OF
BEST LIKED TWEETS PER CONFERENCE

Server Sent
Event
END TO END FLOW CLOUD ENABLED
Cache
API
EventHub CS
API
µ Tweets Likes
Tweet
Count
Likes
Top3

UI µ
Client
Chrome
Tweets µ Likes µ
Client Aggregation Aggregation
Firefox
Key aspects of this demo – What Kafka can do for you

• Bridging Cloud(s) and on premises systems

• Providing decoupled interaction between microservices
• Performing Streaming Analysis
• Bridging technologies (Java, Node, …)
• Bridging the availability (no | one | multiple instances)
• Provide semi-push based synchronization

• Open
• Scalable
• Reliable & Available
• Fast
• Complete historical record

What is Apache Kafka and why is it important? 58

Oracle embracing Apache Kafka

• Event Hub Cloud Service = Managed Apache Kafka platform

• Managed Topics have been announced too
• Kafka as source for Golden Gate and ODI
• Data Pipeline with Data Hub (Apache Cassandra) & Event Hub
• Oracle Service Bus Kafka Adapter
• Integration Cloud
• Stream Analytics (aka Stream Explorer fka Oracle Event Processor)
• Oracle Native Container and Microservices Platform
• Fn Serverless Platform
• JET and ADF real time push based on Apache Kafka
• In general – the bridge between on premises  [public] Cloud

What is Apache Kafka and why is it important? 59

Summary

• => == =>

• Apache Kafka is emerging as platform of choice for message exchange in a world of

• Microservices
• CQRS and Data Source Synchronization
• Clouds
• Fast Data (IoT) and Streaming Analysis
• Real time data integration & distribution
• Oracle is rapidly embracing Apache Kafka on various levels
• Getting started with Apache Kafka is not very hard at all
• The platform is open source – and has broad client support (Java, Node, …)
• Many resources are available – tutorials, blog article, demonstrations, presentation
slides and recordings of conference sessions, samples on GitHub

What is Apache Kafka and why is it important? 60

• Blog: technology.amis.nl

• Email: lucas.jellema@amis.nl Thank you!

• : @lucasjellema

• : lucas-jellema

• : www.amis.nl, info@amis.nl

What is Apache Kafka and why is it important? 61