Working with big data as a consultant developer and architect

for many years, I have come across di!erent variants of big

data strategies. I have seen both sides " projects failing to
meet management expectations and abandoned as well those
labelled thumping success and had a clear and signicant
positive impact on the bottom line. And thus are born the
skeptics and champions of big data.
While interacting with my fellow mates at London Business
School, most of whom are at various levels of management in
their respective organisations, I realised the wide gap in
understanding big data that exists between the management
and the technical big data professionals.
Having the privilege of exploring both sides of the divide, I
put my learnings as a big data consultant in the perspective of
the manager. Here I present what I hope will be an easy read
for the management and help them avoid some common traps
and potholes $sinkholes?% in navigating big data. Hopefully,
this will enable more companies to experience the benets of
big data by learning from others failures " and successes.

1 Ranadip Chatterjee Taming Big Data
What is Big Data
3 Vs, 4 Vs, 5 Vs - you have had
it all! Truth be told, the
denition is little more than an
academic exercise. For the top
management, big data is a big
opportunity to re-evaluate how
more value can be squeezed
out of existing resources,
specically data. Its about
realising data as an asset.
Nearly all mid to large size
businesses, and many small
ones too, have big data in
some form.
Isnt it just a IT
problem
Nope, thats the biggest myth
that is causing a lot of heart
burn. It is, in fact, an
opportunity for the business
managers. Opportunity to
harness some un-captured
value out of its existing data
assets. Opportunity to expand
revenue base, reduce cost or
increase ROI from new
investments by employing new
age techniques, even new
sources of data.

Big data is a business
problem having a
technical solution
ADDRESSING THE CEOS DILEMMA
TAMING BIG DATA
Ranadip Chatterjee
Can my business gain !om big data
Nearly all mid to large businesses are capable of deriving some,
often a lot of, benets out of big data. However, as with any
initiative, this will need some investment. The size of the
investment and the magnitude of the resulting benets, in
other words the return on the investment $ROI%, is dependent
on various factors. Some of these factors are direct, like
industry, operations, sales channels or even the business
model. For example, media and advertising companies have
reaped bigger benets from big data than manufacturing
companies in general. Similarly, B2C companies have found it
easier to extract better benets than B2B companies.
However, some factors are not so direct, like decentralisation
of control and systems, corporate inertia, competitive rivalry
across divisions, even regulations. For example, it can be very
di&cult to consolidate disparate small datasets across business
units in a decentralised rm into a unied big data platform as
business units may perceive such centralised systems as a
threat of loss of control $of data% and hence oppose such
moves. Again, in large behemoths, often corporate inertia can
cause a feeling of complacency such that ground units think all
is good and there is no need to make any changes. Sometimes,
regulations around data movement and privacy can come in
the way of data unication that is common in big data
projects. In these cases, the benets from big data can be
signicant, but di&cult to realise due to such factors. In
summary, the cost and benets can be very rm"specic.
The key is to get an understanding of the potential benets
and likely investment required. This requires people with deep
understanding of the business to come together with those
that understand the capabilities of big data.
Some common easy wins from big data I have seen are:
" Lower data storage costs. Big data storage is generally
cheaper
" Lower licensing and maintenance costs by consolidating
multiple systems into one big data system. This also results in
simpler systems that are easy to maintain and operate.
" Higher returns on promotional and advertisement spends
by better targeting well dened customer segments.
However, these are by no means the biggest or best benets of
big data.
2 Ranadip Chatterjee Taming Big Data
Case Study 1: Tesco
estimates !20m
savings from in
cooling costs
In a pilot conducted by Tesco*, it
identied that by analysing
sensor data from its refrigerators
in a centralised big data system,
it can optimise various aspects
like temperature, run time and
start time of refrigerators at its
stores. This was estimated to
result in 20% savings in the cost
of cooling, which was equivalent
to ! 20 Million throughout its
stores in UK.
* News published on
www.computerweekly.com on
22 May 2013 titled Tesco uses
big data to cut cooling costs by
up to !20m"

Case Study 2: News
company uses big
data to increase
average time
customer spends on
its website
A popular news and media
company in UK consolidated its
user and activity data across
platforms to better track and
analyse what drives customers to
come back and stay on its
website. More time spent by
users equate to more advertising
revenue. Also better customer
insights means better ad
targeting and hence better ROI
for the advertiser, resulting in
more repeat business.
Making the blueprint
By far the most important step in the big data journey is to decide
where to start and how to go forward. Like every important
business initiative, it is important to manage the investors
expectations for returns. Big data is no exception. So it is important
to know how much return to expect, from where and how before
diving deep into it. This is a two"stage process; almost two di!erent
sub projects " the exploration stage and the execution stage.
Exploration In this stage, the potential returns have to be identied.
This includes answering the questions how much return to expect,
from where and how.
Execution When a road map has been clearly established following
the exploration stage and the potential gains identied, its time to
go and execute the how " the deep dive if you will.
It is possible that a company chooses to go through multiple
iterations of the above stages, and quite often that is what happens,
either by choice or otherwise. But it is still quite important to
maintain the distinction between the two at every step. That is
because the two stages need di!erent approaches, di!erent set of
tools and di!erent expertise. Mixing the two is often the cause of
projects falling in the crevice in between the two cli!s.
From the top managements perspective, the exploration stage is
where close involvement, encouragement and monitoring is
important. The steps could be as follows:
" Understanding the revenue and cost sources in the business
" Understand where data"driven decision making exists. These can
be areas of low hanging fruits, potentially providing quick returns
" Understand where manual processes are used for decision
making. These can further be divided into procedural and
judgemental. Procedural processes can also be easy wins $may or may
not be big data use cases but the business gains anyway if additional
e&ciency could be identied%. Judgemental processes can benet
from unstructured data analysis techniques in big data. There could
be signicant benets to reap from such processes though they are
likely to be harder to achieve.
" At each of the processes in the step above, map the current
processes back to P&L and estimate how the added benets of big
data can impact that number $with condence gures%. Also identify
how long and what resources will it take to realise the gain. The cost
can be estimated from that. 'Caution: This will be an intense and
time consuming step, but a very important one(
" Once this information is presented back to management, its up
3 Ranadip Chatterjee Taming Big Data
Case Study 3: Churn
prediction by popular
mobile/VoIP
operator
Many major telecom providers
including Skype, T-mobile and
Celcom have used big data
analytics on Call Data Record
logs to understand and predict
churn and customer life-time
value, among of the biggest
factors affecting revenue in that
sector. This has enabled them to
personalise and focus attention
and promotions precisely where
it matters most.


Case Study 4: Fraud
detection by banking
and insurance firms
Analysing incoming transaction
streams in real-time have helped
banks detect and stop
fraudulent transactions. Similarly,
analysing transaction as well as
social data patterns using big
data have enabled banks,
insurance companies and even
social benets departments in
governments to detect
fraudulent claims. It is estimated
that worldwide retail banking
losses from fraud is over $100
billion. So there is a lot of
potential here.
to the management to pick and choose the appropriate
opportunities. Inputs from a big data expert could help to identify
synergies among di!erent options.
" For each selected option, identify the key KPIs, nancial targets
and deliverables that will help track the execution. Dene scope and
success. That will help eliminate unnecessary digression during
execution. )Digression is allowable during exploration, not during
execution(
That leads us to execution:
" In almost all cases, big data benets are optimal at large scale.
Centralised infrastructure and architecture, consolidated data
storage and computation are often the best ways to harness the
benets of big data. So, it is benecial to create a role that will own
the centralised framework and co"ordinate with various business
units to enable them build their systems on top. Many rms call
this role the Chief Data/Digital O&cer or Chief Architect. Some
even make this person the owner of the execution stage, which is
ne if the role is su&ciently empowered.
" Hand over the targets, deliverables and KPIs to the revenue or
cost centre that will benet from the initiative. Allocate appropriate
funds.
" Monitor the progress, tweak KPIs, if required, based on new
lessons and information. The momentum and enthusiasm needs to
be maintained as is the case for any new business initiative.
" Focus should be on small measurable achievements that can be
linked right up to the nancial statements. Agile development
processes usually help here.
" Encourage the team to announce technical successes to the
technical community. The big data community is pretty vibrant. Not
only does the recognition give a boost to the team, it also allows for
genuine feedback and improvements to make their way into the
project.

I will end with this nal observation: The big data journey usually
turns out to be a signicant step within the organisation. So, strong
backing and encouragement from the top management, including
the CEO, is vital to reap the best possible benets from it. Half"
hearted attempts usually dont go far and are not worth the trouble.
4 Ranadip Chatterjee Taming Big Data
Avoid
Delegating exploration:
Delegating this stage
completely to IT personnel
without high level involvement
from business is likely to have
modest success at best
Jumping into implementation:
Without establishing clear
nancial targets, time lines and
resource requirement, jumping
into implementation can hurt
Slack in monitoring: Financial
targets and time lines should
be closely monitored to avoid
unplanned diversions
Mixing exploration and
execution for reasons
mentioned aside

Do
Allow unrestricted exploration:
Let ideas y, barriers break,
lines blur. This is a discovery
phase after all
Understand the expected
benets to business clearly
before embarking on execution
Promote these benets down
the org chain to get buy-in at all
levels. Inertia will make it
difcult unless strong
encouragement from the top is
there
Celebrate accomplishments in
public. Its a vibrant community
out there with lots to gain.
Ranadip Chatterjee (ranadip.c[at]gmail.com) is a consultant in Big Data strategy and implementation.
An Ex-Yahoo engineer, he has spent over 12 years in IT in engineering, management and consultant
roles. He has a Bachelors in Engineering and holds an MBA from the London Business School.