DO THE ALGORITHMS REPORT

TO THE ROBOTS?
Making decisions in a world of big data; managing in a
world driven by algorithms
Michael Ross

Co-founder and Chief Scientist, DynamicAction

Computers are useless.

They can only give you answers.

PABLO PICASSO

ALGORITHM-AS-EMPLOYEE
Many businesses are getting excited at the possibilities of big data. But data is only useful if it changes the
way you make decisions. Big data may sound exciting, but it is utterly useless if you continue to make
the same decisions, in the same way, with the same data and at the same frequency. Not only will you
have missed the point, you will also have missed the opportunity.
To respond and thrive in this new reality, businesses have to completely rethink how they make decisions, and
then how they execute and manage them.
Algorithms are the answer: the logic of how you make thousands, and even millions, of automated
or semi-automated decisions in the digital commerce world. These algorithms are already amongst us.
Core retail activities such as google paid search, product feeds, product recommendations, on-site search,
retargeting and triggered emails are already powered by algorithms. But just because you have an algorithm,
doesnt mean it is any good.
Algorithms need to be thought of like employees - they need to be managed. They need to be trained,
set the right objectives, reviewed and given feedback. If they consistently underperform, they need to be
replaced.
While humans can be resistant to feedback and may obfuscate or explain away mistakes, algorithms thrive on
feedback and failure. In fact, the essence of machine learning is based on understanding failures and missed
opportunities, and evolving parameters and models accordingly. In the human-computer era, one needs to
establish the right culture, processes and governance that will allow algorithms to develop and thrive.
The management and optimisation of algorithms is a new muscle for retailers. This paper gives practical
guidance on how to rethink business decisions in the digitally-enabled, data-driven, algorithm-powered world.
It covers:
1.
2.
3.
4.

Why we need to rethink decisions

What decisions are we talking about
A new approach to making decisions
The management challenge

2 | Big Data Decisions | Do The Algorithms Report to the Robots?

1. WHY WE NEED TO RETHINK DECISIONS

Decision-making in digital retail, as many retailers have already recognised, is considerably more complex
than in physical retail. Digital commerce is catalysing two tectonic shifts that fundamentally affect decisionmaking:

Consumer behaviour is changing radically. Consumers now have almost unlimited choice, and are
empowered and informed. They are using a portfolio of devices, being bombarded by digital marketing
and are shopping in new and complex ways. The result? Consumers create data, providing managers
with a tsunami of available data to use
Businesses have a vast new set of decisions to make. Firstly decisions on how to become digitally
enabled, and thereafter to take advantage of the ability to personalise and optimise all interactions with
customers and products. The result? Decisions consume data, while at the same time creating more
and more data

THE CHANGING LANDSCAPE OF DATA DECISIONS

DIGITAL

CONSUMER BEHAVIOUR

Devices
eCommerce
Beacons
Marketing

Empowered
Informed
Unlimted Choice
DATA

DECISIONS
Digital enablers
New frequency/logic
Atomised

The consequence is that the logic and economics of many core retail decisions relating to marketing and
merchandising have changed. This is predominantly caused by the transition of marketing from a fixed
to a variable cost (per impression, per click, per transaction). Trivial as this sounds, it has changed the
fundamental equations of retail (do I spend the next on price or marketing?). In addition, it has elided the
marketing-merchandising-store organisational boundaries - silos which made sense in the past, however
dont anymore.
At the heart of the change is the atomisation of both the decisions and the data. We are now faced
with millions of decisions and hundreds of millions of data points.
The key is to start with the decision and then ask how the data (whether big, small or medium) allows you to
make decisions more intelligently, more frequently or more profitably.

3 | Big Data Decisions | Do The Algorithms Report to the Robots?

2. WHAT DECISIONS ARE WE TALKING ABOUT

Businesses are machines for making decisions: to state the obvious, the day-to-day rhythm of business is
about decisions and actions. The decision architecture for making these decisions has evolved, often over
decades, and is part of the fabric of every retailer. But the digital world is catalysing at an unprecedented pace
and scale of change in this architecture.
Altering decision-making may sound overwhelming as a task, but in reality there are specific types of
decisions that require the most immediate attention and adaptation. High priority should be decisions with a
high value, and where data could have a major impact. A good starting point is to conduct a decision audit:
What decisions are being made at the moment and how are they made? What new decisions will need to be
made in the future?
Decisions can be sorted into a hierarchy, based on the nature of the decision and its frequency. It is then
easier to identify what changes need to be made at each level, and where the greatest differences will occur.
Change is required across the hierarchy, but the most profound change will be felt at level 3 (tactical
decisions) and level 4 (programmatic decisions), where increased complexity, volume and frequency of
decisions will necessitate some degree of automation.

DECISION HIERARCHY
Level 1
Strategic

Small volume of high value decisions

New data needs to be applied

Level 2
Planning

Level 3
Tactical

Level 4
Programmatic

New approaches are required.

New data and logic required

New frequency of decisions,

combined with new data/logic

Completely new decisions

and constantly evolving

Level 1. Strategic decisions (annual). Typically familiar decisions that businesses have been making for
years. For example: which new markets to enter, where to locate new shops and what new categories to
develop?
What has changed? - New data is now available to inform the decisions, such as:
-- New retail locations can be informed by looking at the postcode of loyal online shoppers
-- New categories can be informed by searches made on the site by VIP customers for categories that

do not exist

4 | Big Data Decisions | Do The Algorithms Report to the Robots?

Level 2. Planning decisions (bi-annual/seasonal). Again, these are familiar decisions such as: range
planning (breadth and depth), which brands and products to continue or delist, marketing budgets;
channel allocation and technology roadmap
What has changed? The decision logic is different, requiring new data and logic:
-- Size ratio planning can be informed by looking at the size distribution of customers viewing products

and the impact on sell-through
-- Products can be evaluated in terms of their impact on acquiring and retaining high value customers
-- Sales can be planned based on expected sales from existing customers, and consequently how

many new customers need to be acquired
Level 3. Tactical decisions (weekly/daily). Many of these decisions sound familiar, but are actually very
different in the digital world. For example: pricing, promotions, digital marketing, site sort orders, landing
pages and digital touch points
What has changed? The decision logic is different (new trade-offs are required), the level of action is
typically more granular and the frequency of decisions needs to be optimised
-- Retailers need to decide (and can now make the trade-off) whether to invest the next in price,

marketing or a customer promotion
-- CRM actions can be informed by whether the customer is actively browsing the website
-- Sort orders can be personalised and changed every week, day or hour. A key question is what is the

optimal frequency?
Level 4. Programmatic decisions (hourly/real-time). For example: programmatic marketing,
personalised search results, keyword bids and product recommendations
What has changed? The decisions are being made at a very low level of resolution, are increasingly in
real-time (so they need to be automated), are often new and continually evolving
-- Retailers can suggest unique product recommendations for each customer viewing each product
-- Retailers can retarget individual customers who have taken a specific action on their site
-- Landing pages can be personalised based on the customer and the intent of their visit
-- Site search engines can increasingly personalise the sort order (product ranking) of search results

5 | Big Data Decisions | Do The Algorithms Report to the Robots?

3. A NEW APPROACH TO MAKING DECISIONS

Having classified your decisions, how can you make them differently? Essentially this is about introducing
a more disciplined approach to cope with the volume and complexity of decisions. This is similar to the
business process re-engineering revolution fuelled by computers in the 1980s/90s.
Every decision has a control (what you do) and an objective (what you are trying to optimise):
An algorithm determines how to set the control to optimise the objective
This algorithm requires a model, and implicit in the model is a simulation that will have unknowns
(described by parameters)
Feedback will then improve all aspects of this process from the model and parameters to the algorithm

DECISION CHECKLIST

DATA

CONTROL

DECISIONS

ALGORITHM

ACTION

FEEDBACK/
METRICS

OBJECTIVE

MODEL +
PARAMETERS

To help make sense of this, we have developed a decision checklist:

1. Decide what sort of decision it is (the objective and the control) and understand what success
looks like
2. Work out what data is needed
3. Build a model with parameters
4. Work out the algorithm to make the decision
5. Operationalise - turn the decision into an action
6. Review performance, metrics and feedback

6 | Big Data Decisions | Do The Algorithms Report to the Robots?

Step 1: Decide what sort of decision it is

Many decisions can seem ill-defined, such as managing keywords on google or personalising landing pages.
A critical role for any manager in the digital commerce world is to really understand the decision, and be able
to explain it (often to people without the technical expertise).
For each decision, at every level, it is critical to understand the objective (what are you trying to optimise and
what success looks like). The objective will typically be either:
-- Business outcome: profit, revenue, return on capital or customer LTV
-- Interim outcome: product views, new customers acquired or achieved margin
It is also critical to understand the controls (what decisions are actually possible). At a high level, decisions
can be categorised into four main types: binary, discrete, threshold or value (see table for further detail).
Being clear on the controls can help make sense of the decision, and then more clearly articulate it to others.
Moreover, each type of control requires a different approach to management and feedback.
Control

Description

Discrete choices

Decisions where there are Do we mark down a product by 30/50/70%?

a specific set of answers to How often should we change our website
choose from
homepage?
How many different emails should we send per week?

Threshold

The decision is a number

where we will take a
different action above/
below the value

Whats our maximum acceptable threshold for cost

per order on google?
Whats the annual spend threshold for a VIP
customer?
Whats the value threshold for free delivery?

Value

The decision is a specific

value

How many units do we order of a product?

How much should we bid on a paid search keyword?
How much should we spend to acquire a new
customer?

Binary choice

Decisions that have a yes/

no answer

Examples

Do we open or close a store?

Do we list or delist a brand?
Do we accept or reject a potentially fraudulent order?

As well as understanding the objective and controls of each decision, it is important to work out the
interaction between them:
What frequency of control is possible? If wrong, how quickly can the decision be changed?
Whats the feedback time? How quickly will we know if the decision is good or bad?
Whats the cost of a wrong decision?
Whats the sensitivity of the outcome to a bad decision?
This is critical to set the context for making the decision. It is easy to change your mind on decisions that are
low cost and rapid feedback, but more expensive to reverse decisions which are high cost and where feedback
is slow.

7 | Big Data Decisions | Do The Algorithms Report to the Robots?

Step 2: Work out what data is available and needed

There is an almost unlimited amount of data available to make these decisions. The challenge is to work out
what data you really need, and how this might evolve as the decision gets more valuable and the data gets
cheaper.
Figuring this out requires an implicit data-decision trade-off: is it better to make an OK decision with easy
data or a perfect decision with complex data? In practice, the key is to make the trade-off and make a
good enough decision. (This is the essence of satisficing an approach to decision making called bounded
rationality, which recognises that humans are limited in their ability to determine the optimal solution to a
decision. Herb Simon developed the concept in 1956 and won a Nobel Prize for it in 1978).
To make decisions effectively, businesses need to think about making good enough decisions based on the
minimum amount of data required. It is critical to avoid asking for all the possible data available that might be
useful. Or you will quickly become overwhelmed and suffer from crippling analysis paralysis.
For managers to effectively make this data-decision trade-off it is important to really understand:
What data is readily available vs. hard to get?
What data is available now, and how might it evolve?
How good is the data? (e.g., cleanliness, completeness, ability to join, quality)
What data is free vs. at a cost?
How far can you get with siloed data? Whats the value of joining data?
How might the decision logic evolve as new data becomes available?
What the ROI of investing in better data?

EXAMPLE: DATA-DECISION TRADE-OFF FOR PAID SEARCH MANAGEMENT

8 | Big Data Decisions | Do The Algorithms Report to the Robots?

Step 3: Build a model with parameters

At the heart of the new approach to decisions is a model of how the world works (one can also think of the
model as describing the rules of the game). For each decision it is critical to understand what you expect to
happen if you make the decision.
In all decisions, there is some type of uncertainty, and the model and parameters describe this uncertainty.
The uncertainty is typically characterised by one or more parameters, and embedded in the models will
typically be an unknown what-if relationship - if we do X, we hope/expect that Y will happen. For example:
-- How will volume increase if we reduce price? (price elasticity)
-- How will visits/sales increase if we increase marketing? (marketing elasticity)
-- How will customers behave if we ship orders faster? (customer experience elasticity)
In all decisions, there is also some type of trade-off, a cost and a benefit. The model captures the nature of
the trade-off, and the parameters describe the scale of the trade-off. For every decision, there will be one
model that then gets applied by the algorithm (see below) across the thousands/millions of programmatic
decisions.
The key to success is to recognise that a critical aspect to the management role is setting, monitoring
and managing parameters and then evolving the model.
As many managers will recognise, it is incredibly easy to make decisions without having a model or simulation.
In practice, one can get away with this if the business performs well (you could get lucky!). But it is a risky
proposition in the long term because when things dont go so well, it is then hard to know what went wrong,
and what you should do differently.

A SIMPLE MODEL: WHAT IF?

9 | Big Data Decisions | Do The Algorithms Report to the Robots?

Step 4: Work out the algorithm

The algorithm describes how the decision is actually made. Its inputs are the data, the simulation and the
objective. The algorithm answers the question what value of the control will optimise the objective?
The algorithm here is typically described by an equation (or logic statement) that connects the decision
to the objective. Sometimes, one can simply write down the algorithm as a formula (e.g., a replenishment
algorithm); alternatively, some algorithms will be executed using some sort of optimisation routine (e.g.,
finding an optimal sort order).

ALGORITHM EXAMPLE: BIDDING ALGORITHM FOR KEYWORDS

The essence of algorithms is that they operate at a low level of resolution, so one algorithm might make
thousands, if not millions, of programmatic decisions. Algorithms are everywhere. Whether retailers like it
or not, more and more of the critical decisions within their digital commerce business are increasingly being
made by algorithms.

THE STATE OF THE ALGORITHM - EXAMPLES

Example decisions

Control

Level of execution

How orders are

accepted or rejected as
potential fraud

Threshold

All orders are scored

Accept order if score >X, reject
based on a combination of order is score <X
customer and order data,
and 3rd party fraud systems

When to trigger a
promotion to a lapsing
customer

Discrete
choices

Score for every customer

(thousands to millions)

How products are

ranked when an on-site
search is made

Value

Score for every product, for

every search (millions)

10 | Big Data Decisions | Do The Algorithms Report to the Robots?

Algorithm example

Retailer formula based on

inventory, sales, conversion,
newness, availability, etc.

Customers given a score based

on the lifetime spend, recency
and frequency of purchase.
Score is mapped onto a
promotional campaign

Step 5: Operationalise - turn the decision into an action

It is an easy assumption that actions somehow happen automatically in the digital world. In fact, many things
can go wrong when decisions become actions. Since the action is typically not observable, it is often hard to
distinguish between:
-- Execution failure the action simply has not happened
-- Poor execution the action has happened, but has been badly executed (e.g., human error)
-- Unexpected outcomes the action has been properly executed, but results in some sort of technical
failure
Turning a decision into an action should be straightforward, but often it isnt. So it is important to really
understand:
-- Who takes the action? And what happens if they are on holiday?
-- How often do they take it?
-- How complex is the task or action?
-- How is the execution managed (if not observable)?
In traditional human decision-making, the RAPID framework (a variant of the RACI framework) is a helpful way
to understand who does what in making decisions and taking action. While still helpful in the new humancomputer world, it is critical to rethink this approach where some combination of the R, A, P, I and D will be
driven by an algorithm. For the humans involved in this process, its critical to ensure they have the right
information, support and tools required.

WHAT DOES RAPID LOOK LIKE IN AN ALGORITHM-DRIVEN WORLD?

Need a clear owner for

the algorithm

Need a governance
process to review the
algorithm

Need someone to
manage the execution

11 | Big Data Decisions | Do The Algorithms Report to the Robots?

Step 6: Monitor and Review Performance

How do you manage the algorithm-as-employee? A key challenge for any manager (and perhaps also the
Chief Algorithm Officer) is to work out what is going wrong and why it is happening. Reviewing performance
and managing the algorithm requires a portfolio of metrics; metrics for what you did and what happened vs.
what was expected in terms of profit, wastage, long term consequences or unintended consequences.
At a high level, one needs to have an overall measure of performance vs. the objective and then understand:
How did the decision perform overall? How has the decision performed vs. what was predicted? If
performance was good, we can focus on improvement. If performance was bad, we need to understand
whats going wrong:
-- If the prediction is good but performance is bad, it is likely theres a problem with the objective i.e.

weve optimised to an objective that is not correlated with profit
-- If performance is good but the prediction was bad, we need to review the model and parameters
Managing the model, parameters and algorithm requires a disciplined approach to understanding decision
quality. For every decision, one has to define an error which could be: wasted spend, wasted effort or
missed opportunity (a measure of whats gone wrong, or what could be better). One can then look at the
distribution of errors to understand where to focus. For example, if the decisions have a:
-- Consistent error g review parameters. Likely that tweaking a parameter will solve the
problem
-- Inconsistent error g review model. Likely that the model doesnt represent reality
-- Concentrated error g review algorithm. Likely that performance can be improved by making

the algorithm more specific (e.g., if errors are concentrated with specific customers/categories/
outliers)
Much like employees, algorithms have a learning curve. But unlike employees, algorithms thrive on learning
from mistakes (its the essence of all machine learning). When it comes to managing algorithms, leaders
have to celebrate failure (or at least encourage openness). Its too easy to present aggregated averages that
obfuscate the issues and opportunities.

MANAGING THE ALGORITHM: GOOD THINGS TO MEASURE

12 | Big Data Decisions | Do The Algorithms Report to the Robots?

4. The management challenge: co-ordinating all the decisions

We are in the midst of a digital industrial revolution and the before leaders navigating this sort of digitallyled, data-driven transformation face a daunting task. Managing a business with hundreds (or thousands) of
algorithms is hugely complicated. We recommend three critical foundations for success: data, people and
culture.

DATA
Data is often described as the new oil, needing to be mined, extracted and refined. Unfortunately, it is
often an afterthought because the consequences of bad data are not immediately visible. But bad data is
cancerous quietly undermining the quality of decisions.
In reality many retailers are suffering from significant data debt, where years of poor (or non-existent)
data management are now coming home to roost. To introduce good data into your business, we suggest
focusing on the three core building blocks:
Collecting data: Businesses need to recognise the criticality of instrumentation to ensure that data
is collected and is of high quality. A common excuse from retailers is our analytics tagging is not very
good, but when web analytics is the critical glue that joins data in the digital world this excuse should no
longer be tolerated
Joining data: While the data all originates in silos, it has to be joined to have any hope of making profitcentric decisions. The key to joining data is ensuring common fields between systems
Storing data: Data storage has historically been seen as a cost to be optimised, and so data has
been aggregated or thrown away. But storing data is now almost free - Amazon Web Services has
commoditised storage (for example standard storage is now 3 cents per month for 1 GB)

THE INSTRUMENATION CYCLE

Humans
Computers

Automation cycle
ACTIONS

Workow
Tagging

Intrumentation cycle

CUSTOMER
BEHAVIOUR

Alogorithms
Logic

Optimisation cycle

DATA

Digital exhaust
Transactions

13 | Big Data Decisions | Do The Algorithms Report to the Robots?

DECISIONS

Metrics
Inference

MANAGEMENT

Organisation
Incentives

PEOPLE
It is ironic that in a world of big data and automation, many of the toughest management challenges are
people-related. While many aspects of retail decisions can be automated, people are still needed at the
heart of the business it is people who turn the data into something useful.
Retailers need to employ many different types of people to make sense of their data, and it is critical to think
of data as a team effort. Some of the roles we are now seeing in successful data-centric businesses include:
Decision architects: people who understand the business problem you are trying to solve, and can work
out what questions to ask. This is the creative end of data science and these are typically people with a
strategy consulting background
Algorithm developers: people who understand the logic/math to translate the business problem into a
solvable math problem. The key here is to refine the question into something that is answerable given the
available data. These people are typically mathematicians, statisticians and engineers
Data analysts: people who are able to make it work once. These are people with advanced Excel, basic
database and web analytics skills, who are good at cleaning, joining and manipulating data
Data product managers: people who understand the technical requirements to translate the one-off
solution into a technical roadmap. These are people with technical product management experience and
they are the bridge between the business and developers
Big data developers: people who can actually build and operationalise a technical solution. These are
people with skills in database architecture and software development

Action-focus

Decision
science/project
management

Automation/
articial
intelligence

Insight-focus

Business problem

WHAT SORT OF DATA PROBLEM ARE WE TRYING TO SOLVE

Exploratory
data mining

Reporting/
business
intelligence

One-o

Operational

Business process

14 | Big Data Decisions | Do The Algorithms Report to the Robots?

CULTURE, PROCESS AND GOVERNANCE

Digital commerce does not respect organisational boundaries. The inconvenient truth is that many of the
critical decisions of digital commerce are confounded by data from outside the system or organisational silo.
The data creates the challenge and people are a critical part of the solution - but only if the right structure,
process and governance is put in place.
As a result, it is inevitable that some of the traditional ways of managing will need to change. What change
is needed clearly varies across organisations, but will typically require retailers to rethink some or all of the
following:
-- Behaviours: Are we encouraging people to ask the right questions?
-- Skills: Do we have the capability to answer?
-- Resource: Do we have the means to action?
-- Incentives: Are people correctly incentivised?
-- Control: Can we measure success?
-- Organisation: Are the right people making decisions at the right level?
The transition to a world of algorithm-driven retail is a huge challenge for all retailers, but the opportunities are
equally huge for the winners. At the core of this transition is the cultural shift to make an organisation think
differently. Too often retailers revert to making the same decisions, on the same frequency with the same
logic as theyve done in the past. This is not a formula for long-term success.

The role of humans at Amazon is to help computers make better decisions.

Attributed to Jeff Bezos, CEO of Amazon.com

15 | Big Data Decisions | Do The Algorithms Report to the Robots?