Scalable data-driven modeling

for demand response in smart grids

Geert Deconinck, KU Leuven / EnergyVille

bENBIS Energy Demand Forecasting Workshop
18-JAN-2018
 Outline

introduction
what is EnergyVille
smart grids and demand response
aggregate and dispatch: three-step approach
from model-based to data-driven forecasting
SmarThor platform
conclusion

2
 EnergyVille = Flemish energy research partnership by

VITO KU Leuven imec UHasselt

energy technology electrical energy photovoltaics & solid organic PV
sustainable cities thermo-mechanical energy state storage
building physics

3
 Flemish energy research partnership

EnergyVille
Research – Development – Training – Industrial Innovation
4
 Intelligent energy networks

smart grid = “an electricity network that can intelligently integrate

the actions of all users connected to it - generators, consumers and
those that do both - in order to efficiently deliver sustainable, economic
and secure electricity supplies. “

[smartgrids.eu]
5
 Demand response: new or not?

goal: match demand and supply in electric power system

20th century
generate extra load to allow large central plants to run at maximal efficiency
electric accumulation heating, electric boilers, …
 ‘night tariff’, ToU (time-of-use) pricing
21st century
also varying supply from renewables
increase or decrease demand accordingly
electric vehicles, industrial loads, appliances…

6
 Using residential demand response for ...

... market objectives

day ahead market (Time-of-Use pricing)
balancing market (real time)
... grid support objectives
voltage support
congestion support
... prosumer objectives
maximising self consumption from renewables
and combinations thereof

7
 Demand response challenges

scale issues
single large industrial loads
some commercial loads
many residential loads
EnergyVille focus: residential applications in distribution grids
control paradigm
local control (no communication)
coordinated control
direct – centralised – hierarchichal – distributed – peer-to-peer

8
 Challenges (ctd)

uncertainty
user behaviour / grid behaviour / market behaviour / external (eg weather)
worst case – typical case
flexibility providers
homogeneous versus heterogeneous
aggregation provides flattening or peaking? simultaneity?
for market purposes aggregated behaviour
vs local grids require spatial info
…

9
 Outline

introduction
aggregate and dispatch: three-step approach
from model-based to data-driven forecasting
SmarThor platform
conclusion

10
 A first example: Three-step approach for clusters of flex providers

three-step approach
1 aggregate flexibility and constraints
2 centrally determine optimal aggregated setpoint
3 dispatch setpoint to individual devices

applied to plug-in hybrid electric vehicle (PHEV) charging

S. Vandael, B. Claessens, M. Hommelberg, T. Holvoet, G. Deconinck,
"A scalable three-step approach for demand side management of
plug-in hybrid vehicles," IEEE Trans. on Smart Grid, Vol. 4, Iss. 2, June
2013, pp. 720-728.

11
 Three-step approach – step 1: constraint aggregation

power flexibility Power

[W]
Pmax

pt 100 Priority
energy flexibility [%]

Energy
[Wh] E
Req

t deadline
Time

12
 Three-step approach – step 1: constraint aggregation

power
energy

E
Req
Aggregator E
Req
Agent
Pmax

Pmax

Concentrator Concentrator
E
Req
Agent E
Req
E
Req
Agent E
Req

Pmax Pmax
Pmax
Pmax
pt pt pt pt
Device Device Device Device
Agent Agent Agent Agent

13
 Three-step approach – step 2: derive optimal charging schedule
electricity
price

optimisation problem t

E
Req

Pmax t
pm pt
cost minimization for determining optimal demand
Aggregator
Agent
 𝑃𝑡 = cluster power consumption
 𝜆𝑡 = real time pricing tariff
translate optimal demand 𝑃1 to market priority 𝑝∗

14
 Three-step approach – step 3: real-time control

dispatch priority
 all aggregators and device agents know priority pm
and can derive charging power E
Req

Pmax
pm pt

E
Req
Aggregator E
Req
Agent
Pmax
pm
Pmax
pm pm

Concentrator Concentrator
E
Req
Agent E
Req
E
Req
Agent E
Req

Pmax Pmax
Pmax
Pmax
pm pt pm pt pmpt pt pm
Device Device Device Device
Agent Agent Agent Agent
15
 Three-step approach - results

savings

scalability

16
 Outline

introduction
aggregate and dispatch: three-step approach
from model-based to data-driven forecasting
learning market price and flexibility
learning component model
learning cluster behaviour
SmarThor platform
conclusion

17
 Control: from model-based to data-driven

model-based system identification

physical models or greybox models (e.g. ETP - equivalent thermal parameters)
for model-driven control
data-driven system identification
both component and cluster behaviour
 data-driven control
reinforcement learning

data-driven forecasting
of electricity consumption, flexibility, market prices, PV, ...

18
 Learning day ahead market prices and EV-flexibility

S. Vandael, B. Claessens, D. Ernst, T. Holvoet, G. Deconinck, "Reinforcement Learning of Heuristic EV Fleet

Charging in a Day-Ahead Electricity Market," IEEE Trans. on Smart Grid, Vol. 6, Iss. 4, Jul. 2015, pp. 1795-1805.

19
 Machine learning details

reinforcement learning (RL) approach

learns collective charging behaviour of EV fleet
not of individual EVs
together with market prices
decision making process of aggregator modelled as MDP
MDP: markov decision process
objective function based on balancing costs
feature extracted based on charged energy in [t, t+1]
batch RL, because disturbances & EV dynamics unknown
fitted Q iteration
20
 Outline

introduction
aggregate and dispatch: three-step approach
from model-based to data-driven forecasting
learning market price and flexibility
learning component model
learning cluster behaviour
SmarThor platform
conclusion

21
 Learning TCL
control policy

Policy or control map

White: OFF
Black: ON

Price profile

F Ruelens, BJ Claessens, S Quaiyum, B De Schutter,

R Babuska, R Belmans, Reinforcement learning Smoothing
applied to an electric water heater: from theory to techniques
practice, IEEE Trans smart grid, online Dec 2016 Increase the
acceptability of
RL technique
22
 Learning individual TCL flexibility for control policy

why batch reinforcement learning (BRL)?

converges faster than conventional reinforcement techniques (SARSA, Q)
experiences/batch can be reused
improving performance of BRL
model-assisted BRL (optimal building climate control)
ensemble of ELMs allows for faster convergence
validated in simulator (ETP model) as well as EnergyVille HomeLab
BRL with auto-encoder (electric boiler)
auto-encoder used for feature extraction

23
 Outline

introduction
aggregate and dispatch: three-step approach
from model-based to data-driven forecasting
learning market price and flexibility
learning component model
learning cluster behaviour
SmarThor platform
conclusion

24
 Learning cluster behaviour for TCLs with CEM

state-of-charge evolution heterogeneous cluster of 1000 TCLs

S. Iacovella, F. Ruelens, P. Vingerhoets, B. Claessens, G. Deconinck, "Cluster Control of Heterogeneous

Thermostatically Controlled Loads Using Tracer Devices," IEEE Trans. on Smart Grid, Vol. 8, Iss. 2, Mar. 2017,
25 pp. 528-536.
 Cluster control of TCLs and tracing

model aggregated TCL flexibility using tracer

describing population dynamics

26
 Cluster control of TCLs and tracing

three-step approach: step 1: aggregate

i. observe:

ii. assume:

iii. match:
CEM=cross-entropy method

27
 Cluster control of TCLs and tracing

TSA+: step 2: optimization on reduced order tracer models

objective

optimization
TSA

step 3: dispatch
idem as before
28 TSA+
 Cluster control of TCLs and tracing TSA+

results: large scale example (1000 TCLs)

29
 Cluster control of TCLs and tracing TSA+ -- follow up

field trial with fridges - UCB

architecture
1. Modelling and optimization
2. Communication infrastructure
3. Local device module

start-up company ThermoVault bvba

30
 Outline

introduction
aggregate and dispatch: three-step approach
from model-based to data-driven forecasting
SmarThor platform
cloud-based data collection platform
for monitoring and control
of multi-energy systems
conclusion

31
 SmarThor Data Platform

Goals:
Internal platform for data collection, storage, provisioning, and processing
Increase TRL of product development at EnergyVille
Facilitate cooperation with partners

Monitor and control energy production, use, and exchange at Thor park
Determine energy losses
Integrate optimization and control algorithms
Coupling with market models
Standardized development platform for future research projects

32
 SmarThor Data Platform
Control

P
Real-time D Application Environment
r
Captor 1 a o

Thor Park Captor 2 t j

Captor 3 a Project Project e
Relational X Y c
Captor 4 Project
Web DB t
Project Project DB
A
Flux50 Z.1 Z.2 A
Table P
P
Storage I I

33
 SmarThor Data Platform
Control

P
Real-time D Application Environment
r
Captor 1 a o

Thor Park Captor 2 t j

Captor 3 a Project Project e
Relational X Y c
Captor 4 Project
Web DB t
Project Project DB
A
Flux50 Z.1 Z.2 A
Table P
P
Storage I I
Data sources:
EnergyVille/Thor park and external: BMS, PV, weather, market, grid
Historical data and (near) real-time data

34
 SmarThor Data Platform
Control

P
Real-time D Application Environment
r
Captor 1 a o

Thor Park Captor 2 t j

Captor 3 a Project Project e
Relational X Y c
Captor 4 Project
Web DB t
Project Project DB
A
Flux50 Z.1 Z.2 A
Table P
P
Storage I I
Data collection and storage:
Deal with high data volumes and rates
Preprocess and store data in appropriate storage

35
 SmarThor Data Platform
Control

P
Real-time D Application Environment
r
Captor 1 a o

Thor Park Captor 2 t j

Captor 3 a Project Project e
Relational X Y c
Captor 4 Project
Web DB t
Project Project DB
A
Flux50 Z.1 Z.2 A
Table P
P
Storage I I

Data provisioning:
Central data API
Uniform method to retrieve all collected data

36
 SmarThor Data Platform
Control

P
Real-time D Application Environment
r
Captor 1 a o

Thor Park Captor 2 t j

Captor 3 a Project Project e
Relational X Y c
Captor 4 Project
Web DB t
Project Project DB
A
Flux50 Z.1 Z.2 A
Table P
P
Storage I I
Data processing:
Runtime environment for parallel execution of project experiments
Feedback control loop to building management and other energy systems

37
 SmarThor Data Platform
Control

P
Real-time D Application Environment
r
Captor 1 a o

Thor Park Captor 2 t j

Captor 3 a Project Project e
Relational X Y c
Captor 4 Project
Web DB t
Project Project DB
A
Flux50 Z.1 Z.2 A
Table P
P
Storage I I

Data sharing:
Share project results with partners
Share project results across projects

38
 SmarThor Data Platform
Control

P
Real-time D Application Environment
r
Captor 1 a o

Thor Park Captor 2 t j

Captor 3 a Project Project e
Relational X Y c
Captor 4 Project
Web DB t
Project Project DB
A
Flux50 Z.1 Z.2 A
Table P
P
Storage I I

Cloud-based PaaS implementation:

Facilitate scaling with reduced infrastructural management overhead
Multi-tenancy support

39
 SmarThor Data Platform
Control

P
Real-time D Application Environment
r
Captor 1 a o

Thor Park Captor 2 t j

Captor 3 a Project Project e
Relational X Y c
Captor 4 Project
Web DB t
Project Project DB
A
Flux50 Z.1 Z.2 A
Table P
P
Storage I I

Monitoring
Operations & Billing

40
 SmarThor Data Platform
Parameter validation

control strategies
simulations

lab set-ups

API Project x

Project y
databases models
Project z
Multi-energy district Thor

Market data
Weather data
IoT streams

41
 Outline

introduction
aggregate and dispatch: three-step approach
from model-based to data-driven forecasting
SmarThor platform
conclusion

42
 Coordination, control and optimization in smart energy grids –
Challenges and opportunities after 10 years demand response
scalability to millions of devices, interoperability
inspiration from cloud computing, big data
uncertainty and dynamics
stochastic approaches
machine learning
decentralized interaction and interference
multi-agent simulations, peer-to-peer computing
game theory
cybersecurity and dependability
from smart electric power systems to smart multi-carrier energy systems
43
 Relevant recent publications
S. Vandael, B. Claessens, M. Hommelberg, T. Holvoet, G. Deconinck,
"A scalable three-step approach for demand side management of plug-in hybrid vehicles,"
IEEE Trans. Smart Grid, 4(2):720-728, 2013
W. Labeeuw, G. Deconinck, "Residential Electrical Load Model based on Mixture Model
Clustering and Markov Models," IEEE Trans. Industrial Informatics, 9(3):1561-1569, 2013
S. Vandael, B. Claessens, D. Ernst, T. Holvoet, G. Deconinck, "Reinforcement Learning of
Heuristic EV Fleet Charging in a Day-Ahead Electricity Market,"
IEEE Trans. Smart Grid, 6(4):1795-1805, 2015
S. Iacovella, F. Ruelens, P. Vingerhoets, B. Claessens, G. Deconinck, "Cluster Control of
Heterogeneous Thermostatically Controlled Loads Using Tracer Devices,"
IEEE Trans. Smart Grid, 8(2):528-536, 2017
A. van Stiphout, G. Deconinck, “Quantifying the importance of power system operation
constraints in power system planning models: A case study for electricity storage,”
Journal of Energy Storage (Elsevier), Vol. 13, Oct. 2017, pp. 344-358.
F Ruelens, BJ Claessens, S Quaiyum, B De Schutter, R Babuska, R Belmans, “Reinforcement
learning applied to an electric water heater: from theory to practice,”
IEEE Trans. Smart Grid, online since Dec 2016
J. Engels, B. Claessens, G. Deconinck, "Combined Stochastic Optimization of Frequency Control
and Self-Consumption with a Battery," IEEE Trans. Smart Grid, online since Dec 2017.
44
 prof.dr.ir. Geert Deconinck <Geert.Deconinck@kuleuven.be>
full professor KU Leuven
department electrical engineering, faculty engineering science
head of research unit KU Leuven / ESAT / electa on smart grids
6 prof, 13 admin/tech, 7 postdoc, 45-50 PhD students
academic domain responsible EnergyVille
for algorithms, modelling and optimisation

research
coordinated control and ICT in smart grids
teaching
smart distribution systems, industrial automation and control, safety of digital systems,
measurement techniques and high voltage, elektriciteit en domotica, light and lighting
systems, …
service to society
FIET, SMIEEE
45