OpenStack upgrades

OpenStack Summit 2013

Hong Kong

Whoarewe
Sbastien Han
Frdric Lepied
Mehdi Abaakouk
Working for eNovance
Company blog: http://techs.enovance.com/

Worldwide offices
coverage
We design, build and run clouds anytime
-

Problems
State of arts

Key principles
do not log in to the servers
do not manually install packages
do not manually edit configuration files
do not manually restart daemons
Puppet, in combination with GIT is the only one managing
the configuration.

Its all about architecture

design
We need redundancy to perform upgrades without
downtime:
Active / active setup Loadbalancer
Active / passive setup
Databases must be replicated : Galera / MongoDB

Rollback
Even with a good QA system, problems might rise
in production thus we need a rollback mechanism.

Solution
Well, ours

Breaking point
Puppet doesnt install packages anymore.
It only manages configurations. So operating systems are
shipped with all the packages installed.

eDeploy solution
Change the abstraction level
Manage updates using sub-trees
2 kinds of sub-trees: data and programs
Data is not updated only programs

Example:
Data: /var/lib/mysql, /var/log...
Program: /usr, /lib...

Consequences
Prepare trees before installation or upgrade
Debootstrap/yum + chroot magic

Install in 3 phases:
Hardware detection
Hardware configuration
Tree copy

Update:
Rsync Prog sub-trees.
Script to adapt Data and restore config

eDeploy - Overview
Manage system provisioning by software role and
hardware profile
Reproduce provisioning easily
Manage upgrades and rollbacks
Efficient in term of expressiveness and performance

QA
Testing systems

Principles
Everything is versioned:
Jenkins jobs
Puppet modules/manifests
eDeploy system images
Ansible recipes

This is what we get

The upgrade process becomes:
Reproducible
Automated
Testable

Methodology
Upgrades with (almost) no downtime

Things that you must

consider
Architecture design
MySQL schemas
Do backups!

Configuration management
and Orchestration
Puppet is responsible for the configuration of a node:
Upgrade the configuration files only (no packages
upgrades!)
Restart services
Ansible orchestrates the process upgrade.

Components dependency

Base of the process

Database schema
But what if the database schema needs to be updated?
Just upgrade the database schemas at the end of the
orchestration! Not yet (Icehouse?)

But with DB schema upgrade

Summary
Follow best practices
Architecture matters
Automation is mandatory
Tests, tests, tests

Many thanks!
Questions?

Contact:
sebastien.han@enovance.com
frederic.lepied@enovance.com
mehdi.abaakouk@enovance.com