Professional Documents
Culture Documents
LAMP
what happens after you’re scalable
• (right ?)
• But:
• How many servers ?
BUT, um, wait....
• How many databases ?
• How many webservers ?
• How much shared storage ?
• How many network switches ?
• What about caching ?
• How many CPUs in all of these ?
• How much RAM ?
• How many drives in each ?
• WHEN should we order all of these ?
All of these are very legitimate questions. But what is capacity ?
some stats
• - ~35M photos in squid cache (total)
• - ~2M photos in squid’s RAM
• - ~470M photos, 4 or 5 sizes of each
• - 38k req/sec to memcached (12M
objects)
• - 2 PB raw storage (consumed about
~1.5TB on Sunday)
•
capacity
enough
not
Don’t buy too much equipment just because you’re scared/happy that your site will
explode.
3 main parts
Planning includes realizing what you have right NOW, and predicting what you’ll need later.
Deployment includes making sure you can deploy new capacity easily.
Measurement is ongoing, all the time, save as much data as you can.
boring queueing theory
• Forced Flow Law:
• X =Vi i x X0
Little’s Law:
N=XxR
Service Demand Law:
Di = Vi x Si = Ui / X0
•
We can use these...but they’re boring. And take a long time. Too long.
Don’t read books with these equations in them to learn about capacity planning
for web operations.
my theory
I don’t have time for a dissertation on how many MySQL machines we’ll need in an
abstract “future”.
predicting the future
Can’t predict the future until you know what the past and present are like.
Must find out what you have right now for capacity.
• Examples:
•
We get 20-40% more uploads on first work day of the year than any previous peak the
previous year.
40-50% more uploads on Sundays than the rest of the week, on average.
What do you have
NOW ?
• MySQL (disk IO ?)
• SQUID (disk IO ? or CPU ?)
• memcached (CPU ? or network ?)
• boring
• to use in capacity planning...not usually
worth the time
• not representative of real load
Benchmarks are fine for getting a general idea of capabilities, but not for planning.
Artificial tests give artificial results, and the time is better used with testing for real.
• test in production
Don’t be afraid, it’s ok. :)
Best approximation to how it will perform in real life, because it’s real life.
This means build into the architecture mechanisms (config flags, load balancing, etc.) with
which you can deploy new hardware easily into (and out of) production.
what do you expect ?
• define what is acceptable
• examples:
• squid hits should take less than X
milliseconds
• SQL queries less than Y
milliseconds, and also keep up with
replication
• measurement is a necessity.
• it’s not optional.
speed freaks and tweakers and overclockers out there: suck it up.
measurement and pretty graphs are good.
http://ganglia.sf.net
- Uses multicast and/or unicast to squirt xml data into an rrdtool frontend.
- Super super easy to make custom graphs
- originally written to handle stats data from HPC clusters
gmetad
You have redundant machines in clusters, why not use the redundancy for cluster
stats as well ?
gmetad
One box goes away, then another can be used as a spokesperson for that cluster.
super simple graphing
• #!/bin/sh
• /usr/bin/iostat -x 4 2 sda | grep -v ^$ | tail -4 > /tmp/
disk-io.tmp
• UTIL=`grep sda /tmp/disk-io.tmp | awk '{print $14}'`
• /usr/bin/gmetric -t uint16 -n disk-util -v$UTIL -u '%'
Basically, if you can produce a number on the command line, then you can spit it into
rrdtool with ganglia.
memcached
• GraphClick
• http://www.arizona-software.ch/
applications/graphclick/en/
•
This means that you can intuitively tie photos growth to user growth.
Not rocket science, and expected, but knowing how steep that line is...is important.
measurement examples
queries
A week view of one of our MySQL machines. A Dell PE2950 with 6 disk 15K RPM SCSI
drives,
RAID10, with 16GB of RAM. 2x Quad Core CPUs, Intel Clovertown E5320 @ 1.86GHz.
Watch disk IOwait. Any amount of disk utilization is “ok” if IOwait doesn’t increase
dramatically.
By “ok” we mean:
- no slave lag
- query response time is still acceptable
What we know now
Everyone loves caching. Everyone loves memory. Everyone thinks RAM is the answer to
everything.
They might be right.
caching ceilings
squid, memcache
• working-set specific:
• - tiny enough to all fit in memory ?
• - some/more/all on disk ?
• - watch LRU churn
Least Recently Used replacement policy chooses to replace the page which has not been
referenced for the longest time.
churning full caches
• Ceilings at:
• - LRU ref age small enough to affect
hit ratio too much
• - Request rate large enough to affect
disk IO (to 100%)
We MISS on purpose for some larger objects. Usually the hit ratio exists somewhere
between 75-80%.
LRU reference age
Fits with request rates. More requests for more unique objects, more churn.
Low point = 0.15 days = 3.66 hours.
Max point = 0.23 = 5.52 hours.
Squid request rate goes up over time, so week-by-week graphs are made.
This is an example, we know that we can do actually 900-1000 req/sec without
the response time for hits getting above 100ms.
So for every 900 req/sec we expect, we should be adding another squid machine.
not full caches
•http://flickr.com/photos/gaspi/62165296/
•http://flickr.com/photos/marksetchell/27964330/
•http://flickr.com/photos/sheeshoo/72709413/
•http://flickr.com/photos/jaxxon/165559708/
•http://flickr.com/photos/bambooly/298632541/
•http://flickr.com/photos/colloidfarl/81564759/
•http://flickr.com/photos/sparktography/75499095/