Professional Documents
Culture Documents
1
What is DISQUS?
2
What is DISQUS?
dis·cuss • dĭ-skŭs'
http://disqus.com/about/
3
What is Scale?
Number of Visitors
300M
250M
200M
150M
100M
50M
4
Our Challenges
5
Our Challenges (cont’d)
• High availability
• Not a destination site
• Difficult to schedule maintenance
6
Server Architecture
7
Server Architecture - Load Balancing
• Load Balancing • High Availability
• Software, HAProxy • heartbeat
• High performance, intelligent
server availability checking
• Bonus: Nice statistics reporting
• ~100 Servers
• 30% Web Servers (Apache + mod_wsgi)
• 10% Databases (PostgreSQL)
• 25% Cache Servers (memcached)
• 20% Load Balancing / High Availability
(HAProxy + heartbeat)
• 15% Utility Servers (Python scripts)
9
Server Architecture - Web Servers
• Apache 2.2
• mod_wsgi
• Using `maximum-requests` to
plug memory leaks.
• Performance Monitoring
• Custom middleware
(PerformanceLogMiddleware)
• Ships performance statistics
(DB queries, external calls,
template rendering, etc) through
syslog
• Collected and graphed through
Ganglia
10
Server Architecture - Database
• PostgreSQL
• Slony-I for Replication
• Trigger-based
• Read slaves for extra read capacity
• Failover master database for high
availability
11
Server Architecture - Database
12
Server Architecture - Database
13
Our Data Model
14
Partitioning
15
Vertical Partitioning
Vertical partitioning involves creating tables with fewer columns
and using additional tables to store the remaining columns.
http://en.wikipedia.org/wiki/Partition_(database)
16
Pythonic Joins
posts = Post.objects.all()[0:25]
17
Pythonic Joins (cont’d)
18
Designating Masters
19
Routing by Application
class ApplicationRouter(object):
def db_for_read(self, model, **hints):
instance = hints.get('instance')
if not instance:
return None
app_label = instance._meta.app_label
return get_application_alias(app_label)
20
Horizontal Partitioning
Horizontal partitioning (also known as sharding) involves splitting
one set of data into different tables.
http://en.wikipedia.org/wiki/Partition_(database)
21
Horizontal Partitions
22
Routing by Partition
class ForumPartitionRouter(object):
def db_for_read(self, model, **hints):
instance = hints.get('instance')
if not instance:
return None
return get_forum_alias(forum_id)
# What we used to do
Post.objects.filter(forum=forum)
23
Optimizing QuerySets
24
Removing the Cache
# 1 query
qs = Model.objects.all()[0:100]
# 1 query
qs = qs.filter(foo=bar)
25
Removing the Cache (cont’d)
http://gist.github.com/550438
26
Atomic Updates
27
Atomic Updates (cont’d)
post = Post(pk=1)
# a moderator approves
post.approved = True
post.save()
Request 2
post = Post(pk=1)
# the author adjusts their message
post.message = ‘Hello!’
post.save()
28
Atomic Updates (cont’d)
post = Post(pk=1)
# a moderator approves
Post.objects.filter(pk=post.pk)\
.update(approved=True)
Request 2
post = Post(pk=1)
# the author adjusts their message
Post.objects.filter(pk=post.pk)\
.update(message=‘Hello!’)
29
Atomic Updates (cont’d)
http://github.com/andymccurdy/django-tips-and-tricks/blob/master/model_update.py
30
Delayed Signals
31
Delayed Signals (cont’d)
delayed_save.connect(my_func, sender=Post)
32
Caching
• Memcached
• Use pylibmc (newer libMemcached-based)
• Ticket #11675 (add pylibmc support)
• Third party applications:
• django-newcache, django-pylibmc
33
Caching (cont’d)
34
Caching (cont’d)
35
Caching (cont’d)
36
Caching (cont’d)
37
Transactions
38
Transactions (cont’d)
• Tips:
• Use autocommit for read slave databases.
• Isolate slow functions (e.g., external calls,
template rendering) from transactions.
• Selective autocommit
• Most read-only views don’t need to be
in transactions.
• Start in autocommit and switch to a
transaction on write.
39
Scaling the Team
40
Keeping it Simple
41
Setting Up Local
42
Sane Defaults
settings.py
from disqus.conf.settings.default import *
try:
from local_settings import *
except ImportError:
import sys, traceback
sys.stderr.write("Can't find 'localsettings.py’\n”)
sys.stderr.write("\nThe exception was:\n\n")
traceback.print_exc()
local_settings.py
from disqus.conf.settings.dev import *
43
Continuous Integration
44
Continuous Integration (cont’d)
45
Testing
46
Testing (cont’d)
Query Counts
# failures yield a dump of queries
def test_read_slave(self):
Model.objects.using(‘read_slave’).count()
self.assertQueryCount(1, ‘read_slave’)
Selenium
def test_button(self):
self.selenium.click('//a[@class=”dsq-button”]')
Queue Integration
class WorkerTest(DisqusTest):
workers = [‘fire_signal’]
def test_delayed_signal(self):
...
47
Bug Tracking
48
django-sentry
http://github.com/dcramer/django-sentry
49
django-sentry (cont’d)
http://github.com/dcramer/django-sentry
50
Feature Switches
51
Feature Switches (cont’d)
52
Final Thoughts
53
Housekeeping
Birds of a Feather
Want to learn from others about
performance and scaling problems?
Or play some StarCraft 2?
We’re Hiring!
54
Questions
55
References
django-sentry
http://github.com/dcramer/django-sentry
SkinnyQuerySet
http://gist.github.com/550438
django-newcache
http://github.com/ericflo/django-newcache
56