Zenoss Performance and Latency tuning

Andreas Trawoeger 109 posts since Apr 10, 2008 I did some testing over the last couple of days on how to tune the Zenoss Web Gui experience. Our Zenoss Core server is currently running on a recycled 1U server that was never designed for high I/O loads. Especially zenperfsnmp can be very I/O bound and easily saturates our internal raid controller which only provides disk mirroring and doesn't have any write cache. In the original configuration this lead to a situation where the Zenoss Web Gui became practically unusable (wait times of up to 1 minute or more) every 5 minutes. Because zenperfsnmp starved the rest of the system to dead. The suggested modification didn't improve overall performance. But they managed to push the maximum Zenoss page load time into a much more enjoyable time range of 5-10 seconds. Latency tuning involves a loot of Voodoo and can be very hardware dependent. So please take this advice with a big grain of salt: Use Firefox 3.0: Firefox 3.0 has a greatly improved caching and javascript engine. Using Firefox 3 speeds up Zenoss a lot without having to do any low level kernel stuff. Use a battery buffered write cache: Much has been said about Zenoss CPU + Memory requirements. But if you can don't forget to either buy a storage controller that supports a battery buffered write cache or use a high performance storage systems. If you don't you easily end spending a lot of CPU cycles in I/O wait state. Use the Linux Deadline I/O scheduler: This can be done by adding "elevator=deadline" as a kernel parameter in your grub bootloader configuration. The Deadline scheduler can hurt your overall performance, but improves latency for processes that mainly read. Tune your Virtual Memory configuration: Buffer and Virtual Memory tuning is an area of much debate and wild guessing, but the following parameters in /etc/ syctl.conf improved latency on our system: ProxyPass / http://localhost:8080/ retry=5 ProxyPassReverse / http://localhost:8080/ Use Apache mod_proxy and mod_cache: Using mod_proxy can be nice, because it allows you to easily redirect request to Zenoss: CacheEnable mem / But mod_proxy alone doesn't add any performance gains and the same functionality could be gained using Apache rewrite rules. What can improve latency is using mod_proxy in combination with mod_cache and memory web caching: vm.vfs_cache_pressure = 5 vm.swappiness = 5 vm.overcommit_memory = 2 vm.dirty_background_ratio = 5 vm.dirty_ratio = 80 vm.dirty_writeback_centisecs = 3000

Generated by Jive SBS on 2009-11-07-07:00 1

Zenoss Performance and Latency tuning

vm.dirty_expire_centisecs = 9000 net.core.rmem_max=16777216 net.core.wmem_max=16777216 net.ipv4.tcp_rmem=4096 87380 16777216 net.ipv4.tcp_wmem=4096 65536 16777216 net.ipv4.tcp_no_metrics_save=1 net.ipv4.tcp_congestion_control=bic This list is far from complete and different hardware probably will lead to different results. So feel free to comment.

Chet Luther 1,199 posts since May 22, 2007 1. Zenoss Performance and Latency tuning Jun 26, 2008 8:15 AM On Jun 26, 2008, at 7:42 AM, atrawog wrote:

But mod_proxy alone doesn't add any performance gains and the same functionality could be gained using Apache rewrite rules. What can improve latency is using mod_proxy in combination with mod_cache and memory web caching: Code: CacheEnable mem /
Lost of good suggestions, but I just wanted to warn people about a potential gotcha with mod_cache. Don't try to use mod_cache on an Apache reverse proxy to Zenoss unless you have httpd-2.2+. Earlier versions do not play nicely with Zope and will cause very strange behavior. _______________________________________________ zenoss-users mailing list zenoss-users@zenoss.org http://lists.zenoss.org/mailman/listinfo/zenoss-users

jmp242 1,977 posts since Mar 7, 2007 2. Zenoss Performance and Latency tuning Jun 26, 2008 8:45 AM in response to: Chet Luther I'm also enjoying better performance with Opera 9.5 vs 9.2x - just getting an up to date browser I'm sure will improve your web experience, including the Zenoss dashboard.

Generated by Jive SBS on 2009-11-07-07:00 2

Zenoss Performance and Latency tuning

-James Pulver Information Technology Area Supervisor LEPP Computer Group Cornell University _______________________________________________ zenoss-users mailing list zenoss-users@zenoss.org http://lists.zenoss.org/mailman/listinfo/zenoss-users

netdata 203 posts since Jun 22, 2007 3. RE: Zenoss Performance and Latency tuning Jan 29, 2009 8:04 AM in response to: jmp242 Cool tips. I do seem to have a benefit thanks to the caching of apache. But I have a question. On my apache I have also http://myserver/phpmyadmin Enabling reverse proxy in the way explained here will cause that /phpmyadmin is proxyed to zenoss on port 8080 making an URL of: http://myserver:8080/phpmyadmin any idea how I can disable that? So that the proxy ony acts on the / thanks

fdeckert 103 posts since Jul 2, 2008 4. RE: Zenoss Performance and Latency tuning Feb 26, 2009 2:41 AM in response to: netdata I had the same issue with port 8080 being displayed in the javascript code. but with help of thread http://forums.zenoss.com/viewtopic.php?t=8835&highlight=apache+8080 I fixed it :-) The trick lies in ProxyPreserveHost On -Florian Deckert SopraGroup - France

Generated by Jive SBS on 2009-11-07-07:00 3