Re: PostgreSQL Benchmark scoping document

From: Scott Mead <scottm@openscg.com>

To: Robert Bernier <robert7390@comcast.net>
CC: Timothy Harrison <Timothy.Harrison@cisecurity.org>
Date: 06/01/16 11:15 AM

See my comments in-line.

On Wed, Jun 1, 2016 at 10:41 AM, Robert Bernier <robert7390@comcast.net> wrote:

Hi Tim and Scott,

On Wednesday, June 01, 2016 12:42:52 PM you wrote:

> I’m working on building a scoping document

Would this be similar to the table of contents of your typical Benchmarks document?

> and need your input regarding .....

see comments below ...

> operating systems,

The two operating systems mostly commonly used includes:

Linux

Free BSD (BSD family)

Agreed, specifically, for production deployment.

There is a third, Microsoft windows, but one can consider that an edge case.

The only caveat here is that Windows has the largest download base. I would hate to focus too much on it because it is primarily deployed by developers, but Windows tends to be the
very first install vector that postgres gets at an org. This is especially true for large enterprise shops that enforce Windows on laptop / desktop / workstations.

I'm going to add another distinction into our mix, i.e. Linux Distros, :

Red hat/Centos

Debian/Ubuntu

There are of course others but we have to stop somewhere. Both distro families have distinct file system installs that at some point forces us to distinguish them.

> deployment/configuration considerations,

And this is where life gets interesting; deployment/configuration considerations are usually based upon the OS/distro. The five most common, not including cloud configurations, are:

Red hat/Centos

Debian/Ubuntu

Community (binary packages available from postgresql.org )

Source code, custom compiles (this one is really tricky, especially when a secret sauce approach is used)

You'll find that the community packages become extremely important for those using Red hat/Centos. For example, Centos ver 6.0 still allows installing postgres 8.4 even though it
reached end of life back in July 2014.

Agreed, 100%. Typically, the community deploys for os 'families' (RHEL base or Debian base). As Robert points out, you have to add the dimension of install source:

PGDG packaging (Postgres Global Development Group)

Distribution specific packaging
Individual vendor installers (OpenSCG, EDB, etc...)
Source

The one noticeable miss from this list which has seen massive up-take is Amazon's RDS. Their postgres offering is getting deployed many times over what I have seen behind the firewall.
Amazon doesn't speak to the number of deployments, but, back-envelope across our customer base is showing me a high RDS to Custom deployment ratio. Especially for dev / test / qa.

Personally, I think it makes sense to 'draw a line' at the PGDG packaging, for completeness however, I feel it's important to mention that other paths exist. Ubuntu especially has
a heavily modified release that has wildly different security defaults.

> versions of PostgreSQL

Fortunately, this isn't a big deal as the benchmark can be written in such a way as to conform for all versions of postgres that have not reached end of life.

> ... I currently have Generic Linux; RHEL, Suse, Ubuntu, and Amazon Linux; and PostgreSQL 9.1-9.5, respectively. These were an initial estimate of what I determined may be in-scope
and I fully expect them to change. That being the case, I’m open to whatever you think is reasonable and/or most commonly used. Please let me know if you have any questions.

I've noticed the benchmark documentation uses "levels". Perhaps, if it's within the currently defined document standard, the first level centers on common issues. Thereafter,
subsequent levels would focus on further techniques as the installation and implementation diverges.
 It would be constructive to take into consideration the development life-cycle as it evolves through the following phases:

Development (DEV)

Quality Assurance (QA)

Staging

Production (PROD)

We want the database to be easy to work with while it's overall initial design would not be drastically altered as it is morphs from a developer's vision into a production ready datastore.
This suggests that the initial hardening is already in place. Further hardening is expressed in the configuration files , as it moves from one phase to the next.

Agreed, other than this (as previously stated), I think RDS is a critical 'mention'.

FYI, I like the MSSQL benchmark table of contents. I especially like its inclusion of Application Development Practices.

Comments?

--Scott
--
Robert Bernier
robert7390@comcast.net
206-407-9298

--
--
Scott Mead
Sr. Architect
OpenSCG
http://openscg.com