SMP, MPP or NUMA for Your OLAP System?

Article published in DM Direct Newsletter

February 8, 2002 Issue
By Venky Kandaswamy

Many companies have been growing rapidly, and there is a concern that the present data warehousing
architectures will not meet their expanding needs. Several projections indicate that the volume of activity
may go as high as 5 - 10 times the present load due to increased e-commerce or advanced decision support
tools implementation.

Many software engineers are evaluating changes in hardware and software. Several platforms look
promising. Not all architectures are the same, and some exhibit clear advantages over others under certain
circumstances. If the circumstances change, then a solution that previously met the needs may no longer fit.

I am going to create a hypothetical situation: that I was asked to evaluate a real-time data warehousing
architecture, including any potential move to MPP systems at a particular theoretical company, - in the light
of the expected increase in the volume of activity.

This article is a synopsis of my thinking and analysis. It is based on several real-life studies on real-time
data warehousing efforts. I'll describe the company's predominant profile before evaluating hardware and
software changes.

Theoretical Company Profile

This theoretical company expressed several concerns regarding various architectural approaches and the
potential migration to a new platform.

The primary concern is load volume - both insert of raw data and the querying and reporting of the results.
There is also concern about the costs of database licensing and the costs of the hardware platform. Cost may
become unacceptable if additional database licenses are required or an upgrade to new hardware is
necessary. In that situation, changing to a more cost-effective architecture and to better vendor support and
relations is attractive.

This company does not expect to compromise on the quality of service or the accuracy of data reporting
under any circumstance. Any hardware, software or platform changes must not impact the current quality of
service or data accuracy in a detrimental fashion.

Some of the methods for improving performance have been considered and then discarded. Aggregation
does not appear to be a viable alternative due to the requirements for dynamic trimming and the concern that
it might actually add to the workload. Dynamic trimming in the real-time data warehousing world is the
need to eliminate data that is way out of bounds - possibly due to errors. It invariably skews the results and
needs to be eliminated. It also appears that applications may become more complex to write since they have
to be aggregate aware. Also, it is not clear whether detail data can be maintained in addition to aggregates
due to volume.

Clustering has also been considered and discarded due to reports of unsatisfactory performance, reliability
and cost. Also, the current architecture cannot be clustered without partitioning some of the applications or
functionality to run on different portions of the cluster.

Distributed architectures have also been considered and discarded due to manageability and reliability
concerns.

Areas for Enhancement

Load Performance: The company's engineers recognize that insert and query performance can be
improved. The team has already studied this issue and proposed changes. It was reported that up to 10 times
faster speed was possible with array inserts, parallelization and the use of fast loader utilities.

Query Performance: Most, if not all, queries are set to run in a parallel mode today. Successful methods of
improving this query performance are already being addressed efficiently by this company - with
improvements such as partitioning, indexing, high density and potential use of index organized tables (IOT).
Generally, these are very good online transaction processing (OLTP) style performance improvement
techniques. I, however, suggest that queries and query performance be viewed in the context of the
individual business profile of a company. All businesses have a predominant profile that indicates how they
glean information from data.

Scalability: This is an area of major concern for this company. Given the current architecture, scalability is
mostly dependent on the hardware and software - not the actual application architecture. While these
systems can be scaled using larger machines, different hardware architectures and software, it is also valid
to look at what can be accomplished within the application architecture to ensure scalability. Always take
the approach that scalability is predominantly an application architecture issue. If the application
architecture is designed for intelligent scalability, then the system as a whole will be scalable.

Business Profile

It is interesting to study, at a high level, this company's information collection and delivery profile. There
are typically one or more of three different high-level business profiles associated with any company.

The SMP and MPP Machine Architectures

This discussion requires a baseline understanding of symmetric multiprocessing (SMP) and massively
parallel processing (MPP) machine architectures. Stated simply, an SMP machine has memory and disk that
is equally accessible to any processor (hence the term "symmetric"). Each MPP machine's processors have a
dedicated disk and memory. Processors cannot access each other's dedicated memory except by making a
request through a special inter-processor memory link that acts as a token ring network. This link is very
slow compared to the SMP machine's hardware memory bus. Some MPP architectures overcome this
limitation, to an extent, by using a crossbar switch. This is a special hardware switch that connects the
processors' memory in a matrix. This speeds up inter-memory access but is also expensive and is normally
considered a premium machine. MPP machines such as Teradata and Tandem can have crossbar switches.
IBM's SP2 and NUMA-Q come with either high-speed fiber links or crossbar switches (the latter is more
expensive).

This does not mean that SMP machines are better. The physics behind a hardware bus plated into a circuit
board limits how close they can be before electromagnetic interference becomes unmanageable. Most SMP
machines are limited to about 32 processors and cannot scale beyond that.

The MPP machines, since they do not share anything and do not have a hardware bus, can scale
theoretically to a very large size (hence they are called massively parallel).

Some vendors such as SUN have scaled beyond 32 processors by getting rid of the hardware bus and using
a crossbar switch instead. The SUN UE10000 is one such example, and it can scale up to 64 processors.

Generally, both MPP and SMP machines have their own limitations. SMP machines cannot scale and MPP
machines require that applications be partitionable so they can be evenly distributed across the MPP nodes.
This also means that operations on the MPP machines need to be fairly atomic so they do not need data from
other nodes to complete their operation.

The OLTP Profile

This is the online transaction processing profile. In this case, systems are accessing a fairly large database at
random. They retrieve a few records, update some information and store results in the database. All of these
steps may not occur in all instances - but the idea is that data is retrieved, processed and updated in small
packets across the entire database. If one were to put lights where records are being touched, it would appear
as if the database is "twinkling." The Figures 1 and 2 show a typical OLTP environment on both the SMP-
and MPP- style machines.

Figure 1: OLTP on an SMP Machine

The stars represent individual records or small sets of records that are processed by the SMP machine. The
records are read into shared memory, processed and possibly updated back to the disk. SMP machines
eventually hit a limit where the processors start interfering with each other when they compete for the use of
the SMP memory bus (to access shared memory). They machine also stumbles when the I/O volume
saturates the shared channels to the disk.
 Figure 2: OLTP on an MPP Machine

The same OLTP profile on an MPP machine behaves very differently. Each processor has its own
communication path to disk and memory. Therefore, there is never any contention between processors for
disk channels or buses. It is almost as if there are multiple machines doing their own work without paying
any attention to each other.

When they occasionally need data from some other processor's disk or memory they send the request to the
owning processor - which gets the data and sends it to the requesting processor through the inter-processor
link or crossbar switch. There is usually no caching of data between the processors memory due to
difficulties in maintaining coherency.

As can be seen from Figure 2, a purely OLTP profile can scale enormously using an MPP machine. There is
no limitation except for when data for a transaction comes from several disks belonging to other processors.
Then the requests and contentions on the inter- processor slow the system down.

The OLAP Profile

This is the online analytical processing profile (OLAP), where data is still accessed more or less in a
twinkling fashion as in the OLTP case. However, the packet size is a lot larger. While OLTP profiles
process a few rows per transaction, OLAP profiles process a few thousand rows or more per transaction.
OLAP profiles not only retrieve large number of rows but also process them by attempting to derive a
consolidated result. This could be the sum, average or even count etc. These operations are also called
vector operations since they take one or more rows and return one row as the result. Vector operations may
also be applied on sets of data - such as what happens when you use an SQL aggregate function with a
GROUP BY clause. In contrast, a scalar operation is when some process is applied to a record that returns
an updated version of the record or a new record. There is always one-to-one correspondence between input
and output in a scalar operation. There is always a many-to-one correspondence in a vector operation.

The OLAP profile, thus, is different from the OLTP profile in that larger numbers of records are processed
and vector operations are applied on them. The OLTP profile on the other hand processes a few rows and
usually applies scalar operations on them. See Figure 3.
 Figure 3: OLAP on an SMP machine

As we stated earlier, an OLAP profile applies vector operations on data. Thus, the records or stars are read
from disk and then consolidated using the vector function (such as sum or average) in the SMP memory.
Parallel execution of queries on most systems results in one to two levels of parenting. Parenting is when a
set of processes prepares data for a higher-level process which, in turn, sends it to its parent. There always is
at least one level of parenting in parallel query execution on the major relational platforms (Oracle, DB2 and
Informix).

OLAP processing on SMP machines have the same type of considerations and bottlenecks - such as I/O
channel saturation and memory bus contention. OLAP on SMP hits the limits faster than OLTP on SMP.

Figure 4: OLAP on an MPP machine

As can be seen in Figure 4, vector operations require that data eventually be consolidated in a single process
that runs on a single processor. OLAP processing is almost 100 percent vector operations. Interestingly, this
dramatically increases the traffic on the MPP machine's inter-processor crossbar switch or link.
Unfortunately, as the parallelism level gets higher (more processors or nodes in the MPP system) the bigger
the contention effect - eventually leading to a bottleneck. It should be clear that the OLTP profiles do not
suffer from this limitation since they are mostly scalar operations and do not need to use the inter-processor
bus so much.

The DSS Profile

The decision support services (DSS) profile can be thought of as a very dramatic version of the OLAP
profile. The DSS profile processes millions of records to return a few records to the user querying the
system. Most queries touch the entire database along one or more dimensions for most queries. The biggest
difference between an OLAP and DSS profile is that data is only added and seldom updated. In the OLAP
profile data is added and sometimes, but not frequently, updated. In an OLTP system, the data is very
frequently updated.

The key difference between OLAP and DSS is the response time and the expectations of the system users.
OLAP queried are expected to return results in a few seconds to a few minutes. DSS queries are expected to
produce results in a few minutes to a few hours (and sometimes a few days). Most commercial DSS systems
(or data warehouses) are designed to have both OLAP and DSS profiles.

System Architecture Futures

Building any system architecture must take into account the business, application and hardware/software
perspectives. These issues must be correlated, and any potential architecture must satisfactorily provide for
these perspectives. It is also essential not to drastically alter any existing system - which may cause an
enormous amount of code rewrite, regression analysis and testing.

Business Perspective

The theoretical company has so far been a low-price high volume provider of data services, but there are
several initiatives in place for taking this company's service into the enterprise for blue-chip customers. It is
also the intention of this company to figure prominently in the mission-critical road map of its enterprise
customers. These customers will most likely be ready to pay a premium for high information value,
proactive services and enhanced diagnostic services. This is a strategic objective to step into the high-price,
high- value arena. This will and can be accomplished in many different ways. However, it is difficult to
forecast exactly what applications will look like and how they will be built in several years. This type of
information is crucial to the success of the architecture and its ability to carry the company into the future.

The challenge faced by any architect is how to predict and provide a flexible road map that will serve the
company and its strategic thinkers well into the future. In these situations, the company must study how its
information systems may progress in terms of providing more information from the same amount of data.
The progression of OLAP systems is well understood and well documented in the industry. A company will
most likely follow the same industry trend - given the strategic objectives of becoming a premium service
provider and a mission-critical partner for its enterprise customers.

Most OLAP systems start out being a straightforward aggregator of data. Therefore, the information gleaned
from data is presented in a meaningful aggregated fashion. Once this level is reached and stabilized, OLAP
systems tend to progress into more advanced spaces in information processing.

The general progression of OLAP services tend to follow the four different spaces shown in Figure 5.
 Figure 5: The Four Spaces of OLAP Services

Aggregation Space: This is not to be confused with storing aggregated data. This is simply the space in
which information is gleaned from raw data by using sums, averages, deviations and other statistical
functions. The distinguishing characteristic of this space is that operations occur on a set of data and on
individual data elements without reference to other elements. For example, we simply report average DNS
time or average time to download a .gif.

Variance Space: This is the next step in OLAP evolution, where we glean more information by comparing
elements along some predefined dimension, such as aggregation space numbers across backbones or over
time. What was the DNS lookup time from Chicago vs. Atlanta? Or what was it yesterday vs. today? A
distinguishing characteristic is the comparison of individual elements along one or more preselected
dimensions such as time, backbone, agent group or target URL: How am I doing with respect to my
competitor?

Influence Space: This is the yet another step in OLAP evolution, where the interaction between sets of
elements and between elements themselves is studied to glean more information. Questions such as what
happened when I switched my backbone form ATT to Sprint, or what happened when I added these new
.gifs fall into this space. The distinguishing characteristic is the need to see how the variance space of the
data from one or more dimensions influenced each other.

Mapping Space: Ultimately, one needs to map the influences and variances into a set of rules that can be
proactively applied to predict the behavior. Another way of describing these spaces from a more traditional
business perspective is to see how products fit into these increasingly complex categories: performance
reporting, exception analysis, casual analysis and modeling.

Value of Information: The perceived value of information increases with the progression of OLAP systems
into these advanced spaces. It is also general experience that the more advanced spaces are the ones that are
relatively more mission critical (causal analysis, modeling) than others. Everything is important to
customers - otherwise they would not buy it. The big question is which type of information would they see
as absolutely necessary for the efficient operation of their business.

Architectural Support: Irrespective of whether such advanced OLAP spaces are planned for inclusion in
the company's products, it is the responsibility of the architect to ensure that the system architecture has a
way to grow into these spaces if needed. At the same time, it must also support a larger user base and more
data for the traditional spaces such as aggregation and variance.

Application Perspective

Reduce Coding Changes: The most important consideration for applications is to support code reuse, data
model transparency and host environment transparency. Many of this company's applications are already
designed with these features in effect. Any future architecture must preserve and enhance these
considerations. Ensure that code can be reused where applicable and that data model changes do not impact
applications in any way. Also ensure that the applications can run in multiple host environments as far as
possible. We recommend that applications continue to use the model-view-controller pattern in all of the
database-related sections of the program.

Scalability: Applications must be designed to be scalable. This company's applications currently are
designed with the scalability and the ability to fork off several instances if necessary. Any future
architectural changes must preserve this important feature. However, it may also be useful to determine
where database access can be moved to an IPC-based environment rather than TCP/IP based SQLNet. This
can be done by moving end-point SQL programs to the machine that has the database and then use IPC
instead of the network. It will also be good to use array-based operations where possible.

Reliability: Applications must be robust and failsafe. This company's applications are already designed to
be failsafe and robust. Any future architectural changes must maintain this.

Hardware/Software Perspective

Retaining the Ability to Choose: It is important that one resist proprietary architectures unless absolutely
necessary. Even though most hardware and software systems are proprietary - some make a migration easier
than others. It may not always be possible to stick to open systems such as UNIX and relational databases to
satisfy business requirements. The system architecture must provide the ability to choose between various
proprietary or open systems while still providing for the business and application perspectives.

Scalability: Any hardware or software choices must be scalable. The degree of scalability is always a
contentious issue. How far should we able to scale? The reality of the practical world is that all hardware
and software choices have their limitations. In the previous section, we addressed MPP and SMP
architectures and their affect on OLTP, OLAP and DSS profiles. One must consciously determine a realistic
figure for scalability. Is it 10 times? Is it 20 times? Or is it 100 times? Or is it just four times? If we max out
on hardware/software choice, can we migrate easily to another?

Architectural Decisions

Having pondered the issues and considerations discussed earlier, I have come to the conclusion that one
should address these issues as an integrated whole, not piecemeal. View it in terms of the future of the
business, product directions, strategic aspirations, availability of hardware and software, migration paths,
ease of coding, reliability and a host of other issues.

Guidelines

Since the issues are numerous and complex, it is better to set some objectives and guidelines before
dramatically altering the system.

OLTP profiles scale well. How can a system get as close as possible to an OLTP profile? What this
means is that there is a need to reduce the spread of the data, reduce the number of rows processed per
operation and reduce the level of parenting. Aggregate data. This is good irrespective of whether we
choose an SMP or MPP.
 Do as much as possible within the database. This saves external coding efforts and leverages the
research that goes into building Oracle and DB2 today. Does the database support OLAP processing,
aggregation and other OLAP-related performance enhancements? If so, that should be the first choice. Stay
with hardware and software that supports and understands OLAP and DSS.
Can I make the database fairly static so I can build complex aggregations and also provide for
advanced OLAP spaces? What business decisions can I make to allow the architecture to achieve this -
while not compromising my business users and customers? There are always many ways of achieving the
same business objective.
A successful architecture must scale in and of itself. Never depend solely on hardware or software
to do the job.

Direction

It is possible to accomplish many of the suggestions mentioned without hurrying into a hardware or
software change. While a certain degree of experimentation is required to validate the assumptions - it will
be beneficial to look at the following progression toward a more scalable architecture on the current
platform:

Aggregate Data
Move to geometric measures so trimming can be reduced or eliminated.
Separate out real-time analysis from fairly static analysis.
Current hour must be treated differently from prior hours, etc.
Build a proper robust OLAP dimensional model to guide future architecture.

Architectural Curve

There are well understood, industry-standard methodologies for optimizing OLAP systems. Figure 6 shows
the optimization curve. Optimization methods and philosophies are generally classified as tier 1, tier 2 and
tier 3 levels. This company is doing some tier 1 optimization - although not to the extent possible. For
example, one of the key justifications for using partitioning in OLAP systems is to allow the use of partition
elimination during query processing and star joins, star queries and star optimization. This company
partitions its tables but has not investigated or designed for the use of these advanced features.

Figure 6: The Architectural Design Curve

The challenge of any architecture is to stay as far as possible toward the OLAP optimized curve. Practical
systems are usually somewhere in between these two lines. But the challenge still remains as to how we can
stay on the optimized curve.
Staying on the OLAP optimized curve

It is necessary to look at the business as a whole - applications, hardware, software, strategic initiatives,
product directions, evolution paths for the technology etc. to coordinate and build a system architecture that
will carry the company well into the future.

Conclusion

Any company's business profile will encompass OLTP, OLAP and DSS type profiles. It is necessary that
any system architecture prepare itself now and provide a clear understanding and a road map for these
profiles to coexist and perform efficiently. It is also necessary to accept the fact that the very nature of these
different profiles precludes them from being treated as "the same thing." One size never fits all,
unfortunately.

Scalability is primarily an architectural endeavor. Consider staying with established database platforms such
as Oracle or DB2 etc., since they do provide support for OLAP and generate research into providing
facilities for the OLAP and DSS profiles.

If there is a fear that an SMP system may max out, then a logical progression could be a NUMA type of
machine that can scale up similar to MPP systems. The advantage being that NUMA retains some
advantages of SMP systems such as shared disk and some shared memory between groups of processors to
aid OLAP processing. NUMA machines are well proven and are comparable in cost to MPP. They also run
BSD UNIX and Oracle, DB2 or Informix.

...............................................................................

For more information on related topics visit the following related portals...
DW Design, Methodology and Corporate Performance Management (CPM).

Venky Kandaswamy has more than 15 years of experience in information technology with a deep interest in
knowledge management. Kandaswamy has architected many large successful data warehousing efforts, one
of which won TDWI Best Architected Data Mart award in 1999. He is a partner at Adisya Inc.
(www.adisya.com)