Active Memory Expansion

Active Memory Expansion performance
Updated: 2 November 2012
By Dirk Michel Power Systems Performance 2012 updates by Bob Welgan, IBM Systems ISV Enablement SAP Performance Benchmarking
Active Memory Expansion performance Preface Acknowledgments: Disclaimer: 1 Introduction 2 Active Memory Expansion overview 2.1 Active Memory Expansion concept 2.2 Active Memory Expansion value 3 Active Memory Expansion performance considerations 3.1 Memory expansion factor 3.2 Application response time 4 Active Memory Expansion performance measurements 4.1 ERP overview 4.1.1 ERP workload performance measurements 4.1.2 ERP workload single partition throughput 4.1.3 ERP workload summary 5 Conclusion Appendix A: POWER7 ERP workload server throughput Trademarks and special notices 3 3 3 4 5 5 6 7 7 8 9 9 9 11 13 14 15 18
Active Memory Expansion Performance.doc
Page 2
Preface
This document introduces the basic concepts of Active Memory Expansion, showing the principles of operation and performance characteristics of this new component of the AIX operating system. Active Memory Expansion is available on IBM POWER7 systems starting with AIX 6.1 TL04 SP2. This paper is an update to an earlier paper and it contains data collected from experiments on an IBM POWER7+ system. The audience for this document consists of computer users, administrators, programmers, and performance analysts, as well as team leaders and management who need to understand the basics of this technology. You can find a detailed description of design points and detailed instructions for the configuration of Active Memory Expansion in the AIX information center at
http://pic.dhe.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.prftungd/doc/prftungd/intro_ame_process.htm
and the Active Memory Expansion white paper ibm.com/systems/power/hardware/whitepapers/am_exp.html.
Acknowledgments:
We would like to thank the people who made invaluable contributions to this paper as well as the developers of this remarkable technology. Contributions included authoring, insights, reviews, critiques and reference documents. Co-authors of this document are: Thuy Nguyen, Boyd Murrah, Walter Orb, Joerg Droste, and Jose Escalera
Disclaimer:
All performance data contained in this publication was obtained in the specific operating environment and under the conditions described below and is presented as an illustration. Performance obtained in other operating environments may vary and customers should conduct their own testing.
Page 3
1 Introduction
All computers have a limited amount of random access memory (RAM) in which to run programs. Therefore, one of the perennial design issues for all computer systems is how to make the best use of the entire RAM that is physically available in the system, in order to run as many programs concurrently as possible, in the limited space available. Active Memory Expansion, originally a POWER7 feature, supplies a new technique for making better use of RAM: Portions of programs that are infrequently used are compressed into a smaller space in RAM. This, in turn, expands the amount of RAM available for the same or other programs. Starting with POWER7+, Active Memory Expansion memory page compression and decompression is offloaded to a hardware accelerator. This white paper describes the process of Active Memory Expansion in more detail in the next sections, and then reports measurements on a typical workload which illustrates the beneficial effects. Among the benefits of Active Memory Expansion, this paper shows the following scenarios and their performance results: 1. Reducing the physical memory requirement of an logical partition (LPAR) resulting in 125% memory expansion on a POWER7+ system 2. Increasing the effective memory capacity and throughput of a memoryconstrained LPAR, resulting in a 54% increase in application throughput on a POWER7+ system
Page 4
2 Active Memory Expansion overview

RAM is allocated in contiguous units called pages. In an LPAR with Active Memory Expansion enabled, a certain proportion of the RAM is dedicated to serve as auxiliary storage to hold compressed data. When the operating system determines that a page of uncompressed memory is not currently being referenced, and there is demand for memory, then the content of the page can be compressed and migrated to the auxiliary storage dedicated for Active Memory Expansion. Starting with POWER7+ systems, Active Memory Expansion logic is implemented in both hardware and software. Compression of temporarily unreferenced data reduces the total space requirement for that data and frees up more memory than it costs, leaving more RAM to execute programs. Data is expanded when it is referenced again. Capacity of the total system can be increased with the appropriate combination of available RAM and available processor time to compress and uncompress the content of memory pages.
2.1 Active Memory Expansion concept

The effect of memory compression and the expansion of active memory with the Active Memory Expansion feature are illustrated in the following figure:
Memory
AME
Compressed Pool
Expanded Memory
Figure 1. Effect of memory compression and the expansion of active memory
Figure 1 demonstrates an example of taking memory away from an existing LPAR and running it with less physical memory. The figure shows: 1. For a given data set in RAM, compression of data in inactive (or seldom active) pages reduces the total memory requirement for that data set. If the logical memory size in the LPAR is not changed, this expands the space available for active pages in running programs. Alternatively, the extra space can be given to another LPAR in the same frame, or the number of LPARs in the frame can be increased. 2. In view of the fact that compressed memory cannot be directly used by running programs, the diagram illustrates that the logical memory in any LPAR using Active Memory Expansion must be partitioned into active (uncompressed) and inactive (compressed) page pools.
Active Memory Expansion Performance.doc Page 5
Expanded Memory Size
Uncompressed Pool Real Memory Size
Real Memory Size
3. The degree of compression illustrated in the figure above is purely for purposes of illustration. The compressibility of workloads will vary. 4. Pages in the uncompressed pool will be migrated to the compressed pool when they have been unreferenced for a sufficient length of time and there is a demand for memory pages. Pages in the compressed pool are migrated back on demand by referencing them. These operations are transparent to the application. 5. The size of the compressed memory pool is dynamically controlled by the operating system. It is not fixed. Expansion factor and compressibility of the data determine the size of the compressed pool.
2.2 Active Memory Expansion value

As described in 2.1, Active Memory Expansion increases the memory capacity of a partition. The real value is the potential usage of the extra memory to add more workload, leading to a higher throughput and increased utilization of the partition. Figure 2 illustrates another way Active Memory Expansion can be used to increase utilization and throughput:
System Memory LPAR
System Memory LPAR LPAR
LPAR
LPAR
LPAR
LPAR LPAR LPAR LPAR
LPAR
LPAR
Active Memory Expansion
LPAR
LPAR
LPAR
LPAR
LPAR
LPAR LPAR LPAR LPAR
LPAR
LPAR
LPAR
LPAR
LPAR
LPAR
Figure 2. Reducing LPAR physical memory consumption to free memory
In environments where the number of LPARs is limited by the physical amount of memory that is available to the system, Active Memory Expansion reduces the physical memory consumption of the individual LPARs and frees up memory that can be used for additional LPARs to drive more workload on the system.
Page 6
3 Active Memory Expansion performance considerations

Application performance in an Active Memory Expansion environment depends on multiple factors such as the memory expansion factor, application response time sensitivity and how compressible the data is. The graph below illustrates the general relationship between application response time, application throughput, processor utilization and the percentage memory expansion. The processor utilization increases with larger percentage memory expansion due to more compression and decompression activity, which has an impact on the application response time. An increase in application response time often results in lower application throughput.
Application Throughput
CPU Utilization
Application Response Time

% Memory Expansion
3.1 Memory expansion factor

The memory expansion factor has a direct impact on performance because it has an impact on the sizes of the compressed and uncompressed pools and therefore, the amount of active memory that can be held in uncompressed memory. As the amount of active memory becomes larger than the current size of the uncompressed memory pool, more compression and decompression activity occurs. Figure 3 illustrates the impact on the sizes of the compressed and uncompressed pools for reasonable and aggressive memory-expansion factors.
Page 7

Reasonable Memory Expansion Factor Aggressive Memory
Expansion Factor Uncompressed Pool
True Memory Size
Uncompressed Pool
Large Uncompressed Pool
True Memory Size
Small Uncompressed Pool
Compressed Pool
Compressed Pool
Small Compressed Pool
Large Compressed Pool
Expanded Memory
Expanded Memory
Figure 3. Impact on the sizes of the compressed and uncompressed pools for memory-expansion factors
A reasonable memory-expansion factor maintains a small compressed pool and a relatively large uncompressed pool. An aggressive memory-expansion factor leads to a larger compressed pool and a smaller uncompressed pool.
The performance of a workload is less affected by an aggressive memory-expansion factor if the workload has a small active working set and therefore a low amount of compression and decompression activity. However, the amount of compression and decompression activity increases if the memory-compression factor is set too aggressively. The performance of workloads with a large active memory set is more sensitive to an aggressive memory expansion factor. A reasonable memory-expansion factor can be determined by running the Active Memory Expansion Planning Tool, amepat from the command line.
3.2 Application response time

When an application accesses a page that is currently compressed, the operating system automatically decompresses the page and makes it available to the application. This can increase an applications response time because the application has to wait during the decompression process. Although the time to decompress a single page is very short, a large amount of decompression activity can have a noticeable impact on response time.
Page 8
4 Active Memory Expansion performance measurements

You can use Active Memory Expansion for a wide variety of workloads like enterprise resource planning (ERP), online transaction processing (OLTP), online stock trading, and so on. This section describes the performance results for ERP workloads in an Active Memory Expansion environment.*
4.1 ERP overview

ERP workloads such as SAP ERP provide an effective metric to determine systems performance running common business transactions. In the case where the system is constrained by available memory, there are likely to be unused processor resources available. Active Memory Expansion takes advantage of this by using the idle cycles to manage the physical memory in a way that makes more free space available for the application to run. The following two scenarios will be described in more detail in subsequent sections; they show that you can use Active Memory Expansion to: 1. Reduce the physical memory requirement for an LPAR 2. Increase the memory capacity and throughput of a memory-constrained LPAR The application workload chosen for this study represented a sample set of standard SAP ERP business transactions. Multiple simulated users all ran the same set of predefined transactions. The software configuration for these tests was based on SAP ERP 6.0 (Unicode) and the IBM DB2 10.1 database running on the IBM AIX 7.1 operating system and the POWER7+ processor.
4.1.1 ERP workload performance measurements

The following example demonstrates the use of Active Memory Expansion to reduce the physical memory requirements of an LPAR on a POWER7+ server that is running an SAP ERP workload. The SAP ERP workload was configured to run a set number of users to simulate concurrent business transactions. For the baseline measurement, the workload was run with enough users to drive the processor utilization of the system to approximately 60%. Active Memory Expansion was disabled during the baseline measurement. The next step was to reconfigure the partition with a physical memory configuration matching the virtual memory demand of the SAP ERP workload near the 60% utilization level. Then Active Memory Expansion was enabled and a test sequence was run using the same workload as before with a number of different memory configurations. The results are shown in Table 1.
Users on systems with large amounts of memory might experience performance degradation when using Active Memory Expansion if their application used and benefitted from memory pages that are larger than 4 Kb. Active Memory Expansion automatically reconfigures the system to use only 4 Kb memory pages when enabled.
Memory GB Memory expansion Relative performance Processor %
64.5 0% 1.00 59.9
43.25 49% 1.00 59.8
38.75 67% 1.00 60.8
32.25 100% 1.00 62.0
30.25 113% 1.00 64.4
25.75 150% 0.99 80.6
23.25 177% 0.99 86.1
21.25 204% 0.96 83.4
Table 1. Results with Active Memory Expansion enabled
The memory expansion metric in Table 1is calculated by:

((baseline physical memory - new physical memory) / (new physical memory)) * 100
100 90 80 70 60 50 40 30 20 10 0 0% 25% 50% 75% 100% 125% 150% 175% 200% 225% Memory Expansion %
Figure 4. Processor utilization at various memory-expansion rates
Figure 4 demonstrates the processor utilization of the sample SAP ERP workload at different memory expansions. Figure 5 shows the relative performance of the same workload as in the previous graph. There is virtually no measurable performance impact due to Active Memory Expansion until the memory expansion exceeds 125% and even then the performance impact is very slight. This demonstrates how Active Memory Expansion can help maintain application performance even in a reduced physical memory footprint.
CPU %
Page 10
1.00 Relative performance
0.80
0.60 0% 49% 67% 100% 113% 150% 177% 204% Memory Expansion %
Figure 5. Relative performance
As described in the previous section, the SAP ERP workload tests the ability of a configuration to support multiple users following through the simulated steps to complete a number of business transactions. Figure 5 shows the relative application performance of Active Memory Expansion enabled configurations compared to a base configuration that has all the necessary physical resources available. In this memory-constrained environment, as physical memory is reduced, Active Memory Expansion compensates by using available processor cycles to make more memory available. In spite of the additional work dispatched to the processor, the number of transactions per second completed is not affected. The memory expansion, as shown, can continue to increase as long as processor resources are available. After a threshold of processor availability is crossed, performance stability decreases. As long as there are processor cycles available, Active Memory Expansion allows the user to size the configuration for a workload beyond the ordinary limitations imposed by the lack of available physical memory.
4.1.2 ERP workload single partition throughput

Figure 6 and Figure 7 show the results of increasing the workload in a memoryconstrained partition on a POWER7+ server. The first workload was run in a single partition with Active Memory Expansion turned off, and then the same workload was run with Active Memory Expansion enabled. Figure 6 shows the achieved throughput in transactions per second (TPS) at a specific workload level. Figure 7 shows the respective processor utilization for that workload. The columns in the Figure 7 represent the average number of virtual memory page-in operations per second as an indicator of operating system (OS) paging. The paging statistics are only shown for the test sequence with Active Memory Expansion disabled, as no OS paging occurred during the test sequence with Active Memory Expansion turned on.
Page 11
Throughput
600 Transactions per second 500 400 300 200
OS paging
100 0
Figure 6. Throughput without and with Active Memory Expansion turned on
100 90 80 CPU % Busy 70 60 50 40 30 20 10 0 2496 2624 2752 2880 3008 3136 3264 3392 3520 3648 3776 3904 4032 4160 4288 4416 4544 4672 4800 4928 5056 Users Avg pi AME off AME off AME on
24 96 26 24 27 52 28 80 30 08 31 36 32 64 33 92 35 20 36 48 37 76 39 04 40 32 41 60 42 88 44 16 45 44 46 72 48 00 49 28 50 56
Users AME off AME on
CPU Utilization
200 180 160 140 120 100 80 60 40 20 0 page-ins per sec
Figure 7. Processor utilization without and with Active Memory Expansion turned on
Up to the 3,264 user level, the throughput and processor utilization are nearly identical in both cases. With Active Memory Expansion turned off, there is already a significant amount of OS paging at the 3,264 user level starting to affect the throughput. At 3,392 users the throughput actually decreased because of heavy OS paging. At this level, you are using only 60% of the available processor resources; however the simulated users are suffering from increased response times (and thus decreased throughput). After turning Active Memory Expansion on, you can easily run the same workload and then scale the load up all the way to 5,058 users and a processor utilization of 90%. The throughput increased by 54% going from 3,392 users to 5,056 users.
Page 12
4.1.3 ERP workload summary

The test results in the previous sections show that partitions running a standard SAP ERP workload are good candidates for turning Active Memory Expansion on. Note that the simulated environment used SAP ABAP application servers only. SAP Java servers were not part of the test-case. The test showed that you can use Active Memory Expansion to reduce the physical memory requirement of a partition with little impact to transaction throughput. You can also use Active Memory Expansion to overcome memory constraints in a partition and thus enable the partition to handle additional workload. However, the test cases proved that you can add new partitions to the server as well. A common scenario could be to add partitions for additional two-tier test, development, or training systems on servers with spare processor and limited memory capacity. The scenarios used in these tests showed very good compressibility for our workloads two-tier systems running the database and application server instances within the same partition.
Page 13
5 Conclusion
The scenarios of reducing physical memory requirement of a partition, and increasing a partitions memory capacity outlined in this paper have shown that Active Memory Expansion for AIX on POWER7+ technology provides a significant improvement in system throughput and utilization by expanding system memory capacity.
Page 14
Appendix A: POWER7 ERP workload server throughput

This appendix shows the results of testing Active Memory Expansion on a POWER7 server. There were no test updates on POWER7+ for the update of this paper. However, you can use Active Memory Expansion in a similar way on a POWER7+ server to achieve the following goals: Reduce the physical memory requirement of a single partition without impact to the throughput Increase the throughput in a single memory constrained partition
The next set of measurements conducted on a POWER7 system was intended to demonstrate Active Memory Expansion in a production-like setup with multiple partitions running on a single server. This test was conducted on an AIX 6.1 system with DB2 Version 9.5. This test simulated a SAP 3-tier setup with a single SAP system on a server with 48 GB of physical memory. The first partition was running the SAP ERP database server and an application instance. Three more partitions were configured as SAP application servers connected to the database server running in the first partition. Table 2 shows the configuration of the server partitions: Partition Number of processors Partition memory (GB) Active Memory Expansion off 20 14 14 0* Active Memory Expansion on 18 10 10 10 Database sever and application server Application server Application server Application server Role
1 2 3 4
8 8 8 8
* Partition 4 was deactivated during the runs with Active Memory Expansion turned off
Table 2. Configuration of the server partitions
For the runs with Active Memory Expansion disabled, the simulated users were distributed over the first three partitions. The fourth partition was deactivated, as all of the servers installed physical memory was used. After turning Active Memory Expansion on, the physical memory allocation for each of the first three partitions was reduced. This freed up 10 GB of physical memory, which allowed the activation of the fourth partition. The simulated users were then distributed over all four partitions. Figure 8 and Figure 9 show the results of scaling up the workload for both configurations. Note that these are shown as column graphs because the increase in workload was not done in equidistant steps (unlike in the previous section). Figure 8 the combined throughput of all active partitions in transactions per second (TPS) at a specific workload level. Figure 9 shows the respective processor utilization of the physical server for that workload. For the test case with Active Memory Expansion
turned off, the eight deactivated processors of partition 4 were included in the processor utilization calculation as idle processors.
Server Throughput
500
450
400 TPS 350
heavy OS paging
300
250 2900 3200 3780 Users AME off AME on 4500 5000
Figure 8. Server throughput
Server Utilization
100
80
heavy OS paging
CPU % 60
40
20
0 2900 3200 3780 Users AME off AME on 4500 5000
Figure 9. Server utilization
For the first two data points, the throughput and processor utilization are almost identical for both test cases, regardless of whether the simulated users were distributed over three partitions (with Active Memory Expansion turned off) or over four partitions (with Active Memory Expansion turned on). The partitions that had Active Memory Expansion turned off started to page in the operating system at the 2900 user level. Because paging is unwanted in a production environment, the test used the 2900 user level as
baseline for our comparison in this section. At the 3780 user level, the throughput for the configuration with Active Memory Expansion turned off decreased caused by heavy paging in the operating system. The processor utilization actually dropped as well as the runable user threads were slowed down by having to wait for their required memory pages to be paged in. In this particular test-case, the OS statistics showed an average of 400 page-in operations per second in a partition, with peaks of more than 4800 page-ins per second. The average response time of the simulated users went up from below 200 milliseconds to over 4 seconds. In real life, a system with such a heavy amount of OS paging would be basically unusable, even though there is plenty of spare processor capacity to handle additional workload. The configuration with four partitions and Active Memory Expansion enabled was easily able to handle the same workload and subsequently, the workload was scaled up to 5000 simulated users with a subsecond response time. The throughput increased by 60% going from 3200 users to 5000 users. The total virtual memory demand for all partitions at the 5000 user level was a little more than 65 GB, so the Memory Expansion for this test-case was about 35%. This number is lower than the ones shown in section 4.1.1 ERP workload performance measurements for two reasons: The workload was significantly increased and the major part of the spare processor capacity was used to process the increased business workload. The virtual memory footprint of an application server-only partition is relatively smaller than a partition running both a database and an application server. For the same workload, the percentage of actively used memory pages in an application server-only system is higher than in a database and application server partition.
A test case with four partitions each running its own SAP database and application server instance would result in overall higher virtual memory demand and lead to higher memory expansion numbers. A test case similar to the one in section 4.1.1 ERP Workload Performance Measurements using the four-partition setup with a fixed workload of 3600 users was also run. During the test sequence, the physical memory configuration for each partition was reduced. Table 3 shows the test results for this test-case: Memory GB Memory expansion Throughput (TPS) Processor % 48 0% 356 60 44 9% 356 60 39 23% 355 71 34 41% 353 77 32 50% 354 81 28 71% 343 90
Table 3. ERP workload performance with 3600 users
The first row shows the combined physical memory allocation for all partitions. The measured throughput stays virtually the same up to 50% memory expansion and decreased slightly at the 71% memory expansion level.
Trademarks and special notices

Copyright. IBM Corporation 2012. All rights reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. AIX, DB2, IBM, the IBM logo, ibm.com and POWER7+ are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both: Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. The information provided in this document is distributed AS IS without any warranty, either express or implied. The information in this document may include technical inaccuracies or typographical errors. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Photographs shown are of engineering prototypes. Changes may be incorporated in production models.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.
POW03038-USEN-01
Page 19

Active Memory Expansion

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Active Memory Expansion

Uploaded by

Copyright:

Available Formats

Active Memory Expansion performance

Updated: 2 November 2012