The RMF Monitor III Data Portal

z/OS Performance Data on Demand

System z Technical Conference April 16-20, 2007, Munich Session ZP06

Harald Bender IBM Deutschland Entwicklung GmbH Schnaicher Strasse 220 71032 Bblingen, Germany bhbe@de.ibm.com

zSeries Technical Conference 2007

Session ZP06

Content:
Introduction ............................................................................................................... 3 Lesson 1 Setting up the Connection ....................................................................... 4 Lesson 2 Exploring the Overview PerfDesk............................................................ 6 Lesson 3 Understanding the Resource Model ........................................................ 7 Lesson 4 Exploring the Sysplex Configuration........................................................ 8 Lesson 5 Resources and Metrics ......................................................................... 10 Lesson 6 Monitoring the Sysplex Performance..................................................... 12 Lesson 7 Define your own PerfDesk..................................................................... 13 Lesson 8 Advanced Features ............................................................................... 14 Lesson 9 Creating Spreadsheets from DDS Data................................................. 18 Lesson 10 Monitor III Reports............................................................................... 20

Figures:
Figure 1: RMF Distributed Data Server Architecture.................................................. 4 Figure 2: DDS Welcome Screen................................................................................ 5 Figure 3: DDS Overview PerfDesk ............................................................................ 6 Figure 4: RMF Monitor III Resource Model................................................................ 7 Figure 5: Contained Resources of the Sysplex.......................................................... 8 Figure 6: Single valued and List valued Metrics....................................................... 10 Figure 7: Sysplex wide Aggregation of Metrics ........................................................ 11 Figure 8: Creating a new Monitoring Window .......................................................... 12 Figure 9: My View Storage Utilization PerfDesk.................................................... 13 Figure 10: DDS Data in a Spreadsheet Application ................................................. 18 Figure 11: Monitor III Sysplex and System Level Reports........................................ 20 Figure 12: Monitor III Processor Delay Report......................................................... 21 Figure 13: Print and Export Facility.......................................................................... 22 Figure 14: Session with Multiple RMF Sysplex Reports........................................... 23

zSeries Technical Conference 2007

Session ZP06

Introduction
The z/OS RMF Distributed Data Server (DDS) provides a web frontend to sysplex wide RMF Monitor III online performance data. Using a web browser that can display XML documents with XSL style sheets (like Mozilla 1.4 or above, Netscape 7.0 or above or Microsoft Internet Explorer 5.5 or above), you have instantly access to more than 600 z/OS performance metrics. However, the DDS Web Browser interface is not as powerful as the RMF PM Java Client with advanced built-in features: display of multiple intervals at a glance advanced guided-path analysis support for filters and workscopes export to spreadsheet format If you are not using it already, take a look at www.ibm.com/servers/eserver/zseries/zos/rmf/rmfhtmls/pmweb/pmweb.htm RMF DDS technology is available for both z/OS as well as for Linux on zSeries and x86 Linux. If you are interested in the Linux data gatherer, refer to www.ibm.com/servers/eserver/zseries/zos/rmf/rmfhtmls/pmweb/pmlin.htm The DDS is part of the RMF product and doesnt require additional charges. This tutorial explains the DDS architecture and gives you a detailed introduction how to setup and use it. You will learn how you can explore the performance of your system, list metrics, and customize the performance view of your sysplex. In addition the tutorial contains some examples how to utilize the DDS as a web server returning valuable RMF performance data in XML format.

zSeries Technical Conference 2007

Session ZP06

Lesson 1 Setting up the Connection

The core component for the web browser based access to z/OS performance data is the RMF Distributed Data Server (DDS). This is one server address space with sysplex wide scope. That is, this server has only to be started on one single system within your sysplex. However, when you contact this server, it is possible to retrieve data from any system within your sysplex. The DDS automatically contacts all RMF Monitor III instances in the sysplex to provide this sysplex wide data source for reporting functions. RACF services are used to check authorization to access RMF data (by checking TSO user ID and password).

RMF Monitor III RMF Monitor III RMF Monitor III

z/OS RACF TSO User Authentication

RMF Sysplex Data Server

RMF Distributed Data Server (GPMSERVE)

Web Browser Frontend

Figure 1: RMF Distributed Data Server Architecture The DDS is represented by the procedure SYS1.PROCLIB(GPMSERVE). You can simply enter the command start gpmserve on the console to activate the DDS. The userid running the server must have an OMVS segment in RACF and a home directory in the USS filesystem.

zSeries Technical Conference 2007

Session ZP06

No further customization is needed to run the DDS with the defaults. However, in case you want to adjust the DDS runtime parameters, you can edit the member SYS1.PARMLIB(GPMSRV00). The most important DDS parameters are located in the HTTP section of the this member: MAXSESSIONS_HTTP specifies the maximum number of concurrent HTTP requests. Default: 20 HTTP_PORT is the port number where the server is listening for incoming HTTP requests. Default: 8803 HTTP_ALLOW specifies the host names that can use the HTTP interface. Default: HTTP_ALLOW(*) Wildcards * and ? are allowed Examples: HTTP_ALLOW(*.ibm.com) HTTP_ALLOW(9.164.*.*) HTTP_ALLOW(bhbe.boeblingen.de.ibm.com) HTTP_NOAUTH specifies the host names that can use the HTTP interface without authentication (userid/password). Wildcards * and ? are allowed. Default: HTTP_NOAUTH() Now you are ready to connect to the DDS server. Since the DDS behaves like an HTTP server you can open your web browser and simply type the URL: http://<hostname>:8803 Then your browser should display the DDS welcome screen:

Figure 2: DDS Welcome Screen

zSeries Technical Conference 2007

Session ZP06

Lesson 2 Exploring the Overview PerfDesk

The term PerfDesk is derived from the term Performance Desktop, which represents one or more windows containing graphical views of RMF performance metrics. When you click the Overview button on the welcome screen, the following PerfDesk is displayed:

Figure 3: DDS Overview PerfDesk The Overview PerfDesks provides a high level health-check of your sysplex from the following perspectives and its associated metrics: Processor I/O Storage WLM %processor utilization by MVS image i/o intensity by volume % CSA utilization by MVS image performance index by WLM service class period

The time stamp of the RMF interval time is displayed above the metrics. At the moment, when a Monitor III collection interval (mintime) has completed, the window is automatically refreshed with the new data.

zSeries Technical Conference 2007

Session ZP06

Lesson 3 Understanding the Resource Model

If you want to work with the Data Portal efficiently, it is necessary that you understand the concept of RMF Monitor III resources and its associated metrics. Your computing system represents a composition of resources with the sysplex as top-level resource. All other resources are children or grand-children of the sysplex. The entire hierarchical model looks as follows:

Figure 4: RMF Monitor III Resource Model With the Data Portal, you can easily inspect the actual configuration of your sysplex. When you click the Explore button on the welcome screen, the sysplex resource is displayed as ,SHARPLEX,SYSPLEX. A resource identifier is composed of three parts: High level qualifier (HLQ) Resource name Resource type A HLQ is needed, because resources with the same name can exist multiple times in a sysplex, e.g. volumes or channels. For most of the resources, the HLQ is the name of the associated MVS system. For the sysplex resource the HLQ can be omitted. Therefore the identifier starts with a comma. For unique resources, an asterisk is assigned as resource name. As an example, the identifier for the Storage resource of system S1 is: S1,*,STORAGE

zSeries Technical Conference 2007

Session ZP06

Lesson 4 Exploring the Sysplex Configuration

Once you click on the icon or the identifier of a specific resource, the children of this resource are displayed. Thus, you can navigate thru the entire resource tree to any resource you are interested in. Figure 5 shows the children of the sysplex resource for the current connection.

Figure 5: Contained Resources of the Sysplex This sysplex resides on one CPC and consists of 1 MVS image. It is actually a monoplex without any coupling facilities. Depending on the resource type, you can retrieve one or more attributes for a specific resource simply click Show in the Attributes column.

zSeries Technical Conference 2007

Session ZP06

Exercises: What are the children of the S1,*,I/O_SUBSYSTEM? How many LPARs are configured in the CPC? What is the name of the active service policy? What is the processor model of the CPC? How many processors are online to S1? What is the type and model of the storage subsystem #9200? What is the channel type of CHPID #05? What is the device type of volume CICSS2? How many frames are online in the Central Storage for S1?

zSeries Technical Conference 2007

Session ZP06

Lesson 5 Resources and Metrics

Typically, each resource is associated with various metrics or counters. A metric can either consist of exactly one value that is tied to one resource (single value) or a list of name/value pairs (list value). As an example, you can request the metric % total utilization for the specific CHPID #05 or you can request the metric %total utilization by channel path for the parent resource ALL_CHANNELS. As shown in Figure 6, the values displayed for CHPID #05 should be identical in both cases.

Figure 6: Single valued and List valued Metrics You can obtain the list of available metrics for a specific resource by simply clicking on the Metrics item for this resource. The following list shows the metrics that can be selected for the resource type ALL_SSIDS:

10

zSeries Technical Conference 2007

Session ZP06

Please note, that a subset of metrics can be associated with more than one resource type: Figure 7 shows that the metric i/o activity by volume for the resource type SYSPLEX returns the sysplex wide volume activity - in contrast to the single system view for the resource type ALL_ VOLUMES of the single image S1. As a matter of course, the activity rate for the volume S1.SHPUB2 should show identical values, regardless of the scope. However, when the same volume name appears multiple times with different system prefixes, this indicates a shared volume.

Figure 7: Sysplex wide Aggregation of Metrics The meaning of an individual metric might not be self-describing by its pure description. To get extended help for a metric, you can simply click on the Explanation item for this metric.

11

zSeries Technical Conference 2007

Session ZP06

Lesson 6 Monitoring the Sysplex Performance

In general all metrics that appear in the list of available metrics for a specific resource type can be selected for continuous monitoring. Once you click on the Metric description item, the actual values are instantly retrieved and displayed. As an alternative you can use the right mouse button and select the action Open in New Window: this allows you to open one window per metric and thus monitor multiple metrics at the same time. In addition, the complete URL for the actual DDS request is displayed in the Address field of your browser. Figure 8 shows the result for the metric % total physical utilization by partition, selected from the metric list for the CPC resource.

Figure 8: Creating a new Monitoring Window Now you are ready to create your own views to monitor the sysplex performance! Exercise: Identify the resources for the following metrics and monitor this metrics in a new browser window: actual MSU (CP) by partition execution velocity by WLM service class period % cache hits by SSID on system S1 response time for volume SHXCF1 on system S1 % frames available on system S1

12

zSeries Technical Conference 2007

Session ZP06

Lesson 7 Define your own PerfDesk

The monitoring windows you have created in the previous lesson cannot be stored as persistent windows for later use. As a workaround, you might add the URL string to the favorites of your browser. However, the DDS Data Portal allows you to define your own PerfDesk with your preferred metrics. This PerfDesk can be saved and reused in subsequent sessions. To add a metric to your individual PerfDesk, you can simply click on the Add this metric to my view button at the bottom of the corresponding monitoring window. Once you have finished your composition of metrics, the My View button on the left side of the window brings up your individual PerfDesk in auto-refresh mode. Please note, that the information about your individual PerfDesk is stored as standard browser cookie. This means, in case you delete your cookies, this information is lost and you have to redefine this PerfDesk. Exercise: Define your PerfDesk Storage Utilization for the resource STORAGE of system S1 with the following metrics: # frames active by WLM service class period # frames fixed by WLM service class period # frames active by job # frames fixed by job

Figure 9: My View Storage Utilization PerfDesk

13

zSeries Technical Conference 2007

Session ZP06

Lesson 8 Advanced Features

As shown in Lesson 6 Monitoring the Sysplex Performance, the DDS request strings become visible in the browsers address field when we use the Open in New Window action choice. A basic request string is composed of the following parts: protocol name host name document directory - /gpm document name - contained.xml - listmetrics.xml - details.xml - perform.xml resource=<resource identifier> - high level qualifier - resource name - resource type &id=<metric identifier> (only valid for perform requests)

As an example, the following request string returns the metric response time for the volume SHPUB2 located in the i/o subsystem of system S1:
http://mvs1.centers.ihost.com:8803/gpm/perform.xml?resource="S1,SHXCF1,VOLUME"&id=8D10F0

As mentioned before, the DDS behaves like an HTTP server and returns the following XML document:
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="/gpm/include/perform.xsl"?> <ddsml xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="/gpm/include/ddsml.xsd"> <server> <name>RMF-DDS-Server</name> <version>ZOSV1R9</version> <functionality>2344</functionality> </server> <report> <metric id="8D10F0"> <description>response time</description> <format>single</format> <numcols>2</numcols> </metric> <resource> <reslabel>S1,SHXCF1,VOLUME</reslabel> <restype>VOLUME</restype> <reslabelurl>S1,SHXCF1,VOLUME</reslabelurl> </resource> <time-data> <local-start>20070321084500</local-start> <local-end>20070321084600</local-end> <gatherer-interval unit="seconds">60</gatherer-interval> </time-data> <row refno="1" percent="66.6667"> <col></col><col>1.5</col> </row> </report> </ddsml>

14

zSeries Technical Conference 2007

Session ZP06

The value parameter of the row tag in the XML document indicates a response time of 1.5 milliseconds for the current interval that is described by the timerange tag. Beyond the &id parameter, which identifies a specific metric, the DDS supports the following optional parameters: &filter &workscope &range Examples:
Request String Explanation only instances with the name patterns CICS* and SH* are returned specifies a lower bound threshold value specifies an upper bound threshold value specifies the number of list elements (containing highest values) specifies the number of list elements (containing lowest values) specifies the order (ascending/descending names or values) focus on Period 1 of Service Class STCLO focus on Service Class STCLO focus on Report Class DB2 focus on Workload BATCH focus on Job Catalog return data from 16:10:00 to 16:11.40 at 01/28/2007

reduces the amount of data by specifying thresholds and name patterns focuses on individual workscopes (e.g service classes) specifies an individual interval

&filter="PAT=CICS*|SH*" &filter="LB=10" &filter="UB=10" &filter="HI=5" &filter="LO=5" &filter="ORD=NA" (NA/ND/VA/VD) &workscope="STCLO,1,P" &workscope=",STCLO,S" &workscope=",DB2,R" &workscope=",BATCH,W" &workscope=",CATALOG,J" &range=20070128161000,20070128161140

15

zSeries Technical Conference 2007

Session ZP06

Exercises: Specify the URL to obtain the following metrics: 1. response time by volume for all volumes in the sysplex containing the string CICS 2. response time by volume for all volumes in the sysplex containing the string CICS or SH 3. keep the same name pattern, but show only volumes with a response time higher than 1 millisecond 4. response time by volume for all volumes connected to system S1. Show only the 10 volumes with the highest values 5. workflow by job for all jobs running on system S1. Show only the 3 slowest jobs 6. workflow by job for all jobs running on system S1, but in ascending order by value 7. working set by job for all address spaces on system S1 running in service class STCLO 8. % appl (total TCB + SRB) by job for the resource PROCESSOR located on system S1 9. execution velocity by WLM service class for the current date and a 5 minute interval starting at hh.00

16

zSeries Technical Conference 2007

Session ZP06

Solutions:
1. http://mvs1.centers.ihost.com:8803/gpm/perform.xml?resource=,SHARPLEX,SY SPLEX&id=8D1120&filter=PAT=*CICS* 2. http://mvs1.centers.ihost.com:8803/gpm/perform.xml?resource=,SHARPLEX,SY SPLEX&id=8D1120&filter=PAT=*CICS*|*SH* 3. http://mvs1.centers.ihost.com:8803/gpm/perform.xml?resource=,SHARPLEX,SY SPLEX&id=8D1120&filter=PAT=*CICS*|*SH*;LB=1 4. http://mvs1.centers.ihost.com:8803/gpm/perform.xml?resource=S1,*,ALL_VOLU MES&id=8D1120&filter=HI=10 5. http://mvs1.centers.ihost.com:8803/gpm/perform.xml?resource=,S1,MVS_IMAGE &id=8D0560&filter=LO=3 6. http://mvs1.centers.ihost.com:8803/gpm/perform.xml?resource=,S1,MVS_IMAGE &id=8D0560&filter=ord=va 7. http://mvs1.centers.ihost.com:8803/gpm/perform.xml?resource=S1,*,CENTRAL_ STORAGE&id=8D1280&workscope=,STCLO,S 8. http://mvs1.centers.ihost.com:8803/gpm/perform.xml?resource=S1,*,PROCESS OR&id=8D0610 9. http://mvs1.centers.ihost.com:8803/gpm/perform.xml?resource=,SHARPLEX,SY SPLEX&id=8D0F50&range=yyyymmddhh0000,yyyymmddhh0500

17

zSeries Technical Conference 2007

Session ZP06

Lesson 9 Creating Spreadsheets from DDS Data

Once the DDS data is displayed in your browser window, it can be exported to spreadsheet applications by means of standard functions. You can choose any monitoring window as starting point and then select the action Export to Microsoft Excel from the context menu of the free surface within this window. As an example, the data shown in Figure 10 has been exported from the PerfDesk that we created in Lesson 7 Define your own PerfDesk.

Figure 10: DDS Data in a Spreadsheet Application

18

zSeries Technical Conference 2007

Session ZP06

The chart presents the storage utilization of system S1. For this purpose, the following two metrics have been selected: # frames active by WLM service class period # frames fixed by WLM service class period

19

zSeries Technical Conference 2007

Session ZP06

Lesson 10 Monitor III Reports

Basically, a resource can be monitored by the selection of one ore more metrics from the coresponding metrics list. For a comprehensive analysis, you might possibly associate a complete Monitor III report with a specific resource. Therefore, sysplexwide Monitor III reports have been assigned to the SYSPLEX resource and single system reports have been assigned to the MVS_IMAGE resource. Reports assigned to SYSPLEX: CACHDET CACHSUM CFACT CFOVER CFSYS SYSSUM Reports assigned to MVS_IMAGE: CHANNEL CPC DELAY DEV DEVR DSND ENCLAVE IOQ OPD PROC STOR STORC STORCR STORF STORS SYSINFO

Figure 11: Monitor III Sysplex and System Level Reports

20

zSeries Technical Conference 2007

Session ZP06

Once Monitor III reports can be visualized in a Browser window, ISPF based limitations are not a hurdle anymore: the complete report content can be displayed (including the hidden columns) the window is scrollable and can be resized according to your needs

All local adjustments dont consume expensive z/OS CPU cycles - its all on your PC!

Figure 12: Monitor III Processor Delay Report All Monitor III reports supported by the Data Portal are enabled for GO mode. Simply click on the green button on the upper right corner of the window: the color of the button turns to red and the report refreshes automatically with the most recent data. Another click on the same button freezes the report again with the data of the current interval You are also able to acces historical data by navigating forward and backward in the same way as in the ISPF dialog (BREF and FREF commands) In addition, you can obtain field help very easily. Simply move the mouse cursor to the information area of the designated column header - and a popup window with the explanation appears

21

zSeries Technical Conference 2007

Session ZP06

One of the most valuable features of the Data Portal is the Sort capability. Just click on a column header of your choice and the report is sorted according to the values of the selected column The color of this column turns to yellow to indicate the currently active sort criteria By default, the sort order is descending. A second click on the column header switches to ascending sort order Of course, the sort feature is also applicable in GO mode without any limitations. The sort criteria can be changed on the fly without interrupting the GO mode The sort processing does not burn expensive z/OS CPU cycles - it is simply an adjustment of your local view For reports with a high number of columns the right-most columns might be unvisible in the browser window. Although you are able to visualize this data by means of the horizontal slider, it is not possible to display all data at a glance in the standard view Once you click on any instance in the first column, the report layout changes to a vertical view and the report content is focused on this instance. The gained space can now be used to display one metrics per line

There are two possibilties to save the content of a report for later analysis: Print the report: this facility brings up a standard print dialog. By this means, you can either create a hardcopy or a PDF document with any PDF printer driver Export the report content to spreadsheet: this facility creates a CSV (comma separated value) document which can be processed with any kind of spreadsheet application

Figure 13: Print and Export Facility

22

zSeries Technical Conference 2007

Session ZP06

Using the ISPF Monitor III application, you have been limited to view only one report at a time With the Monitor III Data portal you can now run multiple reports in parallel simply by exploiting the Browsers TAB feature: instead of clicking directly on a report type you can click the right mouse button and select Open Link in new Tab from the context menu - and another tab with the requested report is created The data collection for the individual tabs is totally independent. Thus, you are able to run multiple reports in GO mode. At the same time, on another TAB, you can stop the data collection for one selected report in order to examine this report in more detail In case your browser doesnt support tabs, of course you can also open several instances of your browser application to work with multiple reports in parallel

Figure 14: Session with Multiple RMF Sysplex Reports

23