RMF Overview

RMF Development Editionz/OS Resource Measurement Facility RMF Technical Overview March, 6th 2010 © 2010 IBM Corporation Resource Measurement Facility Trademarks The following are trademarks of the International Business Machines Corporation in the United States and/or other countries. APPN* CICS* DB2* DB2 Connect DirMaint e-business logo* ECKD Enterprise Storage Server* ESCON* FICON* GDPS* Geographically Dispersed Parallel Sysplex HiperSockets HyperSwap IBM* IBM eServer IBM e(logo)server* IBM logo* IMS Language Environment* MQSeries* Multiprise* NetView* On demand business logo OS/390* Parallel Sysplex* PR/SM Processor Resource/Systems Manager RACF* Resource Link RMF S/390* Sysplex Timer* System z System z9 TotalStorage* Virtualization Engine VM/ESA* VSE/ESA VTAM* WebSphere* z/Architecture z/OS* z/VM* z/VSE zSeries* z9 * Registered trademarks of IBM Corporation The following are trademarks or registered trademarks of other companies. Java and all Java-related trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countries Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Microsoft, Windows and Windows NT are registered trademarks of Microsoft Corporation. * All other products may be trademarks or registered trademarks of their respective companies. 2 RMF Technical Overview © 2010 IBM Corporation Resource Measurement Facility Anything else? Postprocessor ZZ Session Monitor III Monitor II Mintime & Range Interval & Cycle Time % Workflow 3 RMF Technical Overview Sysplex Data Server Distributed Data Server Spreadsheet Reporter CIM Data Provider % Using / Delay © 2010 IBM Corporation • The RMF product consists of several components for all kinds of performance related disciplines. This session introduces the main RMF components, sheds light on their primary purpose and discusses the most common RMF terms • Even though it’s impossible to become an RMF expert within one hour: let’s get started and let’s see how far we get! Resource Measurement Facility Agenda



Product Structure Address Spaces Controlling the Data Gatherers Historical Reporting ► Postprocessor ► Spreadsheet Reporter
Realtime Reporting ► Monitor III ► Monitor II ► WTO Alerts ► RMF Performance Monitoring ► Data Portal
4 RMF Performance Data APIs RMF Technical Overview © 2010 IBM Corporation The following topics will be covered in this session: • What are the basic functions and what is the primary purpose of these functions? • What address spaces have to be started? What are their names, underlying procedures and responsibilities? • What are the options which determine how the gatheres work in detail? • What are the functions for historical reporting and after the facts analysis? • What are the functions for realtime reporting and problem determination? • Which API's can be user by components or user written programs to retrieve z/OS performance data from RMF Resource Measurement Facility RMF Product Overview Monitor III Data Portal & RMF PM RMF Spreadsheet Reporter Windows, Linux Windows RMF Distributed Dataserver (DDS) RMF Postprocessor Historical Reporting, Analysis and Planning RMF Monitor II and III Real-Time Reporting, Problem Determination RMF Sysplex Data Server and APIs SMF SMF RMF Data Gatherer RMF Monitor I RMF Monitor II background RMF Monitor III VSAM VSAM 5 RMF Technical Overview © 2010 IBM Corporation • z/OS Resource Measurement Facility (RMF) is an optional priced feature of z/OS. It supports installations in performance analysis, capacity planning, and problem determination. For these disciplines, different kinds of data collectors are needed: • Monitor I long term data collector for all types of resources and workloads. The SMF data collected by Monitor I is mostly used for capacity planning and performance analysis • Monitor II snap shot data collector for address space states and resource usage. A subset of Monitor II data is also displayed by the IBM SDSF product • Monitor III short-term data collector for problem determination, workflow delay monitoring and goal attainment supervision. This data is also used by the RMF PM Java Client and the RMF Monitor III Data Portal • Data collected by all three gatherers can be saved persistently for later reporting (SMF records or Monitor III VSAM datasets) • While Monitor II and Monitor III are realtime reporters, the RMF Postprocessor is the historical reporting function for Monitor I data a couple of address spaces have to be activated: • RMF is the root address space.s ZZ SYS1. It is started immediately together with RMF (by default) or later by means of the modify command • RMFGAT is the Monitor III data gatherer. Like Monitor I. it can only be activated with the modify command.PROCLIB(GPMSERVE) RMF Control RMF Monitor III Gatherer s RMF Monitor I (zz Session) SY S ( E 1 . Only one instance is needed per sysplex • RMFM3B is needed for the generation of console messages in case a certain threshold is exceeded .PROCLIB(RMFGAT) SYS1. It is required for any further activities • The Monitor I gatherer is a subtask in the RMF address space.Resource Measurement Facility RMF Address Spaces / Procedures SYS1. one instance per system is needed for RMFGAT • The RMF Distributed Data Server is the data source for the RMF Performance Monitoring java client and the Monitor III Data Portal.PA RB RM RML F0 IB 0) 6 RMF Distributed Data Server RMF Monitor III Batch Reporting s GPMSERVE s RMFM3B SY S (G 1.PROCLIB(RMFM3B) RMF Technical Overview © 2010 IBM Corporation In order to make all RMF functions working.PROCLIB(RMF) SYS1. s III f RMF. Within a sysplex. It cannot be started as standalone address space.PA PM R SR ML V0 IB 0) f RMF.P A RB RM RML F 0 IB 4) SY S (E 1. ..Resource Measurement Facility RMF Control Address Space ► keeps Configuration Tables and other Control Information ► provides the Command Interface to set of modify Options: f RMF. reads and remembers the parmlib options • Provides the command interface to set options or to start and stop the gathering functions • Maintains the SMF Buffer of the RMF Sysplex Data Server © 2010 IBM Corporation .SMFBUF(SPACE(12M).f ZZ.RECTYPE(70:78)) SMF Exits IEFU83x INSTALL COPY Sysplex Data Server SMFBUF (DataSpace) RMF CALL SMF WRITE Write SMF Record SMF SMFWTM Component all SMF Record Types can be maintained by the RMF Sysplex Data Server ! 7 RMF Technical Overview The RMF control address space has 3 major responsibilities: • Builds up configuration tables.MEMBER(99) ► maintains the SMF Data Buffer: s RMF. . CPU seconds.Resource Measurement Facility Data Gathering Methods
exact measurement counts ► pick up consecutive counters ► calculating the difference at the end of an interval Interval Gathering eg..
sampling counts ► inspect variable counters continuously ► building the average at the end of an interval Cycle Gathering eg. 8 RMF Technical Overview © 2010 IBM Corporation In order to collect all the information that is displayed in the various reports. queue counts. device connect time. RMF uses two different techniques of data gathering • The preferred and cheap method is to retrieve exact measurement counts: this can be timers (e.g. dispatch times) or accumalitve counts like the Start Subchannel count for a device • The costly cycling technique is needed for counters which are changing rapidly and frequently: • how long is the dispatcher queue? • how many I/O queue elements are chained to a UCB? • what is the actual enqueue situation? .. frame counts.. 6 XCF OMVS CF HFS 2.PA RB RM RML F 0 IB 0) SMF SMF 74.5 SMF 73 SMF 70.1 SMF 70. User Exits ► NOEXITS 9 RMF Technical Overview © 2010 IBM Corporation • The directives to setup Monitor I data gathering are kept in SYS1.3 SY S (E 1.Resource Measurement Facility Monitor I Data Gathering 1.3 SMF 74.7 SMF 74.4 SMF 74.8 SMF 75 SMF 71 SMF 76 SMF 78.2 SMF 74. loaded by ERBMFIUC) • ERBMFDUC (interval gatherer) • ERBMFRUR (report writer) • ERBMFTUR (termination) . which are the candidates for data gathering • The collected data can be saved to SMF records for later reporting • A subset of SMF records is gathered by Monitor III: since the same data is needed for Monitor III reports. Reporting / Recording ► RECORD ► REPORT(REALTIME) ► SYSOUT(A) 4.1 SMF 77 SMF 78.PARMLIB(ERBRMF00) • There are four different categories of control statements: • Measurements: specifies the kind of resources. duplicate gathering is avoided • Timing: determines the frequency for interval gathering and cycle gathering • Reporting/Recording: defines the type and the creation time of the output • User Exits: specifies routines which can be executed during session processing to gather or report additional data: • ERBMFIUC (initialization) • ERBMFxxx (cycle gatherer.3 SMF 74.2 SMF 72.2 SMF 74. Timing ► CYCLE(1000) ► NOSTOP ► SYNC(SMF) gathered by Monitor III 3. Measurements ► CACHE ► CHANNEL ► CPU ► CRYPTO ► DEVICE ► ENQ ► IOQ ► FCD ► ESS ► PAGESP ► PAGING ► TRACE ► VSTOR ► WKLD SMF 74. M3G. Buffer ► WSTOR(32) ERBVSDEF vsam_ds VSAMVOL(volser) DATASET(ADD(vsam_ds)) DATASET(DEL(vsam_ds)) ERBV2S vsam_ds seq_ds ERBS2V seq_ds vsam_ds 10 RMF Technical Overview © 2010 IBM Corporation • The directives to setup Monitor III data gathering are kept in SYS1. Recording ► DATASET(ADD(RMF. Monitor III VSAM Datasets can be used..DS1)) ► DATASET(ADD(RMF.Resource Measurement Facility Monitor III Data Gathering SY S (E 1.. a subset of gathering activities which does not interfere with address space states can be switched on or off • In order to extend the report range for historical data.&SYSNAME.&SYSNAME. The data collection cannot be controlled on resource granularity: almost all resources must be monitored in order to get the complete picture of the address spaces which are using or delayed for certain resources. the following procedures are provided: • ERBVSDEF defines a Monitor III VSAM dataset • ERBV2S unloads the VSAM data to a sequential dataset • ERBS2V reconverts the sequential dataset to VSAM . Timing ► CYCLE(1000) ► MINTIME(60) ► NOSTOP ► SYNC(00) RMFGAT Address Space In Storage Buffer WSTOR(32) 3.PARMLIB(ERBRMF04) • Only a few measurements can be turned ON or OFF.PA RB RM RML F0 IB 4) 1. However. In this case.DS2)) ► DATASET(START) ► DATASET(NOSWITCH) ► DATASET(WHOLD(7)) DATASET(WHOLD(7)) 4.M3G. Measurements ► IOSUB ► CFDETAIL ► CACHE ► VSAMRLS ► OPD ► HFSNAME ► zFS ► SGSPACE 2. the buffersize which has been specified by the WSTOR option can be reduced to the WHOLD size (since the data which does not fit any more in the buffer is kept at least in the datasets) • For a smart handling of the VSAM datasets. the DDS should run on system with the highest RMF release . • In a mixed release configuration. Monitor III Data Portal.DDS on one TCP/IP image of the Sysplex GPMSERVE (RMF Distributed Data Server) RMF Sysplex Data Server (ERB3XDRS Service) RMF M III Data Gatherer RMF M III Data Gatherer 11 RMF M III Data Gatherer RMF Technical Overview © 2010 IBM Corporation • The RMF Distributed Data Server (DDS) is the component. The DDS uses the ERB3XDRS service to retrieve Sysplex-wide Monitor III data. which serves as data source for the RMF client applications • There is just one additional address space (GPMSERVE) needed per Sysplex.Resource Measurement Facility RMF Distributed Data Server ► has Sysplex-wide scope ► exploited by RMF PM. CIM Provider ► talks XML over HTTP ► no configuration needed! ► just enter S GPMSERVE or F RMF. 3) holds performance data and resource consumption on workload granularity) • Some SMF record types (74. the reporting structure is resource oriented (exception: the WLMGL record (SMF 72. this address space should be active anyway to guarantee access to the Monitor III reports • With OS/390 R10 the Postprocessor has been enabled to accept foreign SMF records: • Type 103 HTTP Server • Type 108 Domino Server • While the RMF Postprocessor generates tabular reports. However.x) are written by the Montior III data gatherer address space RMFGAT.Resource Measurement Facility Historical Reporting Activities measured by Monitor I Other Activities ► Lotus Domino Server ► HTTP Server 12 RMF Technical Overview Lotus 1-2-3 and Excel Spreadsheet Applications ► XCF ► OMVS ► Coupling Facility ► HFS SMF RMF Spreadsheet Reporter Activities measured by Monitor III RMF Data Buffer RMF Postprocessor ► Cache Subsystem ► Channel Path ► CPU / Crypto ► Device ► Enqueue ► Ficon Director ► I/O Queuing ► Page/Swap Data Set ► Paging ► Trace ► Virtual Storage ► Workload © 2010 IBM Corporation • The tool of choice for historical reporting and after the facts analysis is the RMF Postprocessor • Accordingly to the Monitor I data collection options (there is one SMF record type for each resource). the RMF Spreadsheet Reporter can create graphical charts based on Postprocessor data . 7.4.SMFDATA.TYPE(70:78)) //RMFSORT EXEC PGM=SORT //SORTIN DD DISP=SHR. the SMF data is dumped by default to Generation Data Groups (GDGs) • For Sysplex-wide reporting.SYSNAME(-1) SMF reords must be sorted by date and time SORT step is required for sysplex-wide reporting ► Workload Activity Report ► Coupling Facility Report ► Shared DASD Report RMF Technical Overview © 2010 IBM Corporation Before the raw SMF data can be feeded to the Postprocessor.(10.DSN=<input_smfdata_system1> // DD DISP=SHR..PASS).SYSNAME(0) ► RMF.CH.N) 13





SMF data is kept in VSAM datasets Postprocessor requires sequential format use SMF dump utility IFASMFDP to unload the data usuallly GDGs are the prefered target: ► RMF. two preparation steps are needed: • The data has to be transformed from VSAM to Sequential format • This is achieved with the SMF Dump Utility IFASMFDP • In most installations..DSN=<input_smfdata_system2> //SMFDATA DD DISP=(NEW.OPTIONS(DUMP)) OUTDD(SMFDATA.3000. // UNIT=SYSDA.N).E35=(ERBPPE35.36000.Resource Measurement Facility Postprocessor: Postprocessor: Preparing SMF Data //SMFDUMP EXEC PGM=IFASMFDP //IDD1 DD DISP=SHR.DCB=(RECFM=VBS. a preceeding SORT step must be executed • RMF provides the exit routines ERBPPE15 and ERBPPE35 to enable the sorting by RMF interval start time .A. the records arriving from the various systems must be sorted by ascending interval time • Therefore.DSN=<input_smfdata_system1> //IDD2 DD DISP=SHR.OPTIONS(DUMP)) INDD(IDD2.4.BLKSIZE=0) //SYSIN DD * INDD(IDD1.EQUALS MODS E15=(ERBPPE15.SMFDATA.DSN=<input_smfdata_system2> //SYSIN DD * SORT FIELDS=(11.RLSE).LRECL=32760.10).SPACE=(CYL.CH.A). REPORTS //PXSRPTS DD DISP=SHR.SYSPLEX.2400) DINTV(0100) REPORTS(ALL) RMF Technical Overview © 2010 IBM Corporation • Different DD names and control statements must be specified for the different kinds of report output • However.DSN=RMF.SORTOUT //PPRPTS DD DISP=SHR.DSN=*.REPORTS //SYSIN DD * DATE(04212003.DELETE).DSN=*. the user doesn't need to take care about any syntax: the JCL to execute the Postprocessor can be generated with an ISPF dialog or with the RMF Spreadsheet Reporter workstation frontend .04232003) RTOD(0000.INTERVAL.MSG //MFPINPUT DD DISP=(OLD.ALLOC.DSN=RMF.Resource Measurement Facility Postprocessor JCL
DD Names ► ► ► ► ► ► ► MFPMSGDS MFPINPUT PPRPTS PPSUMnnn PPXSRPTS PPORPnnn PPOVWREC Message Output SMF Input Datasets combined Interval Reports Summary Report Output Sysplex Report Output Overview Report Output Overview Record Output
Control Statements ► ► ► ► ► ► 14 DATE RTOD DINTV REPORTS OVERVIEW OVW Start / End Date Start / End Time Duration Interval Length Report Types Report or Record Overview Control Statement JCL can be generated by ISPF Application or Spreadsheet Reporter //RMFPP EXEC PGM=ERBRMFPP //MFPMSGDS DD DISP=SHR.RMFSORT. the RMF Postprocessor is used to produce tabular reports (one per interval).61 29.00 1 CP 100. The chart shows an example for a single system report and for a Sysplex-wide report: • The CPU Activity Report shows the processor utilization from the systems perspective (on logical processor level) • The Workload Activity Report shows Sysplex-wide transaction and resource consumption statistics (on service class period level) .3 RESP 1.Resource Measurement Facility Postprocessor: Postprocessor: Standard Reporting C P U //RMFPP EXEC PGM=ERBRMFPP //SYSIN DD * DATE(10142002.16 % I/O INTERRUPTS HANDLED VIA TPI 0.00 MS ENC 0.93 2.50 A C T I V I T Y START 02/13/2007-11.36 MPL 23.0 SHARED 0.275 HST 0.60 22.0 BLOCK 0.20 TOTAL 67066.00 #SWAPS 106 EXCTD 0 AVG ENC 0.36 ENDED 0 END/S 0.000 AAP 0.048 CYCLE 1.0 Q+PEND 0.00 15 WORKLOAD=SYSTEM TRANS-TIME HHH.43 0.77 2.0 HSP 0.54 2.78 W O R K L O A D z/OS V1R8 SYSPLEX SYSDPLEX RPT VERSION V1R8 RMF A C T I V I T Y DATE 02/13/2007 TIME 11.00 0.42 92.00 3 CP 100.00.15.05 -----------------------------------------------------------------------------------------------------------.SERVICE CLASS PERIODS REPORT BY: POLICY=DEFAULT -TRANSACTIONSAVG 23.00 INTERVAL 15.53 25.6 DISC 0.913 RCT 0.00 END 02/13/2007-11.10142002) RTOD(1100.00 CP TOTAL/AVERAGE LPAR BUSY TIME PERC 3.00.00 N/A --PAGE-IN RATES-SINGLE 0.00.0 0.00 2 CP 100.000 SECONDS B16 MVS BUSY TIME PERC 3.00 INTERVAL 000.97 3.TTT ACTUAL 0 EXECUTION 0 QUEUED 0 R/S AFFIN 0 INELIGIBLE 0 CONVERSION 0 STD DEV 0 SERVICE CLASS=SYSTEM CRITICAL =NONE --DASD I/O-SSCHRT 30.18 2.017 IIT 0.00 MODE = GOAL POLICY ACTIVATION DATE/TIME 02/12/2007 08.SS.1 CONN 0.46 0.43.58 2.1 ---SERVICE---IOC 15835 CPU 5183K MSO 0 SRB 1548K TOT 6747K /SEC 7496 ABSRPTN TRX SERV 321 321 RESOURCE GROUP=*NONE --SERVICE TIMES-CPU 26.09 SHARED 11.17 CPU SERIAL NUMBER 066F7A 066F7A 066F7A 066F7A I/O TOTAL INTERRUPT RATE 14.1300) REPORTS(CPU) SYSRPTS(WLMGL(SCPER)) SYSOUT(H) z/OS V1R8 CPU 2084 SYSTEM ID SYSD RPT VERSION V1R8 RMF MODEL 314 H/W MODEL ---CPU--.86 0.MM.498 SRB 7.69 0.ONLINE TIME NUM TYPE PERCENTAGE 0 CP 100.15.3 IOSQ 0.49 0.000 RMF Technical Overview © 2010 IBM Corporation In most cases.00 REM ENC 0.00 AAP IIP 0.23 2.61 3.00 -----STORAGE----AVG 2871.000 IIP N/A PROMOTED PERIOD=1 IMPORTANCE=SYSTEM ---APPL CP AAPCP IIPCP %--3. 9 33.0 5.00 CYCLE 1.MM.0 5.00.1 38.SS 16 RMF Technical Overview © 2010 IBM Corporation • When the user wants to focus on specific metrics without analyzing an entire report.00.00 INTERVAL 01.00 12. Chapter 17.00.00 PROCS CPUBSY APPL TRANS RTIME 5.00 11.5 16.00 INT HH.SS 11.00 01.REPORT) OVW(PROCS(NUMPROC)) OVW(CPUBSY(CPUBSY)) OVW(APPL(APPLPER(POLICY))) OVW(TRANS(TRANS(POLICY))) OVW(RTIME(RTIME(POLICY))) DINTV(0100) (via Spreadsheet Reporter) R M F z/OS V1R8 NUMBER OF INTERVALS 4 DATE MM/DD 08/07 08/07 08/07 08/07 TIME HH.00.SS to HH.00 01.0 5.66 44.00.00 O V E R V I E W SYSTEM ID SYSF RPT VERSION V1R8 RMF R E P O R T START 08/07/2006-11.9 24. he can extract specific values and produce his own customized reports • Multiple intervals are visible at a glance: Thus.MM. Overview reporting is well suited for trend analysis over long periods • For all SMF 70-78 record types.000 SECONDS TOTAL LENGTH OF INTERVALS 04.0 30.5 40.8 33. For more details.Resource Measurement Facility Postprocessor: Postprocessor: Overview Reporting OVERVIEW(RECORD.00.0 9.00.6 24.2 8. the Postprocessor Overview feature provides a smart solution: by means of overview control statements. refer to the RMF Users Guide (SC33-7990).00 12. Overview and Exception Conditions .99 Format of the INTERVAL column changes from MM.92 25.00.00.SS 01.56 35.MM.5 8.00 01.00.00. the user can choose from a rich set of Overview Control statements.8 28.00 END 08/07/2006-15.00.3 8. Resource Measurement Facility RMF Spreadsheet Reporter ► ► ► 17 converts SMF Data to Spreadsheet Format creates graphical Views for Trend Analysis can be downloaded from the RMF Homepage RMF Technical Overview © 2010 IBM Corporation • In extension to the RMF Postprocessor.com/servers/eserver/zseries/zos/rmf/. You will find the Spreadsheet Reporter in the Tools category of the homepage.ibm. which are shipped as samples together with the Spreadsheet Reporter main application. The installable image is shipped with the host file SYS1. • The Spreadsheet Reporter is a basic part of the RMF product and doesn’t require additional charges. RMF offers a graphical workstation frontend for historical reporting: the RMF Spreadsheet Reporter • The visualization of the data is performed by spreadsheet macros. .SERBPWS(ERB9R2SW) or the most recent version can be downloaded from the RMF homepage: http://www. Resource Measurement Facility RMF Spreadsheet Reporter SMF raw Data RMF Postprocessor Report Listings Overview Records Download Report Listings Overview Records Extract *.WK1 Files Generate Working Sets Display Spreadsheet 18 RMF Technical Overview © 2010 IBM Corporation The transition from the SMF raw data to graphical charts is a complex process with multiple steps.OVW Files Convert CPU Contention: System SYS1 *.the RMF Spreadsheet Reporter does it all for you by magic: • Generates Report Listings or Overview Records from the SMF data with the RMF Postprocessor: on the Resource notebook page of the Spreadsheet Reporter. specify as input the SMF dump data sets from all the systems of your sysplex. which allows you to run the Postprocessor from your workstation! • Downloads the Postprocessor output to your workstation • Extracts the reports or records • Convert the reports or records to spreadsheet format and generates the Working Set . The good news . Now you are ready to exploit the Spreadsheet Reporter’s remote job execution engine. Then check the desired reports on the Reports selection dialog.RPT Files *. the user can select from a rich set of macros for all performance relevant areas • Three basic types of macros can be distinguished: • Macros based on data contained in one Postprocessor Report (for detailed analysis of one single interval) • Macros based on data contained in Overview Records (for trend analysis with multiple intervals) • Macros based on data contained in multiple Postprocessor Reports (“hybrid” macros) • The new XCF Trend Report macro belongs to the family of “hybrid” macros .Resource Measurement Facility RMF Spreadsheet Reporter Macros for all Types of Performance relevant Areas !! 19 RMF Technical Overview © 2010 IBM Corporation • Once a Working Set has been created from SMF data. best suited for snapshot reporting ➨ single job and resource monitoring ➨ ► RMF PM / Data Portal enterprise-wide reporting of z/OS systems ➨ based on RMF Monitor III data ➨ 20 RMF Technical Overview RMF offers three different Monitors for realtime reporting: • Monitor III Reporter (ISPF): • Is designed for workflow analysis and goal attainment supervision • Can be used in realtime mode.Resource Measurement Facility Realtime Reporting CF

covers all Sysplex related aspects two monitors and a workstation extension ► Monitor III. real-time and historical reporting ➨ online performance analysis ➨ goal attainment supervision ➨ sysplex-wide and single-system reporting ➨ monitoring of exceptional conditions ➨ WLM ► Monitor II. but the GUI is totally different © 2010 IBM Corporation . best suited for short-term. but supports historical reporting as well • Monitor II (ISPF and native TSO): • Is a snapshot monitor. The interval length can be determined by the user • Consists of a set of address space and resource oriented reports • RMF Performance Monitoring / Monitor III Data Portal: • Allows to manage multiple Sysplexes from the workstation • The raw data is identical with Monitor III ISPF. JES . Storage.g. Operator. there are lots of resource oriented reports with similar statistics than the RMF Postprocessor reports • Together with hardware or software related innovations new reports have been added with each z/OS release .Resource Measurement Facility Monitor III Reporting Monitor III Delay Monitoring





Processor Storage Device Enqueue Operator ► Message ► Tape Mount Subsystem ► HSM .and software resources and assign those resources on address space level • Possible states for an address space are: • Delayed for Processor. Device. using an unmonitored waiting mechanism) • Activity–Resource Monitoring • In addition to the workload–delay concept.XCF Monitor III Features Monitor III Activity Monitoring













21 Common Storage Page/Swap Data Sets Storage Frames Device Data Set Level by Job and Volume Cache Coupling Facility Goal Attainment VSAM RLS UNIX System Services Enclaves zFS Diskspace Spin/Suspend Locks











Cursor-Sensitive Navigation Workflow/Exceptions Monitoring Automatic Customization Support of WTO Messages Continuous Monitoring Hardcopy Reports On-Line Tutorial On-Line Help Adaptive Reports User Reports Sysplex-wide Reports Remote Reporting RMF Technical Overview © 2010 IBM Corporation The Monitor III ISPF Reporter provides two basic types of monitoring: • Workload–Delay monitoring • The state samplers are inspecting all hard. Enqueue. Subsystem • Using Processor or Device • Idle (timer wait or terminal input wait) • Unknown (e. all address spaces can be associated with the following qualities: • Using% • Delay% • Workflow% • Since not all address spaces can be catched up every cycle. Possible reasons for unknown are: • The address space is delayed for unmonitored resource (other than DASD or Tape) • The address space is using an unmonitored waiting mechanism (most Started Tasks!) • statistical reasons = 60% .Resource Measurement Facility States of a Job Proc Idle Dev Dev Dsn Dsn JES Enq Oper HSM Stor XCF Proc ? Unknown 22 Using(%) = using samples number of samples 100 = 50% Delay(%) = delay samples number of samples 100 = 33% Workflow (%) = using samples using samples + delay samples RMF Technical Overview 100 © 2010 IBM Corporation • Address spaces are either using resources or they are delayed for resources or they are idle • The RMF state samplers are trying to monitor continuously all kinds of hard. the remaining states are counted as unknown.and software resources • Depending on the result. 5 Jobname USG DLY Reason % % A B C D 25 25 50 0 75 25 25 25 C C A C © 2010 IBM Corporation RMF Technical Overview The example helps to understand what is calculated and reported by RMF Monitor III • From the jobs perspective: • Job A 1x Using(25%) 3x Delayed(75%) 0x Idle(0%) Workflow: 1/4 = 25% • Job B 1x Using(25%) 1x Delayed(25%) 2x Idle(50%) Workflow: 1/2 = 50% • Job C 2x Using(50%) 1x Delayed(25%) 1x Idle(25%) Workflow: 2/3 = 66% • Job D 0x Using(0%) 1x Delayed(25%) 3x Idle(75%) Workflow: 0/1 = 0% Workflow = 4/10 = 40 % Average Delayed User (ADU): 6/4 = 1.5 • From the resource perspective: • 4x using.Resource Measurement Facility Example: Using and Delay D B A Job A B C D D Res I C C U D * * A Why C C * * Job A B C D I D Res U D * Why B * * * Job A B C D I A U * Why * A * D I C U D * Why C * C * * RMF Monitor III Resource Delay Report Samples: 4 Jobname WFL USG DLY IDL Primary % % % % Reason WFL Resource % C C A C Job A B C D * Time: 06.20 Range: 4 Sec 25 25 75 0 50 25 25 50 66 50 25 25 0 0 25 75 Res A D Samples: 4 A B C D B Res C B RMF Monitor III Delay Report 23 B Res Time: 06.28.28.20 Range: 4 Sec ADU 40 1. 6x delay . % Delayed for ---.20 Range: 100 Sec ► ► 24 Name Service CX Class SUSANK CONSOLE RRSSERVQ BHBE MORABIT RONDA2A D24JAP1 RRSSERVQ GRSARTSQ RRSSERVQ CATALOG ANFWPROC GRSARTSQ SMFDRS JES2 GRSARTSQ ARTXESQ DFRMM T S B T T B T B B B S SO B S S B B S Cr WFL USG DLY IDL UKN ---. the critical candidates with low workflow appear on top of the address space list • Beyond the basic statistics (workflow. idle.28. unknown). using. the delay reason information is displayed in the rightmost column: • For DEV the critical volume • For PRC the competing address space with the highest usage • For SUB the subsystem name • For ENQ the enqueue resource name . a delay type breakdown for the following resources is provided: • PRC processor • DEV device • STR storage • SUB software subsystem • OPR operator • ENQ enqueue • Additionally. delay.Resource Measurement Facility Monitor III: Job Delays RMF V1R7 Samples: 100 System: AQTS Delay Report Date: 07/18/06 Line 1 of 326 Time: 06.Primary % % % % % PRC DEV STR SUB OPR ENQ Reason TSOPRIME SYSTEM WLMSHORT TSOPRIME TSOPRIME COMBUILD TSOPRIME WLMSHORT WLMSHORT WLMSHORT SYSTEM SYSSTC WLMSHORT STCMED SYSSTC WLMSHORT WLMSHORT SYSSTC 0 0 0 40 41 42 49 50 50 50 63 67 71 71 73 80 80 83 0 100 0 15 0 1 4 6 37 56 29 42 22 23 2 2 1 1 1 1 57 39 2 1 5 2 5 2 16 6 4 1 4 1 81 18 0 0 0 90 0 0 56 0 0 0 0 0 0 0 0 0 0 0 0 85 1 0 7 29 0 0 0 0 12 2 0 93 79 0 0 1 0 0 0 6 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 56 41 23 2 0 0 39 1 1 0 6 0 1 18 0 100 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 ► Address Space Performance at a Glance ! sorted by ascending Workflow Delay Type Breakdown Delay Reason Information HSM Message JES JHUGO COMPK5 COMPK5 PRIPK5 SPOL1J JES JES MCATTS SPOL1J SPOL1J SYSZVVDS SPOL1J JES SPOL1L SL3061 RMF Technical Overview © 2010 IBM Corporation • The Address Space Delay report is the basic report that visualizes the overall Monitor III concept • The report is sorted by ascending workflow: thus. *SYSTEM PROC-SUSANK 12.0 % delay Awaiting reply to operator request 52.1 users BHBE SUBS-JES 100.28. Possible cause or action *SLIP* SLIP PER TRAP SLIP ID=U427 is active. individual address spaces or resources • The Exceptions section displays alerts when critical thresholds are exceeded • The content of both sections can be customized individually: • For the Workflow section.0 % delay Processing TSO OUTPUT command request.20 Line 1 of 32 Range: 100 Sec --------------------------.Exceptions ------------------------------------Name Reason Critical val.Resource Measurement Facility Monitor III: Workflow/Exceptions RMF V1R7 Samples: 100 System: AQTS Workflow/Exceptions Date: 07/18/02 Time: 06. ALL TSO DEV -RMFLIB 1.0 % delay Device may be overloaded.Speed (Workflow) --------------------------------Average CPU Util: 81 % Speed of 100 = Maximum. 0 = Stopped Name Users Active Speed Name Users Active Speed *SYSTEM 560 35 42 *DEV 53 15 46 ALL TSO 19 4 17 *MASTER* 1 1 100 ALL STC 485 6 52 ALL BATCH 39 24 37 ALL ASCH Not avail ALL OMVS 17 1 92 *PROC 292 8 43 -----------------------------. up to 14 candidates for continuous display of workflow can be selected • For the Exceptions section. HSM Not avail Job HSM is not running. MORABIT PROC-SUSANK 75.5 users Short block size or excessive seeks. CONSOLE OPER-Message 13. alerts for almost all Monitor III metrics (as well as combination of metrics) can be defined . SYSPAG Not avail Volume SYSPAG is not mounted.0 % delay GANDALF DEV -SMS001 20. 25 Bottleneck Detection at a Glance ! ► Workflow for Groups and Resources ► Exception Lines for individual Conditions RMF Technical Overview © 2010 IBM Corporation • The Monitor III Workflow Exception Report consists of two independent sections: • The Workflow section provides an overview about groups of work. 000 0.000 0. Otherwise the bar is extended with a green square.0 85 N/A 47 Line 1 of 14 Scroll ===> CSR 0.000 0.000 0.Resource Measurement Facility Monitor III: Goal Attainment RMF 2.33 ------.SCLMPLEX Command ===> WLM Samples: 240 Systems: 3 Date: 05/15/02 Time: 13. 4. Thus.000 0.928 0.080 0.07.46 N/A N/A 1. In case the Performance Index (PI) for a certain Service Class Period is higher than 1.000 AVG N/A Sec Installed at: 12/06/00. coloured lines are generated: • A red line for Service Class Periods with high Importance (1.080Importance > 2 0. Actuals on Service Class Period level.000 0.0 Sysplex Summary .100 1.000 0.000► 80 Intervals in GO Mode 0.000 RMF Technical Overview ! © 2010 IBM Corporation • The Sysplex Summary Report is the standard report for goal attainment supervision • The report provides Goals vs. a coloured bar shows the state for the last 80 intervals: when at least one line for the current interval becomes red or yellow.000 2.000 0. a red or yellow square will be added. 10. Resp.--Actual-.928 0. 10.000 0.000 0.567 0.000 0.109 2.928 AVG AVG AVG 0.24 Activated at: 12/06/00. 5) • In GO Mode.000 0.000 0.000 0.000► colored Indication for PI > 1 0.109 2.000 0.608 0.000 0.000Sysplex Performance at a Glance 0.608Importance = 1+2 0.07 1.00 Range: 60 Service Definition: SCLM Active Policy: STANDARD Name T STC STCCMD SYSTEM SYSSTC SYSTEM TSO PRDTSO W S W S S W S 1 2 3 R MASTER 26 I 3 1 1 1 40 0. 2) • A yellow line for Service Class Periods with low Importance (3.Indx 88 88 69 N/A 68 N/A 70 84 84 60 0.100 2.000 0.000 0.383 0.000 AVG 1.000 0.07.10.000 0.Goals versus Actuals -------Exec Vel --.000 0.500 AVG 2.000 0.608 0. the report provides a direct link to Response Time Distribution Report by means of cursor sensitivity .08 0.00.080 0.000 0.Response Time --.109 2.000 0.Perf Goal Act ---Goal--.150 0.608 0. TimeEnded WAIT EXECUT ACTUAL Rate Time Time Time 0.46 Trans --Avg. the Sysplex performance is visible at a glance for an 80-minute period (proposed a 60 second reporting range) • For Service Classes with response time goals. SERBTENU SYS1. up to 99 VSAM datasets can be preallocated (DDNAME = RMFDSxx) .g exception definitions).sys2.data) TSO Address Space VSAM Datasets SYS1.ADMGDF 27 RMF Technical Overview © 2010 IBM Corporation • The options for a Monitor III reporter session are stored in tables (e.&REL.m3.m3. The initialization procedure ERBRMF3X performs the table allocations when a reporter session is going to be started • By default.ISPTABLE SYS1.SERBCLS (ERBRMF3X) Monitor III Reporter Type text Type text ERBTLIB &USER.&REL. it is also possible to establish a standalone reporter session independent from the Monitor III gatherer: in order to analyze historical data.Resource Measurement Facility Monitor III: Session Setup VSAM Archives alloc f(rmfds00) da(rmf.data) alloc f(rmfds01) da(rmf.sys1.SERBT Type text RMFGAT In Storage Buffer &USER. a reporter session accesses directly the actual data collected by the RMFGAT (via Cross Memory) • However. . . Color... for example: • initial mode (GO or STOP) • initial screen • range length • By means of report options (RO command). Current.Resource Measurement Facility Monitor III Reporter Usage RMF Session Color etc..... the content of almost all reports can be customized • A very detailed tutorial provides lots of information about the Monitor III concepts and features • Cursor sensitive HELP (including detailed metric descriptions) is always available via the PF1 key © 2010 IBM Corporation .. ICU.. Short Commands F1 System Information Samples: 100 System: AQTS Group WFL *SYSTEM 30 *TSO 20 DMN001 28 PG002 20 User 250 22 16 22 . Enter F3 Scroll Monitor II Enter Tutorial F7 Postprocessor Selection Menu F8 BREF/FREF F3 F10 F11 Find... GO. and more . 28 RMF Technical Overview • The Monitor III reporter session can be invoked directly from the RMF Main Menu • Various session options (SO command) can be defined and stored persistently. ... . 3 RMF Main Menu 1 Report Options Session Options RO F3 M III Main Menu 2 Tutorial Help Report Selections. g.Resource Measurement Facility Monitor III Report Overview Sysplex Reports Coupling Facility SYSSUM CFSYS SYSRTD CFOVER SYSWKM CFACT Cache Subsystem VSAM RLS RLSSC CACHSUM User Reports SYSENQ RLSDS CACHDET DSINDEX RLSLRU RG System/Workload Reports Monitoring Reports GROUP SYSINFO CPC WFEX Delay Reports Job/Workload oriented PROC DEV DSND DSNV STORS HSM ENCLAVE MSI OPD SYSTREND DELAY Resource Utilization Reports DEVN STORCR DEVT STORC DSD Resource oriented ENQR ENQ STOR Other DEVR STORR STORF STORCR JES CHANNEL PROC XCF Job Reports IOQUEUE Modified panel User-written report Sortable report Detailed Job Analysis 29 JOB RMF Technical Overview © 2010 IBM Corporation • Monitor III offers a complete report suite which covers all performance relevant areas. The reports can be grouped by categories: • A family of reports with sysplex wide scope • System level reports for various kinds of measurements • Job/address space oriented reports • Reports from the perspective of the major hard-and software resources • User reports which can be adapted or customized • In order to improve the Monitor III capabilities. e. • OS/390 R4 DSND DSNV • OS/390 R6 CFSYS CFOVER CFACT • OS/390 R7 CACHESUM CACHDET ENCLAVE • OS/390 R10 RLSSC RLSDS RLSLRU OPD • z/OS V1R2 CPC • z/OS V1R7 ZFSSUM ZFSACT SPACEG SPACED • z/OS V1R8 XCFOVW XCFSYS XCFPATH XCFGROUP (Data Portal only) . new reports have been added frequently. 0) +RMF304I 3B: MSU Consumption of critical LPARs: +RMF305I 3B: SYS1 : 82 (WTO Limit: 60) +RMF305I 3B: SYS2 : 12 (WTO Limit: 10) $HASP100 BMGU ON TSOINRDR $HASP373 BMGU STARTED IEF125I BMGU .TIME=12.38. This environment is well suited.LOGGED ON .LOGGED ON . the user can select from all metrics contained in one of the Monitor III reports: the procedure accepts any report name as input parameter.g. In other words.. +RMF301I 3B: Local Partition Capping State: +RMF302I 3B: WLM Capping %: 24.SERBCLS): • ERBR3WFX processes the WFEX report • ERBR3SYS processes the SYSINFO report • ERBR3CPC processes the CPC report . when a threshold is exceeded and a corresponding message should be displayed on the console • For threshold processing.. the specified report is the one and only report that is produced by the batch reporter session (multiple reports are not supported) • The scanning of the report tables and the comparison against thresholds is performed by exit routines. +RMF301I 3B: Local Partition Capping State: +RMF303I 3B: Time until Capping (sec): 40 (WTO Limit: +RMF304I 3B: MSU Consumption of critical LPARs: +RMF305I 3B: SYS1 : 64 (WTO Limit: 60) +RMF305I 3B: SYS4 : 48 (WTO Limit: 32) IEF126I BMAI . Depending on the result of this comparison..Resource Measurement Facility Monitor III: Setup for WTO‘ WTO‘s
Monitor III Batch Address Space creates Reporting Tables Monitor III Gatherer
Thresholds can be defined via ► Workflow Exception Options Dialog ► Reporter Phase Exit Module
Console Message is generated by Exit Module (e.40 30 Set of Samples Monitor III Batch Reporter 600) RMF Technical Overview WTO Exit Module Sample Exits provided for: ➨ Workflow Exception Report ➨ Sysinfo Report ➨ CPC Capacity Report © 2010 IBM Corporation • The procedure RMFM3B is supplied to run a Monitor III reporter session in batch mode. a WTO can be generated • RMF ships the following REXX exit routines as examples (in SYS1.2 (WTO Limit: 10.39..LOGGED OFF .20 +RMF300I 3B: Processing CPC Report. WLM Capping) $HASP100 BMAI ON TSOINRDR $HASP373 BMAI STARTED IEF125I BMAI .TIME=12.TIME=12.00 $HASP395 BMAI ENDED $HASP250 BMAI PURGED +RMF300I 3B: Processing CPC Report.36. . paging activity. . devices. by means of a background session.g.s aa 10:12:08 JOBNAME *MASTER* PCAUTH 31 DEV CONN 2618 1264 SMF 79..x SMF 79. when immediate feedback about address spaces or resources is required • However. CHANNEL) • Resource reports (e. ASRM) • I/O subsystem related reports (e. ASD.x RMF Technical Overview © 2010 IBM Corporation • Monitor II complements the host based RMF reporting suite. SPAG) • Monitor II is the appropriate function. These records can be feeded to the RMF Postprocessor for after the facts analysis . ARD.g. Monitor II can also produce SMF Type 79 records.Resource Measurement Facility Monitor II: Overview
Monitor II is a Snapshot Reporter ► collects the status of system resources (CPU.g. SENQ. It is the RMF snapshot reporting feature • Three basic groups of reports can be distinguished: • Address space oriented reports (e. state information)
use Monitor II to ► ► ► ► continuously monitor resource usage determine the state of any address space in the system track CPU usage of problem address spaces collect supplemental information when analyzing performance problems with Monitor III
choose Background Session ► to collect SMF records for archiving and later postprocessing ► to automate snapshot reporting
choose Display Session Postprocessor ► for immediate feedback ► for online analysis f rmf. DEV.) ► collects the status of address spaces (resource usage. ) • ISPF is the recommended Monitor II reporting environment (although Monitor II reports can be invoked from native TSO as well) .Resource Measurement Facility Monitor II Reporting Activities measured by Monitor II:
Address Space Data










32 ► Resource ► State ► SRM Channel Path Device I/O Queuing Enqueue HFS IRLM Long Locks Paging Page/Swap Data Set SRM Resource Data Sysplex Data Server Library Display ISPF ► for most comfortable usage ► supports sorting and finding ► started from TSO READY or from RMF main menu TSO ► don't use it anymore! 3270 ► dropped with z/OS V1R2 RMF Technical Overview © 2010 IBM Corporation • Monitor II collects statistics about activities and resources (but no delays like Monitor III) • The most heavily used Monitor II reports are the address space oriented reports • Address Space Resource Data(ARD): Resource consumption of address spaces (CPU. Service Units.g IN or OUT) • Address Space SRM Data(ASRM): Details from the SRM point of view (Transaction Active Time. I/O Storage) • Address Space State Data(ASD): State Information for address spaces (Service Class. Dispatching Priority. Location e... .000 1.ASD e. ARD RMF Selection ===> Reset Selection RO RMF . Workloads TSO. Batch.113 Other commands: Sort H D Print Sys Keys 33 Prints all reports Delta-mode Prints current screen Remote reporting View/Assign PF-keys F6 FF PRIV LSQA BEL FF CSF 0 0 0 0 455 2 3 2 106 32 105 62 GO nn for auto update <Enter> for manual update RMF Technical Overview © 2010 IBM Corporation • Monitor II reports can be selected either by menu choice or by report name • With the report options command RO..g. ASD Monitor II Primary Menu Enter selection number or command RMF ..g.3 0.) • Use the PF7/PF8 keys for scrolling (in case not all report lines are visible) • Note.000 0.Resource Measurement Facility Monitor II Commands TSO-Command: RMF MON2 e. the report contents can be customized (e. the reports from all remote systems in the Sysplex can be accessed . that all reports are sortable (place the cursor on the desired coulmn header and press PF6) • The GO command provides the automatic refresh of the report data • With the SYS command.g. ASD report options Find string RFind Cancel Report Options e.g.1K 1 Address Spaces 2 I/O Subsystem 3 Resource L Library Lists U User Return F4 11:15:27 DEV JOBNAME CONN *MASTER* PCAUTH TRACE DUMPSRV 364.ARD Command ===> report-name F7 Scroll up F8 Scroll down MIG=63. 0 0.0 0.0 0.2 174.0 0.5 81.0 0.00 0.0 0.0 0.0 0.15 771.00 0.65 115.00 0.0 0.0 0.65 FF PRV LSQA LSQA X SRM 16M FF CSF ESF M CR ABS 1 4 136 2 23 160 0 731 102 0 2K 1059 0 29 109 1 20 65 9 66 185 1 205 764 0 2 53 5 131 630 0 2 234 0 3 53 0 36 79 0 2 56 0 40 67 1 5 148 0 5 87 6 12 18 459 10 6 43 85 12 60 1 9 67 8 14 31 48 X X X X X X X X X X X X X 0.0 0.0 0.0 0.00 0.0 0.Resource Measurement Facility Monitor II: ARD Report RMF .00 0.0 0.0 0.00 0.79 66.0 0.0 0.02 0.9 45.0 0.0 0.0 0.00 0.0 0.55 2.06 0.0 0.00 0.7 99.2 697.20 3.0 0.00 0.0 0.0 0.ARD Address Space Resource Data Line 1 of 48 Scroll ===> PAGE Command ===> CPU= 45/ 30 UIC=2540 PR= 16:04:11 DEV JOBNAME CONN HSM ANTMAIN *MASTER* XCFAS IOSAS RMFGAT JES2 SMSVSAM SMS OMVS CATALOG HSMMON NET SMF LLA NETVSCLM IXGLOGR 79920 28880 8616 4485 3774 3144 773.0 0.0 0.93 52.0 0.0 0.0 117.67 2.8 1556 3.0 0.00 0.0 0.77 111.0 0.7 2652 796.0 0.0 0.04 0.73 24.0 0.73 27.0 0.0 0.41 0.0 0.0 0.0 0.1 38.8 434.3 550.0 0.0 0.00 0.04 2.00 0.0 0.0 0.0 0.0 0.0 0.7 46.0 0.0 0.0 0.0 0.5 46.0 0.00 0.0 0.0 0.00 0.00 0.0 0.1 85.0 TCB TIME 8508 360.0 0.0 0.0 0.0 0.0 0.0 0.6 409.00 0.0 0.0 0.0 0.00 0.0 Address Space Resource Consumption at a Glance ! ► I/O Activity ► Frame Counts ► CPU Time all Table Reports are sortable ! 34 RMF Technical Overview © 2010 IBM Corporation • The ARD report provides a quick check about address spaces and their resource usage (e.0 0.0 0.19 1.0 0.0 0.0 0.0 0.0 0.2 296.00 0.01 0.00 0. Device Connect Time or TCB Time) • The report header displays the following information: • MVS view of the CPU Utilization • LPAR view of the CPU utilization • Unreferenced Interval Count • Pagein Rate • The tabular part of the report contains similar information than the SDSF DA screen .0 0.0 0.0 0.0 0.07 5160 360.1 548.0 0.0 0.8 166.88 0.00 0.28 0 System= SCLM Total CPU EXCP SWAP LPA CSA NVI V&H TIME RATE RATE RT RT RT RT 10575 427.00 0.0 0.g.4 376.19 0.0 0.4 101.2 103.89 5236 455.02 23.0 0.09 125.0 0.4 771.0 0.21 0.0 0.00 0.00 0.70 4. ISPTABLE © 2010 IBM Corporation .LRECL=137) 2.REPORTS') disp DCB=(RECFM=VBA.REPORTS 35 RMF Technical Overview • In case the user wants to save the reports for later analysis he can use the Hardcopy and Print commands • The reports are written either to SYSOUT or to the preallocated dataset RMFDMTSO • For most of the reports you can specify options or filters.RMF&rel.M2.ISPTABLE BHBE.ARD Command ===> 1.g.M2.3 0.BLKSIZE=1693.113 FF PRIV LSQA BEL FF CSF 0 0 0 0 455 2 3 2 106 32 105 62 ERBROPT2 Report options BHBE. e. device numbers or storage groups • For persistence. ALLOC F(RMFDMTSO) DA('BHBE.RMFOS260. the report options are written to the dataset &uid.Resource Measurement Facility Monitor II Session Setup H | Print RMF . RMF MON2 MIG=63. • ARD: address space types BATCH or TSO only • DEV: volume names.000 0.000 1.1K 11:15:27 DEV JOBNAME CONN *MASTER* PCAUTH TRACE DUMPSRV 364. g.1 0.9 0.20 7.28 0.ARDJ Address Space Resou Command ===> RMF .22 CPU= 13:20:26 DEV JOBNAME CONN *MASTER* PCAUTH RASP TRACE DUMPSRV XCFAS GRS SMXC SYSBMAS ➙ 36 Current status in highlighted line 8 UIC=2540 FF PRIV LSQA LSQA X SRM BEL FF CSF ESF M ABS 409.00 2 X 0.000 0.94 16.01 3 0.89 16.92 16.21 7.64 X 0.93 16.0 30.Resource Measurement Facility Monitor II Display Modes Specific resource or job.22 7.88 16.91 16.0 0.21 7.00 X 0.0 34.05 X 0.87 16.21 7.20 7. e.000 Row Report 9/ 0 503 0 2 0 4 0 3 0 2 0 1420 0 40 0 2 0 41 106 32 12 105 62 749 692 33 73 0 0 0 0 2 5 0 0 0 TCB TIME 0. job BHOL All resources or jobs at a specific point in time RMF .ARD Address Space Resour Command ===> CPU= 19/ 19 UIC=2540 BHOL TIME DEV CONN 13:16:46 13:16:47 13:16:48 13:16:49 13:16:50 13:16:51 13:16:52 13:16:53 13:16:54 16.113 273.00 0.000 1.0 0.0 241.95 FF PRIV LSQA LSQA X SRM BEL FF CSF ESF M ABS 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 79 79 79 79 79 79 79 79 79 0 0 0 0 0 0 0 0 0 65K 65K 65K 65K 65K 65K 65K 65K 65K TCB TIME 7.000 0.0 0.20 7.90 16.94 Table Report ➙ Current status in whole report RMF Technical Overview The address space and volume oriented reports are available in two flavours: • Table reports: one sample with all address spaces or all volumes at a glance is displayed • Row reports: multiple samples for one address space or one volume are displayed © 2010 IBM Corporation .000 0.0 28.000 0.22 7.0 0.00 0.0 3. • Those reports are not available when the ZZ session is inactive (reason: Supervisor state and Key zero is required for certain gathering activities) • The SORT capability as well as the HFS report is only available in the ISPF environment . SRM RMF Technical Overview © 2010 IBM Corporation • A subset of Monitor II reports is based on Monitor I gatherer data (DEV. IOQUEUE. PGSP). processor.Resource Measurement Facility Monitor II Report Overview Name Mon I Row ISPF ARD Address space resource data ARDJ Y ASD Address space state data ASDJ Y ASRM Address space SRM data ASRMJ Y CHANNEL Channel path activity data DEV Y DEVV Y Device activity data Y HFS Y ILOCK IOQUEUE Y I/O queuing activity data Library lists Y Page/swap data set activity SDS Sysplex data server statistics SENQ Enqueue contention SENQR 37 HFS statistics IRLM locking data LLI PGSP Explanation Enqueue reserve activity SPAG Y Paging activity SRCS Y Central storage. Resource Measurement Facility RMF Performance Monitoring ► Enterprise-wide performance monitoring ► ► ► ► ► ► ► 38 of z/OS hosts Platform independent Java Edition Linux gathering support Graphical user interface Flexible definition of data Persistent definition of views Powerful data reduction Analysis support RMF Technical Overview © 2010 IBM Corporation • RMF Performance Monitoring (RMF PM) is the graphical workstation frontend for Monitor III data.html .ibm. The most recent version can be downloaded from the RMF homepage: http://www. It allows to manage multiple Sysplexes from one single point of control • RMF PM is free of charge.com/servers/eserver/zseries/zos/rmf/rmfhtmls/pmweb/pmweb. Resource Measurement Facility RMF PM: Perfdesk Concept Perfdesk Folder Perfdesk 1 Dataview 1 Dataview 2 Series 1 Series n Series 1 Series n Dataview 3 Dataview 4 Series 1 Series n Series 1 Series n Perfdesk 2 39 RMF Technical Overview Dataview 1 Dataview 2 Series 1 Series n Series 1 Series n © 2010 IBM Corporation • The main window is of the RMF PM application is divided into 2 parts: • The left side contains the PerfDesk notebook page. the Distributed Data Server (DDS) address space must be started in order to provide the data for the PM client by means of a TCP/IP connection . The data itself is represented by the so called Series • On the host side. which allows to control the content of the right side of the window • The right side contains the Open Perfdesk window: one Perfdesk consists of one or more views with the actual data (Data Views). ESQA Expanded Storage Enqueue Operator Subsystems JES. Individual metrics (or counters) can be selected for data collection and continous display . SQA. ECSA. XCF. The Monitor III data model maps those resouces in terms of a hierarchical scheme • Each resource is associated with a set of metrics. HSM CPC LPAR Coupling Facility CF Structure 40 RMF Technical Overview © 2010 IBM Corporation • Resources are components of a computing system.Resource Measurement Facility RMF PM: Resources and Metrics Sysplex MVS Image I/O Subsystem All SSIDs SSID All LCUs LCU All Channels Channel All Volumes Volume Processor Storage Auxiliary Storage Central Storage CSA. This is achieved by clicksensitive bars whereby the analysis menu choice offers a selection of various pathes for further drill-down . he can navigate to context information or additional views in order to detect the reason of a problem.Resource Measurement Facility RMF PM: Intelligent Analysis ► data views with click-sensitve bars ► link to predefined Analysis PerfDesks 41 RMF Technical Overview © 2010 IBM Corporation One of the most powerful features of RMF PM is the Intelligent Analysis: once the user has detected an abnormal value. two additional components are needed: • The Linux compiled version of the Distributed Data Server infrastructure • The Linux data gatherer backend (collects data out of the procfile system) • The Linux support can be downloaded free of charge: http://www.) are also available for Linux environments • For the RMF PM client. filter. Thus. etc.Linux support available on Internet Common View for z/OS and Linux Data 42 RMF Technical Overview © 2010 IBM Corporation • RMF PM has been extended to support the Linux operating system using the existing RMF PM infrastructure. it is transparent whether the host is a z/OS or Linux system • On the server side.ibm. the PM capabilities (spreadsheet export.com/servers/eserver/zseries/rmf/pmlin.Resource Measurement Facility Linux Performance Monitoring RMF PM Java TM Technology Edition . historical data collection. graphical trend reports.htm • Please note that the Linux support is not an official part of z/OS RMF . the client can be either a “fat” client like RMF PM or the data can also be accessed directly by means of a web browser. the RMF DDS server behaves like a webserver: the data can be requested by means of the standard HTTP protocol. That’s what we call the RMF Monitor III Data Portal • In other words. In this case.Resource Measurement Facility RMF Monitor III Data Portal direct connection to the RMF Distributed Data Server just specify http://<hostname>:8803 ► all resources and metrics supported (same than RMF PM) ► ► 43 RMF Technical Overview © 2010 IBM Corporation • Since the RMF Distributed Data Server talks XML over HTTP. the presentation is managed by associated formatting instructions in terms of Style Sheets (XSL documents) • The chart shows the Overview Performance Desktop (=PerfDesk) with the following metrics: • % processor utilization by MVS image • % total physical utilization (CP) by partition • performance index by important WLM service class period • % CSA utilization by MVS image . the user can customize his own private PerfDesk (aka MyView). The definitions are stored persistently in terms of browser cookies • The example shows a Coupling Facility PerfDesk with the following metrics: • sync request rate by CF structure • sync service time by CF structure • async request rate by CF structure • async service time by CF structure .Resource Measurement Facility RMF Monitor III Data Portal… Portal… select favorite Metric from List click Add this Metric to My View for Persistence select any Resource from Configuration View ✔ Coupling Facility PerfDesk with: Request rates ► Service times ► 44 RMF Technical Overview © 2010 IBM Corporation • Beyond the Overview PerfDesk. Thereby he can select from a rich set of metrics for any kind of resource he wants to monitor contionuously. the strengths of the Monitor III ISPF reporting are combined with a state of the art frontend: • The window is resizable and scrollable • All report columns can be displayed (including ISPF hidden fields and the content of popup windows) • The columns can be sorted by clicking on the column header • Support of automatic refresh (GO mode) • Multiple reports can be displayed in parallel (tabbed browsing) • Print facility and Spreadsheet export .Resource Measurement Facility RMF Monitor III Data Portal… Portal… ► Combine strengths of Monitor III with state-of-the-art frontend ► View full Reports and exploit the SORT capability Report name Sortable! Scrollable! All columns visible! Resizeable! 45 RMF Technical Overview © 2010 IBM Corporation • In the past. the Data Portal was limited to display single metrics for selected resources • This concept has been extended: with z/OS V1.8 RMF. complete Monitor III reports can be displayed in the browser window • Thus. ERBDSREC ERB3XDRS ERB2XDGS RMF Provider for z/OS CIM Server RMF Technical Overview • RMF offers a couple of powerful APIs to retrieve z/OS performance data.Resource Measurement Facility RMF Performance Data API‘ API‘s
ERBSMFI . The exploiters can be: • Other z/OS components or systems management applications • User written applications • All RMF APIs are documented in the RMF Programmers Guide (SC33-7994) © 2010 IBM Corporation .Monitor II Data (SMF Type 79)
RMF Sysplex Data Server (SDS) ► SMF Data: ► Monitor III Data: ► Monitor II Data:
46 ERBDSQRY. Resource Measurement Facility Sysplex Data Server: Services
Sysplex-wide ► ERBDSQRY ► ERBDSREC access to SMF data requests a directory of available SMF data in the sysplex requests SMF record data in the sysplex
Sysplex-wide access to Monitor III data ► ERB3XDRS requests a set_of_samples of Monitor III data ✘ does not require an ISPF and Monitor III reporter environment ✘ provides data reduction features to transfer only the necessary data
Sysplex-wide access to Monitor II data ► ERB2XDGS requests Monitor II data according to the specified SMF type 79 subtype ✘ returns Monitor II snapshot data ✘ provides data reduction features like ERB3XDRS all Services are available as High-Level-Language APIs 47 RMF Technical Overview © 2010 IBM Corporation • The RMF Sysplex Data Server provides a set of callable services to retrieve performance data from all RMF Monitors • Please note. the record images can be obtained with the RMF services ERBDSQRY and ERBDSREQ . that the SMF data related services are not limited to the SMF 70-78 record types: once the user has specified any SMF record type for buffering by the Sysplex Data Server (via the SMFBUF parameter). Resource Measurement Facility Groovy! Postprocessor ZZ Session Monitor III Monitor II Mintime & Range Interval & Cycle Time % Workflow 48 RMF Technical Overview Sysplex Data Server Distributed Data Server Spreadsheet Reporter CIM Data Provider % Using / Delay © 2010 IBM Corporation That’s the freshly baked RMF expert! Probably he feels a bit more skilled now. At least he came out of his hiding-place! . presentations.com/systems/z/os/zos/bkserv/ 49 RMF Technical Overview © 2010 IBM Corporation .ibm. ■ Downloads


Spreadsheet Reporter RMF PM Java Edition RMF data collector for Linux RMF email address: [email protected]/systems/z/os/zos/features/rmf/ ■ Product information.Resource Measurement Facility Information and Tools RMF homepage: www..ibm. SC33-7991 ■ RMF User‘s Guide.com Documentation and news ■ RMF Performance Management Guide.ibm. SC33-7992 ■ RMF Report Analysis. newsletters.. . SC33-7990 ■ Latest version of PDF files can be downloaded from: http://www. Resource Measurement Facility RMF Redbook !!! 50 RMF Technical Overview © 2010 IBM Corporation .