BW Systemadministration – PortfolioExtraction and Data Transfer Data Load Monitoring, Reload and Repair ODS and Datamart Handling Process Chain Architecture & Functionality Process Chain Modelling Aspects Process Chain Monitoring & Error Handling Data Modelling from the technical point of view BW Transport Management System Periodical jobs and tasks Monitoring Changerun Load Balancing & Number Range Buffering BW Statistics and Query Tuning with Aggregates(RSRT) Usage of Basis Transactions (SM50, …) for applicational people SAP AG 2005, BW System Administration / 1 1 Process Chain Architecture & Functionality First steps & Basics Specific settings Background Processing & Load Distribution to different Servers Transporting Process Chains Tips & Tricks Implementing new Processes SAP AG 2005, BW System Administration / 2 2 Introduction: Typical Data Load Cycle Load into PSA Load into ODS Activate Data in ODS Data Load Monitor Start Drop Indices Roll up Aggregates Build DB Statistics Load into Cube Data Target Maintenance SAP AG 2005, BW System Administration / 3 Build Indices 3 Motivation Automating the BW administration Graphical modelling and monitoring Central monitoring of all BW processes Openness for all kinds of processes Moving responsibility into the individual processes SAP AG 2005. BW System Administration / 4 4 . Yellow: Process with warnings . change of design.Grey: unplanned processes (e. .Green: Error-free processes . new process.Yellow: planned but unknown processes ...Yellow: running .Red: aborted or failed note that the log information is a usually mixture of .Green: planned processes (process chain is active and start process is released) .g.g.Grey: Not yet run . activation) 5 .Green: Finished without error . missing index deletion and recreation.Red: Process with errors Log view .log information of maintenance activites (e. wrong references in variants. BW System Administration / 5 There are three main views in the icon bar with different views: Planning view (shows if the process chain is active) .3 Different Views to Process Chains Views related to the working area Planning view for checking the plan status of the process chain Checking view for check consistency of the process chain Log View for monitoring maintenance activities and executions Different views to the the activities which are possible: Process chains Process types Data targets InfoSources Logs working area SAP AG 2005.) .Red: multiple planned processes Checking view (consistency check like double used start variants. not activated process chain) . For reassignment of process chain to different display component For creation of new display component attributes 6 . BW System Administration / 6 Folders in the process chain are called display components.Different Object Trees for Process Chain Administration For easy administration on process chains different object trees can be displayed. For mainentance of display components you have to use process chains display components. Display component Available Process Types Process chain SAP AG 2005. BW System Administration / 7 7 .Different object trees when building process chains For easy administration on process chains different object trees can be displayed. Log Display the log tree InfoAreas Search in InfoProvider tree InfoSources Search in InfoSources tree for InfoPackages In InfoPackage reference to process chain is displayed Possible processes on data target are displayed SAP AG 2005. BW System Administration / 8 A starter process is part of every process chain. Examples include: Data load Attribute/Hierarchy Change run Aggregate rollup Reporting Agent Settings Other special types of application processes exist: Starter – process that exists to initiate process chain execution ABAP program Another process chain Customer built process SAP AG 2005. 8 .Application Processes Application processes represent BW activities that are typically performed as part of BW operations. only Standard-Out is given back to BW Remote process chains Supports process chains in a data mart environment Master data reorganization Reporting Agent Settings Exception reporting Print queries in background Pre-calculating web templates Pre-calculating characteristic variables of type precalculated value sets SAP AG 2005.Application Process Types Operating system command Access to OS commands like ‘dir’ (list contents of application server file system) Note: There is only an call “out” to OS. BW System Administration / 9 9 . etc. BW System Administration / 10 Collector processes allow the designer of a process chain to trigger a subsequent process based on whether certain conditions are met by multiple predecessor processes.Collector Processes Collectors are used to manage multiple processes that feed into the same subsequent process. even if several predecessor processes raise an event SAP AG 2005. this event chain is internal and cannot be edited. Application processes are the other type of processes – these represent BW activities such as aggregate rollup. The collectors available for BW are: AND: All of the processes that are direct predecessors must send an event in order for subsequent processes to be executed OR: A least one predecessor process must send an event The first predecessor process that sends an event triggers the subsequent process EXOR: Exclusive “OR” Similar to regular “OR”.x. 10 . Although the “AND” condition is implemented for process chains using the event chain functionality from 2. but there is only ONE execution of the successor processes. 11 .RSPC User Interface: Building a Process Chain Load Process and Subsequent Processing Data Loading Process Read PSA and update data target Save Hierarchy Further Processing of ODS Object Data Data Export into External Systems Delete Overlapping Requests from InfoCube Data Target Administration Delete Index Generate Index Construct Database Statistics Initial Fill of New Aggregates Roll Up of Filled Aggregates Compression of the InfoCube 1. BW System Administration / 11 Start TR Load Data drop TR Texts 2. Change TR Characteristics 3. Drag and Activate ODS Object Data Complete Deletion of Data Target Contents Other BW Processes Attribute Change Run Adjustment of Time-Dependent Aggregates Deletion of Requests from PSA General Services Start Process AND (Last) OR (Each) EXOR (First) SAP AG 2005. Draw line to connect processes Attrib. Press ‚Refresh‘ When you draw a line to connect processes. In other words. you are prompted to indicate whether the subsequent process should execute based on the success or the failure of the predecessor process. it is possible to schedule a process to run only if the predecessor process fails. BW System Administration / 12 The process variant could be used just by one process chain. Each process chain could be started manually or via RFC connection with the function module ‚RSPC_API_CHAIN_START‘. 12 .Start Process Variant When and how process chain will be scheduled Note: For each process chain the start process is mandatory. It is not possible to use the same variant more than once !!! SAP AG 2005. g.Structure of a process – Example on variant Maintenance of variant is specific for every process type A variant is the configuration of a process of a particular type set at the time of definition (e. the variant for a data load process is an InfoPackage) Type – Kind of task Attribute Change Run ps_attrib2_long PA_ATTRIB2 Execute InfoPackage: ZPAK_4QVBBF65GZONPAP96APH2P3KQ Variant – Configuration Process = process type + variant SAP AG 2005. BW System Administration / 13 13 . BW System Administration / 14 If standard process types can be changed depends on the changeability of name ranges (transaction SE03) 14 .Structure of a process – Example of process type Type – Attribute Change Run Execute the hierarchy and attribute change run Process types can be maintained via Settings – Process Types Do not change standard process types (if allowed) SAP AG 2005. BW System Administration / 15 15 .Structure of a process – Example of variant Variant – Configure the Hierarchy and Attribute Change run Execute the specific hierarchy and attribute change run for which hierarchy. which InfoObject or which data loading process SAP AG 2005. BW System Administration / 16 16 .Process Types – Example Execute InfoPackage SAP AG 2005. Maintain Message Planning view context menu Write a message and fill in recipient and type. Info saved within process variant. Send with Note [email protected][email protected] SAP AG 2005, BW System Administration / 17 Internet Mail Access has to be configured (with SAPCONNECT) for sending Emails from SAP-System to the Internet via SMTP. 17 Process Chain Architecture & Functionality First steps & Basics Specific settings Background Processing & Load Distribution to different Servers Transporting Process Chains Tips & Tricks Implementing new Processes SAP AG 2005, BW System Administration / 18 18 Automatic Insertion of Corresponding Process Types If a process is inserted into the process chain the corresponding process variants are inserted into the process chain automatically: Example: Example You drag and drop a data load process to your process chain, the Index drop and the Index create process are automatically inserted. If you want no corresponding processes to be inserted automatically, flag the (user specific) setting in the menu under Settings Default Chains SAP AG 2005, BW System Administration / 19 19 Although the CPU is not under pressure during the waiting time. Distributed processes. are also informed about the status of distributed processes. The main job run all the time in parallel (flag is set) or is only triggered by a finished process of the process chain (flag is not set: default). You require one more background process. With the polling flag you determine whether the main process needs to be kept as long as the actual process has ended. SAP AG 2005. it is during the status checks that run at two minute intervals. are characterized as having different work processes involved in specific tasks. BW System Administration / 20 With setting the polling flag you can control the behavior of the main process when you have distributed processes. 20 . Setting the indicator also has the following disadvantages: Increased demand on resources. such as the loading process.Polling Flag With setting the polling flag you can control the behavior of the main process when you have distributed processes (Process Chain Attributes Polling). Setting the indicator has the following advantages: A high level of process security is guaranteed External scheduling tools that react only to the SAP internal event 'Batch Process Complete'. SAP AG 2005.WAIT Time . BW System Administration / 21 21 .. Reason: Sometimes there are alos other processes influenced by this wait time. in order to have some time to catch a process for debugging in SM50! Don‘t use this feature to getting some wait time for a productive process chain.. Note: This wait time is just for Debugging purposes. . BW System Administration / 22 22 . SAP AG 2005..WAIT Process Sometimes it is necessary to have some wait time before the subsequent process is started . Please refer to notes: 604448 24092 519059 Collection note on job management Distribution of background jobs on application servers FAQ: Background processing system 23 . SAP AG 2005. Due to the fact that the different processes of this chain are scheduled via event. they are usually started also on this server. if the process chain is not started directly. BW System Administration / 23 For the case that you start the process chain with directly the the setting for background server is given as proposal.Background Server Note: This setting is just valid. when you execute the process chain. But you can change the background server within this step. Only if there are to less background processes they are distributed to other servers. but as sub chain! With this setting you can assign this process chain (especially the Process BI_PROCESS_TRIGGER to a specific background server. Process Chain Architecture & Functionality First steps & Basics Specific settings Background Processing & Load Distribution to different Servers Transporting Process Chains Tips & Tricks Implementing new Processes SAP AG 2005. BW System Administration / 24 24 . Execution of a direct scheduled process chain After process chain is started. For subchains just process ‚BI_PROCESS_CHAIN‘ is released. BW System Administration / 25 Time scheduled 25 . the different processes are scheduled and released. Event triggered Start process chain All processes except the BI_PROCESS_TRIGGER are event scheduled All other processes are event triggered SAP AG 2005. . BW System Administration / 26 Recommendation is just useful for SAP Systems with more than one SAP Instance 26 .61.. This means there is a fixed realationship for the pickup times between the different SAP Instances. like 59. Result: one instance is usually the favourite execution server Recommendation: Set rdisp/btctime on the different SAP Instances to different values.. With this setting there is no fixed relationship for the pickup times anymore SAP AG 2005..General Rules for Time Scheduled Jobs Description of Time scheduled Jobs On each SAP Instance with BTC processes a system program runs periodically (rdisp/btctime=(60)) and picks up as much jobs from the ‚time scheduling‘ queue until all the BTC processes of the Instance are occupied After installation: periodicity of the time scheduler is et to 60s on each SAP Instance. 60. the system checks the SAP parameter rdisp/btcname on the actual SAP Instance and starts the event trigger program on the Instance which is specified with this parameter (default: Central Instance). BW System Administration / 27 27 . this Instance will take almost all event triggered BTC Jobs (this means no load distribution for Process Chain Jobs!) SAP AG 2005. the Job is converted to ‚time scheduled‘ If the Job is converted to time scheduled the mechanism for time scheduled jobs is working (see slide before!) Result: If rdisp/btcname is the same on all Instances and there are sufficient BTC processes running on this Instance. After this the event scheduler tries to start the Job immediately on this Instance or in case of that all BTC processes occupied.General Rules for Event Triggered Jobs (1) Description of Event Triggered Jobs When an event is triggered. so that usually these BTC processes are not sufficient for all jobs which are triggered by event -> conversion to time scheduled takes place Set rdisp/btctime on the different SAP Instances to different values. but not smaller! Disadvantage: Delay times are occuring for event triggered jobs .. BW System Administration / 28 28 .. 60. like 59. but the more Instances the less delay times.General Rules for Event Triggered Jobs (2) Case 1: Typically for more than 2 SAP Instances Recommendation: Use one btcname – Server for all Instances.31.61. in order to minimize delay times: rdisp/btctime could be set to 30. …..4). With this setting you get an almost equal distribution of BTC Jobs over all other servers except rdisp/btcname No Backgound Server have to be assigned to the START processes of the different process chains. 2 . This BTC name server (rdisp/btcname) must have less BTC processes (e. SAP AG 2005.g.32.. BW System Administration / 29 29 . Set rdisp/btctime on the different SAP Instances to different values. 60. After this you have to assign dedicated server names for the START processes of the process chains.General Rules for Event Triggered Jobs (3) Case 2: 2 SAP Instances Recommendation: Assign for rdisp/btcname in the Instance profile the own Instance name. like 59. Disadvantage: Unflexible and getting optimal Job distribution on the different servers is more difficult.. SAP AG 2005. .. Therefore you get different settings for this parameter on different Instances. If there are sufficient BTC processes are configured on the Instances the entire process chain runs on the same Instance. BW System Administration / 30 30 .Goal: Equal distribution of BTC Jobs over all servers RZ01: SAP AG 2005. Dedicated server for a direct scheduled process chain After starting a process chain you are able to assign a dedicated server for the Job ‚BI_TRIGGER_PROCESS‘ Due to the fact that the other processes of the chain are event triggered. BW System Administration / 31 604448 24092 519059 700479 636854 Collection note on job management Distribution of background jobs on application servers FAQ: Background processing system Load balancing of background jobs does not work Jobs remain in the 'Ready' status for a long time 31 . they are usually running on the server which is determined with the SAP Parameter /rdisp/btc_name (instance specific!) (see note 24092) SAP AG 2005. If subchain are used there is a minimum number of BTC workprocesses necessary in order to avoid dead lock situations # BTC (minimum) = number of parallel starting subchains at the widest part of the chain + 1 details: see note 621400 SAP AG 2005. you can improve the overall runtime of the process chains.Number of required BTC Processes If no subchains are used there is no special requirement to the number of BTC work processes. in order to avoid unnecessary delay times. BW System Administration / 32 rough calculation for the number of BTC processes in order to get best performance and avoid delay times in the batch queue: # BTC (optimal) = number of parallel processes at the widest part of the chain + number of parallel subchains at the widest part + 1 32 . Sometimes it makes sense to configure more than 2 BTC processes per CPU. If complex chains (also in parallel) are executed. if there are sufficient BTC processes configured. Number of required BTC Processes Widest part: 2 subchains 3 processes Minimum mumber of BTC processes: 2+1=3 SAP AG 2005. BW System Administration / 33 Optimal number of BTC Processes: 3+2+1=6 33 . 34 . there is one free BTC process missing for BI_Process_Trigger of the second subchain. • Due to the fact that the there are just 5 BTC configured on the SAP Instance. This process starts subsequent processes and finishes after the subsequent processes are sending status information back to BI_Process_Trigger •PI_Process_Chain processes are starting subsequently the BI_Process_Trigger jobs of their assigned local process chains. because there are just 5 BTC configured on the SAP Application Server SAP AG 2005. This job is still waiting in the batch queue for free resources on this or further SAP Instances. BW System Administration / 34 meaning of this slide •there are 5 BTC work processes configured for the BW System (in this case just Central Instance exists) •After execution of the process chain all BTC processes are allocated •BI_Process_Trigger for the Process Chain itself.Number of required BTC Processes SM50: Workprocess Overview: 5 BTC configured SM37: Job Overview: active jobs Note: BI_Process_Trigger for the second sub chain could not be started immediately. BW System Administration / 35 Please also refer to the note: 621400 35 .Subchains If you have a complex structure of subchains which are running in parallel. you have to provide a minimum number of BTC processes: # BTC (minimum) = number of parallel subchains at the widest part of the chain + 1 Otherwise you sometimes get a Deadlock situation…… SM 37: Job Overview SM50: Configuration with 3 BTC processes All BGD processes are blocked by ‚BI_PROCESS_CHAIN‘ SAP AG 2005. process chains with many parallel running master data loads) SAP AG 2005.Number of BTC processes If process chains are used intensively and complex scenarios with lots of subchains are designed. then a sufficient number of BTC processes shall be configured Rule of thumb: #BTC = 2 * #CPUs not valid in all cases anymore In specific cases the runtime for process chains could be optimized by increasing number of BTC processes (e.g. BW System Administration / 36 36 . Optimize Load distribution Apply different parameter values for rdisp/btctime on different servers (e.g. which checks periodically after rdisp/btctime the central BTC Queue for waiting BTC jobs Event triggered and immediately started jobs are usually not started view batch scheduler I f subchains are used. 37 . 61) (see note 519059) Each server has its own batch scheduler. 60. 1-2 BTC per CPU) SAP AG 2005.g. a different background server can be assigned to the subchains with attribute ‚background server‘ in the maintenance view Usually the processes within one process chain are running on the same server (because they are event-scheduled!) BTC process definition on the application server General rule: don‘t apply too much BTC work processes for an SAP instance (e. BW System Administration / 37 rdisp/btctime: Start Interval for Background Scheduler Specifies the frequency in seconds with which the background processing job scheduler is to be started in a server which can run jobs in the background. 59. Remark: General Rules for immediately scheduled Jobs Usually immediately scheduled Jobs are not used in BW. The jobs mostly run on the same server Result: The jobs mostly run on the same server Recommendation: apply note 700479 in order to get a round robin procedure SAP AG 2005. Description of Immediately scheduled Jobs default setting: Jobs that have the 'Immediate start' start condition and no target server are not distributed consistently on the different batch servers.. BW System Administration / 38 38 . besides exceptional cases like restart of process chains.. . Process Chain Architecture & Functionality First steps & Basics Specific settings Background Processing & Load Distribution to different Servers Transporting Process Chains Tips & Tricks Implementing new Processes SAP AG 2005. BW System Administration / 39 39 . BW System Administration / 40 40 .Process Chains: Transport Process chains are transportable Button for writing to a change request when maintaining a process chain in RSPC Process chains available in the transport connection wizard (administrator workbench) Tlogo Object R3TR RSPC SAP AG 2005. BW System Administration / 41 41 .Transport – Import Post-Processing Destination SAP AG 2005. Changeability Note 337950 “Queries (and other objects) cannot be edited” This option allows to deactivate the Change and Transport System for specific BW components This setting is only intended for systems in production or test systems that are set to not changeable ! SAP AG 2005. BW System Administration / 42 42 .Transport . Process Chain Architecture & Functionality First steps & Basics Specific settings Background Processing & Load Distribution to different Servers Transporting Process Chains Tips & Tricks Implementing new Processes SAP AG 2005. BW System Administration / 43 43 . Consideration of automatisms in process chains The following loading process automatisms are ignored within process chains: Flag for automatic ODS-activation in ODS-maintenance Flags for automatic deletion/creation of infocube indexes/statistics (Performance-Tabstrip in Infocube Manage) Flags for automatic Infocube Content Deletion or Deletion of overlapping requests in infopackage Flag in Infopackage: PSA + update subsequently in datatargets There are differences between manual execution and execution in process chain For all these cases there are own process types to get the same functionality and to avoid interdependency / deadlock problems. SAP AG 2005. BW System Administration / 44 44 . BW System Administration / 45 Only the BW Quality flag is processed. otherwise unknown kinds of error messages can be displayed… 45 .Consideration of automatisms in process chains These flags aren‘t processed within a Process Chain ! Instead of this: Process Type „Complete Deletion of Data Target Contents“ Instead of this: Process Type „Read PSA and Update Data Targets“ Instead there is: Process Type „Delete Overlapping Requests from InfoCube“ SAP AG 2005. but it is a prerequisite for Process Chains that this flag is set. Start Process Variant name and description Direct scheduling: Job BI_PROCESS_TRIGGER will be scheduled when the process chain is executed. BW System Administration / 46 46 . Start of the process chain have to be done via FM ‚RSPC_API_CHAIN_START‘ or with another process chain SAP AG 2005. Start using Meta Chain of API: No BI_PROCESS_TRIGGER will be scheduled. Technical Name of process chain SAP AG 2005.Manual Start of Process Chain Function Builder: SE37 Sometimes you want to start a process chain immediately. BW System Administration / 47 Each process chain could be started always manually or via RFC connection with the function module ‚RSPC_API_CHAIN_START‘. You don‘t want to change the variant of the start process. 47 . Display Component Typical situation: Folder structure missing! Proposal: Apply your own display components SAP AG 2005. BW System Administration / 48 48 . Restartability of Process Types For each process type (self defined or standard) re-starting is a feature which is set in table: RSPROCESSTYPES SAP AG 2005. BW System Administration / 49 49 . BW System Administration / 50 50 .Copying Process Chain – Step 1 + 2 Transaction Code: =copy Assign new name: SAP AG 2005. Copying Process Chain – Step 3 + 4 Exchange start variant: Assign new unused start variant: SAP AG 2005. BW System Administration / 51 51 . The simple view displays the processing chain as it exists logically. The detail view displays the processing chain with the extra collector processes. there are actually several background jobs scheduled with events in order to construct the conditional nature of collector processes. a collector process is displayed as multiple collector processes (needed for conditions) SAP AG 2005.Maintain Process Chains – Detail View On RSPC -> View -> Detail View On: Technical names Ability to move boxes to re-design the process chain Note: In detail view. 52 . BW System Administration / 52 Hidden collector processes are displayed When collector processes are built into a process chain. BW System Administration / 53 53 .Master Data Reorganization Available as process type in process chains Minimize table size: improve performance Attribute/Hierarchy Change Run Query performance Often required in HR Two features supported: Compressing time intervals for time-dependent master data Compressing attributes with same A.and M-version SAP AG 2005. I Compressing time intervals for time-dependent MD Combining records with same attributes and adjacent time intervals Example: An employee‘s position (time-dependent master data) 01/2001 11/2001 Reorganization 01/2001 11/2001 Development manager Development manager Development manager 2 records 1 record SAP AG 2005. BW System Administration / 54 54 .Master Data Reorganization . and M-version Deletion of equal A.Master Data Reorganization . BW System Administration / 55 55 .II Compressing attributes with same A.and M-version of master data records Scenario: Active data (A-version) already exists Load of new (different) data M-version inserted No activation of loaded master data Load of new data again which equals A-version active data modified data blue green 1st upload blue blue Reorganization Example: Master data for article 4711 (attribute: colour) blue blue time no activation 2nd upload SAP AG 2005. Process Chain Architecture & Functionality First steps & Basics Specific settings Background Processing & Load Distribution to different Servers Transporting Process Chains Tips & Tricks Implementing new Processes SAP AG 2005. BW System Administration / 56 56 . which should be displayed in the process monitor ..Have implemented an own monitor .. ..Appendix: Openness: Implementing a Process If you..Have a simple.then it is recommended to custom develop own process type SAP AG 2005. which is planned in batch by another program or user .. BW System Administration / 57 57 .. stand alone program to run in a chain ..... However if you.Want to ask predecessor processes of your program .Write a log ...Have a program.. which is more difficult than the “parameters“ of a program ....then you can use the process type “ABAP” to run your program in a chain.. ..Have a customizing screen... BW System Administration / 58 58 .Appendix: Implementing a Custom Designed Process Interfaces: IF_RSPC_* EXECUTE GET_INFO GET_LOG MAINTAIN GET_VARIANT GET_DEFAULT_CHAIN CHECK GET_STATUS CALL_MONITOR TRANSPORT CONTEXT_MENU – – – – – – – – – – – Execution of the process Give Information to successors Give back messages Maintenance of variants F4 for variants Give default-chain for a process Check consisteny Give status of instance own monitoring tool give back TLOGO-entry Enhance context-menu Make entry in view RSPROCESSTYPES The interfaces are documented in the system SAP AG 2005. : Interface: IF_RSPC_EXECUTE is executed at runtime of the Process Chain.: Interface: IF_RSPC_Get_Status is executed at runtime and when startig log view. It is not mandatory to use this interface within your own process type.g.g. mandatory alternatively optional Interfaces: EXECUTE GET_INFO GET_LOG MAINTAIN GET_VARIANT GET_DEFAULT_CHAIN Maintainance Runtime Log View X X X X X X X X X X X X X X X X X X X X X X X X X CHECK GET_STATUS CALL_MONITOR TRANSPORT CONTEXT_MENU SAP AG 2005. 59 .. e..Appendix: Implementing a Custom Designed Process Method called at.. Implementing is. Usage of this Interface within your self defined process type is mandatory Process Type ABAP just uses IF_RSPC_EXECUTE and IF_RSPC_GET_Variant.. No status is given back. BW System Administration / 59 means: e. Appendix: Generic Services There are several services in Paket RSPC which shall help you implementing a process type: Generic variant storage class: CL_RSPC_VARIANT Generic dynpro service for variant maintainance: Function module RSPC_VARIANT_MAINTAIN Generic instance storage class: CL_RSPC_INSTANCE Wrapper class for application log for unambigous assignment of logs to instance CL_RSPC_APPL_LOG Service function group: RSPC_SERVICES SAP AG 2005. BW System Administration / 60 60 .