SAP HANA Troubleshooting and Performance Analysis Guide En

PUBLICSAP HANA Platform SPS 11 Document Version: 1.0 – 2015-11-25 SAP HANA Troubleshooting and Performance Analysis Guide Content 1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.1 Related Information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2 Symptoms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1 Performance and High Resource Utilization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Authorization, Authentication and Licensing Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3 Common Symptoms and Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Slow System-wide Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Slow Individual SQL Statements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Frequent Out of Memory (OOM). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3 Root Causes And Solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.1 Memory Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Memory Information in SAP HANA Studio. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Memory Information from Logs and Traces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Memory Information from SQL Commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Memory Information from Other Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27 Root Causes of Memory Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.2 CPU Related Root Causes and Solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 Indicators of CPU Related Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 Analysis of CPU Related Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Resolving CPU Related Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38 Retrospective Analysis of CPU Related Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Controlling Parallelism of SQL Statement Execution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Bind Processes to CPUs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.3 Disk Related Root Causes and Solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Check Internal Disk Full Event (Alert 30). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.4 I/O Related Root Causes and Solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Analyzing I/O Throughput and Latency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Savepoint Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.5 Configuration Parameter Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52 Issues with Configuration Parameter log_mode (Alert 32 and 33). . . . . . . . . . . . . . . . . . . . . . . 53 3.6 Delta Merge. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Inactive Delta Merge. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Retrospective Analysis of Inactive Delta Merge. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .56 Indicator for Large Delta Storage of Column Store Tables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 Analyze Large Delta Storage of Column Store Tables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 2 PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. SAP HANA Troubleshooting and Performance Analysis Guide Content Failed Delta Merge. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Delta Storage Optimization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.7 Security-Related Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 System Locked Due to Missing, Expired, or Invalid License. . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 License Problem Identification and Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .65 Resolution of License Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Troubleshooting Authorization Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .66 Troubleshooting Problems with User Name/Password Authentication. . . . . . . . . . . . . . . . . . . . 71 3.8 Transactional Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Blocked Transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Troubleshooting Blocked Transaction Issues that Occurred in the Past. . . . . . . . . . . . . . . . . . . 78 Multiversion Concurrency Control (MVCC) Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Version Garbage Collection Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 3.9 Statement Performance Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .86 SQL Statement Optimization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 SQL Statements Responsible for Past Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87 SQL Statements Responsible for Current Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 SQL Statements Reported by Traces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Analysis of Critical SQL Statements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 SQL Plan Cache Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Example: Reading the SQL Plan Cache. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Detailed Statement Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .95 Optimization of Critical SQL Statements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Outlier Queries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Data Manipulation Language (DML) Statements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 SQL Query Tuning Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Creation of Indexes on Non-Primary Key Columns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Create Analytic Views. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 Developing Procedures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 Application Function Library (AFL). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 About Scalability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Further Recommendations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 3.10 Application Performance Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 SQL Trace Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Statement Measurement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 Data Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 Source Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Technical Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 3.11 System Hanging Situations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 Transparent Huge Pages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 CPU Power Saving. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 SAP HANA Troubleshooting and Performance Analysis Guide Content PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 Blocked Transaction Monitoring.1 SAP HANA Dynamic Tiering. . . . . . . . 164 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 SQL Statement Analysis. . . . . . . . . . . . . . . . . . 131 4 Tools and Tracing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 Analysis Tools in SAP HANA Web-based Developer Workbench. . . . . . . . . . . . . . . . . . . . . . . . .4 Advanced Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 4. . . . . . . 163 Analyzing Column Searches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Query Plan Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Analyzing SQL Execution with the SQL Plan Cache. . . . . . . . 140 4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .139 Load Monitoring. . . . . . . . . . . . 137 Session Monitoring. . . . . . . . . . . . . . . . . . . . .12 Troubleshoot System Replication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 6. . . . . . . . . . . . . . . . . . . . . . . . . 168 Diagnosis Information. . . . . . . 166 Performance Trace Options. . . . . . . . . . . . . . . . . . . 166 Kernel Profiler. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 Kernel Profiler Options. . . . . . . 165 Performance Trace. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . All rights reserved. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .147 Analyzing SQL Execution with the Plan Explanation. . . . . . . . . . . 134 Thread Monitoring. . 147 Analyzing SQL Execution with the Plan Visualizer. . . . . . . . . . . . . . . . 191 Tools and Tracing. 192 Data Loading Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 4. . . . . . . . . . . . . . . . . . 191 Query Plan Analysis. . . . . . . . . . . . . . . . . . . . . . . SAP HANA Troubleshooting and Performance Analysis Guide Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 Additional Analysis Tools for Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 Job Progress Monitoring. . . . .1 System Performance Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Analyzing Table Joins. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 4 PUBLIC © 2015 SAP SE or an SAP affiliate company. . . . . . . . . . . . . . . . . . . . . . . . 172 6 HANA Options: Performance Analysis. . . . . . . . 141 Analyzing Expensive Statements Traces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 4. . . . . . . . . 169 5 Alerts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 Analyzing SQL Traces. . . . . . . . . . . . . . . . . . . . . . . . . ● The latest version of SAP HANA studio is required. see the following: Content Location SAP Data Services Performance Optimization Guide http://help.pdf Performance of SAP Software Solutions http://service. for example. disk) ● Size and growth of data structures ● Transactional problems ● SQL statement performance ● Security. you can analyze data at incredible speeds. HA200).sap. However.1 Related Information For more information about identifying and resolving performance issues. All rights reserved. 5 .sap. 1. memory.1 Introduction With SAP HANA.com/businessobject/product_guides/ sboDS42/en/ds_42_perf_opt_en. The functionality discussed in this document may not be available in previous versions of SAP HANA. such results are only possible if the system is monitored and performance issues are kept to a minimum. with scans of 1 billion rows per second per core and join performance of 10 million rows per second. and licensing ● Configuration The guide focuses on SAP HANA SPS 08 and subsequent releases.com/performance Related Information SAP HANA Administration Guide SAP HANA Troubleshooting and Performance Analysis Guide Introduction PUBLIC © 2015 SAP SE or an SAP affiliate company. Prerequisites ● Knowledge of the relevant functionality of the SAP HANA database (for example courses HA 100. authorization. ideally matching the version of SAP HANA on the server. This guide describes what steps you can take to identify and resolve specific performance issues and what you can do to enhance the performance of your SAP HANA database in the following areas: ● Host resources (CPU. SAP HANA SQL and System Views Reference 6 PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. SAP HANA Troubleshooting and Performance Analysis Guide Introduction . See Memory Problems for more information on alert configuration. High Memory Consumption You observe that the amount of memory allocated by the SAP HANA database is higher than expected. review Alerts to find the part of this guide which addresses the problem. a problem can have its roots in a number of seemingly unrelated components. Checking the alerts is a good starting point if you experience any trouble with your SAP HANA system. All rights reserved.2 Symptoms In some cases. The following alerts indicate issues with high memory usage: ● Host physical memory usage (Alert 1) ● Memory usage of name server (Alert 12) ● Total memory usage of Column Store tables (Alert 40) ● Memory usage of services (Alert 43) ● Memory usage of main storage of Column Store tables (Alert 45) ● Runtime dump files (Alert 46) See the section Memory Problems for information on analyzing the root cause. Related Information Memory Problems [page 20] Alerts [page 172] 2. problems can occur without triggering an alert and alerts do not always indicate a serious problem. high memory usage. paging or column store unloads we can start to narrow down the possible causes as a first step in analyzing the issue. SAP HANA Troubleshooting and Performance Analysis Guide Symptoms PUBLIC © 2015 SAP SE or an SAP affiliate company. However. 7 . most notably performance issues. Our goal is to help you narrow down the probable root cause and find the right part of the guide to proceed with your analysis. If the system issues an alert. since alerts are configurable and do not cover all aspects of the system.1 Performance and High Resource Utilization By observing the general symptoms shown by the system such as poor performance. If you see a constant high usage of memory or CPU. Column Store Unloads You observe unloads in the column store. but standard network analysis tools can also be helpful to determine whether the network is the main bottleneck. Typical reasons for a slow system are resource shortages of CPU. See the section Memory Problems for information on analyzing the root cause. when a table has to be reloaded into memory. Another reason for poor performance. I/O Related Root Causes and Solutions provides ways to check for disk I/O related problems. 8 PUBLIC © 2015 SAP SE or an SAP affiliate company. are other processes running on the same host that are not related to SAP HANA. You can use the operating system tools to check for such processes. which affects future SQL statements.Out of Memory Situations You observe trace files or error messages indicating an Out of Memory (OOM) situation. proceed with sections Memory Problems or CPU Related Root Causes and Solutions respectively. These include not only maintenance related tasks such as savepoints (disk I/O. Note that SAP only supports production systems running on dedicated hardware. this can lead to unloads of tables. see I/O Related Root Causes and Solutions) or remote replication (network I/O). If performance issues only appear sporadically. memory. SAP HANA Troubleshooting and Performance Analysis Guide Symptoms . which in many cases cannot be detected by the SAP HANA instance itself. network performance. but also SQL statements dispatched by other users. See the section Memory Problems for information on analyzing the root cause. Check Administration Overview or Administration Performance Load . Permanently Slow System Issues with overall system performance can be caused by a number of very different root causes. The following alerts indicate issues with high memory usage: ● Column store unloads (Alert 55) See the section Memory Problems for information on analyzing the root cause. disk I/O and. for distributed systems. Basic network I/O data is included in the Load graph and in the M_SERVICE_NETWORK_IO system view. the problem may be related to other tasks running on the database at the same time. which can block a lot of resources. Paging on Operating System Level You observe that paging is reported on operating system level. In the case of memory. All rights reserved. see Memory Problems as well. Note that operating system tools can also provide valuable information on disk I/O load. In this case. 9 . The alert for the licensed memory usage (Alert 44) is issued. SAP HANA Troubleshooting and Performance Analysis Guide Symptoms PUBLIC © 2015 SAP SE or an SAP affiliate company. Authentication and Licensing Issues There are a number of issues that can occur which prevent you from accessing the system. a detailed Statement Performance Analysis might reveal ways to optimize the statement. which are related to Authorization. License Memory Limit Exceeded You observe that the licensed amount of memory is exceeded.2 Authorization. but since they depend on configurable thresholds. there could be an issue with the delta merge operation. some queries are inherently complex and require a lot of computational resources and time. so most of the previous information also applies to statement performance. proceed with Delta Merge. For troubleshooting. In principle. a statement can trigger all the resource problems that also lead to an overall slowdown of the system. All rights reserved. For troubleshooting. ) in the If the runtime of a statement increases steadily over time. blocked transactions. proceed with Transaction Problems. A transactionally blocked thread is indicated by a warning icon ( Status column. Related Information Memory Problems [page 20] CPU Related Root Causes and Solutions [page 35] Disk Related Root Causes and Solutions [page 43] I/O Related Root Causes and Solutions [page 45] M_SERVICE_NETWORK_IO Transactional Problems [page 73] Delta Merge [page 54] Statement Performance Analysis [page 86] 2. In addition. However. Authentication and Licensing. Alerts should be issued for most problems occurring with the delta merge. that is. Blocked transactions can be checked in the Threads tab.Slow Individual SQL Statements or with Increasingly Long Runtimes Issues with the performance of a particular statement can be caused by a number of very different root causes. statement performance can suffer from transactional problems. but the statement is still too slow. If you have none of the above problems. this is not always the case. 2. Typically.3 Common Symptoms and Troubleshooting Typical symptoms and the related troubleshooting information are described in this section. please gather runtime dump more than twice with 3 minute intervals for further investigation by SAP as per SAP Note 1813020 . Generally.How to generate a runtime dump on SAP HANA. all DML statement will fall into wait status. check the following. a full log volume is the cause for this. For each section.1 Slow System-wide Performance Slow system-wide performance issues are problems that could be caused by excessive use of CPU. slow individual statement performance.3. Run "df -h" in OS shell or 10 PUBLIC © 2015 SAP SE or an SAP affiliate company. misuse of SAP HANA database resource such as locks or configuration of OS parameters. System-side slow performance.1. 2. Check DISKFULL event in indexserver trace or 2. Root cause: Log volume full either caused by disk full or quota setting Required action: 1. there is case where we need to check sar files or /var/log/messages because the cause could be at the OS level. Related Information SAP Note 1813020 2.1 System Appears to Hang with no new Connections and no Response from DML In cases where logs cannot be written. All rights reserved.3. Since the performance issue could persist as time goes by or could happen sporadically. you may see continuing high usage of CPU according to OS commands or load graph or many pending or waiting threads in the thread view (both are visible from SAP HANA studio Administration Performance ) If the entire system looks hung or works extremely slower than before. this document will cover the known symptoms and the corresponding troubleshooting steps to follow depending on the causes. when you encounter a performance issue. This can lead to a failure of opening new connection because the system internally executes DML statements during the process. SAP HANA Troubleshooting and Performance Analysis Guide Symptoms . The runtime dumps should be collected while the system is experiencing the issue for later analysis in case the situation is already resolved Additionally. and OOM problems are issues that you might experience while using the SAP HANA database. to analyze root cause. With certain revision and conditions. Check whether the symptoms match the description in SAP Note 2214279 . 3. please check backint. All rights reserved.Blocking situation caused by waiting writer holding consistent change lock.log (located at /usr/sap/<SID>/HDB<Instance#>/ <Host>/trace ) to see whether it includes ERROR information. Check quota setting in file system 4. … DataAccess::SavepointLock::lockExclusive() DataAccess::SavepointImpl::enterCriticalPhase(…) … 2. Then. Root cause: log volume full caused by failure of log backup Required action: 1.log (located at /usr/sap/<SID>/HDB<Instance#>/<Host>/trace ) to see whether it includes ERROR in log backup. Check backup.3.How to generate a runtime dump on SAP HANA and look for following call stacks in many threads. the conflict between savepoint lock and DML lock blocks subsequent statements when long running update/insert statements exist. If so. M_LOG_SEGMENTS 2. Create a support ticket attaching a runtime dump for further analysis. Please contact SAP support if you encounter the next case described below. apply the parameter in the Note. follow the procedure described in SAP Note 1679938 . If log backup uses backint. 11 . Root cause: Savepoint lock conflict with long running update Required action: 1.Analyzing log volume full situations. Related Information SAP Note 1679938 SAP Note 2083715 SAP Note 1813020 SAP Note 2214279 SAP HANA Troubleshooting and Performance Analysis Guide Symptoms PUBLIC © 2015 SAP SE or an SAP affiliate company.Log Volume is full. … DataAccess::SavepointLock::lockShared(…) DataAccess::SavepointSPI::lockSavepoint(…) … And one of following call stack that is in the savepoint phase. Check M_BACKUP_CATALOG. You can find additional information in SAP Note 2083715 . Collect runtime dump as per SAP Note 1813020 . and contact backint vendor support. Linux paging improvements Root cause: Problem caused by translation lookaside buffer (TLB) shootdown Required action: 1.ini -> execution -> max_concurrency (>SPS08) Related Information Controlling Parallelism of SQL Statement Execution [page 39] SAP Note 2031375 SAP Note 1557506 SAP Note 2206354 System Hanging Situations [page 128] 12 PUBLIC © 2015 SAP SE or an SAP affiliate company. see SAP Note 2206354 SAP HANA DB: High System CPU Consumption Caused by Plan Trace Root cause: Due to high context switches (High SYS CPU) by many SqlExecutor threads 1. Check sar file ( /var/log/sa ) whether kbcached usage ( sar -r ) is higher than 10% of physical memory and high page in/out occurred 2.ini/indexserver. All rights reserved. SAP HANA Troubleshooting and Performance Analysis Guide Symptoms .ini -> parallel -> num_cores (<=SPS07) and refer to Controlling Parallelism of SQL Statement Execution 2. Apply KBA 2031375 . For more information.1. Check and apply SAP Note 1557506 . Check load graph and indexserver. it consumes large amounts of memory. available from SAP HANA studio Administration Performance Root cause: Problem caused by the configuration of transparent huge page Required action: 1. Therefore. Check Transparent Huge Page whether it is set to [never] by running the command "cat /sys/ kernel/mm/transparent_hugepage/enabled" 2.2 System Appears to Hang with High System CPU Usage The SAP HANA database is an in-memory database and by its nature. you can see high usage of SYSTEM CPU that can be monitored in the load graph. some performance issues of SAP HANA can be caused by the OS's memory management. indexserver.2.3.ini -> sql -> sql_executors / max_sql_executors and refer to Controlling Parallelism of SQL Statement Execution Root Cause: Due to high context switches (High SYS CPU) by many JobExecutor threads. Required action: Check that the following parameters are set to be bigger than the number of logical CPU cores 1.SAP HANA: Transparent HugePages (THP) setting on Linux Root cause: Problem caused by the configuration of OS page cache Required action: 1. global. Check plan trace. For this case. please collect the explain plan.3.1. please create an SAP support ticket. In this case. All rights reserved.2. Root cause: Not many executor threads but high CPU in a few nodes and a few tables accessed Required Action: In an SAP Business Warehouse system. For more information. Root Cause: Performance degradation by huge MVCC versions Required Action: Check for Active Version in the load graph to find the MVCC garbage blocker. the application configuration or its usage. to analyze further. Related Information SQL Statement Analysis [page 140] Query Plan Analysis [page 147] SAP Note 2000002 SAP HANA Troubleshooting and Performance Analysis Guide Symptoms PUBLIC © 2015 SAP SE or an SAP affiliate company. SAP KBA 2000002 FAQ: SAP HANA SQL Optimization explains general information about SQL optimization. If the following doesn’t help to resolve the problem. performance trace and catalog export. see Troubleshooting Blocked Transactions Related Information Indicator for Large Delta Storage of Column Store Tables [page 56] Troubleshooting Blocked Transactions [page 76] SAP Note 1819123 2.2 Slow Individual SQL Statements This section looks at the causes of slow individual SQL statements although there is no significant performance issue in system level and the associated troubleshooting steps.3.3 Slower Response with High User CPU Usage The performance can degrade as the usage of the CPU can be increased by the SAP HANA database configuration. then kill it. Also please refer to SQL Statement Analysis and Query Plan Analysis for more information.BW on SAP HANA SP5: landscape redistribution and Indicator for Large Delta Storage of Column Store Tables. For more information. 13 . check for non-even partitioning of huge column tables. plan vizualizer file. see SAP Note 1819123 . If a long running compiled query plan has been evicted frequently from the plan cache. Required Action: Check the execution time after adding 'with hint (ignore_plan_cache)' at the end of query. For more information.1 A Statement is Sometimes Slow and Sometimes Fast There are a number of things to check when you experience inconsistent query execution time. see Delta Merge Related Information SQL Plan Cache Analysis [page 90] Delta Merge [page 54] 2. a query can have a different plan. Related Information Expensive SQL Statements [page 29] Analyzing SQL Execution with the Plan Visualizer [page 148] 14 PUBLIC © 2015 SAP SE or an SAP affiliate company. And create an SAP support ticket. For more information.2. For more information.2 A Statement is Slower After an Upgrade After upgrade. Required Action: If you have an instance running on an older revision. Check the following possibilities: Root Cause: If a related table was unloaded. All rights reserved. and LAST_MERGE_TIME columns of M_CS_TABLES whether there is large amount of data in delta Check M_DELTA_MERGE_STATISTICS to check when the last merge occurs. You can refer to LOADED column of M_CS_TABLES. Root Cause: merge status of column table can affect query plan Required Action: Check MEMORY_SIZE_IN_DELTA.2. the query execution time can be different because of changes in the query execution plan.3.3. it takes some time to load tables Required Action: Check unload trace and execute after table loaded fully. increase the query cache size. Root Cause: Query compilation time is long. see Expensive SQL Statements and Analyzing SQL Execution with the Plan Visualizer.2. SAP HANA Troubleshooting and Performance Analysis Guide Symptoms . RAW_RECORD_COUNT_IN_DELTA. Root cause: After an upgrade. see SQL Plan Cache Analysis. please compare the plan and collect the plan vizualizer file. which leads to a different execution time. This hint will always cause the query to be compiled. please generally refer to SAP Note 1662726 . Required Actions: Apply parameters for ABAP opt. All rights reserved..Configuration of SAP HANA internal network Root cause: Due to statement routing and huge data transfer among distributed nodes. For more information see Performance Trace and Analyzing SQL Execution with the Plan Visualizer. Related Information Network View [page 156] Performance Trace [page 166] Analyzing SQL Execution with the Plan Visualizer [page 148] SAP Note 2183363 2. Root cause: Indexes are missing Required Action: Check WHERE clause and check concat indexes for all fields used in WHERE clause. However. Root cause: Due to DBSL behavior.2. query execution can be routed to other nodes for better performance. However. Required Action: Check how much data is transferred among distributed nodes and the network performance. For more information see Network View and SAP Note 2183363 .4 Slow Select for all Entries (FAE) Query There are a number of points to check if you have performance issues with SELECT FOR ALL ENTRIES (FAE) from an ABAP query. 15 . slower operator can be chosen.2. the network used may impact performance. there is chance of having slow performance in the case where the network used for transferring intermediate results generated during query execution is slow or where there is an inefficient network configuration. it can be slower than having no filter.SAP HANA: Parameter setting for SELECT FOR ALL ENTRIES Root cause: When using less than or greater than ('<' or '>') filters in FAE query. Root cause: A misconfigured network can lead slow queries. Required Action: Check your network configuration and its bandwidth/latency among SAP HANA servers.3 A Query on Multiple Nodes can be Slower In distributed system. SAP Note 1987132 . Required Actions: Do not use less than or greater than ('<' or '>') filters in an FAE query SAP HANA Troubleshooting and Performance Analysis Guide Symptoms PUBLIC © 2015 SAP SE or an SAP affiliate company. it can lead to the difference of query execution by the difference of anchor nodes. For more information see. Then consider the locations of joined tables to reduce transferred intermediate result size. For FAE on SAP HANA. In distributed system.Optimization of select with FOR ALL ENTRIES on SAP HANA database.3.2.3. query execution can be routed to other nodes for better performance. FAQ: SAP HANA Memory as they provide information on SAP HANA memory and its problems. Root cause: If there are too many versions of single records.3. To verify further whether there are too many versions for specific table. the number of system-wide MVCC versions is in acceptable range. This section introduces common problems and their troubleshooting steps. 16 PUBLIC © 2015 SAP SE or an SAP affiliate company. Root cause: Huge intermediate results during query processing. All rights reserved.2. Required Action: Applications should commit as early as possible or close any cursors that are not needed. select * from m_rs_table_version_statistics where table_name = <TABLE_NAME> This can be caused by a cursor unnecessarily being held on a query result or a long-lived transaction without a commit/rollback. SAP HANA Troubleshooting and Performance Analysis Guide Symptoms .Related Information SAP Note 1662726 SAP Note 1987132 2.3.5 All Statements Accessing a Specific Table are Slow Check if there are too many versions if queries run on specific tables are slow. 2.1 Out of Memory Caused by Sudden Increased Memory Usage Check the memory consumption of statements in the event of OOMs caused by suddenly increased memory usage.3 Frequent Out of Memory (OOM) If Out Of Memory situations happen frequently. First check Memory Problems and SAP KBA 1999997 . check the result of this query:. Related Information Memory Problems [page 20] SAP Note 1999997 2.3. Check the application logic to see whether it really needs to update single records frequently. it can also lead to performance drop by unloading tables or shrinking memory jobs.3. In this case. accessing the table can be slow. Reduce the amount of data in column store or increase physical memory. check the memory consumption of statements using M_EXPENSIVE_STATEMENTS. 17 . the system cannot reclaim the memory used for query execution because its related transactions become dangled. Root cause: Commit/rollback within stored procedure can lead to memory leakages. out of memory (OOM) can happen frequently. see SAP HANA SQLScript Reference for more information. If you would like to use commit/rollback within stored procedure.3. If you find a suspicious component keeping allocating memory. mm trace. create a support ticket attaching a full system dump.3.Required Actions: Enable memory tracking by setting the following parameters to on in the global. ● enable_tracking = on ● memory_tracking = on Then. Root cause: Undersized memory Required Action: 1. Check top allocators in [MEMORY_OOM] section of OOM dump whether they are for column store components as described in section 13 of KBA 1999997 . 2.3. Root cause: Due to memory leakage caused by a programming error Required Action: Check the performance load graph in SAP HANA studio Administration Performance Load to determine whether used memory is continuously increased without a significant increased data size. Check the unload trace whether frequent unloading of tables took place. 3. If this syntax is used. Related Information SAP Note 1999997 SAP HANA Troubleshooting and Performance Analysis Guide Symptoms PUBLIC © 2015 SAP SE or an SAP affiliate company.FAQ: SAP HANA Memory 2.2 Out of Memory Caused by Continuously Increased Memory Usage Check if many statements trigger an Out Of Memory in a system where used memory is continuously increased.MEMORY_SIZE. 2.HOST_HEAP_ALLOCATORS_BASE. After your analysis you can optimize any problematic queries that were found. and the output of _SYS_STATISTICS.3. All rights reserved.3 Out of Memory While High Usage Of Column Store Components' Allocators If a system is under sized. Required Actions: Remove exec(“commit”) or exec(“rollback”). Do not use exec(“commit”) or exec(“rollback”) within a stored procedure.ini file resource_tracking section and enable the Expensive Statement trace. SAP HANA DB: Big statistics server table leads to performance impact on the system Root cause: Big Pool/Statistics allocator size Required Action: Check SAP Note 2147247 .3. Required Action: Check table size of _SYS_STATISTICS. an OOM can occur.5 Out of Memory Occurs due to High Usage of Shared Memory Shared memory is space where system information and row store tables are stored. SAP HANA Troubleshooting and Performance Analysis Guide Symptoms . 18 PUBLIC © 2015 SAP SE or an SAP affiliate company.STATISTICS_ALERTS_BASE and truncate STATISTICS_ALERTS_BASE from hdbsql based on solution from SAP Note 2170779 . Root cause: Due to big STATISTICS_ALERTS_BASE table size.3.FAQ: SAP HANA Statistics Server first Related Information SAP Note 2170779 SAP Note 2147247 2. or else increase the system memory. Root cause: Severely fragmented row store tables Required action: 1.4 Out of Memory Caused by Large Memory Usage of Statistics Server This case can happen if the majority of used memory by the statisticsserver occurs because of many alerts or undersized memory.SAP HANA DB: Row store reorganization Root cause: Due to memory shortage caused by high used memory of row store tables in an SAP Business Warehouse (BW) system Required action: In an SAP Business Warehouse system the used amount of shared memory is high and SAP Note 1813245 doesn't recommend row store reorganization. Check the following if the used amount of shared memory is high.SAP HANA DB: Row store reorganization again Root cause: Due to memory shortages caused by high used memory of row store tables in non SAP Business Warehouse (BW) systems Required Action: Check if you can convert some row store tables into column store tables or archive old data to reduce the memory size. Apply SAP Note 1813245 . All rights reserved. Check whether SHARED_MEMORY in [MEMORY_OOM] -> IPMM short info of OOM dump is relatively high 2.3. first apply SAP Note 706478 .3.2.Preventing Basis tables from increasing considerably. then apply SAP Note 1813245 . All rights reserved. 19 .Related Information SAP Note 1813245 SAP Note 706478 SAP HANA Troubleshooting and Performance Analysis Guide Symptoms PUBLIC © 2015 SAP SE or an SAP affiliate company. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . For more general information on SAP HANA memory management. SAP Note 1898317 – How to Handle Alert ‘Check host free physical memory’ In order to understand the current and historic SAP HANA memory consumption you can use the following tools and approaches: ● Memory information in SAP HANA studio ● Memory information from logs and traces ● Memory information from SQL commands ● Memory information from other tools Related Information SAP HANA Administration Guide SAP HANA Memory Usage Explained SAP Note 1840954 SAP Note 1898317 20 PUBLIC © 2015 SAP SE or an SAP affiliate company.3 Root Causes And Solutions This section provides detailed information on the root causes of problems and their solutions. This SAP Note provides information on how to analyze out-of-memory (OOM) dump files. All rights reserved. It also explains the correlation between Linux indicators (virtual and resident memory) and the key memory usage indicators used by SAP HANA. see the SAP HANA Administration Guide and the whitepaper SAP HANA Memory Usage Explained which discusses the memory concept in more detail. Further overview information can be found in SAP Note 1840954 – Alerts related to HANA memory consumption. 3. For more information on the SAP HANA alerts see the following documents: ● SAP HANA Administration Guide ○ Monitoring Overall System Status and Resource Usage ○ Monitoring System Performance ● Alerts 1 and 43: See.1 Memory Problems This section discusses the analysis steps that are required to identify and resolve memory related issues in the SAP HANA database. 3. allocation limit. All rights reserved. used memory and resident memory open Administration Overview Open Landscape Services memory for each service. used memory and resident memory.1. 21 . for high level information about physical memory. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company.1 Memory Information in SAP HANA Studio There are a number of sources of information in SAP HANA studio that can assist you in understanding memory utilization. allocation limit. To get high level information about physical memory. allocation limit and used Open Administration Performance Load for high level history information about physical memory. select Configuration and Monitoring Open Memory Overview to drill-down into memory utilization (Physical Memory / SAP HANA Used Memory / table and database management memory). Open Landscape Services and right click a service and choose Memory Allocation Statistics to drill-down into used memory grouped by different main components like “Statement Execution & Intermediate Results” or “Column Store Tables” which are further divided by sub components: When you choose a main component in the upper part of the screen its sub components are shown in the lower part. All rights reserved.From the Systems open the context menu of a system. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . 22 PUBLIC © 2015 SAP SE or an SAP affiliate company. including statement execution details. see Performance Trace. The trace output is written to a trace file perftrace.2 Memory Information from Logs and Traces In case of critical memory issues you can often find more detailed information in logs and trace files. To collect this information. and Job Selection (SM37). launch the Database Trace wizard and select the Show all components checkbox. If specific SAP HANA system components need deeper investigation. Enter the search string. ● In the SAP HANA system alert trace files on the Diagnosis tab. for example. All rights reserved. SAP Customer Support can ask you to raise the corresponding trace levels to INFO or DEBUG.ini file and change the System Trace Level to the appropriate values.1. the following information is valuable and should be added to the ticket: ● Diagnosis Information (full system info dump). 23 . which must be sent to SAP Customer Support. can create many megabytes of trace information and require an increase of the values maxfiles and maxfilesize in the [trace] section of the global. Search for the strings “memory”. ABAP Runtime Error (ST22).Choose Show Graph to show historic information for component memory usage: 3. Send the indexserver trace file(s) to SAP Customer Support. If help from SAP Customer Support is needed to perform an in-depth analysis of a memory-intensive SQL statement. good starting points for analysis are System Log (SM21). try to identify memory-related errors. ● Check if an out-of memory (OOM) trace file was created. select the found component in the indexserver. To do so. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. ● Performance Trace provides detail information on the system behavior. or “OOM” (case-insensitive). ● Investigate error messages seen on the application side that occurred at times of high memory usage.tpt. join_eval = DEBUG. “allocat”. Some trace components.ini file. see Diagnosis Information. If the application is an SAP NetWeaver system. To enable this trace. The aggregation comprises both Column Store and Row Store tables. The most important memory related analysis commands are in the following files: 24 PUBLIC © 2015 SAP SE or an SAP affiliate company. To do so.py to SAP Customer Support. Based on your needs you can configure restrictions and parameters in the section marked with /* Modification section */.3 Memory Information from SQL Commands There are a number of ways to analyze memory usage based on pre-defined and modifiable SQL queries. The System Information tab of SAP HANA studio provides a set of tabular views to display the memory consumption of loaded tables based on pre-defined SQL queries: ● The view Schema Size of Loaded Tables displays the aggregated memory consumption of loaded tables in MB for different database schemas. edit the parameter trace in the [pythontrace] section of the executor. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . SAP Note 1969700 – SQL Statement Collection for SAP HANA contains several commands that are useful to analyze memory related issues.Internal details about SQL statement execution can be collected by enabling the Executor Trace.ini file and change its value to on. Order by the schema size column and find the largest consumers. Related Information Diagnosis Information [page 168] Performance Trace [page 166] 3. Upload the trace file extrace. on the Configuration tab. All rights reserved. The Executor Trace provides the highest detail level and should only be activated for the short time of query execution.1. ● The view Used Memory by Tables shows two values: the total memory consumption of all Column Store tables in MB and the total memory consumption of all Row Store tables in MB. ● “HANA_Memory_Overview”: Overview of current memory information Figure 1: Example: Overview of Current Memory Information SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. 25 . SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . All rights reserved.● “HANA_Memory_TopConsumers”: Current top memory consuming areas Figure 2: Example: Current Top Memory Consuming Areas ● “HANA_Memory_TopConsumers_History”: Historic top memory consuming areas ● “HANA_Tables_LargestTables”: Overview of current memory allocation by tables Figure 3: Current Memory Allocation by Table 26 PUBLIC © 2015 SAP SE or an SAP affiliate company. 4 Memory Information from Other Tools There are a number of tools available to analyze high memory consumption and out of memory situations. The tool hdbcons provides expert functionality to analyze memory issues.Related Information SAP Note 1969700 3. Based on the results from the analysis approaches you are now able to answer the following questions: ● Is it a permanent or a sporadic problem? ● Is the memory consumption steadily growing over time? ● Are there areas with critical memory consumption in heap.1.1. Out-of-memory (OOM) dumps can be analyzed as described in SAP KBA 1984422 – Analysis of SAP HANA Out-of-memory (OOM) Dumps. Typical memory consumers are: ● Operating system (for example.Required information to investigate high memory consumption. mapping structures) SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company.1 Significant External Memory Consumption If the database resident memory of all SAP HANA databases on the same host is significantly smaller than the total resident memory you have to check which processes outside of the SAP HANA database(s) are responsible for the additional memory requirements. All rights reserved. caches.5 Root Causes of Memory Problems Once you have completed your initial analysis you have the information required to start the next phase of your analysis. row store or column store? ● Is there a big difference between used memory and allocated memory? In the following you can find typical root causes and possible solutions for the different scenarios.1. 27 . 3. Related Information SAP Note 1786918 . For more information see SAP Note 1786918 .Required information to investigate high memory consumption SAP KBA 1984422 – Analysis of SAP HANA Out-of-memory (OOM) Dumps 3.5. 1. backup. ● General recommendations for avoiding and reducing data can be found in the Data Management Guide available at: http://service.2 Space Consumed by Large Tables If particularly large tables consume significant amount of space in the row store or column store you should check if the amount of data can be reduced.5. All rights reserved. see: SAP HANA Administration Guide: Managing Tables. in the case of CONCAT attribute columns that were created in order to support joins. For more information see SAP Note 1986747 – Internal Columns in Column Store .sap. for example. In some situations a cleanup is possible. Related Information SAP Note 198674 28 PUBLIC © 2015 SAP SE or an SAP affiliate company.1.3 Internal Columns in Column Store For several reasons SAP HANA creates internal columns in the Column Store. ● SAP Note 706478 .5. when you are able to identify the reason for the increased memory consumption of the external program you can check if it is possible to optimize its configuration. required for communication.com/ilm > Data Archiving > Media Library > Literature and Brochures For more information on memory management for resident table data. 3.Preventing Basis tables from increasing considerably describes archiving and deletion strategies for typical SAP tables with a technical background for example. Related Information SAP Note 706478 SAP HANA Administration Guide 3. virus scanner) How to identify top memory consumers from non-SAP HANA processes is out of scope of this guide. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .● Third party tools (for example. logging or administration). However. ● If you can’t resolve the problem yourself.5 Large Heap Areas Some heap areas can be larger than necessary without being a memory leak. open a SAP customer message and use the component HAN-DB. SAP Note 1840954 – Alerts related to HANA memory consumption contains an overview of heap allocators with a potentially large memory consumption and possible resolutions. Related Information SAP Note 1840954 3.5.1. If you have identified a suspicious area proceed as follows: ● Check for SAP Notes that describe the memory leak and provide a solution. All rights reserved.4 Memory Leaks A memory leak is a memory area (typically a heap allocator) that grows over time without any apparent reason.5. 3.6 Expensive SQL Statements SQL statements processing a high amount of data or using inefficient processing strategies can be responsible for increased memory requirements. 29 .5.1. Related Information SQL Statement Analysis [page 140] Set a Statement Memory Limit [page 32] SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. See SQL Statement Analysis for information on how to analyze expensive SQL statements during times of peak memory requirements.1.3. ● Check if the problem is reproducible with a recent SAP HANA revision. 8 Used Space Much Smaller than Allocated Space In order to optimize performance by minimizing the memory management overhead or due to fragmentation.3.5. used memory is not released any more. SAP HANA may allocate additional memory rather than reusing free space within the already allocated memory.7 Transactional Problems High memory consumption can be caused by problems with transactions. ● Hanging threads. This can lead to undesired effects that the SAP HANA memory footprint increases without apparent need. All rights reserved. In particular. see Transactional Problems. In order to limit the amount of allocated space you can set the parameter global_allocation_limit to a value not larger than the maximum memory that should be allocated See Set the global_allocation_limit Parameter in the SAP HANA Administration Guide. ● Blocked transactions. For more information. ● Long-running or unclosed cursors. high memory consumption is caused by wait situations. the number of table versions can grow up to more than 8. Related Information SAP HANA Administration Guide 3. In some cases. which can have different reasons.000 which is considered the amount where an action is required. In order to minimize fragmentation of row store tables you can proceed as follows: 30 PUBLIC © 2015 SAP SE or an SAP affiliate company. The SAP HANA license checks against allocated space.5. so from a licensing perspective it is important to keep the allocated space below the license limit.1.1.5.9 Fragmentation Fragmentation effects are responsible for inefficiently used memory.1. They can occur in different areas. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .000. Related Information Transactional Problems [page 30] 3. As one of the negative impacts. Related Information SAP Note 1969700 Delta Merge [page 54] SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. If the delta store size is larger than expected. Related Information SAP Note 1813245 3. To check for memory LOBs and switch to hybrid LOBs see SAP Note 1994962 – Activation of Hybrid LOBs in SAP HANA.5. Related Information SAP Note 1994962 3. All rights reserved.1.1. SAP Note 1813245 .● If the fragmentation of row store tables in the shared memory segments of indexserver processes reaches 30% and the allocated memory size is greater than 10GB.SAP HANA DB: Row Store reorganization describes how to determine fragmentation and perform a table redistribution. You can identify the current size of the delta store by running the SQL command: “HANA_Tables_ColumnStore_Overview” (SAP Note 1969700 – SQL Statement Collection for SAP HANA). 31 . a table redistribution operation is needed. proceed as described in the section Delta Merge.10 Large Memory LOBs LOB (Large Object) columns can be responsible for significant memory allocation in the row store and column store if they are defined as memory LOBs.11 Large Delta Store The delta store can allocate a significant portion of the column store memory.5. 5. Statement executions that require more memory will be aborted when they reach the limit.MEMORY_SIZE. Set a statement memory limit (integer values only).13 Set a Statement Memory Limit The statement memory limit allows you to set a limit both per statement and per SAP HANA host. 32 PUBLIC © 2015 SAP SE or an SAP affiliate company.ini file.1. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .ini file. Enable statement memory tracking. In this case you should perform a sizing verification and make sure that sufficient memory is installed on the SAP HANA hosts.REUSED_MEMORY_SIZE is not used as of SPS 09. 3. By default. Prerequisites You have the system privilege INIFILE ADMIN.5. Procedure 1. In the global. expand the resource_tracking section and set the following parameters to on: ○ enable_tracking = on ○ memory_tracking = on You can view the (peak) memory consumption of a statement in M_EXPENSIVE_STATEMENTS.12 Undersized SAP HANA Memory If a detailed analysis of the SAP HANA memory consumption didn’t reveal any root cause of increased memory requirements it is possible that the available memory is not sufficient for the current utilization of the SAP HANA database.1. expand the memorymanager section and set the parameter statement_memory_limit . Context You can protect an SAP HANA system from excessive memory usage due to uncontrolled queries by limiting the amount of memory used by single statement executions per host. All rights reserved. there is no limit set on statement memory. Note M_EXPENSIVE_STATEMENTS. In the global. 2.3. during the night). 33 . (Optional) Set a user specific statement limit. You can set a threshold for statement memory limit. No statements have to be canceled if total memory is below the threshold. To exclude certain users from the statement memory limit (for example. the user specific limit takes precedence. The statement is aborted. All rights reserved. ○ The parameter is shown in USER_PARAMETERS (like all other user parameters) Note To reset a user specific statement limit use the SQL statement: ALTER USER <user_name> CLEAR PARAMETER STATEMENT MEMORY LIMIT 4. expand the memorymanager section and set the parameter statement_memory_limit_threshold The statement memory limit will only take effect if the total memory used in the system (as per the global_allocation_limit parameter) is above the set threshold (in %). ○ Setting the statement memory limit to 0 will disable any statement memory limit for the user. Related Information Parameters that Control Memory Consumption [page 34] SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. ○ The user specific statement memory limit is active even if resource_tracking is disabled. which sets the minimum time difference (in seconds) between two dumps of the same kind (and the same process). 3. a dump file is created with "compositelimit_oom" in the name. If a second limit hits in that interval. This allows expensive statements that consume more than the allowed statement memory limit to finish successfully during periods when a system runs under no load (for example. but otherwise the system is not affected. to ensure an administrator is not prevented from doing a backup) use the SQL statement: ALTER USER <user_name> SET PARAMETER STATEMENT MEMORY LIMIT = <gb> ○ If both a global and a user statement memory limit are set.ini file using the oom_dump_time_delta parameter. The interval can be configured in the memorymanager section of the global. In the global. ○ If the user specific statement memory limit is removed the global limit takes effect for the user.ini file. no dump file is written. Statements that exceed the limit you have set on a host are stopped by running out of memory.When the statement memory limit is reached. This means the statement_memory_limit parameter is taken into account only when total memory usage reaches the threshold. By default only one dump file is written every 24 hours. regardless of whether it is higher or lower than the global statement memory limit. ini 34 memorymanager allocationlimit PUBLIC © 2015 SAP SE or an SAP affiliate company.13. The default allocation limit is calculated in the same way as the de­ fault global allocation limit. Parameters configured at the database level apply to the specified database only. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . global. The global allocation limit limits the amount of memory that can be used by the system as a whole.1 Parameters that Control Memory Consumption The memorymanager section of the global. All rights reserved. A missing entry or a value of 0 results in the system using the default settings.1. The allocation limit limits the amount of memory that can be used by individual processes.3. Table 1: INI file Section Parameter Default Description global. Note In a system that supports multitenant database containers. in the case of small phys­ ical memory. the global allocation limit con­ figured at the sys­ tem layer of the global. Or.ini file is always effective regardless of any value configured at the database layer.5. Parameters configured at the system level apply to the complete system and all databases. Collectively.ini file contains parameters that allow you to control the memory consumption of SAP HANA. Note In a system that supports multiten­ ant database con­ tainers. The global allocation limit is calculated by default as follows: 90% of the first 64 GB of available physical memory on the host plus 97% of each further GB. physical memory minus 1 GB. The value is the maximum alloca­ tion limit in MB. all proc­ esses cannot consume more memory than the global allocation limit.ini at both the system level and the database level. you can configure the global.ini memorymanager global_allocati on_limit A missing entry or a value of 0 results in the system using the default settings. The value is the maximum allo­ cation limit in MB. the software will parallelize queries as much as possible in order to provide optimal performance. Note that a proper CPU utilization is actually desired behavior for SAP HANA.ini memorymanager statement_memor y_limit 0 (no limit) When reaching the statement memory limit. The unit of measure is GB. 3. a dump file is created with "compo­ sitelimit_oom" in the name. It also does not automatically indicate a performance issue. The following alerts may indicate CPU resource problems: ● Host CPU Usage (Alert 5) ● Most recent savepoint operation (Alert 28) ● Savepoint duration (Alert 54) SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. but other­ wise the system is not affected. The statement is aborted. SAP HANA is optimized to consume all memory and CPU available. A constantly high CPU consumption will lead to a considerably slower system as no more requests can be processed.2 CPU Related Root Causes and Solutions This section covers the troubleshooting of high CPU consumption on the system.2. All rights reserved.INI file Section Parameter Default Description global. From an end user perspective. is unresponsive or can even seem to hang. 3.1 Indicators of CPU Related Issues CPU related issues are indicated by alerts issued or in views in the SAP HANA Studio. More concretely. global. so this should be nothing to worry about unless the CPU becomes the bottleneck. 35 .ini memorymanager statement_memor y_limit_thresho ld 0% (statement_mem­ ory_limit is always re­ spected) The statement mem­ ory limit is applied if the current SAP HANA memory consumption exceeds the statement memory limit thresh­ old as a percentage of the global allocation limit. the application behaves slowly. So if the CPU usage is near 100% for a query execution it does not always mean there is an issue. All rights reserved. non-SAP HANA processes on the host. the CPU consumption of other processes running on the same host is not 36 PUBLIC © 2015 SAP SE or an SAP affiliate company.2 Analysis of CPU Related Issues The following section describes how to analyze high CPU consumption using tools in the SAP HANA studio tools. When analyzing high CPU consumption.2. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .You notice very high CPU consumption on your SAP HANA database from one of the following: ● Alert 5 (Host CPU usage) is raised for current or past CPU usage ● The displayed CPU usage on the overview screen ● The Load graph is currently showing high CPU consumption or shows high consumption in the past 3. While the CPU consumption of SAP HANA will be addressed here in detail. you need to distinguish between the CPU resources consumed by HANA itself and by other. All rights reserved. Note that setting a maximum duration or memory limit for profiling is good practice and should be used if appropriate values can be estimated. the issue is most likely related to a non-SAP HANA process. However. If SAP HANA CPU usage is low while total CPU usage is high. 37 .. It contains a section that displays SAP HANA CPU usage versus total CPU usage. A good starting point for the analysis is the Overview tab in the SAP HANA studio. To find out what is happening in more detail. A high CPU time of related threads is an indicator that an operation is causing the increased CPU consumption. add further restrictive criteria such as database user or application user to narrow down the amount of information traced. More information about this tool can be found in Kernel Profiler. and keeps track of the maximum CPU usage that occurred since the last restart of SAP HANA. If possible. Another tool to analyze high CPU consumption is the Kernel Profiler. taking a Full System Info Dump requires resources itself and may therefore worsen the situation. Figure 4: Thread Monitor Showing CPU Time In order to identify expensive statements causing high resource consumption. turn on the Expensive Statement trace and specify a reasonable runtime (see Expensive Statements Trace). The Thread Monitor shows the CPU time of each thread running in SAP HANA in microseconds.ini file set the following parameters to on. perform the following steps: tab (see Thread Monitoring). Related Information SAP Note 1730928 Thread Monitoring [page 134] Expensive Statements Trace [page 144] SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. Note that the CPU time for each statement is shown in the column CPU_TIME if resource_tracking is activated. which includes all processes on the host. Such situations are often caused by additional programs running concurrently on the SAP HANA appliance such as anti-virus and backup software. To capture the current state of the system for later analysis you can use Full System Info Dump. open Performance Threads order to prepare it for CPU time analysis. For more information see SAP Note 1730928. open Diagnosis Files Diagnosis Information and choose Collect (SQL Procedure) if the system is up and accepting SQL commands or Collect (Python Script) if it is not. ○ cpu_time_measurement_mode ○ enable_tracking ● Display the CPU Time column by using the Configure Viewer button on the outer right side of the Threads tab. In ● To switch on resource tracking open the Configuration tab and in the resource_tracking section of the global. To get a Full System Info Dump.covered. To do so. two options are available: ● On the client side. if available. Client IP. In order to resolve the situation contact him and clarify the actions he is currently performing: Figure 5: Identify Application User As soon as this is clarified and you agree on resolving the situation. Actually stopping the operation causing the high CPU consumption can be done via the Thread Monitor (see Thread Monitoring). right-click on the thread in the Threads tab and choose Cancel Operations. Client PID and Application User it is possible to identify the user that triggered the operation. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . which may complicate identifying the root cause Issue resolution should aim to bring the system back to a sane state by stopping the operation that causes the high CPU consumption. end the process calling the affected threads ● Cancel the operation that is related to the affected threads. 38 PUBLIC © 2015 SAP SE or an SAP affiliate company. after resolving the situation it might not be possible to find out the actual root cause. please open a ticket to SAP HANA Development Support and attach the Full System Info Dump. Related Information Analysis of CPU Related Issues [page 36] Thread Monitoring [page 134] 3. However.2.3 Resolving CPU Related Issues The first priority in resolving CPU related issues is to return the system to a normal operating state. For further analysis on the root cause. Therefore please consider recording the state of the system under high load for later analysis by collecting a Full System Info Dump (see Analysis of CPU Related Issues).2.Kernel Profiler [page 167] 3.4 Retrospective Analysis of CPU Related Issues There are a number of options available to analyze what the root cause of an issue was after it has been resolved. All rights reserved. With the columns Client Host. turn on the Expensive Statements trace during that time to record all involved statements (see Expensive Statements Trace ). too much parallelism of single statements may lead to suboptimal performance. search through the trace files of the responsible process. 39 . check the following statistics server table which includes historical host resource information up to 30 days: HOST_RESOURCE_UTILIZATION_STATISTICS (_SYS_STATISTICS schema) With this information.2. Be careful to choose the correct host when SAP HANA runs on a scale-out landscape. Caution Altering the settings described here can only be seen as a last resort when traditional tuning techniques like remodeling and repartitioning as well as query tuning are already fully exploited. On systems with highly concurrent workload. If the phenomenon is recurrent due to a scheduled batch jobs or data loading processes. Furthermore. check for concurrently running background jobs like backups and Delta Merge that may cause a resource shortage when run in parallel. All rights reserved. Playing with the parallelism settings requires a deep understanding of the actual workload and has severe impacts on the overall system behavior. Using the alert time or the Load graph. Two subsystems control the parallelism of the statement execution. SQLExecutors and JobExecutors. Related Information Expensive Statements Trace [page 144] M_BACKUP_CATALOG M_DELTA_MERGE_STATISTICS HOST_DELTA_MERGE_STATISTICS 3.A retrospective analysis of high CPU consumption should start by checking the Load graph and the Alerts tab. For simple (OLTP-like) SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. determine the time frame of the high CPU consumption. For each statement execution. ● SqlExecutor These thread types handle incoming client requests and execute simple statements. Historical information about such background jobs can be obtained from the system views: ● M_BACKUP_CATALOG ● M_DELTA_MERGE_STATISTICS ● A longer history can be found in the statistics server table HOST_DELTA_MERGE_STATISTICS (_SYS_STATISTICS schema). that control the parallelism of statement execution.5 Controlling Parallelism of SQL Statement Execution There are two subsystems. so be sure you know what you are doing. The information contained in the trace files will give indications on the threads or queries that were running during the affected time frame. If you are not able to determine the time frame because the issue happened too long ago. The parameters below allow you to adjust the CPU contention in the system. an SqlExecutor thread from a thread pool processes the statement. ini/ indexserver. if available. memory garbage collection. 40 PUBLIC © 2015 SAP SE or an SAP affiliate company. which were delegated by the SqlExecutor. JobExecutor settings do not solely affect OLAP workload. Caution Lowering the value of this parameter can have a drastic effect on the parallel processing of the servers and reduce the performance of the overall system.ini sql max_sql_executo rs 0 (disabled) Sets the maximum number of threads that can be used Table 3: JobExecutor Parameters Ini file Section Parameter Default Description global. but also other SAP HANA subsystems (for example. Table 2: SqlExecutor Parameters INI file Section Parameter Default Description indexserver. ● JobExecutor The JobExecutor is a job dispatching subsystem. savepoint writes). For both SqlExecutor and JobExecutor. The JobExecutor executes also operations like table updates and backups. indexserver.ini sql sql_executors 0 (number of available threads) Sets the minimum number of threads that can be used The parameter sql_executors de­ fines the number of parallel (SQL) queries that can be processed by the system. The de­ fault value is the num­ ber of hyperthreads in a system (0). JobExecutor settings are soft limits.statements against column store as well as for most statements against row store. This can be used to keep some room for OLTP workload on a system where OLAP workload would otherwise consume all the CPU resources. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . reducing the value of this parameter can help to avoid memory footprint. As each thread allocates a par­ ticular amount of main memory for the stack. meaning the JobExecutor can “loan” threads.ini execution max_concurrency Number of available threads Sets the maximum number of threads that can be used. Almost all remaining parallel tasks are dispatched to the JobExecutor and its associated JobWorker threads. and then fall back to the maximum number of threads when done. All rights reserved. a separate limit can be set for the maximum amount of threads. this will be the only type of thread involved. Sysfs is a virtual file system and is used to access information about hardware. The commands listed below can be used to discover this setting. a reasonable default value for the max_concurrency parameter is the number of cores divided by the number of tenant databases. This is achieved by assigning an affinity to logical cores. 41 . you can bind SAP HANA processes to logical cores of the hardware. Context Tip Instead of binding processes to CPUs as described here. then only those will be available to SAP HANA processes. see Managing Resources in Multiple-Container Systems. Do not specify a value of 0. the affinity for the container also applies to the complete SAP HANA system. Prerequisites ● You have access to the operating system of the SAP HANA instance and are able to read the directories and files mentioned below. Note that the logical cores used in the configuration do not have a fixed association to the physical cores on the hardware.ini configuration file. All rights reserved. So the quality of the reported information depends on the configuration of the VM guest. For Linux containers (like Docker) /proc/cpuinfo reports more cores than are actually available. If the physical hardware on a host needs to be shared with other processes. For more information. This is more convenient to do and does not require the system to be offline. ● You have the privilege INIFILE ADMIN.Tip In a system that supports multitenant database containers. General information on sysfs is available in the Linux kernel documentation. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company.2. As Linux containers use the same mechanism to limit CPU resources per container as SAP does for the affinity.6 Bind Processes to CPUs For better workload management in a system that supports multitenant database containers. For Xen and VmWare. for example to partition the CPU resources of the system by tenant database. 3. you can achieve similar performance gains by changing the parameter [execution] max_concurrency in the indexserver. it may be useful to assign a set of cores to an SAP HANA process. the users in the VM guest system see what is configured in the VM host. This means that if the Linux container is configured to use logical cores 0-4. The information on the cores provided by sysfs is used to restrict the cores available to SAP HANA in the daemon. Therefore SAP cannot give any performance guarantees in this case.ini file. use the following affinity setting: Example ALTER SYSTEM ALTER CONFIGURATION ('daemon. Restrict CPU usage of SAP HANA processes to certain CPUs. Procedure 1. they will be assigned to distinct logical cores. where c1 and so on are logical cores. In the daemon. Get sibling cores.ini file the relevant sections are: nameserver.ini'. With sched_setaffinity (similar to numactl) the Linux OS will make sure that whatever number of threads are executed by a process. 'affinity') = '0. Here the parameter affinity can be set to c1. 'SYSTEM') SET ('nameserver'. Results After following these steps you have the information required to assign an affinity to the logical cores. c4-c5. The affinity setting only takes effect after a restart of the affected SAP HANA processes.16' 42 PUBLIC © 2015 SAP SE or an SAP affiliate company. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . Get cores available for scheduling: cat /sys/devices/system/cpu/present Typical output: 0-31 The system exposes 32 logical cores (or cpu). Example affinity settings: ● To restrict the nameserver to two logical cores of the first CPU of socket 0 (derived in step 3). compileserver. that is the cores on the same socket: cat /sys/devices/system/cpu/cpu0/topology/core_siblings_list Typical output: 0-7. c3. 2. All rights reserved. 16 4. preprocessor and xsengine. indexserver.There are a number of steps to analyze the topology of sockets and cores. c2. Get the socket of a specific core: cat /sys/devices/system/cpu/cpu0/topology/physical_package_id Typical output: 0 5. You can use this information to define the desired setting for the affinity of SAP HANA processes. 16-23 3. Get all logical cores assigned to the same physical core (hyperthreading): cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list Typical output: 0. For each of them there is a /sys/devices/system/cpu/ cpu# with # denoting the core-id in the range 0-31. All rights reserved. backup or trace files reaches a critical size.ini'. Low Disk Space This issue is usually reported by alert 2 which is issued whenever one of the disk volumes used for data. Note Even on a system with 32 logical cores and two sockets the assignment of logical cores to physical CPUs and sockets can be different. 'SYSTEM') SET ('compileserver'. 'affinity') = '9-15.25-31'. 'SYSTEM') SET ('indexserver'. log. 'SYSTEM') SET ('indexserver.DB1'. ALTER SYSTEM ALTER CONFIGURATION ('daemon.3 Disk Related Root Causes and Solutions This section discusses issues related to hard disks and lack of free space. 43 . all except 0 and 16) on socket 0 use the following affinity setting (derived in steps 2 and 3): Example ALTER SYSTEM ALTER CONFIGURATION ('daemon.ini'. 'SYSTEM') SET ('indexserver.ini'. 'affinity') = '1-7. It is important to collect the assignment in advance.● To restrict the preprocessor and the compileserver to all remaining cores (that is.DB2'. 3. Use the following tools in the SAP HANA studio to examine the situation and try to free some disk space: ● The Volumes tab ● Open Performance Load .24-31' ● To set the affinity for two tenant databases called DB1 and DB2 respectively in a multitenant database container setup: Example ALTER SYSTEM ALTER CONFIGURATION ('daemon.17-23' ● To restrict the indexserver to all cores on socket 1 use the following affinity setting (derived in steps 1 and 2): Example ALTER SYSTEM ALTER CONFIGURATION ('daemon.ini'.Check SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions Host Disk Used . 'SYSTEM') SET ('preprocessor'. (See also Load Monitoring ) PUBLIC © 2015 SAP SE or an SAP affiliate company. 'affinity') = '1-7.17-23' ALTER SYSTEM ALTER CONFIGURATION ('daemon. 'affinity') = '1-7.ini'. 'affinity') = '8-15.17-23'. there are more possible causes that may prevent SAP HANA from writing to disk. log.1 Check Internal Disk Full Event (Alert 30) Alert 30 is issued when it is not possible to write to one of the disk volumes used for data. Check if the system is running out of inodes (NFS): 44 PUBLIC © 2015 SAP SE or an SAP affiliate company. All of them will lead to this alert. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . Determine the file system type: df -T 2. Related Information Load Monitoring [page 140] I/O Related Root Causes and Solutions [page 45] SAP HANA Administration Guide 3. Check for disk space using file system specific commands Option Description XFS/NFS df GPFS mmfscheckquota 3.● Under the System Information tab.3. backup or trace files. Context Note that besides running out of disk space. the following information is helpful too. The commands have to be executed from the command line on the SAP HANA server: Procedure 1. All rights reserved. Example causes include: ● File system quota is exceeded ● File system runs out of inodes ● File system errors (bugs) Besides doing an analysis via the tools described in Disk Related Root Cause and Solutions. open Size of Tables on Disk More information about the tools can be found in I/O Related Root Causes and Solutions and in the SAP HANA Administration Guide. All rights reserved.df -i 4. such as files and I/O statistics. Although SAP HANA is an in-memory database. 45 . contact SAP Support. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. From an end user perspective. Related Information Disk Related Root Causes and Solutions [page 43] 3.4 I/O Related Root Causes and Solutions This section covers troubleshooting of I/O performance problems. In the Volumes tab in SAP HANA studio you can see the attached volumes and which services use which volumes. is unresponsive or can even seem to hang if there are issues with I/O performance. Check quota Option Description XFS/NFS quota -v GPFS mmfscheckquota Next Steps If it is not possible to track down the root cause of the alert. I/O still plays a critical role for the performance of the system. an application or the system as a whole runs sluggishly. Attached volumes In the lower part of the screen you can see details of the volumes. the new main storage is per­ sisted in the data volume. for example during the transaction commit. Savepoints run asynchronously to SAP HANA update operations. SAP HANA asynchronously writes the redo log with I/O or­ ders of 4 KB to 1 MB size into log segments. Updates on Column Store tables are stored in the delta storage. that is. Write Transactions All changes to persistent data are captured in the redo log. Depending on the type of data the block sizes vary between 4 KB and 16 MB. The SAP HANA data­ base triggers savepoints in 5 minutes intervals by default. Snapshot The SAP HANA database snapshots are used by certain op­ erations like backup and system copy.In certain scenarios data is read from or written to disk. Table 4: Scenario Description Savepoint A savepoint ensures that all changed persistent data since the last savepoint gets written to disk. They are created by triggering a system wide consistent savepoint. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . Right after the delta merge. Examples are included in table. 46 PUBLIC © 2015 SAP SE or an SAP affiliate company. where they are stored read optimized and compressed. which is usually taking some microseconds. Transactions writing a commit into the redo log wait until the buffer con­ taining the commit has been written to the log volume. Detailed information about snapshots can be found in the SAP HANA Administration Guide. Data is automatically saved from memory to the data vol­ ume located on disk. written to disk. All rights reserved. During the delta merge these changes are applied to the main stor­ age. The system keeps the blocks belonging to the snapshot at least until the drop of the snapshot. Database update transactions only wait at the critical phase of the sa­ vepoint. The delta merge does not block parallel read and update transactions. Most of the time this is done asynchronously but at certain points in time synchronous I/O is done. Delta Merge The delta merge itself takes place in memory. Even during asynchronous I/O it may be that important data structures are locked. The log entries have to be replayed. the persistence is read from the storage. the longer it takes until the sys­ tem is available for operations again. then most of the column store tables are already in memory. If table preload is used. For writing a data backup it is essential that on the I/O connection there are no collisions with other transactional operations running against the database. including the block sizes that are read or written. Failover (Host Auto-Failover) On the standby host the services are running in idle mode. The bigger the row store is. Database Recovery The restore of a data backup reads the backup content from the backup device and writes it to the SAP HANA data vol­ umes. Additionally the redo log entries written after the last savepoint have to be read from the log volume and replayed in the data area in memory. Also the redo log can be replayed during a data­ base recovery. All rights reserved. 47 . When this is finished the database is accessible. During takeover the replicated redo logs that were shipped since the last data transport from primary to secondary have to be replayed. that is the serv­ ices are active but cannot accept SQL and thus are not usa­ ble by the application. The I/O write orders of the data recovery have a size of 64 MB. In the below table the I/O operations are listed which are executed by the above mentioned scenarios. Just like in the database restart (see above) the row store tables need to be loaded into memory from persistent storage. Takeover (System Replication) The secondary system is already running. Data Backup For a data backup the current payload of the data volumes is read and copied to the backup storage. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. Row as well as column store tables (the latter on de­ mand) must be loaded into memory. which then has read and write access to the files of the failed active host. Upon failover.Scenario Description Database restart At database startup the services load their persistence in­ cluding catalog and row store tables into memory. Log Backup Log backups store the content of a closed log segment. that is. the data and log volumes of the failed host are automatically assigned to the standby host. that is the log backups are read from the backup device and the log entries get replayed. They are automatically and asynchronously created by reading the payload from the log segments and writing them to the backup area. All rights reserved. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . up to Read block sizes from Read block sizes from 64 MB (clustered Row Store backup file headers à copy backup file headers à copy super blocks) blocks into “[data] backup blocks into “[data] backup buffer” of size 512 MB buffer” of size 128 MB Related Information SAP HANA Administration Guide 48 PUBLIC © 2015 SAP SE or an SAP affiliate company. WRITE Snapshot. up to in up to 64 MB blocks from 64 MB (clustered Row Store “[data] backup buffer” super blocks) are asynchro­ nously copied to “[data] backup buffer” of 512 MB Log Backup READ WRITE asynchronously copied to in up to 64 MB blocks from “[data] backup buffer” of 128 “[data] backup buffer” MB Database Recovery WRITE READ READ 4 KB – 16 MB blocks.Table 5: I/O pattern Data Volume Savepoint. 4 KB – 16 MB blocks. up to 64 MB (clustered Row Store super blocks) Write transactions WRITE OLTP – mostly 4 KB log write I/O performance is relevant OLAP – writes with larger I/O order sizes Table load: READ DB Restart. READ 64 MB (clustered Row Store super blocks) Takeover Data Backup READ WRITE 4 KB – 16 MB blocks. 4 KB – 16 MB asynchronous Delta merge Log Volume (redo log) Backup Medium bulk writes. up to Failover. max_trigger_async_write_time as "Max Trigger Async Write Time in Microsecond".volume_id. The latency is important for LOG devices.total_write_size / 1024 / 1024.1 Analyzing I/O Throughput and Latency When analyzing I/O the focus is on throughput and latency. port type. round(s.3. round(s. With this statement you get the log write wait time (for data of type LOG) with various buffer sizes written by the indexserver. Explanation of Ratio: I/O calls are executed asynchronously.total_write_time end. v. trigger_write_ratio as "Write Ratio" from "PUBLIC". s.total_write_size / case s. v.volume_id and type not in ( 'TRACE' ) and v. To analyze the latency. a ratio close to 1 means that the thread waits until I/O request is completed because the asynchronous call is blocked (time for triggering I/O time for I/O completion). alter system reset monitoring view M_VOLUME_IO_TOTAL_STATISTICS_RESET.total_read_time when 0 then -1 else s. 3) as "Read Througput in MB". More information can be found in SAP Note 1930979. if they are drifting towards 1. round(max_io_buffer_size / 1024. max_write_time as "Max Write Time in Microsecond" from "PUBLIC".total_read_size / case s. 3) as "Writes in MB". PUBLIC. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company.total_write_time / 1000 / 1000. It is possible to reset the view and analyze the I/O throughput for a certain time frame by running the reset command below and query again after the desired time frame.total_read_time end. You can analyze the I/O throughput with this SQL statement: select v. Monitoring views and SQL statements help with your analysis. A ratio close to 0 says that the thread does not wait at all. avg_trigger_async_write_time as "Avg Trigger Async Write Time in Microsecond". service_name. The Ratio fields indicate bad performance.total_read_time / 1000 / 1000. 3) as "Read Time in Sec". 3) "Maximum buffer size in KB". 3) as "Write Througput in MB". 3) as "Write Time in Sec" . use: select host.port.total_write_time when 0 then -1 else s."M_VOLUME_IO_DETAILED_STATISTICS_RESET" where type = 'LOG' and volume_id in (select volume_id from m_volumes where service_name = 'indexserver') and (write_count <> 0 or avg_trigger_async_write_time <> 0). that is the thread does not wait for the order to return.volume_id = v. They should tend towards 0.4."M_VOLUME_IO_TOTAL_STATISTICS_RESET" s. All measures are the periods of time between enqueueing and finishing a request.host.type. 49 . trigger_read_ratio as "Read Ratio". round(s. round(s.service_name. round(s. s. All rights reserved. The system view M_VOLUME_IO_TOTAL_STATISTICS_RESET is used to get the size of reads and writes and the throughput in MB for the indexserver since the last reset of the counters. avg_write_time as "Avg Write Time in Microsecond".volume_id in (select volume_id from m_volumes where service_name = 'indexserver') order by type.M_VOLUMES v where s. 3) as "Reads in MB". trigger_async_write_count. round(s.total_read_size / 1024 / 1024. write_count. mm. If the state of SAP HANA has to be recovered. round(critical_phase_duration / 1000000) as "Critical Phase Duration in Seconds". sum(case when (critical_phase_duration > 1000000 and critical_phase_duration <=2000000) then 1 else 0 end) as "<= 2 s".4. Savepoints are used to implement backup and disaster recovery in SAP HANA. round(duration / 1000000) as "Duration in Seconds". MB/sec". round(total_size / duration) as "Appro. volume_id. You can analyze the savepoint performance with this SQL statement: select start_time. the database log from the last savepoint will be replayed. The critical phase duration should not be longer than a second. round (flushed_rowstore_size / 1024 / 1024) as "Row Store Part MB" from m_savepoints where volume_id in ( select volume_id from m_volumes where service_name = 'indexserver') . In the example below the times are significantly higher due to I/O problems. All rights reserved.dd') as "time". sum(case when (critical_phase_duration <= 1000000) then 1 else 0 end) as "<= 1 s". This statement shows how long the last and the current savepoint writes took/are taking.2 Savepoint Performance To perform a savepoint write operation. poor I/O performance can extend it to a length that causes a considerable performance impact. must be observed carefully. Figure 6: Savepoints The following SQL shows a histogram on the critical phase duration: select to_char(SERVER_TIMESTAMP. in which savepoints need to take a global database lock. This period is called the “critical phase” of a savepoint. SAP HANA needs to take a global database lock. Especially the critical phase duration. While SAP HANA was designed to keep this time period as short as possible. round(total_size / 1024 / 1024) as "Size in MB".Related Information SAP Note 1930979 M_VOLUME_IO_TOTAL_STATISTICS_RESET 3. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .'yyyy. 50 PUBLIC © 2015 SAP SE or an SAP affiliate company. dd') order by to_char(SERVER_TIMESTAMP. sum(case when (critical_phase_duration > 4000000 and critical_phase_duration <=5000000) then 1 else 0 end) as "<= 5 s". sum(case when (critical_phase_duration > 40000000 and critical_phase_duration <=60000000) then 1 else 0 end) as "<= 60 s". All rights reserved."HOST_SAVEPOINTS" where volume_id in (select volume_id from m_volumes where service_name = 'indexserver') and weekday (server_timestamp) not in (5. 6) group by to_char(SERVER_TIMESTAMP.mm.'yyyy. sum(case when (critical_phase_duration > 5000000 and critical_phase_duration <=10000000) then 1 else 0 end) as "<= 10 s".'yyyy. count(critical_phase_duration) as "ALL" from "_SYS_STATISTICS". sum(case when (critical_phase_duration > 60000000 ) then 1 else 0 end) as "> 60 s".mm. sum(case when (critical_phase_duration > 10000000 and critical_phase_duration <=20000000) then 1 else 0 end) as "<= 20 s".dd') desc. sum(case when (critical_phase_duration > 3000000 and critical_phase_duration <=4000000) then 1 else 0 end) as "<= 4 s".sum(case when (critical_phase_duration > 2000000 and critical_phase_duration <=3000000) then 1 else 0 end) as "<= 3 s". Figure 7: Savepoint Histogram SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. sum(case when (critical_phase_duration > 20000000 and critical_phase_duration <=40000000) then 1 else 0 end) as "<= 40 s". 51 . file_name.sys_start_time order by mbc. mbc. m_backup_catalog mbc where mbc.host.file_name = b. round(sum(backup_size) / SECONDS_BETWEEN (mbc. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . 2) speed_mbs from m_backup_catalog_files mbcf .backup_id = mbc. layer_name.tenant_name. section. round(sum(backup_size) / 1024 / 1024 / 1024. 52 PUBLIC © 2015 SAP SE or an SAP affiliate company. b.ini – transaction – lock_wait_timeout Transactional Problems 32 Log mode legacy Global.sys_end_time) seconds.key.sys_end_time) / 1024 / 1024.backup_id. mbc.file_name and a. b.value <> b.ini – persistence – log_mode Issues with Configuration Pa­ rameter log_mode (Alert 32 and 33) To check for parameters that are not according to the default settings.sys_start_time 3. select a. value as currentvalue from sys.currentvalue Note Default values of parameters may change when updating the SAP HANA database with a new revision. b.layer_name. The following table lists the monitored parameters and related alerts.The performance of the backup can be analyzed with this statement: select mbc. SECONDS_BETWEEN (mbc.2) size_gb.section = b. Table 6: Alert ID Alert Name Parameter Further Information 10 Delta merge (mergedog) configuration Indexserver.backup_id. mbc. Custom values on the system level and on the host level will not be affected by such updates. b.sys_start_time.backup_id group by mbc.section and a.sys_end_time. the following SQL statement can be used. host. b.currentvalue from sys.key and a.m_inifile_contents a join ( select file_name.ini – mergedog active Delta Merge 16 Lock wait timeout configura­ tion Indexserver.key = b. tenant_name. a.section.m_inifile_contents b where layer_name <> 'DEFAULT' ) b on a. b.state_name = 'successful' and mbcf.sys_start_time.ini – persistence – log_mode Issues with Configuration Pa­ rameter log_mode (Alert 32 and 33) 33 Log mode overwrite Global. All rights reserved. key.value as defaultvalue.5 Configuration Parameter Issues The SAP HANA database creates alerts if it detects an incorrect setting for any of the most critical configuration parameters. mbc.entry_type_name = 'complete data backup' and mbc. Context To ensure point-in-time recovery of the database the log_mode parameter must be set to ‘normal’ and a data backup is required. Most of the parameters can be changed online and do not require any further action. unless you have received a specific recommendation from SAP to change the parameter to another value. All rights reserved. Change the value of the parameter log_mode in SAP HANA studio to normal SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. 53 . In most cases the suggested correct value will be the default value. The following steps are recommended when facing this alert: Procedure 1.1 Issues with Configuration Parameter log_mode (Alert 32 and 33) Alerts 32 and 33 are raised whenever the write mode to the database log is not set correctly for use in production. you can fix the issue by changing the parameter from the Configuration tab of SAP HANA studio. You can filter on the parameter name to find it.Correcting Parameter Settings Usually alerts on incorrect parameter settings include information about correct setting of the parameter. Note Make sure that you change the parameter in the correct ini-file and section.5. So. since the parameter name itself may be not unique. Related Information Delta Merge [page 54] Transactional Problems [page 73] Issues with Configuration Parameter log_mode (Alert 32 and 33) [page 53] SAP Note 1891582 3. Exceptions for common parameters are documented in SAP Note 1891582. From an end user perspective. This synchronization operation is called delta merge. because read times from delta storage are considerably slower than reads from main storage. performance issues may occur if the amount of data in the delta storage is large. Write operations on the compressed data are costly as they require reorganizing the storage structure and recalculating the compression. since the data to be merged is hold twice in memory during the merge operation.2. all changes are at first written into a separate data structure called the delta storage and at a later point in time synchronized with the main storage. Backup the database configuration For information on how to perform a backup of database configuration files see SAP Note 1651055. In addition the merge operation on a large data volume may cause bottleneck situations. The following alerts indicate an issue with delta merges: ● Delta merge (mergedog) configuration (Alert 10) ● Size of delta storage of Column Store tables (Alert 29) 54 PUBLIC © 2015 SAP SE or an SAP affiliate company.6 Delta Merge This section covers troubleshooting of delta merge problems. The Column Store uses efficient compression algorithms to keep relevant application data in memory. Restart the database 5. Instead. 6. Therefore write operations in Column Store do not directly modify the compressed data structure in the so called main storage. Schedule an initial data backup 3. Schedule a regular data backup Related Information SAP Note 1651055 3. All rights reserved. Test successful completion of the backup 4. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. In a production system this alert needs to be handled with very high priority in order to avoid performance issues.3. Alert 10 Delta Merge (mergedog) Configuration is raised. Check the value of active in the mergedog section of the indexserver.6. Context Whenever issues with delta merge are suspected. this alert should be checked first.ini. Figure 8: Delta Merge Set To Inactive Procedure 1.1 Inactive Delta Merge In case the delta merge is set to inactive. Check the current parameter value in the Configuration tab of SAP HANA studio and filter for the parameter mergedog. 55 . To correct the value. double-click on active and choose Restore Default. Figure 9: Check Mergedog Active 2. An inactive delta merge has a severe performance impact on database operations. This will delete all custom values on system and host level and restore the default value system-wide. All rights reserved. 6. SAP Solution Manager) that create a snapshot of configuration settings at regular intervals. read operations on the table will slow down. the Alert Size of delta storage of Column Store tables (Alert 29) can be raised. When the delta storage of a table gets too large. The thresholds can be customized in the SAP HANA studio to take into account the configured size 56 PUBLIC © 2015 SAP SE or an SAP affiliate company.6. refer to the SAP HANA Security Guide. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . All rights reserved.Figure 10: Restore Defaults Note Depending on the check frequency (default frequency: 15 minutes) the alert will stay in the Alert inbox until the new value is recognized the next time the check is run.3 Indicator for Large Delta Storage of Column Store Tables If the delta storage of a table gets too large. For details about configuring security auditing and for analyzing audit logs. Related Information SAP HANA Security Guide 3. This usually results in degraded performance of queries reading from the affected table.2 Retrospective Analysis of Inactive Delta Merge Retrospective analysis of the root cause of the parameter change that led to the configuration alert requires the activation of an audit policy in SAP HANA that tracks configuration changes Other sources of information are external tools (for example. 3. Alert 29 is raised when the amount of memory consumed by the delta storage exceeds the configured thresholds. Check if the alert is raised for a small number of tables or if it is raised for multiple tables. Procedure 1. Corrective action needs to be taken in one of the following areas: ● Change of an application ● Changed partitioning of the table ● Configuration of delta merge 3. Usually this problem occurs because of mass write operations (insert.4 Analyze Large Delta Storage of Column Store Tables Analyze and interpret issues related to delta storage with help from alerts in SAP HANA studio. This may indicate a specific usage pattern from application side that might have room for optimization. All rights reserved. If an alert was raised. a. Note that if the alerts are not configured properly.of the delta storage. Check whether it occurs regularly at a certain time. Alerts raised with low or medium priority usually don’t need immediate action. 57 . update.6. go to the Alerts Tab in the SAP HANA studio and filter for "delta storage". since read-access on delta storage may be one reason for slow performance. Focus on tables where the alert has high priority. or there may be no symptoms. Double-click on an alert and check the alert details about its previous occurrences. delete) on a column table. the symptoms can occur without raising an alert. when many inserts and deletes are performed during a load process. it SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. b. the alert Check delta storage record count * table column count will be triggered. Also these alerts should be taken into account when specific performance issues with end-user operations on these tables are reported. For each affected table a separate alert is created. Figure 11: Check Alert Details 2. even though an alert is raised. If the total count of records (record count * column count) in the delta storage exceeds the threshold of this alert before the next delta merge. If the alert occurred several times. check since when this started. but should be taken as one indicator for checking the sizing. For example. M_CS_TABLES where table_name='<name of table>' and schema_name='<name of schema>'. To determine the usage of the table by applications. SELECT * FROM SYS. 3. Check the following attributes: ○ LAST_MERGE_TIME ○ MERGE-COUNT ○ READ_COUNT. you can check for the tables with the most records in the delta. All rights reserved. If there are many deleted records. A large difference between RAW_RECORD_COUNT_IN_MAIN and RECORD_COUNT suggests that the table has not been compressed properly. update and delete operations occur. A high WRITE_COUNT suggests that many insert. Figure 12: M_CS_TABLES Information SELECT * FROM SYS. WRITE_COUNT ○ RECORD_COUNT ○ RAW_RECORD_COUNT_IN_MAIN ○ RAW_RECORD_COUNT_IN_DELTA ○ MEMORY_SIZE_IN_MAIN ○ MEMORY_SIZE_IN_DELTA a.Merge. If no alert was raised.or CRITICAL. then it usually will be sufficient to trigger the merge for this table manually. See Perform a Manual Delta Merge Operation in the SAP HANA Administration Guide. but only in case of AUTO-. 4. Information regarding the delta merge operation on specific tables can be obtained from the system view M_CS_TABLES. If MERGE_COUNT is high then this is an indicator that the delta merge works properly.M_CS_TABLES where record_count>0 order by raw_record_count_in_delta desc. SMART. it is also required to trigger a compress of the table with the following command: UPDATE <name of table> WITH PARAMETERS('OPTIMIZE_COMPRESSION'='YES'). Note that compression is not triggered when a merge is triggered from an SQLScript. c. the data in the Expensive Statements Trace and Load monitor can be employed. Confirm the delta merge operation has succeeded in the following ways: 58 PUBLIC © 2015 SAP SE or an SAP affiliate company. b. then start with checking current attributes of this table.might be possible to replace these operations with a suitable filter in the source system. If the occurrence of the delta merge problem is rare. Check the time stamp of the alert if it is current. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . while a low MERGE_COUNT suggests a need for corrective action. Unless other recommendation has been provided by SAP the default values shall be applied to the system. then it is likely that the alert occurred due to incorrect configuration of the alerting rather than due to issues with tables.000.000.000 ○ Medium: 1. applications or delta merge functions. If WRITE_COUNT is low. check the threshold value of "Check currently utilized percentage of main memory" in Configure Check Settings of the Alerts tab in SAP HANA studio. 59 .600.Open the table definition in the table editor and on the Runtime Information tab and check the relevant values: ○ LAST_MERGE_TIME ○ MERGE_COUNT ○ RAW_RECORD_COUNT_IN_DELTA ○ LAST_COMPRESSED_RECORD_COUNT 5.000 If you find other (lower) settings. Default values are: ○ Low: 800. To resolve this. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company.000. This can be done in System Information > Merge Statistics. where you can put a filter on the table name and schema name. change the settings back to the default values: Figure 13: Configure Check Settings 6.000 ○ High: 4.000. If problems with the delta storage re-occur frequently for a specific table. All rights reserved. check Merge Statistics for this table. Check the columns START_TIME.M_DELTA_MERGE_STATISTICS where table_name='<name of table>' and schema_name='<name of schema>'. c. 7. If you need to analyze the delta merge statistics for a longer period. MOTIVATION and MERGED_DELTA_RECORDS. EXECUTION_TIME. A value of MERGED_DELTA_RECORDS = -1 suggests that no records were merged but that a compression optimization was performed. A typical error is 2048 and ERROR_DESCRIPTION shows error 2484 which indicates that there was not enough memory to compress the table after the merge. a. b. Check the column LAST_ERROR for records with value other than 0.HOST_DELTA_MERGE_STATISTICS where table_name='<name of table>' and schema_name='<name of schema>'. All rights reserved. 60 PUBLIC © 2015 SAP SE or an SAP affiliate company. For other error codes please refer to the SAP HANA Administration Guide.Figure 14: Merge Statistics in SAP HANA studio Alternatively you can run the following SQL statement and perform the following checks: select * from SYS. than use the equivalent select on table HOST_DELTA_MERGE_STATISTICS of the statistics server: SELECT * FROM _SYS_STATISTICS. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . Check column SUCCESS for records with value other than TRUE. For cases where MERGED_DELTA_RECORDS becomes excessively large the trigger function for the MOTIVATION type should be reviewed and the LOAD should be analyzed for that time frame ( Performance Load ). To analyze the error codes. To change the trace configuration go to the Trace Configuration tab in SAP HANA studio and change the configuration of the Database Trace. However. Related Information SAP HANA Administration Guide M_CS_TABLES 3. Note that the merge only failed if SUCCESS is not TRUE. the error codes need to be analyzed in more detail. 61 .000 ms). All rights reserved. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. you should increase the trace level to INFO for the components mergedog and mergemonitorin the INDEXSERVER section of the Database Trace. optimization with regards to memory consumption can only be done by adjusting the decision function of the corresponding merge type and the corresponding priority function. Note You need to select the Show all components checkbox to display the mentioned trace components. In any other case the error code describes a noncritical condition during a successful merge. changes should be done very carefully and always with involvement of experts from SAP.5 Failed Delta Merge If many cases are identified where auto merge has failed.The delta merge configuration can checked in the SAP HANA studio by opening indexserver. Parameters of the functions are documented in the SAP HANA Administration Guide.ini Configuration mergedog Figure 15: Merge Dog Parameter Since the default value for the frequency of delta merges is already 1 minute (check_interval = 60.6. 62 PUBLIC © 2015 SAP SE or an SAP affiliate company.Figure 16: Database Trace The following table lists error codes and typical corrective actions. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . No action required. No action required. Table 7: Error Codes Error Description Recommended Action 1999 General error (no further information available) Check the indexserver trace for more errors regarding the exception 2450 Error during merge of delta index oc­ curred Check in diagnosis files for an Out-OfMemory dump that occurred during the delta merge operation 2458 Table delta merge aborted as cancel was manually requested by a kill ses­ sion call. 2480 The table in question is already being merged. All rights reserved. No action required. 2481 There are already other smart merge requests for this table in the queue. While the number of records can be kept low by triggering a smart merge from the application. If no knowledge about application is available. TYPE=SPARSE.6. optimization with regards to the size of the table SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. If it happens frequently: A) Analyze change operations on the table and consider table partitioning to minimize the size of the delta storage. Hash Partitioning with a size of 500.000. the number and memory size of columns and the memory size of the table.6 Delta Storage Optimization Table partitioning allows you to optimize the size of tables in memory and their memory consumption as each partition has its own delta storage. (parameter smart_merge_decision_func) 2483 2484 Smart merge is not active (parameter smart_merge_enabled=no) Change the parameter Memory required to optimize table ex­ ceeds heap limit (for failed compres­ sion optimization operations. All rights reserved.00 records is a good initial choice. SUCCESS=FALSE). Internal error.Error Description Recommended Action 2482 The delta storage is empty or the evalu­ No further action required if this occurs ation of the smart merge cost function occasionally. B) Analyze change operations on the table and consider adjusting the pa­ rameter auto_merge_decision_func C) Increase delta storage D) Review sizing 6900 Attribute engine failed 29020 ltt::exception caught while operating on Internal error. Check the indexserver trace for more errors regarding the ex­ ception. Check the indexserver $STORAGEOBJECT$ trace for more errors regarding the ex­ ception. 63 . indicated that a merge is not neces­ sary. The memory consumption of a table in memory during a merge operation depends on the number of records. If it happens frequently: Check M_DELTA_MERGE_STATISTICS and review smart merge cost function with SAP experts. No further action required if this occurs smart_merge_enabled=yes) occasionally. 3. and authentication. or Invalid License New installations of SAP HANA are equipped with a temporary license that expires after 90 days. For more information. The system goes into lockdown in the following situations: ● Your first temporary license of 90 days has expired. If no application knowledge is available. See. ● You have deleted all license keys installed in your database.000.7. authorization. In this case. When a merge is performed. Partitions should be created in a way that avoids as much as possible that single statements need to access multiple partitions. All rights reserved. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . then hash partitioning with a partition size of about 500.1 System Locked Due to Missing. Table Partitioning in the SAP HANA Database in the SAP HANA Administration Guide. 64 PUBLIC © 2015 SAP SE or an SAP affiliate company. To keep the system functional after this period. This is due to the fact that each partition holds a separate delta storage. Note that such licenses are only used in some scenarios. Improper licensing may lead to a lockdown of your SAP HANA system. ● The installed license key is an enforced license key and current memory consumption exceeds the amount specified in the license key.000 records is a good initial choice. you have to install a permanent license. see Managing SAP HANA Licenses in the SAP HANA Administration Guide. the only allowed action is to install a valid license. When considering partitioning it is recommended to analyze the typical usage of this table.can be achieved by table partitioning. Expired. Related Information SAP HANA Administration Guide 3.7 Security-Related Issues This section looks at issues to do with security like licensing. ● An old backup was used for recovery and the license key in the backup has expired in the meantime. 3. the data from the main storage has to be loaded into memory which is a considerably less amount when only a single partition is handled rather than the full table. ● Your permanent license has expired and you do not renew it within 28 days. 3 Resolution of License Issues If your license becomes invalid.2 License Problem Identification and Analysis The first signs of problems related to licensing will be visible by Alert 31 or Alert 44 being issued. All rights reserved. The M_LICENSE system view provides you with the following information: ● License data: ○ SID ○ Hardware key ○ Installation number ○ System number ○ Product limit (licensed amount of memory) ○ Validity start date ○ Expiration date ○ Last successful check date ● License status (permanent. you must have the system privilege LICENSE ADMIN. you need to install a new license. right click on a system in the Systems view. You can install a new license either in the SAP HANA studio or using SQL. Alternatively.Related Information SAP HANA Administration Guide 3.7. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. To check your current license using SAP HANA studio. Information on previously installed licenses is available. only SID and hardware key are displayed. 65 . Related Information SAP HANA SQL and System Views Reference 3. Note To be able to query license information. choose Properties and then License. or enforced) Note that in case of system lockdown. valid.7. you can retrieve the same information using SQL: select * from m_license. make sure to enable multi-line statement support (command line option -m or \mu ON when within SAP HANA HDBSQL). You install a license key with the following SQL statement: SET SYSTEM LICENSE '<license file content goes here. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . General Analysis The system view EFFECTIVE_PRIVILEGES is useful for checking the privileges of a specific user. ● Your installed license key is correct.7. check the following: ● The current system time is within the validity period of the license. the M_LICENSE view displays only one row with a valid license for the product SAP HANA. 66 PUBLIC © 2015 SAP SE or an SAP affiliate company.Note To install a license key. line breaks matter>'. Related Information SAP Note 1704499 .4 Troubleshooting Authorization Problems SAP HANA implements its authorization concept based on the entities user. If you have successfully installed a license but your system is still locked down. see Install a Permanent License in the SAP HANA Administration Guide. For more detailed information about how to install a license key. you need the LICENSE ADMIN system privilege. ● The SAP Notes in the Related Links section.License request and installation for SAP HANA database SAP Note 1699111 . Note Line breaks are essential for interpretation of the license key text. privilege. It includes information about all privileges granted to a specific user (both directly and indirectly through roles). The command will fail if the license key has a different installation number or system number than the current ones in the database instance. in particular. as well as how the privileges were obtained (GRANTOR and GRANTOR_TYPE column).26 SAP HANA Administration Guide 3. All rights reserved.License key update in SAP HANA prior to Rev. If you use the command line tool SAP HANA HDBSQL to install the license. hence they must not be removed. and role.System Measurement for License Audit SAP Note 1634687 . you need to find out which privileges the user is missing and then grant them to the user. Related Information SAP HANA Security Guide SAP HANA Administration Guide SAP HANA SQL and System Views Reference 3. All rights reserved. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. Procedure 1. see System Views for Verifying Users' Authorization. For more information about the authorization concept in SAP HANA. 67 . Prerequisites You have the system privilege TRACE ADMIN.Figure 17: Output of Effective Privileges For more information about using this view and other system views related to authorization. On the Trace Configuration tab of the Administration editor.1 Troubleshoot the Error "Insufficient Privilege: Not Authorized" If the error Insufficient privilege: Not authorized occurs during statement execution. set the database trace level for the component authorization of the indexserver service to INFO. see the SAP HANA Security Guide.7.4. oid=<oid>) followed by a structure showing which privileges are checked on which schemas and objects. you require the following system privileges: ● CATALOG READ ● TRACE ADMIN Procedure ● Check which analytic privileges have been granted to the user using the system view EFFECTIVE_PRIVILEGES. Set the database trace level for the component authorization of indexserver service back to DEFAULT. 2. you need to determine which privileges have been granted to the user and whether or not they are correct. In many cases. Related Information SAP HANA Administration Guide 3. Execute the statement that triggered the error. the trace will also list all relevant analytic privileges that have not been granted to the user. you will find something like: UserId(<nnnn>) is not authorized to do SQL_ACT_abc on ObjectId(m. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . the name belonging to UserId is given below that structure. Prerequisites To troubleshoot this issue. Usually. In this structure. choose Choose All Components. All rights reserved. 68 PUBLIC © 2015 SAP SE or an SAP affiliate company. If the user cannot access a view due to a missing analytic privilege. Use that information to grant the missing privilege. To see it. On the Diagnosis Files tab of the Administration editor. you will find the name which belongs to <oid>. even though he has been granted an analytic privilege.2 Troubleshoot the Display of Unrestricted or Incorrect Results for a View Secured with Analytic Privileges If a user has unrestricted access to a view or sees results that he should not. 3.7. 4. examine the indexserver trace to find out about the failed authorization check.n.4.Note The component is not visible by default. MODELING. Information about filter conditions generated from the relevant analytic privileges can be traced in the indexserver trace file. verify that the user does not have the analytic privilege _SYS_BI_CP_ALL. ● Identify wrong filters specified in the analytic privileges granted to the user. Prerequisites To troubleshoot this issue. On the Trace Configuration tab of the Administration editor. This can help you to identify wrong filters specified in the analytic privileges granted to the user.4.3 Troubleshoot the Error "Insufficient privilege: Not authorized" Although User Has Analytic Privileges Even if a user has the correct analytic privileges for a view. Related Information SAP HANA Administration Guide SAP HANA SQL and System Views Reference SAP HANA Security Guide 3. regardless of any other analytic privileges that apply. Caution The MODELING role is very privileged and should not be granted to users. for example. This analytic privilege potentially allows a user to access all the data in all activated views. you require the following system privileges: ● CATALOG READ ● TRACE ADMIN SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. In particular. All rights reserved. set the database trace level for the component analyticprivilegehandler of the indexserver service to DEBUG. The MODELING role should only be used as a template.7. particularly in production systems. he still may receive the error Insufficient privilege: Not authorized if there is an issue with privileges at another level. the user will have this analytic privilege through a role. 69 .Execute the following SQL statement: SELECT * FROM EFFECTIVE_PRIVILEGES WHERE USER_NAME = '<user>' AND OBJECT_TYPE = 'ANALYTICALPRIVILEGE'. Usually. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . check the indexserver_alert_<host>. SELECT) on all base objects (for example. You can do this by selecting from the EFFECTIVE_PRIVILEGES system view: SELECT * FROM EFFECTIVE_PRIVILEGES WHERE USER_NAME = '_SYS_REPO'. ● If the analytic privilege uses a database procedure to define dynamic value filters at runtime. You can do this by selecting from the EFFECTIVE_PRIVILEGES system view: SELECT * FROM EFFECTIVE_PRIVILEGES WHERE USER_NAME = '_SYS_REPO'. tables) of the view.7. Note that analytic privileges have to contain at least a view attribute with or without filter condition in order to grant access to the view. 70 PUBLIC © 2015 SAP SE or an SAP affiliate company. You can verify a user's privilges by selecting from the EFFECTIVE_PRIVILEGES system view: SELECT * FROM EFFECTIVE_PRIVILEGES WHERE USER_NAME = '<user>' AND OBJECT_TYPE = 'ANALYTICALPRIVILEGE'. you require the following system privileges CATALOG READ. All rights reserved. check for errors in the execution of the underlying procedure. SELECT) with GRANT OPTION on the base tables of the view. Procedure ● Verify that the _SYS_REPO user has all required privileges (for example. In addition. thee data preview for an activated view does not show any data. ● Verify that the analytic privileges required for any underlying views have been granted to the user. Prerequisites To troubleshoot this issue.4. the granted analytic privilege grants access only to this top-level view. To find out the actual error during procedure execution for analytical privileges. Analytic privileges are required for all underlying views.4 Troubleshoot the Error "Invalidated View" During SELECT Statement Execution A user may receive the error Invalidated view when executing a SELECT statement against a view that was activated from the repository. 3. If the view is a top-level view (calculation view) with underlying views.trc trace file (accessible on the Diagnosis Files tab of the Administration editor).Procedure ● Verify that the _SYS_REPO user has all required privileges (for example. 71 . Follow the procedure as described in detail in the SAP HANA Administration Guide. All rights reserved. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. Figure 18: User Editor For more information about managing users in the User editor.7.5. Related Information SAP HANA Administration Guide 3. see Security Administration in the SAP HANA Administration Guide. As a prerequisite. <sid>adm access to the (primary) SAP HANA host is required.3. it can be reset.5 Troubleshooting Problems with User Name/Password Authentication Common problems with regards to authentication are related to incorrect or expired passwords.7. User administrators can change users' password in the User editor of the SAP HANA studio.1 Resetting the Password for the SYSTEM User In case the password for the SYSTEM user is not available anymore. This is defined by the password policy parameter maximum_invalid_connect_attempts. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .5. Prerequisites You have system privilege USER ADMIN. try again later: lock time is 1440 minutes. The user can now log on again. For more information about this and other password policy parameters. see Password Policy Configuration Options in the SAP HANA Security Guide. Context An example of this error might look like: Error "user is locked. the user logged on too many times with the wrong password.2 Troubleshoot the Error "User is locked" A user receives the error User is locked after too many failed log on attempts. Most likely. Related Information SAP HANA Security Guide 72 PUBLIC © 2015 SAP SE or an SAP affiliate company. The default maximum number of failed logon attempts is 6.7. Procedure Reset the invalid connect attempts with the following SQL statement: ALTER USER <user> RESET CONNECT ATTEMPTS. user is locked until 2014-05-28 21:42:24.Related Information SAP HANA Administration Guide 3. All rights reserved.12214212" (the time is given in UTC). 1 Blocked Transactions Blocked transactions are write transactions that are unable to be further processed because they need to acquire transactional locks (record or table locks) which are currently held by another write transaction.Long-running blocking situations ● Alert 59 . 73 .1 Identify and Assess Blocked Transaction Issues The first signs of blocked transactions are poor application response or alerts 49 or 59 are raised.8. is unresponsive or can even seem to hang if there are issues with uncommitted transactions.3. 3. an application runs sluggishly.8 Transactional Problems This section covers troubleshooting of transaction problems.Percentage of transactions blocked SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. a high number of active versions or blocked transactions. 3.1. All rights reserved.8. Those situations are not covered in this section. From an end user perspective. Note that transactions can also be blocked waiting for physical resources like network or disk. long-lived cursors blocking garbage collection. The initial indicators of blocked transactions are given by: ● Users reporting bad application responsiveness ● Alert 49 . look at the Job Progress monitor. 74 PUBLIC © 2015 SAP SE or an SAP affiliate company. of Transactions” are of special interest. and how many transactions are affected. See Job Progress Monitoring and Savepoint Performance. The latter gives you the corresponding information if a session blocks other sessions. It shows currently running SAP HANA background processes like Delta Table Merge. The first tells you whether the session is blocked by another session and identifies the ID of the blocking one. Since the Delta Table Merge needs to lock tables to proceed.To confirm the database performance is harmed by blocked transactions. Another job display by this monitor is the savepoint write which needs to pull a global database lock in its critical phase. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . you should check the following SAP HANA studio monitors under the Performance tab: Load Monitor Figure 19: SAP HANA Studio Load Monitor The Blocked Transactions graph shows how many blocked transactions currently exist and existed in the past to a certain extent. See Session Monitoring. In the context of blocked transaction troubleshooting. it is a common cause for blocked transactions. See Load Monitoring. the columns “Blocked by Connection Id” and “Blocks No. Job Progress Monitor To further track down the issue. All rights reserved. Session Monitor The Session Monitor lists all currently opened SQL sessions (meaning user connections). Figure 20: SAP HANA Session Monitor Blocked Transaction Monitor The Blocked Transaction Monitor is the next drill down step. It only lists those transactions that are currently blocked. The ordering is done via a blocking/blocked relation. That means transactions that are blockers are highlighted. Directly beneath the blocked transaction are displayed: Figure 21: Blocked Transaction Monitor Example: In the figure above, you see transaction 126 (green) blocking multiple other transactions (red). Note that in this example transaction 126 was initiated by remote transaction 77 on another node. That means transaction 77 is the root of the blocked transaction chain. See Blocked Transaction Monitoring. Thread Monitor The Thread Monitor allows the most fine-grained view into the current situation by listing all threads in the system. Note that it is usually not necessary to drill into that level of detail. Threads contributing to a transaction that is currently blocked are marked by a warning sign in the “Status” column. To get additional information about the blocking situation, hover the mouse over the warning sign. See Thread Monitoring. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. 75 Figure 22: Threads contributing to Blocked Transactions Related Information Load Monitoring [page 140] Job Progress Monitoring [page 139] Savepoint Performance [page 50] Session Monitoring [page 138] Blocked Transaction Monitoring [page 137] 3.8.1.2 Troubleshooting Blocked Transactions When troubleshooting blocked transactions, it is helpful to differentiate between situations where only single or a few transactions are blocked from the situation where a high percentage of all transactions are blocked. 3.8.1.2.1 Single or Few Transactions are Blocked If you identified only a single or a few blocking transactions, there is likely an issue on application side. A usual pattern is a flaw in the application coding that does not commit a write transaction. Such a transaction will be a blocker for any other transaction that needs to access the same database object. To release the situation you have to close the blocking transaction. 76 PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions There are several possibilities to achieve this: ● Contact the Application User The Session Monitor allows you to identify the user of the application. You can find this information in the “Database User” column or, in case the application has its own user management (for example, SAP BW), in the “Application User” column. Contact the user and ask him whether he can close the application. ● Contact the Application Developer As a follow-up, the author of the application should be contacted whether such situations can be avoided in the future by changing the application code. 3.8.1.2.1.1 Cancel the Session If you are not able to contact the user to have them cancel the session, you can also cancel the session in the context menu of the Session Monitor. The current transaction will be rolled back. The session cancellation may take some time to succeed. If it takes longer than 30 seconds, consider this as a bug and contact development support. 3.8.1.2.1.2 Kill the Client Appication In case the session cancellation takes too long or does not complete at all, you can kill the client process that opened the session. This will terminate the blocking transaction as well. As a prerequisite, you have to have access to the client machine. The information needed for this task can be retrieved from the Session Monitor. See columns “Client IP” and “Client Process ID” to determine the host and process to be killed. Note that killing the client application is safe from a database consistency standpoint, the current transaction will be rolled back gracefully. 3.8.1.2.2 Many Transactions are Blocked In the case that a large amount of transactions are blocked, the troubleshooting should take a slightly different approach. First you need to determine whether there is a single or few blocking transactions that block a large amount of other transactions. For this, open the Blocked Transaction Monitor and check the amount of blocking transactions. If you assess there is only a few blocking transactions, use the techniques described in Single of Few Transactions are Blocked to resolve the situation. If there are many transactions in a blocking state, you need to find out whether a specific access pattern causes the situation. In case that multiple transactions try to access the same database objects with write operations, they block each other. To check if this situation exists, open the Blocked Transaction Monitor and analyze the “Waiting Schema Name”, “Waiting Object Name” and “Waiting Record Id” columns. If you find a fair amount of blocking transactions that block many other transactions you need to investigate if the following is possible: SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. 77 ● Change the client application(s) to avoid the access pattern ● If a background job is running that issues many write transactions (for example, a data load job): Reschedule to a period with a low user load ● Partition tables that are accessed frequently to avoid clashes. See the SAP HANA Administration Guide for more details on partitioning. In case you cannot identify specific transactions or specific database objects that lead to transactions being blocked, you have to assume a problem with the database itself or its configuration. One example is an issue with long savepoint durations. See Savepoint Performance for troubleshooting such issues. Related Information Single or Few Transactions are Blocked [page 76] Savepoint Performance [page 50] SAP HANA Administration Guide 3.8.2 Troubleshooting Blocked Transaction Issues that Occurred in the Past Finding the root cause of blocked transaction situations that you have resolved is more difficult than troubleshooting issues that are currently happening. Tools such as the Load Monitor, system views and the SQL Plan cache are available to help you. First use the Load Monitor to isolate the exact time frame where the issue happened. Using that information, investigate what happened at this specific time frame. You should check the following monitoring and StatisticServer views: ● _SYS_STATISTICS.HOST_BLOCKED_TRANSACTIONS: Analyze the columns “WAITING_SCHEMA_NAME”, “WAITING_TABLE_NAME” and “WAITING_RECORD_ID” to identify the database objects that lead to blocked transactions ● SYS.M_DELTA_MERGE_STATISTICS: The column “START_TIME” and “EXECUTION_TIME” provide you with the information if there was a Delta Table Merge running A longer history can be found in the StatisticServer table _SYS_STATISTICS.HOST_DELTA_MERGE_STATISTICS ● SYS.SAVEPOINTS: Check if a savepoint was written during the time period. A longer history can be found in _SYS_STATISTICS.HOST_SAVEPOINTS 78 PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions In addition the SAP HANA studio SQL Plan Cache monitor may be able to provide information about the statements that were involved in the situation: Figure 23: Plan Cache Monitor Only check entries that have “TOTAL_LOCK_WAIT_COUNT” > 0. For those entries, compare the column “MAX_CURSOR_DURATION” against “AVG_CURSOR_DURATION”. If there is a significant difference, there was at least one situation where the transactions took much longer than average. This can be an indication that it was involved in the situation. 3.8.3 Multiversion Concurrency Control (MVCC) Issues In this section you will learn how to troubleshoot issues arising from MVCC. Multiversion Concurrency Control (MVCC) is a concept that ensures transactional data consistency by isolating transactions that are accessing the same data at the same time. To do so, multiple versions of a record are kept in parallel. Issues with MVCC are usually caused by a high number of active versions. Old versions of data records are no longer needed if they are no longer part of a snapshot that can be seen by any running transaction. These versions are obsolete and need to be removed from time to time to free up memory. This process is called Garbage Collection (GC) or Version Consolidation. It can happen that a transaction is blocking the garbage collection. The consequence is a high number of active versions and that can lead to system slow-down or out of memory issues. 3.8.3.1 Row Store Tables Garbage collection is triggered after a transaction is committed and also periodically (every hour by default). A transaction that is currently committing can be identified in the Threads tab (see System Performance Analysis). The Thread type will be “SqlExecutor” and the Thread method “commit”. The periodic garbage collection can be identified by Thread Type” MVCCGarbageCollector”. Note that the periodic garbage collection interval can be configured in the indexserver.ini file transaction section with the parameter mvcc_aged_checker_timeout. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. 79 Related Information System Performance Analysis [page 134] 3.8.3.2 MVCC Problem Identification There are a number of indicators of MVCC problems to check for. Problems with high number of active versions can be identified by ● Users report an increase of response times ● The indexserver trace contains "There are too many un-collected versions. The transaction blocks the garbage collection of HANA database." ● By checking “Active Versions” in the Load Monitor (see Performance tab) Figure 24: Load Monitor Showing Garbage Collection Transactions blocking garbage collection can originate from: ● Long-running or unclosed cursors ● Long-running transactions with isolation mode “serializable” or ”repeatable read” ● Hanging threads In order to validate there is a problem with MVCC, check the number of active versions in the Row StoreMVCC manager monitoring view. Note that in multihost environment, you have to check the master host. select * from m_mvcc_tables where host='ld9989' and port='30003' and (name='NUM_VERSIONS' or name='MAX_VERSIONS_PER_RECORD' or name='TABLE_ID_OF_MAX_NUM_VERSIONS'); Figure 25: MVCC Information on a Healthy System 80 PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions 81 . transaction_id. connection_id.3 Analysis of MVCC Issues You have to find which transactions are blocking the garbage collection and to which connection they are related.8. use the following query to find out which other transaction spawned the candidate and determine its connection ID. You have to check both. Similarly. isolation_level FROM M_TRANSACTIONS WHERE MIN_MVCC_SNAPSHOT_TIMESTAMP > 0 order by min_mvcc_snapshot_timestamp desc.primary_transaction_id = t. transaction_id. this should be treated as a problem and a slowdown of accesses to a specific table is expected. SELECT top 1 host. it is considered a problem and an overall slowdown of the system can be experienced.000. Use TABLE_ID_OF_MAX_NUM_VERSIONS and join it against the SYS. The following queries will return the transaction that may block the Garbage Collection. transaction_type.connection_id AS "Kill this connection id". All rights reserved. Figure 26: User Transaction Possibly Blocking Garbage Collection In case of a user transaction being the candidate (TRANSACTION_TYPE=’USER TRANSACTION’). connection_id. update_transaction_id.transaction_id AND e. if the maximum number of versions per record (MAX_VERSIONS_PER_RECORD) exceeds 8. primary_transaction_id.volume_id = t. primary_transaction_id. port. you can directly determine the connection ID the transaction belongs to (see an example in the Figure above).If the number of active versions (NUM_VERSIONS) is greater than eight million. SELECT top 1 host. isolation_level FROM M_TRANSACTIONS WHERE MIN_MVCC_SNAPSHOT_TIMESTAMP = (SELECT MIN(VALUE) FROM M_MVCC_TABLES WHERE NAME = 'MIN_SNAPSHOT_TS') order by min_mvcc_snapshot_timestamp desc. t.transaction_id AS "To get rid of external transaction id" FROM m_transactions t JOIN m_transactions e ON e. port.TABLES system view to determine the table which is having the problem. Figure 27: External Transaction Possibly Blocking Garbage Collection If the candidate’s transaction type is ‘EXTERNAL TRANSACTION’.transaction_id AS "Belonging to user transaction id". transaction_type.3.000. e. SELECT t. Related Information Performance Trace [page 166] 3. update_transaction_id.volume_id SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. 3.4 Version Garbage Collection Issues Alerts 73. Cancel connection 4. Context The following steps allow you to check whether or not the issue you have is related to a “version space overflow/skew” situation. Contact the user to stop his activity 2. Use the following approaches in the given order for transactions where you know the connection ID. contact development support immediately to get further help Related Information Resolving CPU Related Issues [page 38] 3.WHERE e. Kill the client application Note There is no guaranteed that these measures will stop a transaction which blocks the garbage collection. All rights reserved. Cancel the statement/cancel internal transaction 3. 82 PUBLIC © 2015 SAP SE or an SAP affiliate company. If that is the case.3. 74.4 Solution of MVCC Issues Solving MVCC issues is similar to solving blocked transaction issues.8.8. 1. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .transaction_id = <GC blocker transaction id>.transaction_type = 'EXTERNAL TRANSACTION' and e. Procedure 1. and 75 help you to identify and resolve version space overflow issues. The first step is to check the alerts. The following alerts are indicators that could be the problem: ○ Alert 73 Rowstore version space overflow. determines the overflow ratio of the rowstore version space. investigate what the statement is doing and take the necessary corrective action. 83 . then use MVCC Blocker Transaction. a. In this case. In this case you can query the following monitoring view to check how many versions the related table TEST2 has. However. ○ The view MVCC Blocker Statement shows statements that may be blocking garbage collection. All rights reserved. determines whether the rowstore version chain is too long. The System Information tab of SAP HANA studio provides a set of tabular views to display blocking connections based on pre-defined SQL queries. If there is no blocker in MVCC Blocker Statement. Identify the statement and connection blocking garbage collection.000 its impact on performance is negligible. In the first row of the table shown below if there is a "global" garbage collection blocker whose IDLE_TIME > 3600 seconds (1 hour) then investigate what the statement is doing and take the necessary corrective action. See Resolve Out of Space Errors for Delta Dbspace in SAP HANA Dynamic Tiering: Administration Guide. a table level garbage collection blocker whose IDLE_TIME > 3600 seconds (1 hour) can be seem. In the second row. Table 8: Type IDLE_TIME_SEC CONNEC­ TION_ID START_MVCC_ TIMESTAMP STATE­ MENT_STRING INFO GLOBAL 4000 200285 142113 SELECT * FROM TEST1 Check VER­ SION_COUNT in M_MVCC_OVER­ VIEW TABLE 5000 200375 142024 SELECT * FROM TEST2 ORDER BY A Check VER­ SION_COUNT in M_MVCC_TA­ łè”ï³Ił%Łû˝ÔÇ¡Jù®° SHOTS SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. a version space overflow may manifest as an out of space error for the delta dbspace. SELECT * FROM M_TABLE_SNAPSHOTS WHERE TABLE_NAME = 'TEST2' AND START_MVCC_TIMESTAMP = 142024 If the results of the query shows that VERSION_COUNT > 1 million. determines the overflow ratio of the metadata version space. The blocker would be likely one of internal/external/serializable transaction. 2. Note that the table only shows selected columns from the results of running MVCC Blocker Statement for the purposes of this example.○ Alert 74 Metadata version space overflow. and will not trigger these alerts. If you use extended storage. ○ The view MVCC Blocker Transaction shows transactions that may be blocking garbage collection. ○ Alert 75 Rowstore version space skew. Identify the connection or transaction that is blocking version garbage collection. if VERSION_COUNT is less than 10. the blocking statement can cause a performance drop for table updates. ○ The view MVCC Blocker Connection shows connections that may be blocking garbage collection. So make sure you are connected to the right node before you cancel the transaction. Kill the statement and/or connection that is blocking garbage collection. a. MVCC Blocker Statement WITH STORED_MVCC_BLOCKER_STATEMENTS AS (SELECT 84 PUBLIC © 2015 SAP SE or an SAP affiliate company. To cancel a transaction that you have identified as the cause of the blockage open the Emergency Information tab in the SAP HANA studio. All rights reserved. right click on the transaction in the tabular view and choose Cancel Transaction <TRANSACTION ID>. If the TYPE column is GLOBAL. With M_TABLE_SNAPSHOTS. The column LIFE_TIME_SEC column will help your decision.. To kill the blocking connection use ALTER SYSTEM DISCONNECT SESSION 'CONNECTION_ID' b. then it is a global version garbage collection blocker. if killing the connection does not succeed or CONNECTION_ID does not exist. If TYPE is TABLE. The column IDLE_TIME_SEC will help your decision. Kill the connection Check MVCC Blocker Statement to find out which statement is blocking version garbage collection and investigate what the blocker is doing. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . You can find a version garbage collection blocker with thread info using these queries. you can find out the number of versions for each table. it’s a blocker of the specific table. Note Be aware that TRANSACTION ID is only unique on a specific node and therefore the ID might be reused. Check MVCC Blocker Transaction to find out which transaction blocks global version garbage collection and investigate what the blocker is doing.Type IDLE_TIME_SEC CONNEC­ TION_ID START_MVCC_ TIMESTAMP STATE­ MENT_STRING INFO TABLE 100 200478 142029 SELECT * FROM TEST0 ORDER BY A Check VER­ SION_COUNT in M_MVCC_TA­ ™8Ù� ×*‰Fö-ᨣZAc/ SHOTS 3. Kill the transaction. MVCC Blocker Connection WITH STORED_MVCC_BLOCKER_CONNECTIONS AS (SELECT * FROM M_CONNECTIONS WHERE CONNECTION_ID IN (SELECT CONNECTION_ID FROM SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. 85 .START_TIME.START_MVCC_TIMESTAMP > 0 AND A.HOST = B.START_MVCC_TIMESTAMP = B.HOST = B.PORT WHERE A.HOST AND A.START_MVCC_TIMESTAMP <= B.CASE WHEN A.MIN_MVCC_SNAPSHOT_TIMESTAMP ORDER BY LIFE_TIME_SEC DESC) SELECT * FROM STORED_MVCC_BLOCKER_TRANSACTIONS AS A.MIN_MVCC_SNAPSHOT_TIMESTAMP = B. CASE WHEN A.* FROM M_ACTIVE_STATEMENTS AS A JOIN M_MVCC_OVERVIEW AS B ON A. CURRENT_TIMESTAMP) LIFE_TIME_SEC.PORT = B. MVCC Blocker Transaction WITH STORED_MVCC_BLOCKER_TRANSACTIONS AS (SELECT SECONDS_BETWEEN(A.START_MVCC_TIMESTAMP = B. M_SERVICE_THREADS AS B WHERE A.MIN_MVCC_SNAPSHOT_TIMESTAMP > 0 AND A.MIN_MVCC_SNAPSHOT_TIMESTAMP ORDER BY IDLE_TIME_SEC DESC) SELECT * FROM STORED_MVCC_BLOCKER_STATEMENTS AS A. CURRENT_TIMESTAMP) IDLE_TIME_SEC.PORT WHERE A.CONNECTION_ID.HOST AND A.* FROM M_TRANSACTIONS AS A JOIN M_MVCC_OVERVIEW AS B ON A. SECONDS_BETWEEN(A. M_SERVICE_THREADS AS B WHERE A. A.CONNECTION_ID. All rights reserved.CONNECTION_ID = B.MIN_MVCC_SNAPSHOT_TIMESTAMP THEN 'GLOBAL' ELSE 'TABLE' END TYPE.STATEMENT_STATUS <> 'NONE' AND A.MIN_MVCC_SNAPSHOT_TIMESTAMP THEN NULL ELSE 'check M_TABLE_SNAPSHOTS for details' END INFO.PORT = B.LAST_ACTION_TIME. A.CONNECTION_ID = B. MIN_MVCC_SNAPSHOT_TIMESTAMP <= B.HOST = B. 3. M_SERVICE_THREADS AS B WHERE A.M_TRANSACTIONS AS A JOIN M_MVCC_OVERVIEW AS B ON A.9.CONNECTION_ID = B.MIN_MVCC_SNAPSHOT_TIMESTAMP)) SELECT * FROM STORED_MVCC_BLOCKER_CONNECTIONS AS A. Identification of Critical SQL Statements A key step in identifying the source of poor performance is to understand how much time is spent in the SAP HANA engine for query execution.9 Statement Performance Analysis This section gives an overview of issues and solutions concerning SQL statement performance. By analyzing SQL statements and calculating their response times. you can better understand how the statements affect application and system performance.HOST AND A. 3. Before you are able to analyze and optimize a SQL statement you have to identify the critical SQL statements.1 SQL Statement Optimization This section provides an overview of tools. The most critical are the following areas: ● A long runtime can result in delays of the business activities ● A high CPU consumption can lead to system-wide CPU bottlenecks ● High memory consumption can be responsible for out-of-memory situations and performance penalties due to unload of tables from memory SQL statements consuming significant resources are called expensive SQL statements.CONNECTION_ID.MIN_MVCC_SNAPSHOT_TIMESTAMP > 0 AND A.PORT WHERE A. traces and SAP HANA studio areas that can be used to identify critical SQL statements. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .PORT = B. SQL statements that are not executed efficiently can cause local and system-wide problems. We can distinguish between the following scenarios: ● SQL statements that have caused problems in the past ● SQL statements that are currently causing problems 86 PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. if available.2 SQL Statements Responsible for Past Problems Sometimes it is not possible to identify a critical SQL statement during runtime. you can check the SQL plan cache history (HOST_SQL_PLAN_CACHE). In this case you can use the following approaches to identify one or several SQL statements that can have contributed to the problem. You can use the SQL statement: “HANA_Threads_ThreadSamples_FilterAndAggregation” available from SAP Note 1969700 – SQL Statement Collection for SAP HANA in order to check. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. Figure 28: SQL Statements With High Runtimes ● To determine the top SQL statements that were executed during a dedicated time frame in the past. You can identify a SQL statement either by its SQL text (“statement string”) or by the related statement hash that is uniquely linked to an individual SQL text. All rights reserved. 87 . can also be used to determine the top SQL statements. so that the SQL statements with the highest elapsed time from SAP HANA are returned. Figure 29: SQL Statements With Highest Elapsed Time ● The thread sample history (tables M_SERVICE_THREAD_SAMPLES.3. The mapping of the statement hash to the actual SQL text is described later. You have to specify a proper BEGIN_TIME / END_TIME interval and use AGGREGATE_BY = ‘STATEMENT_HASH’ to identify the top SQL statements during the time frame. HOST_SERVICE_THREAD_SAMPLES). You can use the SQL statement: “HANA_SQL_SQLCache_History” available from SAP Note 1969700 – SQL Statement Collection for SAP HANA in order to check for top SQL statements during a specific time frame: You have to specify a proper BEGIN_TIME / END_TIME interval and typically use ORDER_BY = ‘ELAPSED’. ● To determine SQL statements with a particularly high runtime you can check for the top SQL statements in terms of TOTAL_EXECUTION_TIME in the SQL plan cache in SAP HANA Studio on the SQL Plan Cache Performance tab.9. Often you need to have a look at some more SQL statements.9. ● SAP HANA Alert 39 (“Long running statements”) reports long running SQL statements and records them in the table _SYS_STATISTICS.3 SQL Statements Responsible for Current Problems If problems like high memory consumption.Figure 30: SQL Example Output In this case the SQL statement with hash 51f62795010e922370bf897325148783 is executed most often and so the analysis should be started with it. All rights reserved. Related Information SAP Note 1969700 SAP Note 1989031 SAP Note 1984422 3. for example the statements related to the next statement hashes fc7de6d7b8942251ee52a5d4e0af728f and 1f8299f6cb5099095ea71882f84e2cd4. high CPU consumption or a high duration of individual database requests are currently happening. Check the contents of this table to determine details of the SQL statements that caused the alert.sh” ● In case of an out-of-memory (OOM) situation you can determine potentially responsible SQL statements by analyzing the OOM dump file(s) as described in SAP Note1984422 – Analysis of HANA Out-of-memory (OOM) Dumps. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .sh script to regularly collect thread samples as described in SAP Note 1989031 – Scheduling of Shell script “thrloop. you can determine the active SQL statements with the help of SAP HANA studio. ● In cases where the M_SERVICE_THREAD_SAMPLES / HOST_SERVICE_THREAD_SAMPLES information is not usable you can use the thrloop.HOST_LONG_RUNNING_STATEMENTS. Check for the currently running SQL statements in SAP HANA studio on the 88 PUBLIC © 2015 SAP SE or an SAP affiliate company. Performance Threads tab. All rights reserved. Related Information Tools and Tracing [page 134] 3. you can also analyze query plans from a logical perspective and consider the following questions to gain the insight you need: ● Does SAP HANA read data from multiple tables when only one is required? ● Does SAP HANA read all records when only one is required? SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. From a technical perspective.4 SQL Statements Reported by Traces Additional traces can be activated to help identify critical SQL statements. and see whether data is processed in parallel. Activating traces in addition to the normally available performance information can be useful for the following reasons: ● Capturing of more detailed information ● Determination of SQL statements that are not obvious on a global level. you can collect more information about the SQL statement in order to identify the root cause and optimize the SQL statement.9. but important for business critical applications.Figure 31: Running SQL Statements in Threads Detail 3.9. if you understand the idea and purpose behind the query. However. The following traces are available to determine SQL statements: ● SQL trace ○ Capturing of performance data for all SQL statements ○ Filter conditions can be used to limit the trace activities ● Expensive statements trace ○ Captures all SQL statements with a runtime beyond a defined threshold Further details can be found in Tools and Tracing. 89 . understand how much data is processed by the operators.5 Analysis of Critical SQL Statements When you have identified the SQL text and the related statement hash based on the tools and traces. analyzing query plans allows you to identify long running steps. The following standard traces exist to identify critical SQL statements. The available analysis approaches are described here. The historic execution details for a particular SQL statement can be determined with the SQL statement: “HANA_SQL_StatementHash_SQLCache_History” included with SAP Note 1969700.● Does SAP HANA start with one table even though another table has a much smaller result set? 3. 90 PUBLIC © 2015 SAP SE or an SAP affiliate company. Also here the appropriate STATEMENT_HASH has to be specified as input parameter. Figure 32: SQL PLAN Cache Example Output Alternatively you can check the SAP HANA view M_SQL_PLAN_CACHE or the Cache Performance SQL Plan tab in SAP HANA studio. All rights reserved.9. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .6 SQL Plan Cache Analysis The SAP HANA SQL Plan Cache can be evaluated in detail for a particular statement hash. You can use the SQL statement: “HANA_SQL_StatementHash_KeyFigures” available in SAP Note 1969700 – SQL Statement Collection for SAP HANA in order to check for the SQL Plan Cache details of a specific SQL statement (the related STATEMENT_HASH has to be maintained as input parameter). ● If the value for Cursor duration is very high and at the same time significantly higher than the value for Execution time. further analysis should be done on the application side in order to check if it is possible to reduce the number of executions. ● If the value for Preparation time is responsible for a significant part of the Execution time value you have to focus on optimizing the parsing (for example. See section “Query Plan Analysis” for more information. Related Information SAP Note 1969700 Query Plan Analysis [page 147] Example: Reading the SQL Plan Cache [page 92] SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company.Figure 33: Statement Hash Example output: Based on the results of this evaluation you can distinguish the following situations: ● If the value for Executions is unexpectedly high. you have to check which processing steps are executed on the application side between the individual fetches. A high value for Cursor duration can negatively impact the database in general because open changes may impact the MVCC mechanism. All rights reserved. reuse of already parsed SQL statements). ● If the value for Records is unexpectedly high. sufficient SQL plan cache size. ● If Execution time is much higher than expected (that can be based on the statement complexity and the number of processed rows). 91 . the SQL statement has to be checked more in detail on technical layer to understand the reasons for the high runtime. further analysis should be done on the application side in order to check if it is possible to reduce the number of selected records. Why is this simple SQL taking so long? Does it have problems processing? ● The execution times for statement 3 are fine for one-off execution.833 181. 92 PUBLIC © 2015 SAP SE or an SAP affiliate company. The following example aims to show you how to interpret information about locking situations. but it runs too frequently. Table 9: USER_ NAME STATMENT_ STRING TOTAL_ EXECUTION_ AVG_ EXECUTION_ TIME TIME EXECUTION_ COUNT SYSTEM SELECT "REQUEST" .. "STEPCOUNT" .604 – <Further 6832 records> – – – You could read these top 3 results as follows: ● Statement 1 takes the longest time overall but it is also executed frequently.556.7 Example: Reading the SQL Plan Cache These examples aim to show you how to gain useful insights by analyzing the SQL plan cache.367. Execution in a Single-Host System This example aims to show you how to interpret information about execution time. Why? Is there a problem in application code? Sorting by AVG_EXECUTION_TIME or EXECUTION_COUNT provides a different angle on your analysis. "DATAPAKID" .. "INTREPORT" ..877 60. The information in columns TOTAL_LOCK_WAIT_COUNT and TOTAL_LOCK_WAIT_DURATION lets us know which statement is waiting for others and how much time it takes.987 20.266 4. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . ● Statement 2 is not executed very frequently but has the second highest total execution time.672.3. over 20. . The following table is sorted by TOTAL_EXECUTION_TIME. "JOBCOUNT" .000 times.9.. All rights reserved. "RECORD" .272 SYSTEM SELECT * FROM "/BIC/ AZDSTGODO40" WHERE "SID" = ? 726. "CALDAY" . 774. "JOBGROUP" .620.073 12 SYSTEM SELECT "JOBNAME" .452 22. "PARTNO" . 473. Table 10: USER STATEMENT_STRING TOTAL_LOCK_ WAIT_COUNT TOTAL_LOCK_ WAIT_DURATION TOTAL_ EXECU­ TION_TIME SYSTEM SELECT "FROMNUMBER","TONUMBER" ,"NRLEVEL" FROM "NRIV" ... FOR UPDATE 11,549,961 210,142,307,207 676,473 SYSTEM UPDATE "NRIV" SET "NRLEVEL" = ? WHERE "CLIENT" = '000' ... AND "TOYEAR" = '0000' 0 0 3,706,184 SYSTEM SELECT "DIMID" FROM "/BIC/DZDSTGCUBE4" WHERE "/B49/S_VERSION" = ? 0 0 460,991 Here, it is clear that the first statement is waiting almost all the time. Known as pessimistic/optimistic locking, the SELECT...FOR UPDATE code locks the resulting columns and may be replaced by a non-locking variant, which can result in poor performance. If the application is critical, it may be necessary to revise the SELECT...FOR UPDATE code for better resource utilization and performance. Execution in a Distributed System In distributed SAP HANA systems, tables and table partitions are located on multiple hosts. The execution of requests received from database clients may potentially have to be executed on multiple hosts, depending on where the requested data is located. The following example illustrates statement routing and how, if it is not enabled, requests from the database client are executed on the contacted index server (in this case the master index server) and the required data is fetched from the index server on the relevant host(s). However, if statement routing is enabled, after initial query compilation, request execution is routed directly to the host on which the required data is located. Figure 34: Distributed Execution with Statement Routing Off and On SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. 93 Execution times should be better with statement routing enabled. You can use the SQL plan cache to compare the execution statistics of statements with statement routing enabled and disabled and thus confirm the effect. Statement routing is controlled by the client_distribution_mode parameter in the indexserver.ini file. It is enabled by default (value=statement). The following SQL plan cache examples show the execution times of sample statements based on the scenario illustrated above. Note The column IS_DISTRIBUTED_EXECUTION indicates whether or not statement execution takes place on more than one host. 94 PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions The TOTAL_EXECUTION_TIME for a statement is the sum of execution times on all hosts, therefore: Table 11: Statement Request Path Total Execution Time UPSERT "RSBMONMESS_DTP" ( "MSGNO", "MSGTY", "MSGID", ... seltera12 = execution time on seltera12 SELECT * FROM "/BI0/SIOBJNM" WHERE "IOBJNM" = ? seltera12 > selbld13 = execution time on seltera12 + execu­ tion time on selbld13 SELECT * FROM "/B49/SCUSTOMER" WHERE "/B49/S_CUSTOMER" = ? seltera12 > seltera13 > selbld16 = execution time on seltera12 + execu­ tion time on selbld16 Statement Request Path Total Execution Time UPSERT "RSBMONMESS_DTP" ( "MSGNO", "MSGTY", "MSGID", ... seltera12 = execution time on seltera12 SELECT * FROM "/BI0/SIOBJNM" WHERE "IOBJNM" = ? selbld13 = execution time on selbld13 SELECT * FROM "/B49/SCUSTOMER" WHERE "/B49/S_CUSTOMER" = ? selbld16 = execution time on selbld16 Table 12: 3.9.8 Detailed Statement Analysis When you have identified a critical SQL statement and identified its overall key figures from the SQL plan cache analysis you can have a closer look at the actual runtime behavior of the SQL statement. The following tools can be used for a more detailed analysis: ● Plan Explanation ○ Creation of an execution plan ● Plan Visualizer ○ Detailed graphical execution plan ○ Temporal breakdown (timeline) available ● QO Trace SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. 95 no large data transfers to the application are required and you can use performanceenhancing features such as parallel execution. Procedures are used when other modeling objects. are not sufficient. you must create them manually. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .9 Optimization of Critical SQL Statements You can improve the general performance of the SAP HANA database by implementing various best practices. however. available features. All rights reserved. 96 PUBLIC © 2015 SAP SE or an SAP affiliate company. To enhance the performance of the SAP HANA database. SAP HANA can automatically recognize star queries and enable the performance benefits of using star schemas. design principles. SAP HANA automatically creates indexes for primary key columns. ● Create virtual star schemas on SAP HANA data by joining attribute views to tables that contain measures within the definition of an analytic view By using analytic views. if you need indexes for nonprimary key columns. we recommend you do the following: ● Optimize outlier queries Queries that sometimes take much longer than expected can be caused by query-external factors (for example. and add-ons. UPDATE and DELETE can be impacted by lock waits. Related Information Tools and Tracing [page 134] 3. such as analytic or attribute views. ● Create indexes for any non-primary key columns that are often queried.9. ● Develop procedures to embed data-intensive application logic into the database.○ Query optimizer trace ○ Advanced tool that can be useful to understand the decisions of the query optimizer ○ Particularly helpful to understand column searches ● JE Trace ○ Join evaluation trace ○ Advanced tool to analyze table join operations ● Performance trace ○ Low level recording of key performance indicators for individual SQL statement processing steps ○ Advanced tool that should only be used in collaboration with SAP support ● Kernel profiler ○ Sample based profiling of SAP HANA process activities ○ Advanced tool that should only be used in collaboration with SAP support All these tools are described in Tools and Tracing. ● Check data manipulation commands (DML) DML operations like INSERT. resource bottlenecks) that have to be determined and eliminated. ● Improve SQL query design You can significantly improve the performance of your SQL queries by knowing how the SAP HANA database and SAP HANA engines process queries and adapting your queries accordingly. such as reducing dependencies for query processing. With procedures. follow the best practices for using SQLScript. the SAP HANA default settings should be sufficient in almost any application scenario.○ If you use SQLscript to create procedures.9. parallelized approach. In general. ○ For statistical computing. Any modifications to the predefined system parameters should only be done after receiving explicit instruction from SAP Support. All rights reserved. Besides heavy load on the machine by non-SAP HANA processes (which should not be the case on production systems) SAP HANA itself can be under heavy load. Note With SAP HANA. Application functions are like database procedures written in C++ and called from outside to perform data intensive and complex operations. the better overall system performance is. This usually happens because extra work has to be performed during execution. Reasons include: ● Many users are connected and issue a large amount of statements ● Extraordinary expensive statements are executed ● Background tasks are running SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. ● Scale SAP HANA to improve performance. 3. SAP HANA's performance is derived from its efficient. ● Download and install the available application function libraries. The more computation cores your SAP HANA server has. 97 . you do not need to perform any tuning to achieve high performance.10 Outlier Queries Outlier queries are database statements that take much more time than usual and expected. such as Predictive Analysis Library (PAL) and Business Function Library (BFL). create procedures using the open source language R. All rights reserved. ● Savepoints: Savepoints consume resources and write-lock the database during their critical phase. To resolve the situation consider restricting the number of users on SAP HANA.Use the Load Monitor to determine the number of statements issued and the indexserver CPU usage while the slow statement execution was perceived ( See the figure Load Monitor in SAP HANA Studio. or get in touch with SAP Support if scalability can be improved in this case. ● Column Unloads: See Load Monitor Column Store Column Unloads and the Monitoring View M_CS_UNLOADS to look for signs of column unloads. upgrading the hardware. During that time frame you would look for something that consumed a lot of resources or blocked the statements (locking). If there was a high statement load in the same period when you experienced the slow execution the root cause is likely a lack of resources. If a column used in the problematic statement had to be loaded before execution. fast SQL statements the workload changed to a few "heavy" (complicated calculation requiring many CPU cycles) SQL statements. just after 15:35 you see that the CPU consumption increases while the SQL throughput decreases. Check M_SAVEPOINTS and look for savepoints during the time frame of the problem. the execution itself will take significantly longer. In that case try to improve the delta merge strategy to prevent merges happening in phases where users are disturbed (see the SAP HANA Administration Guide for details). Here. check for background activities: ● Delta Merges: Use Load Monitor Column Store Merge Requests and the monitoring view M_DELTA_MERGE_STATISTICS to check if delta merges happened. the lower line (red) is the CPU consumption in percent (%). the upper line (orange) is the SQL throughput / s): Figure 35: Load Monitor in SAP HANA Studio You can see that during the period in the red rectangle both CPU consumption and SQL throughput decreased. 98 PUBLIC © 2015 SAP SE or an SAP affiliate company. If it happens often. If a savepoint slowed down your execution. please contact SAP Support. a possible case would be a change in usage: instead of many small. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . If you did not experience a high statement load during the time frame of the problem. the chance of having the same problem again is very low. If there is a significant difference. All rights reserved.9. there was at least one situation where the transactions took much longer than average. 99 .Related Information SAP HANA Administration Guide M_DELTA_MERGE_STATISTICS M_CS_UNLOADS M_SAVEPOINTS 3. compare the column MAX_CURSOR_DURATION against AVG_CURSOR_DURATION. See Transactional Problems for information on how to deal with such issues. For those entries. Check under issues: Performance SQL Plan Cache and the view M_SQL_PLAN_CACHE to determine such Figure 36: Plan Cache Monitor Note Only check entries that have TOTAL_LOCK_WAIT_COUNT greater than 0. Related Information M_SQL_PLAN_CACHE Transactional Problems [page 73] SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company.11 Data Manipulation Language (DML) Statements Data Manipulation Language (DML) statements are often slowed down by lock-wait situations. 9. For instance. One way to avoid the cost of implicit type casting is by using explicit type casting on an inexpensive part of the query. The system can generate type castings implicitly even if the user did not explicitly write type casting operation. it should be noted that adding generated columns improves query performance at the expense of increased memory consumption. You can find the type precedence rule in the SAP HANA SQL and System Views Referencel.3. it is better to enforce that the two columns have the same data type from the beginning. 'YYYYMMDD'). it could be compared with a string generated from a DATE value in the form of 'YYYYMMDD'. However. the system generates an implicit type casting operation that casts the VARCHAR value into a DATE value. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . In the example below. If two columns are frequently compared by queries. and increased insertion and update cost. Implicit type casting is done from lower precedence type to higher precedence type. if there is a comparison between a VARCHAR value and a DATE value. 100 PUBLIC © 2015 SAP SE or an SAP affiliate company. one way to avoid implicit type casting entirely is to add generated columns. As a general guideline for improving SQL query performance. date_string is a VARCHAR column. if a VARCHAR column is compared with a DATE value and the user knows that casting the DATE value into a VARCHAR value produces what the user wants. Note that comparison result between strings is different from that between dates in general and identical only in some cases. adding generated columns will be mentioned as a possible workaround to improve performance of the query. 3. we recommend avoiding operations that are not natively supported by the various SAP HANA engines since they can significantly increase the time required to process queries.9.1 Avoid Implicit Type Casting in Queries You can avoid implicit type casting by instead using explicit type casting or by adding generated columns. Please note that following specific recommendations may help to improve the performance of SQL queries involving column tables.12. it is recommended to cast the DATE value into a VARCHAR value. All rights reserved. SELECT * FROM T WHERE date_string < TO_VARCHAR(CURRENT_DATE. You should be aware of this trade-off before deciding to add generated columns.12 SQL Query Tuning Tips Performance of SQL queries can be improved significantly by knowing how the SAP HANA database and SAP HANA engines process queries and adapting queries accordingly. For example. Caution Throughout this section. If there is no way to avoid implicit type casting. if the VARCHAR column contains only date values in the form of 'YYYYMMDD'. Table 13: Problematic Query Workaround SELECT * FROM T WHERE date_string < CURRENT_DATE. hashed-range) after materializing the intermediate results from both children.0' and '1. When such join predicates are used. PUBLIC © 2015 SAP SE or an SAP affiliate company. M. Table 15: Problematic Query SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions SELECT M. 101 . SELECT * FROM T WHERE num_value = 1. if equijoin predicates are connected by AND with non-equijoin predicates.year. Depending on the condition.9. An example of rewriting a non-equijoin predicate into an equijoin predicate is shown below. intermediate results will be materialized and consumed by the row engine or column engine.12. It is always a good practice to try to avoid intermediate result materialization since materialization can be costly if results are large and have significant impact on the performance of the query.2 Avoid Inefficient Predicates in JOIN This section will list out predicate conditions that are not natively supported by column engine. the row engine executes the join operation using the appropriate join algorithm (nested-loop.9.month.last_date GROUP BY M. In the example.12.2. The column engine does not natively support an outer join with join predicates other than equality condition (that is. range. Table 14: Problematic Query Workaround SELECT * FROM T WHERE varchar_value = 1 ALTER TABLE T ADD (num_value DECIMAL GENERATED ALWAYS AS varchar_value). Also.first_date AND M. SUM(T. M is a table containing first and last dates of months of interest.year. it is processed in the same way as the case with only non-equijoin predicates. '1.1 Non-equijoin Predicate in Outer Join The column engine natively support an outer join with a join predicate of equality and join predicates connected by OR.00' stored in a VARCHAR column using the revised query and avoid implicit casting. 3. non-equijoin) and join predicates connected by OR (even if each predicate is an equality condition). All rights reserved. user can find '1'. 3. M.In the example below.ship_date BETWEEN M.month.ship_amount) FROM T LEFT OUTER JOIN M ON T. An example of avoiding calculated join columns using generated columns is shown below.12. 3. For such queries. M.ship_amount) FROM T LEFT OUTER JOIN M ON EXTRACT(YEAR FROM T. Calculation with parameters are not natively supported by column engine. most of calculations are natively supported by column engine.year. ALTER TABLE T ADD (next_day DATE GENERATED ALWAYS AS ADD_DAYS(ship_date.last_date. SELECT * FROM T JOIN M ON T. intermediate results will be materialized and will be consumed by column engine.year AND EXTRACT(MONTH FROM T. M. We recommend trying to rewrite the query to avoid such conditions.last_date. All rights reserved.9.target_amount. As of SPS10. SELECT * FROM T JOIN M ON T. This will result in materialization of the intermediate results which will be consumed by row engine. One way to avoid calculation on query processing is to add generated columns. calculation involving multiple tables are not natively supported by column engine. For such cases. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .year.Workaround SELECT M. M is a table containing first and last dates of months of interest.ship_date) = M. the intermediate results from both children are materialized and consumed by row engine.ship_date) = M.2. Table 17: Predicate with parameter is not natively supported 102 PUBLIC © 2015 SAP SE or an SAP affiliate company. there can be performance impact if the amount of materialization is large. 1)).2 Calculated Column Predicates Intermediate results from calculations that are not natively supported by column engine are materialized but can be avoided using generated columns.month.next_day = M. Table 16: Problematic Query Workaround SELECT * FROM T JOIN M ON NEXT_DAY(T. SUM(T.month. But for a calculation that is not natively supported by column engine.month GROUP BY M. One way to avoid such calculation is to maintain needed columns in different tables by changing schema design.ship_amount + ? = M. if possible. Also.ship_date) = M. In the example. nation. WITH HINT(CYCLIC_JOIN) --prefer cyclic join WITH HINT(NO_CYCLIC_JOIN ) --break cyclic join SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company.ship_amount. The following hints can be tried to optimize performance for cyclic inner join. One way of breaking cycle is to maintain needed columns in different tables by changing schema design. nation column of supplier table is moved to a line-item table. For the acyclic join in the example below. lineitem L WHERE L.supp_key = S. WITH HINT(CS_EXPR_JOIN) --prefer column engine calulation WITH HINT(NO_CS_EXPR_JOIN) --avoid column engine calculation 3.key AND S.9. Column engine does supports cyclic inner join natively.cust_key = C. lineitem L WHERE L. 103 .Calculation involving multiple tables is not natively sup­ ported SELECT * FROM T JOIN M ON IFNULL(T.nation = C. SQL optimizer selects cyclic inner join based on cost and it is sometime worthwhile to try hints to guide optimizer to break cyclic join into two column searches and vice versa. but it is better to avoid it because its performance is inferior to acyclic inner join. the following hints can be tried to optimize performance for predicates involving calculated columns.supp_key = S.key AND L.key AND L. customer C. Table 18: Cyclic join Acyclic join SELECT * FROM supplier S.supp_nation = C. customer C.key AND L.target_amount. All rights reserved. the result from a child of the join that completes the cycle is materialized to break the cycle. If applicable.2.12.pre_order_amount) = M. M.nation.cust_key = C. SELECT * FROM supplier S.3 Cyclic Join The column engine does not natively support join trees that have cycles in join edges if an outer join is involved in the cycle (also known as cyclic outer join) If there is such a cycle involving an outer join. b = 1) S ON T. Moving those predicates upwards in the outer join produces the same results and such a move is automatically done by the SQL optimizer. 104 PUBLIC © 2015 SAP SE or an SAP affiliate company. If filter predicates are used inside outer join predicates and they cannot be moved below join. Filter predicates over the right child of left outer join and filter predicates over the left child of a right outer join are exceptions because moving those predicates below a join produces the same results.9. Below is an example of the filter predicate that triggers materialization of intermediate result. 3.b = 1. Table 19: Filter that is natively supported Filter that is not natively supported SELECT * FROM T LEFT JOIN S ON T.9.3.a AND T. (equivalent to SELECT * FROM T LEFT JOIN (SELECT * FROM S WHERE S.a = S.b = 1.a = S.a AND T. the result from the child including the predicate is materialized before executing the join.a AND S.a = S. Table 20: Problematic query Workaround SELECT * FROM T LEFT JOIN S ON T.4 Filter Predicate Inside an Outer Join The column engine does not natively support filter predicates inside outer join predicates.a) SELECT * FROM T LEFT JOIN S ON T. row engine executes the join operation after materializing intermediate results from both children. All rights reserved.5 Subquery Filter Predicate Accessing Multiple Tables Inside Outer Join Filter predicates over multiple tables are not natively supported by column engine if they are under an outer join.2.one.b = 1.b = S. One way of avoiding such materialization in the example would be maintaining priority column in the lineitem table instead of orders table.a = S.2.12.a AND T.12. Filter predicates over the left child of a left outer join and filter predicates over the right child of a right outer join are exceptions. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . SELECT * FROM T LEFT JOIN S ON T. An example of rewriting such filter predicates into equijoin predicates using a generated column is shown below. ALTER TABLE S ADD (one INTEGER GENERATED ALWAYS AS 1).a = S. If such a filter predicate exists. order_key.order_key ) ON C. if a calculation cannot guarantee that NULL will not be returned in the result.6 Subquery Projection Of Constant Or Calculated Value Below Outer Join The column engine does not natively support constant or calculated value projection below outer joins.9.cust_key = L.order_key = L.order_key ) ON C. Constant or calculated value projection over left child of left outer join or right child of right outer join are exceptions because moving such projections above join produces the same results and such move is automatically done by SQL optimizer.12. 0) FROM orders O JOIN lineitem L ON O. All rights reserved.shipmode = 'AIR' OR O.2. then intermediate result will not be materialized. SELECT * FROM customer C LEFT OUTER JOIN ( SELECT coalesce(L. Also.order_key WHERE L. Below is an example where coalesce was used to guarantee that order_key will not return NULL which triggers materialization. If constant or calculated value projection exists.cust_key = L.order_key. A possible workaround of avoiding materialization of intermediate result is by adding a generated column for the constants or calculated values.cust_key = L. SELECT * FROM customer C LEFT OUTER JOIN ( SELECT * FROM lineitem L WHERE L.Table 21: Problematic query Workaround SELECT * FROM customer C LEFT OUTER JOIN ( SELECT * FROM orders O JOIN lineitem L ON O. 105 . Table 22: Calculation that will not trigger materialization Calculation that will trigger materialization SELECT * FROM customer C LEFT OUTER JOIN ( SELECT L.order_key = L.shipmode = 'AIR' OR L.cust_key = L.cust_key. the result from the child including the projection is materialized before executing the join. 3.order_key.order_key FROM orders O JOIN lineitem L ON O.priority = 'URGENT') ON C.order_key = L.cust_key.priority = 'URGENT') ON C. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. 3 Avoid Inefficient Predicates in EXISTS/IN This section will list inefficient predicate conditions.const ALTER TABLE ORDERS ADD (const INTEGER GENERATED ALWAYS AS 1).9.12.a = T. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .order_key ) ON C.Table 23: Problematic query Workaround SELECT * FROM JOIN ( SELECT 1 JOIN lineitem L. Since left outer join processing is more expensive than inner join processing in general.12. Table 24: Problematic query Workaround SELECT * FROM T WHERE EXISTS (SELECT * FROM S WHERE S.3. we recommend avoiding such disjunctive EXISTS predicates if possible.const.order_key = ON C.order_key ) customer C LEFT OUTER const FROM orders O L ON O.a = T. it is internally mapped to left outer join.b = 2). 3.cust_key = L. Below is an example of avoiding disjunctive EXISTS predicate.9. Also.b = 2)). SELECT * FROM T WHERE EXISTS (SELECT * FROM S WHERE S. avoid using OR to connect EXISTS or NOT EXISTS predicates with other predicates if possible. SELECT * FROM customer C LEFT OUTER JOIN ( SELECT * FROM orders O JOIN lineitem L ON O. When an EXISTS or NOT EXISTS predicate is connected with other predicates through OR. 3.1 Disjunctive EXISTS Predicate If possible avoid disjunctive EXISTS predicates.a AND S.a AND S. This tuning tip also applies to the following cases: ● Filter predicate inside NOT EXISTS predicate accessing multiple tables ● Filter predicate inside disjunctive EXISTS predicate accessing multiple tables 106 PUBLIC © 2015 SAP SE or an SAP affiliate company.a AND (S.b = 1 OR S. All rights reserved.b = 1) OR EXISTS (SELECT * FROM S WHERE S.order_key = L.a = T.cust_key = L. 9. All rights reserved. S. SELECT * FROM T WHERE NOT EXISTS (SELECT * FROM S WHERE S.b) b FROM T FULL OUTER JOIN S ON T.3. SELECT a. b FROM T UNION SELECT a. However. for NOT IN.a). with NOT EXISTS. true or false will be returned while checking for provided condition. Unless subquery results is very small.a) a.b. S. The transformation is automatically applied by SQL optimizer if all columns of interest have NOT NULL constraints declared explicitly. It is valid only if there is no null values in the columns of interest. PUBLIC © 2015 SAP SE or an SAP affiliate company. Please note that the transformation in the example is not valid in general. avoiding them may improve performance. INTERSECT and EXCEPT are not natively supported by the column engine.2 NOT IN Predicate The NOT IN predicate is much more expensive to process than NOT EXISTS. INTERSECT and EXCEPT are shown below. UNION. The transformations for INTERSECT and EXCEPT are automatically applied by SQL optimizer if all columns of interest have NOT NULL constraints declared explicitly.b. using NOT EXISTS is much faster than NOT IN.a AND T.a. entire subquery is processed first before the processing the query as a whole. Below is an example of avoiding the NOT IN predicate. Examples of avoiding UNION.9. Note that the transformations in the examples are not valid in general.a = T. b FROM S. Table 25: NOT IN query Possibly equivalent query in some cases SELECT * FROM T WHERE a NOT IN (SELECT a FROM S).3. b FROM T INTERSECT SELECT a.b = S. 107 .4 Avoid Set Operations UNION ALL.12. SELECT DISTINCT COALESCE(T.a = S. b FROM S. matching up based on the condition provided. Table 26: UNION query Possibly equivalent query in some cases INTERSECT query SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions SELECT a. COALESCE(T. it is recommended to use NOT EXISTS instead of NOT IN if possible. In general.12. They are valid only if there is no null value in the columns of interest. 3. ( Same applies for EXISTS/IN ). 3.9. please note that creation of the index will increase memory consumption. M.a = S.b = S. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .year = M. However. Below is an example of query that needs concatenated columns and syntax to create the needed concatenated columns.ship_amount) FROM T JOIN M ON T. T.Possibly equivalent query in some cases EXCEPT query Possibly equivalent query in some cases SELECT DISTINCT T.a AND T. CREATE INDEX M_year_month ON M(year. SUM(T.a = S. month).6 Using Hints to Alter a Query Plan This section lists hints that may be used to alter a query plan.a a.month = M.5 Improve Performance For Multiple Column Join Creating a concatenated column index can improve query performance when multiple columns are involved in a join. M.12. Table 27: Problematic Query Workaround SELECT M.year AND T.month.b b FROM T WHERE NOT EXISTS (SELECT * FROM S WHERE T.12. month). T. The following hints can be tried to optimize performance for multiple column join predicate: WITH HINT(OPTIMIZE_METAMODEL) --prefer creation of concatenated attribute during query processing time WITH HINT(NO_OPTIMIZE_METAMODEL) --avoid creation of concatenated attribute 3. SELECT DISTINCT T.9. All rights reserved.a a.a AND T. b FROM T EXCEPT SELECT a.b.b).b b FROM T JOIN S ON T. CREATE INDEX T_year_month ON T(year. SELECT a. 108 PUBLIC © 2015 SAP SE or an SAP affiliate company.year. One way to optimize this type of query is to create concatenated column index explicitly.b = S.year. b FROM S.month GROUP BY M.month. 3.9.12.6.1 Changing an Execution Engine Decision You can use hints to change which engine is used to execute your query. The SQL optimizer chooses a column search execution engine (Join Engine or Olap Engine) based on the costmodel. For various reasons, optimal plan may not be executed using the best engine. The following hints can be applied to change the execution engine used. WITH HINT(USE_OLAP_PLAN) -- to prefer OLAP engine (if possible) WITH HINT(NO_USE_OLAP_PLAN) -- to prefer JOIN engine 3.9.12.6.2 Changing Query Transformation Hints can be used to apply or remove query transformation rules. By examining the plan, you can apply different hints to alter query transformation as needed. By using NO_<hint>, then can be disabled to produce opposite effect. For a full list of available hints, please refer to HINT Details in the SAP HANA SQL and System Views Reference. Table 28: Hint: GROUPING_SIMPLIFICATION Before After Table 29: Hint: GROUPING_REMOVAL Before SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions After PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. 109 Related Information SAP HANA SQL and System Views Reference 3.9.12.6.3 Changing Operator Order Hints can be used to change the order of operators during plan generation. By examining the plan, you can apply different hints to change the operator order as needed. By using NO_<hint>, they can be disabled to produce opposite effect. For a full list of available hints, please refer to HINT Details in the SAP HANA SQL and System Views Reference. Table 30: Hint: AGGR_THRU_JOIN Before After Table 31: Hint: PREAGGR_BEFORE_JOIN Before 110 PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. After SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions Table 32: Hint: FILTER_THRU_JOIN Before After Related Information SAP HANA SQL and System Views Reference 3.9.12.6.4 Choosing Preferred Algorithm Hints can be used to select preferred algorithms for execution ( column engine vs row engine ). By using NO_<hint>, they can be disabled to produce opposite effect. For a full list of available hints, please refer to HINT Details in the SAP HANA SQL and System Views Reference. Table 33: Hint: CS_JOIN Before SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions After PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. 111 Table 34: Hint: CS_ORDERBY Before After Table 35: Hint: RANGE_JOIN Before After Related Information SAP HANA SQL and System Views Reference 3. If you want to persist the binding between a query and hints or you are unable to append hints to a query during runtime (that is application generated queries ). 112 PUBLIC © 2015 SAP SE or an SAP affiliate company. please refer to HINT Details in the SAP HANA SQL and System Views Reference. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .6. All rights reserved. a hint table can be used.5 Using Alter System Command To Pin Hints To Problematic Query A hint table can be used to persist the binding between a query and hints.9. For a full list of available hints.12. The indexAdvisor.py script is part of a SAP HANA system installation and runs from the command line. Use the index adviser to find out for which tables and columns indexing would be most valuable. To create indexes on non-primary columns. The higher the selectivity is. the higher are the performance gains from indexing the columns. particularly highly selective queries on non-primary key columns. in the Plan Visualizer. There is a trade-off between indexing and memory consumption: While indexing non-primary key columns can make query execution faster. in the properties of a column. It is located in the $DIR_INSTANCE/exe/python_support directory. you can also see the system view M_INDEXES. that is. SAP HANA automatically creates indexes for all primary key columns. When filter conditions are on non-key fields and tables have many records. 113 . The index adviser takes this trade-off into account: In dynamic mode. the more different values are in the column.13 Creation of Indexes on Non-Primary Key Columns Create indexes on non-primary key columns to enhance the performance of some queries using the index adviser. Alternatively.9. use the following SQL statement: CREATE INDEX <name> ON <table> (<column>) Related Information SAP HANA SQL and System Views Reference SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. You can create indexes on non-primary key columns to enhance the performance of some queries. Tip To check whether there is an index for a column. the index adviser looks for the tables and columns that are used most often.Related Information SAP HANA SQL and System Views Reference 3. creating an index on the non-primary key columns may improve the performance. the downside is that memory consumption increases. Indexing the primary key columns is usually sufficient because queries typically put filter conditions on primary key columns. All rights reserved. see the Inverted Index entry. Select the star join node. Choose OK. price. In the context menu of the package. d. In Find Data Sources dialog. In Find Data Sources dialog. In the context menu. Define the central fact table in the data foundation node. enter the name of the data source. Provide name and description. 3. star join node and the data foundation. e. 6. c. a. Launch SAP HANA studio. and so on.14 Create Analytic Views Analytic views are used to model data that includes measures. a. Procedure 1. 8. d. All rights reserved. enter the name of the data source and select it from the list. Choose OK. Modeler launches a new analytiv view editor with the semantics node. Note You can to add the same table again in Data Foundation using table aliases in the editor. 7. choose Add Objects. For example. Choose Finish. select a package where you want to create the new calculation view. 5. c. b. Add data sources. b. transactional fact table representing sales order history would include measures for quantity. In SAP HANA System view. expand the content node. Select the data foundation node. Continue modeling the analytic view with a cube structure. 114 PUBLIC © 2015 SAP SE or an SAP affiliate company. Note You can only dimensions (attribute views) as a data source in star join node of analytic views. select New Analytic View . Note You cannot add column views to the Data Foundation. In the navigator. which includes attributes and measures. choose Add Objects. The input to the star join node must provide the central fact table.3. Select the required attribute view from the list. 4. In the context menu. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .9. 2. a. in the context menu of the data source. 9. create joins to join the attribute views with the fact table (star schema). Selec the Data foundation node and define the property. A join path is the set of joins that links the fact table to other tables. If you want to hide the measure of attribute in the reporting tool. e.to activate the current view and redeploy the affected objects if an active version of the affected object exists. b. All rights reserved. choose Add To Output. choose Add All To Output. If there are more than one table in the data foundation node.to activate the current view along with the required and affected objects. Define output columns. select a measure or attribute. Activate the analytic view: ○ If you are in the SAP HANA Modeler perspective: ○ Save and Activate . c. d. select a output column. In the context menu. Select the data foundation node or star join node. Select the Semantics node. 12. The activation triggers validation check for both the client side and the server side rules. select the columns that you want to add to the output of the node. In the Local section. Note The Shared tab page shows attributes from the attribute views that are used in the analytic view. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. ensure that a table does not appear twice in any join path. Choose the Columns tab. Note You can also activate the current view by selecting the view in the SAP HANA Systems view and choosing Activate in the context menu. Note If you want to add all columns from the data source to the output. While creating joins. the object activation fails. a. While creating joins between analytic view and attribute view: ○ The same table cannot be used in the join path of analytic view and attribute view ○ The table of the attribute view which is linked to the fact table should not have an alias table 11. Define attributes and measures a. Restriction Self-joins are not supported. 10. Otherwise only current view gets activated. select the Hidden checkbox. ○ Save and Activate All . In the Details pane.e. You can also create a temporal joins between date fields of the fact table to an interval (to and from) field of the attribute view. If the object does not meet any validation check. In the Star Join node. specify the central table (fact table) from which the modeler must derive the measures. In the Type dropdown list. Central Entity in the Properties pane. 115 . b. then close and reopen the analytic view editor to see any changes that you have made to the attribute view. in the SAP HANA Development perspective. the current view activation might go through. Create Counters 116 PUBLIC © 2015 SAP SE or an SAP affiliate company. 13. Note If an active version of the affected objects exist. However. select the required object. Table 37: Working With Attributes and Measures Requirement Task to perform If you want to count the count the number of distinct values for a set of attribute col­ umns. Table 36: Working With View Nodes Requirement Task to Perform If you want to filter the output of the data foundation node. if any of the affected objects redeployment fails. Hence. For more information. Choose Finish. In the Project Explorer view. Note While modeling an analytic view.○ If you are in the SAP HANA Development perspective: 1. In the context menu. Assign Changes a. if there are any errors on the client side. they are skipped and the object activation goes through if no error found at the server side. either create a new ID or select an existing change ID that you want to use to assign your changes. select Team Activate . 14. activating the current view redeploys the affected objects. Note The activation triggers the validation check only for the server side rules. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . Choose Finish. In the SAP HANA Modeler perspective. the current view activation also fails. see chapter SAP HANA Change Recording of the SAP HANA Developer Guide. All rights reserved. b. you can perform certain additional tasks to obtain the desired output. even if the affected object redeployment fails. Filter Output of Data Foundation Node. In the Select Change dialog. For more information on assigning changes. see SAP Note 1783668 . Next Steps After creating an analytic view. if you have also opened and edited an attribute view that is used in the analytic view. The table below lists the additional tasks that you can perform to enrich the analytic view. 2. such as for financials. ● SQL does not provide for imperative logic. All rights reserved. Associate Measures with Currency If you want associate measures with unit of measures and perform unit conversions. are not sufficient. for example. Assign Variables If you want associate measures with currency codes and perform currency conver­ sions.15 Developing Procedures SQL in SAP HANA includes extensions for creating procedures. Enable Information Views for Time Travel Queries If you want to invalidate or remove data from the cache after specific time intervals. eliminating the need to explicitly create temporary tables for intermediate results. ● SQL queries return one result set.Requirement Task to perform If you want to create new output columns and calculate its values at runtime using an expression. ● Complex SQL statements can be hard to understand and maintain. Create Input Parameters If you want to. such as analytic or attribute views. Procedures can return multiple result sets. Associate Measures with Unit of Measure If you want to group related measures together in a folder. Create Restricted Columns If you want to assign semantic types to provide more meaning to attributes and measures in analytic views. Some of the reasons to use procedures instead of standard SQL: ● SQL is not designed for complex calculations. Invalidate Cached Content If you want to maintain object label texts in different languages. Filter Data for Specific Clients If you want to execute time travel queries on analytic views. filter the results based on the values that users provide to attributes at runtime. Group Related Measures. ● Procedures can have local variables. Procedures can be written in the following languages: SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. Maintain Modeler Objects in Multi­ ple Languages If you do not recommend using an anlytic view. Assign Semantics If you want to parameterize attribute views and execute them based on the values users provide at query runtime. Table 38: Working With Ananlytic View Properties Requirement Task to perform If you want to filter the view data either using a fixed client value or using a session client set for the user. which enables you to embed data-intensive application logic into the database. Deprecate Information Views 3. 117 . Procedures are used when other modeling objects. where it can be optimized for performance (since there are no large data transfers to the application and features such as parallel execution is possible).9. Create Calculated Columns If you want to restrict measure values based on attribute restrictions. context specific schemas. and textual input. and debug stored procedures. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . Local Variables 2. Tools for Developing Procedures The following describes the tools integrated with SAP HANA studio that allow you to evaluate. Keywords Note Objects selected from the proposed list might be automatically inserted as quoted identifier based on the SQLScript language guidelines For example. synonyms ● Local Variables: such as input and output parameters. scalar functions. declared scalar variables ● Database Artifacts Suggested objects are listed alphabetically. procedures. Code completion proposals take into consideration SQLScript grammar. All rights reserved. table functions. You can quickly identify valid objects reducing errors during activation. edit. which can be installed and integrated with SAP HANA.artifact_type (artifact_context). according to the following format: [icon] artifact_name .● SQLScript: The language that SAP HANA provides for writing procedures. views. or lower and upper case characters. There are additional libraries of procedures.Table (<Schema name>) The list of proposals contain syntactic and semantic proposals listed in the following order: 1. Within the SQLScript editor you can use the Semantic Code Completion feature. revise. 118 PUBLIC © 2015 SAP SE or an SAP affiliate company. called Business Function Library and Predictive Analysis Library. if the object contains special characters. for example DUMMY . The semantic code completion feature is a context based search tool that lists suggested catalog object and local variables that assist you with developing accurate stored procedures in a faster and more efficient matter. ● R: An open-source programming language for statistical computing and graphics. The suggested objects are derived from the following origins: ● Catalog Objects: such as schemas. that can be called via SQL or from within another procedure. and optimize stored procedures: SQLScript Editor The SQLScript editor studio allows you to create. Catalog Objects (maximum of 50 suggestions) 3. name VARCHAR(50). The LANGUAGE clause specifies the language you are using to code the procedure. NULL). ● CREATE PROCEDURE: Creates a procedure. A local table type is created within the SAP HANA Systems procedure tool and for only the current procedure. END. Related Information SAP HANA SQL and System Views Reference SAP HANA SQLScript Reference SAP HANA R Integration Guide SAP HANA Business Function Library (BFL) Reference SAP HANA Predictive Analysis Library (PAL) Reference SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. the SQLScript editor includes debugging capabilities. For example: CREATE PROCEDURE ProcWithResultView(IN id INT. For example: CREATE TYPE tt_publishers AS TABLE ( publisher INTEGER. then use the SQLScript Editor. You should only use this method: ● If you need to develop a procedure using a local table type as an input or output parameter. NULL. but there are no debugging capabilities. 'EUR'. For example: CALL getOutput (1000. SQL Extensions for Procedures SQL includes the following statements for enabling procedures: ● CREATE TYPE: Creates a table types. which are used to define parameters for a procedure that represent tabular results. ● CALL: Calls a procedure. All rights reserved. ● If you need to edit a procedure previously created in the Navigator view that contains table type parameters. You can also use the Navigator view in the Modeler perspective to build procedures. price DECIMAL. OUT o1 CUSTOMER) LANGUAGE SQLSCRIPT READS SQL DATA WITH RESULT VIEW ProcView AS BEGIN o1 = SELECT * FROM CUSTOMER WHERE CUST_ID = :id. cnt INTEGER). 119 .Debugger In addition to the creating and editing procedures. If you can use a global table type. the system creates the user _SYS_AFL. packages. even when an area with all its objects is dropped and recreated during system start-up. The role for each library is named: AFL__SYS_AFL_<AREA NAME>_EXECUTE. You must be assigned this role to execute the functions in the library. Note There are 2 underscores between AFL and SYS. Note The user and its schema _SYS_AFL are created during a new installation or update process if they do not already exist. all these objects have fully specified names in the form of _SYS_AFL. All AFL objects. and must be installed separately by an administrator. SAP HANA provides several techniques to move application logic into the database. there is a role. All rights reserved. In other words. the user still keeps the role that was previously granted. ● Roles For each AFL library. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . see the SAP HANA Server Installation and Update Guide. are created under this user and schema.SAR). Application functions are like database procedures written in C+ + and called from outside to perform data intensive and complex operations. whose default schema _SYS_AFL. Note Once a role is created.9. Functions for a particular topic are grouped into an application function library (AFL). Note The AFL archive is not part of the SAP HANA appliance. one SAR file with the name AFL<version_string>. such as the Predictive Analytical Library (PAL) or the Business Function Library (BFL).<object name>. the role for executing PAL functions is AFL__SYS_AFL_AFLPAL_EXECUTE. functions. Security Considerations ● User and Schema During startup. 120 PUBLIC © 2015 SAP SE or an SAP affiliate company. Currently. all AFLs are delivered in one archive (that is. For example.16 Application Function Library (AFL) You can dramatically increase performance by executing complex computations in the database instead of at the application sever level. it cannot be dropped. and one of the most important is the use of application functions. such as areas. For more information about installing the AFL archive. Therefore. and procedures.3. applications.17 About Scalability Scalability has a number of aspects that need to be considered such as scaling data. a rough prediction of scalability can be established by measuring the average CPU utilization while the workload is running.Related Information SAP HANA Server Installation and Update Guide 3. For example. Partitioning is transparent for most SQL queries and other data manipulations. All rights reserved. for example. Using simulations and estimations of your typical query workloads. the better overall system performance. an average CPU utilization of 45% may indicate that the system can be loaded 2X before showing a significant reduction in individual query response time. parallelized approach. you can scale out over multiple machines to create a distributed SAP HANA system. performance. The most important strategy for scaling your data is data partitioning. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company.9. Particular care must be taken in distributing the data so that the majority of queries match partitioning pruning rules. to set the allocation limit according to your needs. this is not always possible. and then to increase it over time to adapt to your data. Partitioning supports the creation of very large tables (billions of rows) by breaking them into smaller chunks that can be placed on different machines. The more computation cores your SAP HANA server has. Scaling Performance SAP HANA's performance is derived from its efficient. when a single fact table is larger than the server's RAM size. This accomplishes two goals: directing different users to different hosts (load balancing) and avoiding the network overhead related to frequent data joins across hosts. Once you have reached the physical limits of a single server. Scaling the Data One technique you can use to deal with planned data growth is to purchase more physical RAM than is initially required. 121 . and hardware. you can determine the expected load that a typical SAP HANA installation may comfortably manage. However. Scaling the Application Partitioning can be used to scale the application as it supports an increasing number of concurrent sessions and complex analytical queries by spreading the calculations across multiple hosts. Scaling performance requires a more detailed understanding of your workload and performance expectations. You can do this by distributing different schemas and tables to different servers (complete data and user separation). At the workload level. 122 PUBLIC © 2015 SAP SE or an SAP affiliate company.000 records or greater. When you have a SAP HANA-based application with unsatisfying performance you should start a systematic investigation considering the various layers that are involved.9. for column tables. You can also use TABLE LOCK. ● When inserting or loading a large number of rows into a table that has a TEXT or SHORTTEXT column or uses a FULLTEXT INDEX. you can use further features and best practices to improve performance. SAP recommends that you scale up as far as possible (acquire the configuration with the highest processor and memory specification for the application workload). Therefore. Tables locks are only recommended for initial loads.10 Application Performance Analysis This section provides a high-level guide for analyzing the performance of SAP HANA-based applications. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . delivered in a number of different configurations and "sizes" by certified hardware partners or by using the tailored data center integration model. ● For multicolumn join scenarios. If appropriate. This creates different system design options with respect to scale-up and scale-out variations. use dynamic joins rather than standard joins. the elements of a join condition between two data sources are defined dynamically based on the fields requested by the client query. In general. approaches for analyzing UI and application server performance are described in the respective documents. ● When loading data from CSV files using the IMPORT FROM command. Note The SAP HANA hardware partners have different building blocks for their scale-out implementations. use THREADS and BATCH to enable parallel loading and commit many records at once. with a commit frequency of 10. Here we focus on the database layer. 3. It is used to improve the performance by reducing the number of records to be processed by the join node. you should always consult with your hardware partner when planning your scale-out strategy. you can take the following actions to improve performance: ● For any required long-running transactions. which locks the entire table and bypasses the delta table. and as part of a cloud-based service. you can use the SQL command ALTER SYSTEM RECLAIM VERSION SPACE to trigger the row-store garbage collector to free up memory space and enhance system responsiveness. In a dynamic join. merge the delta of the table for better search performance. a good setting to use is 10 parallel loading threads. To maximize performance and throughput. before scaling out (for deployments with even greater data volume requirements). All rights reserved. for specific scenarios. 3.Scaling Hardware SAP HANA is offered in a number of ways – in the form of an on premise appliance.18 Further Recommendations In addition to the general recommendations for improving SAP HANA database performance. Start the tracing of database calls. SAP HANA studio and SAP HANA Web-based Development Workbench provide two main tracing tools. so you should further investigate those calls Related Information Plan Trace [page 160] Analyzing SQL Traces [page 141] Analysis Tools in SAP HANA Web-based Developer Workbench [page 169] SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. Both.3. 123 . statements that need to be optimized over and over again ○ One or more database calls with unexpected bad performance. This should be done within the context of a given user interface step or transaction.1 SQL Trace Analysis The first step in application performance analysis is to figure out if the database layer is causing performance problems for your application at all. Deep tracing (including complete execution plans) is provided by Plan Trace in SAP HANA studio. 2. A convenient way to narrow the trace analysis to the scope of a user interface step or transaction is to use the passport-based filtering of Expensive statements trace in the SAP HANA Web-based Development Workbench. As a result of this investigation you might see some indicators for bad application logic creating excessive load on the database such as: ○ Too many database calls (per transaction/UI step) ○ Many identical executions. All rights reserved. that is. Procedure 1. Terminate the tracing and review aggregated and individual results.10. 3. namely SQL trace and Expensive statements trace. 4. Run the application from its user interface or with any other driver. which also offers aggregated statistics to quickly answer above questions. Context You should analyze how many and which database calls are made and what their contribution to the overall application performance is. for example repeated identical selects ○ Too many records returned (per execution or in total) ○ Too many columns or all columns of a row selected ○ Inefficient statement reuse. 2.3. 4. Procedure 1. such as high load. In SAP HANA studio you have to use the administration perspective in order to examine the system state. 5. high resource usage and so on 3. Execute your statement(s) and measure their performance (in particular response time). Repeat your measurements until you get stable results without major variations (for example. Results As a result of this activity you have reliable data for your query performance. Once you have a stable result you may also acquire a detailed SAP HANA engine trace which will allow for a deeper analysis. Both. In case of disturbing conditions repeat your measurement from step 1. In addition. the SAP HANA Web-based Development Workbench implicitly supports observing the system state and also executing repeated measurements. Related Information Statement Performance Analysis [page 86] 124 PUBLIC © 2015 SAP SE or an SAP affiliate company. SAP HANA studio and SAP HANA Web-based Development Workbench offer basic measurement of SQL statements. Check your SAP HANA system status for disturbing conditions. both for initial query execution performance (possibly cold execution) and stabilized execution performance (warm execution). All rights reserved. Note Higher initial response times could be an indicator of caches that are not properly filled. 3 stable executions in a row). SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .10. Depending on your business needs you can decide whether this is acceptable or not.2 Statement Measurement Once you have determined which SQL statements are problematic you should first perform a sound measurement in order to get reliable performance numbers and to make sure that indeed your statement(s) are causing the issues and not the current state of your SAP HANA system. As a result of this investigation you may identify: ○ inefficient join orders (starting with table A and not with table B. the order in which tables are joined. meaning the set of tables used. returned by your SQL statement and compare it with the actual business needs. SAP HANA studio offers data flow analysis with the Graphical Plan view in the SAP HANA Plan Visualizer perspective. Procedure 1. the number columns selected in the field list or by applying additional filters in the WHERE clause) with regards to its database call. 2. Check the data flow of your statement. Check the result size. As a result of this investigation you might identify tables that should not be involved at all and adapt your statement or the underlying database logic accordingly (for example.10. how filters are applied and the size of intermediate results. Related Information Tables Used [page 158] Analyzing SQL Execution with the Plan Visualizer [page 148] SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. As a result of this investigation you might restrict the result size by changing the application logic (for example. SAP HANA studio offers data flow analysis with the Tables Used view in the SAP HANA Plan Visualizer perspective. You can then adapt your statement or underlying database logic as necessary. their size and the number of entries selected from those tables.3. that is. intermediate selections which seem too broad). All rights reserved. by checking joins and join types). 125 . Check the usage of underlying tables.3 Data Analysis The easiest way to analyze a poorly performing SQL statement is to look at the data flow as this can be matched with the understanding of the business/application needs. that is the number of records and number of columns. when that is the much smaller result set) ○ unexpectedly missing filters (that is. 3. and compare it with your understanding of the business needs. Check that the information is combined and aggregated in the correct way as required by the business scenario. Context The actual activities depend on the respective artifact (models. a. views) for possible performance-impacting aspects. scripts. a. Procedure 1.4 Source Analysis Before conducting a detailed technical analysis it is recommended to first analyze source artifacts (models. Check whether modeling constructs are used that are known to be costly. for example: ○ Is the aggregation behavior consistent with the intended semantics? ○ Is the join cardinality correct for each join? ○ Are dynamic joins used? ○ Is the result always aggregated? ○ Has Join Optimization been enabled? ○ Do analytical privileges influence performance? c. Check whether any performance-relevant execution hints are activated. 2. parameters and filters used in the query on the execution time. scripts. Check the effect of the attributes.10. for example: ○ Enforce execution via a specified database engine ○ Enforce upper bound for parallelization d. furthermore the Plan Visualizer perspective provides deeper inspection of these models. Check if your statement fits with the guidelines for SQL Query Design For example. All rights reserved. see SQL Query Design 126 PUBLIC © 2015 SAP SE or an SAP affiliate company. SQL Script The Statement Statistics view (part of the Plan Visualizer) support analysis of SQLScript. Graphical Calculation Models SAP HANA studio offers dedicated performance debugging capabilities as part of the Modeler for graphical calculation models. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . Check if your procedure fits with the given guidelines For example. Analyze your script for most expensive steps / statements 3. Check whether intermediate steps produce reasonable results (in terms of size).3. b. see Developing Procedures b. Plain SQL a. for example: ○ Calculated join attributes ○ Complicated filter expressions ○ Mixed models that involve engine switches e. views). 127 . Dedicated views and filters support the analysis along numerous dimensions. track down the KPI to the deepest possible level manifesting the symptom. 2. All rights reserved. For any suspicious KPI. Procedure 1.Results As a result of this activity you either figured out and solved problems at the source level or you now have to continue with an even deeper technical analysis. Related Information Developing Procedures [page 117] Technical Analysis [page 127] Statement Statistics [page 152] 3. try to correlate the symptom with the cause.10. 3. distribution characteristics) in order to figure out in which aspect or dimension you might look for possible issues. or on a specific time interval. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. in higher level statement elements. You might restrict the scope of your analysis by focusing on the critical path. From there. engine. Context You should follow this in order to track down problematic performance symptoms from which you can derive possible root causes at higher levels.5 Technical Analysis The deepest level of performance analysis addresses the technical details of a database statement execution. 4. Inspect aggregated execution KPIs (execution time. or execution plan operator. There are a number of tools that can support you: ● SAP HANA studio offers the Plan Visualizer perspective which allows for deep technical analysis. resource consumption. there are multiple tools allowing for even deeper analysis such as Explain Plan and the numerous tracing tools. ● Furthermore. system node. The database is said to be hanging when it no longer responds to queries that are executed against it. Context The source of the system standstill might be related to any of the components involved. Procedure 1.How to generate a runtime dump on SAP HANA) 2. memory and other information about threads currently running in the system by executing the command top -H and taking a screenshot of the output. OS and hardware. you should perform the following steps if it is still possible to log on to the OS of the master host (for example. As SQL statements cannot usually be executed for analysis. Collect CPU. network. All rights reserved. the storage. Please provide this information when logging an incident with SAP support and avoid restarting the database as otherwise retrospective analysis might not always be possible. SAP HANA database or the application layer. Collect a runtime dump (see SAP Note 1813020 . SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions .Related Information Analyzing SQL Execution with the Plan Visualizer [page 148] Analyzing SQL Execution with the Plan Explanation [page 147] 3. Related Information SAP Note 1999020 SAP Note 1813020 Transparent Huge Pages [page 129] CPU Power Saving [page 130] 128 PUBLIC © 2015 SAP SE or an SAP affiliate company.11 System Hanging Situations This section explains what checks to carry out and how to perform those checks if your SAP HANA instance is hanging. Also see SAP Note 1999020 on "SAP HANA: Troubleshooting when database is no longer reachable" for further specific steps and guidance on pro-active or reactive actions you can take. For troubleshooting it is essential to collect information about the context of the active threads in the SAP HANA database. 3. for example. as the <sid>adm user). SAP strongly recommends that you disable Transparent Huge Pages on all your SAP HANA servers.5 ● SAP Note 2136965 SAP HANA DB: Recommended OS settings for RHEL 6. Additionally. Due to the special manner of SAP HANA's memory management.Transparent Huge Pages (THP) on SAP HANA Servers ● SAP Note 1824819 SAP HANA DB: Recommended OS settings for SLES 11 / SLES for SAP Applications 11 SP2 ● SAP Note 1954788 SAP HANA DB: Recommended OS settings for SLES 11 / SLES for SAP Applications 11 SP3 ● SAP Note 2013638 SAP HANA DB: Recommended OS settings for RHEL 6. They have to be executed as the root user in the Linux shell. 129 . cat /proc/meminfo | grep AnonHugePages If the first output shows "[always] madvise never" that is "always" is shown inside square brackets. Until further notice. If the second output shows a value greater than 0 Kb (Kilobytes). using THP may lead to hanging situations and performance degradations.1 Transparent Huge Pages Transparent Huge Pages (THP) is a feature that is generally activated for the Linux kernel shipped with SUSE Linux Enterprise Server (SLES) 11 and Red Hat Enterprise Linux (RHEL) 6 versions certified for usage with SAP HANA.6 Note the following checks and steps should be performed on all host of the affected SAP HANA system. then THPs are activated according to the value of the relevant Linux Kernel parameter. that the CPU load is high and/or severe performance issues. See the SAP Notes below for further reference: ● SAP Note 2131662 . then this is evidence that some processes use THPs. To check whether Transparent Huge Pages are activated and currently being used by processes.11. If both steps show that Transparent Huge Pages are activated or in use. cat /sys/kernel/mm/transparent_hugepage/enabled b. 2. execute the below commands in the Linux shell: a. Note Transparent Huge Pages is supported on Intel-based hardware platforms only. deactivate them by executing the commands below in the Linux shell: SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. Context You experience that your SAP HANA database does not react anymore.3. All rights reserved. Procedure 1. Transparent Huge Pages are activated on your SAP HANA servers. use one of the below strategies to persist it: a. All rights reserved.a. /etc/rc.6 130 PUBLIC © 2015 SAP SE or an SAP affiliate company.conf file of the Bootloader.. To prevent toggling the value from [never] back to [always] causing THPs to be activated again. 3. Right after executing this command the hanging situation will be resolved.local). Context See the SAP Notes below for further reference: ● SAP Note 1824819 SAP HANA DB: Recommended OS settings for SLES 11 / SLES for SAP Applications 11 SP2 ● SAP Note 1954788 SAP HANA DB: Recommended OS settings for SLES 11 / SLES for SAP Applications 11 SP3 ● SAP Note 2013638 SAP HANA DB: Recommended OS settings for RHEL 6. You can verify whether the Kernel parameter has the correct value by executing the second command which should show the output "always madvise [never]" where the new value is in square brackets. b. Related Information SAP Note 2131662 SAP Note 1824819 SAP Note 1954788 SAP Note 2013638 SAP Note 2136965 3. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . Note that unless the value of the Kernel parameter has not been persisted. Integrate the parameter "transparent_hugepage=never" within your system boot scripts (for example.11.2 CPU Power Saving The Linux Kernel shipped with SUSE Linux Enterprise Server (SLES) 11 and Red Hat Enterprise Linux (RHEL) 6 versions certified for usage with SAP HANA contain a new cpuidle driver for recent Intel CPUs. it is only set/valid until the host is restarted the next time. Add the parameter "transparent_hugepage=never" to the kernel boot line in the /etc/grub. echo never > /sys/kernel/mm/transparent_hugepage/enabled b. cat /sys/kernel/mm/transparent_hugepage/enabled The first command will deactivate Transparent Huge Pages by setting the relevant Kernel parameter to "never". This driver leads to a different behavior in C-states switching and causes performance degradations.5 ● SAP Note 2136965 SAP HANA DB: Recommended OS settings for RHEL 6. Procedure 1. Often due to network related issues or a disk I/O related issue on the secondary site. system replication can slow down transaction processing in the following cases: ● Asynchronous replication is configured over long distances. 131 . cat /sys/devices/system/cpu/cpuidle/current_driver The correct value for the cpuidle driver should be "acpi_idle". ● Multitier system replication is configured and a tier 3 system is attached. Check if the recommended driver is enabled and whether the CPU power safe mode is activated. Check the CPU power save mode by running the following command. cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor The correct value for the CPU power save mode should be "performance". Execute the following command as root user in the Linux shell. Check If Log Can Be Shipped In Time ● Check if log shipping is significantly slower than local log write (SYNC/SYNCMEM) ● Check Async Buffer Full Count (ASYNC) SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. If the output shows at lease one line with "ondemand" please follow the steps in SAP Note 1890444 .12 Troubleshoot System Replication If system replication appears to slow down transaction processing you can check the network and disk I/O on the secondary site. 2.Slow HANA system due to CPU power save mode. If so no further steps are required. Related Information SAP Note 1824819 SAP Note 1954788 SAP Note 2013638 SAP Note 2136965 SAP Note 1890444 3. All rights reserved. ● SYNC/SYNCMEM replication mode is configured over short distances The following troubleshooting steps can help you determine and resolve the underlying cause. In case the output shows the wrong value "intel_idle" please follow the steps in SAP Notes. ● Collect network information on bandwidth and latency ● If the bandwidth can handle load check if the network is shared and whether other applications may be interfering with performance.ipv4. For example. which is based on the I/O statistics from the primary site.core|net.tcp_rmem ● The BDP tells you what TCP window size is needed to use the network link fully. ● Check the network utilization profile for the network link to see if the maximum capacity of the network has been reached ● If the network is not fully utilized check the linux kernel TCP configuration with sysctl –a | egrep “net.tcp_wmem and net. We recommend executing this SQL statement when system replication is disabled. Tile Catalog: SAP HANA System Replication in the SAP HANA Administration Guide. use HANA_Replication_SystemReplication_Bandwidth.txt (from SAP Note 1069700). use the open source IPERF tool or similar to measure the real application network performance ● To estimate the required bandwidth for Data/Log shipping. Check If Data Load Can Be Handled By Network Link ● Estimate the amount of data/log shipped from the primary site ● Compare this with the available bandwidth (the recommended bandwidth is 10 Gbit/s) ● Do a network performance test.2 MB). net.tcp” ● Check that windows scaling is set to the default value of 1. All rights reserved. Check Disk I/O on Secondary Site Slow disk I/O on the secondary can postpone releasing log buffers on primary. Calculate the Bandwidth Delay Product (BDP): Bandwidth * Latency (for example.ipv4. Check Network Configuration (long distance) Increasing the TCP window size can result in better network utilization and higher throughput.ipv4. For more information see. You can: ● Use a Disk Performance Test Tool ○ Execute fsperf on log volume 132 PUBLIC © 2015 SAP SE or an SAP affiliate company. SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions . BDP = 50ms * 3 Gbps = 19. It is based on the system view M_SERVICE_REPLICATION and can be used to compare log shipping time to local log write time.tcp_window_scaling = 1 ● Check whether the max size can be increased for net.txt (from SAP Note 1069700).● These checks can be used to verify that the problem is really related to system replication ● For an overview of basic system replication KPIs see HANA_Replication_SystemReplication_Overview. which results in wait situations on the primary.ipv4. ● The SAP HANA Cockpit can also be used to check system replication KPIs. ● Check I/O relevant tables in the proxy schema of the corresponding secondary site. Related Information SAP Note 1069700 SAP HANA Troubleshooting and Performance Analysis Guide Root Causes And Solutions PUBLIC © 2015 SAP SE or an SAP affiliate company. which provide SQL access on the primary site on statistic views of the secondary. use command line tools (this has to be done for each individual service) ○ For example: $ hdbcons "statreg print -n M_VOLUME_IO_TOTAL_STATISTICS" ○ A runtime dump also contains I/O statistics. All rights reserved. 133 . use hdbcons only with the guidance of SAP HANA development support. which you can see with: $ hdbcons “runtimedump dump” Caution Technical expertise is required to use hdbcons. see Monitoring Secondary Sites in the SAP HANA Administration Guide. For more information.○ For example: $ fsperf /usr/sap/TST/SYS/global/hdb/log/mnt00001/hdb00002 ● Check the Monitoring view ○ If SQL is not available. To avoid incorrect usage. and are any of these threads blocked? ● Are any sessions blocking current transactions? ● Are any operations running for a significantly long time and consuming a lot of resources? If so. This arranges the information as follows: ● Threads with the same connection ID are grouped. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . It may be useful to see. you can take certain actions to improve performance.4 Tools and Tracing This section gives you an overview of the available tools and tracing options that are available. then the callee). On big systems with a large number of threads. 134 PUBLIC © 2015 SAP SE or an SAP affiliate company. when will they be finished? ● How do different hosts compare in terms of performance? On the Performance tab. the information provided on the Performance tab enables you to focus your analysis on the following questions: ● What and how many threads are running. including canceling the operations that cause blocking situations. how long a thread is running. or if a thread is blocked for an inexplicable length of time. To revert to an unstructured view. deselect the Group and sort checkbox or change the layout in some other way (for example. sort by a column). for example.1 System Performance Analysis As a first step to resolving SAP HANA performance issues. All rights reserved. Thread Display By default. ● Within each group. ● Groups are displayed in order of descending duration. the Threads sub-tab shows you a list of all currently active threads with the Group and sort filter applied. 4. you can analyze detailed aspects of system performance in the SAP HANA studio on the Performance tab of the Administration editor.1 Thread Monitoring You can monitor all running threads in your system in the Administration editor on the Performance Threads sub-tab. what are they working on. 4.1. the call hierarchy is depicted (first the caller. this arrangement provides you with a more meaningful and clear structure for analysis. When analyzing system performance issues. They manage code-level access to a range of resources (for example. while changes to the table definition (DDL operations) require an exclusive table lock. PlanExecutor threads are used to process column-store statements and have an SqlExecutor thread as their parent. Exclusive row-level locks prevent concurrent write operations on the same record. Table-level locks prevent operations on the content of a table from interfering with changes to the table definition (such as drop table. Note With revision 56. Thread Method. and Thread Status columns is helpful for analyzing what a thread is currently working on. 135 . internal SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. Intentional exclusive locks can be acquired if no other transaction holds an exclusive lock for the same object. Note The information in the Thread Detail. All rights reserved. or metadata) is indicated in the Transactional Lock Type column. for example. For more detailed analysis of blocked threads. Exclusive locks require that no other transaction holds a lock for the same object (neither intentional exclusive nor exclusive).Thread Information Detailed information available on the Threads sub-tab includes the following: ● The context in which a thread is used This is indicated by the thread type. information about low-level locks is available in the columns Lock Wait Name. There is also a LOCK TABLE statement for explicitly locking a table. or result fetching issued by applications on top of SAP HANA. You can see detailed information about the blocking situation by hovering the cursor over this icon. the SQL statement currently being processed is displayed. Lock Wait Component and Thread ID of Low-Level Lock Owner. The lock mode determines the level of access other transactions have to the locked record. details about the execution plan currently being processed are displayed. Transactional locks may be held on records or tables. Transactions can also be blocked waiting for other resources such as network or disk (database or metadata locks). PlanExecutor threads were replaced by JobWorker threads. Important thread types are SqlExecutor and PlanExecutor. alter table). In the case of SqlExecutor threads. ● Information about transactionally blocked threads A transactionally blocked thread is indicated by a warning icon ( ) in the Status column. The lock mode is indicated in the Transactional Lock Type column. Thread Method. ● What a thread is currently working on The information in Thread Detail. table. In the case of PlanExecutor threads (or JobWorker threads as of revision 56). table. The type of lock held by the blocking thread (record. statement execution. and Thread Status columns is only useful to SAP Support for detailed analysis. A transactionally blocked thread cannot be processed because it needs to acquire a transactional lock that is currently held by another transaction. Low-level locks are locks acquired at the thread level. Note The information in the Thread Type column is only useful to SAP Support for detailed analysis. DML operations on the table content require an intentional exclusive lock. They are acquired implicitly by update and delete operations or explicitly with the SELECT FOR UPDATE statement. SqlExecutor threads handle session requests such as statement compilation. or database. that is those with a connection ID of -1. The Blocked Transactions sub-tab provides you with a filtered view of transactionally blocked threads. All rights reserved. ● Jump to the following related objects by right-clicking the thread and choosing Navigate To <related object> : ○ Threads called by and calling the selected thread ○ Sessions with the same connection ID as the selected thread ○ Blocked transactions with the same connection ID as the selected thread ● View the call stack for a specific thread by selecting the Create call stacks checkbox. The Trace Configuration dialog opens with information from the selected thread automatically entered (application and user). the new settings overwrite the existing ones. disk). Note The information contained in call stacks is only useful to SAP Support for detailed analysis. Related Information SQL Trace [page 141] Performance Trace [page 166] Expensive Statements Trace [page 144] SAP HANA SQL and System Views Reference 136 PUBLIC © 2015 SAP SE or an SAP affiliate company. you can perform the following actions on the Threads sub-tab: ● See the full details of a thread by right-clicking the thread and choosing Show Details. Monitoring and Analysis Features To support monitoring and analysis. Note This option is not available for threads of external transactions. refreshing the page. If the performance trace is already running. you must stop it before you can start a new one. Lock wait components group low-level locks by engine component or resource. and then selecting the thread in question. or performance trace by choosing Configure Trace <required trace> . ● End the operations associated with a thread by right-clicking the thread and choosing Cancel Operations. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing .data structures. network. ● Activate the expensive statements trace. Note If the SQL trace or expensive statements trace is already running. SQL trace. They are indicated in the Administration editor on the Performance Threads tab. or database. SQL trace. Exclusive locks require that no other transaction holds a lock for the same object (neither intentional exclusive nor exclusive). You can see another representation of the information about blocked and blocking transactions on the Blocked Transactions subtab. They manage code-level access to a range of resources (for example. Lock Wait Component and Thread ID of Low-Level Lock Owner. or transactionally blocked threads.2 Blocked Transaction Monitoring Blocked transactions. The Trace Configuration dialog opens with information from the selected thread automatically entered (application and user). Intentional exclusive locks can be acquired if no other transaction holds an exclusive lock for the same object. Information About Blocked Transactions Blocked transactions are transactions that are unable to be processed further because they need to acquire transactional locks (record or table locks) that are currently held by another transaction. while changes to the table definition (DDL operations) require an exclusive table lock. or expensive statements trace for the blocking transaction (that is the lock holder) by choosing Configure Trace <required trace> .4. There is also a LOCK TABLE statement for explicitly locking a table. internal data structures.1. Low-level locks are locks acquired at the thread level. table. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. Lock wait components group low-level locks by engine component or resource. Monitoring and Analysis Features To support monitoring and analysis. ● Activate the performance trace. information about low-level locks is available in the columns Lock Wait Name. Exclusive row-level locks prevent concurrent write operations on the same record. DML operations on the table content require an intentional exclusive lock. For more detailed analysis of blocked transactions. 137 . alter table). Table-level locks prevent operations on the content of a table from interfering with changes to the table definition (such as drop table. They are acquired implicitly by update and delete operations or explicitly with the SELECT FOR UPDATE statement. The type of lock held by the blocking transaction (record. can impact application responsiveness. Transactions can also be blocked waiting for other resources such as network or disk (database or metadata locks). The lock mode is indicated in the Transactional Lock Type column. or metadata) is indicated in the Transactional Lock Type column. table. All rights reserved. The lock mode determines the level of access other transactions have to the locked record. network. disk). you can perform the following actions on the Blocked Transactions subtab: ● Jump to threads and sessions with the same connection ID as a blocked/blocking transaction by rightclicking the transaction and choosing Navigate To <related object> . 3 Session Monitoring You can monitor all sessions in your landscape in the Administration editor on the Sessions Performance sub-tab.Note If the SQL trace or expensive statements trace is already running. If the performance trace is already running. the new settings overwrite the existing ones. You can see the following information: ● Active/inactive sessions and their relation to applications ● Whether a session is blocked and if so which session is blocking ● The number of transactions that are blocked by a blocking session ● Statistics like average query runtime and the number of DML and DDL statements in a session ● The operator currently being processed by an active session (Current Operator column). Related Information SQL Trace [page 141] Performance Trace [page 166] Expensive Statements Trace [page 144] SAP HANA SQL and System Views Reference 4. you can get this information from the SYS. you can analyze the SQL statements being processed in the session. All rights reserved.1.M_CONNECTIONS monitoring view with the following statement: SELECT CURRENT_OPERATOR_NAME FROM M_CONNECTIONS WHERE CONNECTION_STATUS = 'RUNNING' Tip To investigate sessions with the connection status RUNNING. Note In earlier revisions. ensure that the Last Executed Statement and Current Statement columns are visible. you must stop it before you can start a new one. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . Session Information The Sessions sub-tab allows you to monitor all sessions in the current landscape. You can then copy the statement into the SQL console and analyze it using 138 PUBLIC © 2015 SAP SE or an SAP affiliate company. To see the statements. It is also possible to use the SQL plan cache to understand and analyze SQL processing. for example: SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. Monitoring and Analysis Features To support monitoring and analysis. for example. or expensive statements trace by choosing Configure Trace <required trace> .1. Note If the SQL trace or expensive statements trace is already running. Related Information SQL Trace [page 141] Performance Trace [page 166] Expensive Statements Trace [page 144] SAP HANA SQL and System Views Reference 4. By monitoring the progress of long-running operations. All rights reserved. The following information is available. the new settings overwrite the existing ones. you must stop it before you can start a new one. 139 . You can monitor long-running jobs in the Administration editor on the Progress Performance Job sub-tab. The Trace Configuration dialog opens with information from the selected session automatically entered (application and user). SQL trace. you can perform the following actions on the Sessions sub-tab: ● Cancel a session by right-clicking the session and choosing Cancel Session. see how far along they are. you can determine whether or not they are responsible for current high load. ● Jump to the following related objects by right-clicking the session and choosing Navigate To <related object> : ○ Threads with the same connection ID as the selected session ○ Blocked transactions with the same connection ID as the selected session ● Activate the performance trace..the Explain Plan and Visualize Plan features..4 Job Progress Monitoring Certain operations in SAP HANA typically run for a long time and may consume a considerable amount of resources. If the performance trace is already running. and when they will finish. delta merge operations and data compression. All rights reserved. Related Information SAP HANA SQL and System Views Reference 4. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . 140 PUBLIC © 2015 SAP SE or an SAP affiliate company.● Connection that triggered the operation (CONNECTION_ID) ● Start time of the operation (START_TIME) ● Steps of the operation that have already finished (CURRENT_PROGRESS) ● Maximum number of steps in the operation (MAX_PROGRESS) For more information about the operations that appear on the Job Progress sub-tab. ● Expensive statements trace On the Performance Expensive Statements Trace exceed a specified response time. You can use the load graph for performance monitoring and analysis. which provides a statistical overview of what statements are executed in the system. see system view M_JOB_PROGRESS. Related Information SAP HANA Troubleshooting and Performance Analysis Guide 4. you can better understand how the statements affect application and system performance. you can use it to get a general idea about how many blocked transactions exist now and in the past.5 Load Monitoring A graphical display of a range of system performance indicators is available in the Administration editor on the Performance Load sub-tab. You can analyze the response time of SQL statements with the following traces: ● SQL trace From the trace file. you can view a list of all SQL statements that In addition to these traces. tab. you can analyze the response time of SQL statements. For example. or troubleshoot the root cause of slow statement performance. By analyzing SQL statements and calculating their response times. you can analyze the SQL plan cache.1.2 SQL Statement Analysis A key step in identifying the source of poor performance is understanding how much time SAP HANA spends on query execution. In the Trace Configuration dialog box.1 Analyzing SQL Traces The SQL trace allows you to analyze the response time of SQL statements within an object. and so on. the database connection being used.4. 5. This means that the SQL trace in the system database runs on the name server. Note In the system database of a multiple-container system. calculate the difference between the times given for # tracing PrepareStatement_execute call and # tracing finished PrepareStatement_execute.2. 4. Choose Show Entire File. unique constraint violations) that were reported. So the SQL trace is a good starting point for understanding executed statements and their potential effect on the overall application and system performance. the number of records affected. set the trace status to Active. as well as for identifying potential performance bottlenecks at statement level.1 SQL Trace The SQL trace collects information about all SQL statements executed on the index server and saves it in a trace file for further analysis. 6. Choose Finish. SQL trace information is saved as an executable python program that you can access on the Diagnosis Files tab of the Administration editor. and specify the required trace and user filters. It is inactive by default. the name server functions as the index server. Run the application or SQL statements you want to trace. Information collected by the SQL trace includes overall execution time of each statement. 141 .1. Here. The SQL statements in the trace file are listed in order of execution time. All rights reserved. To calculate the response time of a specific SQL statement. Choose the Diagnosis Files tab and open the trace file you created. 4. 7. choose the Trace Configuration trace and edit the SQL trace. Procedure 1. Re-open the SQL trace configuration and set the trace status to Inactive. In the Administration editor. 3. 8. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. specify a name for the trace file. Choose Finish. 9. potential errors (for example.2. Analyze the response time of the relevant SQL statements to identify which statements negatively affect performance. there is no index server. 2. Therefore.2. Trace Levels You can configure the following trace levels: Table 39: Trace Level Description NORMAL All statements that have finished successfully are traced with detailed information such as executed timestamp. connection ID and statement ID. You activate and configure the SQL trace in the Administration editor on the Trace Configuration tab.Note Writing SQL trace files can impact database performance significantly. thread ID. and statement ID.2 SQL Trace Options Several options are available for configuring the SQL trace. error. and rollback are traced with detailed infor­ mation such as executed timestamp. connection ID and statement ID. They also consume storage space on the disk. thread ID. and statement ID. for example. thread ID. 142 PUBLIC © 2015 SAP SE or an SAP affiliate company. Caution An SQL trace that includes results may expose security-relevant data. ERROR_ROLLBACK All statements that are rolled back are traced with detailed information such as executed timestamp. connection ID. Note An SQL trace that includes results can quickly become very large. Related Information SQL Trace Options [page 142] 4. it is not recommended that you leave the SQL trace enabled all the time. thread ID. connection ID.1. query result sets. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . ALL All statements including status of normal. ALL_WITH_RESULTS In addition to the trace generated with trace level ALL. ERROR All statements that returned errors are traced with detailed information such as executed timestamp. All rights reserved. the result returned by select state­ ments is also included in the trace file. py. 4. All statements matching the filter criteria are recorded and saved to the specified trace file. 2.2 Analyzing Expensive Statements Traces The expensive statements trace allows you to identify which SQL statements require a significant amount of time and resources. As soon as the flush limit num­ ber of messages is buffered (or if the connection is closed).Other Configuration Options The following additional configuration options are available: Table 40: Option Description Trace file User-specific name for the trace file If you do not enter a user-specific file name. those messages are written to the trace file. 143 . The system will identify any statements that exceed this threshold as expensive statements. PUBLIC © 2015 SAP SE or an SAP affiliate company. 3. object. Procedure 1. Choose Finish. choose the Trace Configuration trace and edit the expensive statements trace. In the Trace Configuration dialog box. Run the application or SQL statements you want to trace. User. All rights reserved. In the Administration editor. and Filters to restrict traced statements to those of particular database or application users and statement filters applications. where: ● DB_<dbname> is the sub-directory where the trace file is written if you are running on a tenant database ● $HOST is the host name of the service (for example.2. 5. 4. Flush limit During tracing. the messages of a connection are buffered. indexserver) ● $PORT is the port number of the service ● $COUNT:3 is an automatically generated 3-digit number starting with 000 that incre­ ments by 1 and serves as a file counter when several files are created. set the trace status to Active and specify a threshold execution time in microseconds. Choose the Performance Expensive Statements Trace SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing tab. as well as to certain statement types and tables. the file name is generated according to the fol­ lowing default pattern: DB_<dbname>/sqltrace_$HOST_${PORT}_${COUNT:3}. application. 6. It is inactive by default. the following columns are especially useful for determining the statement's impact on performance: ○ START_TIME ○ DURATION_MICROSEC ○ OBJECT_NAME (names of the objects accessed) ○ STATEMENT_STRING ○ CPU_TIME 4. Information about recorded expensive statements is displayed on the Performance tab.2. Analyze the displayed information to identify which statements negatively affected performance.2.000. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . and object filters Filters to restrict traced statements to those of particular database or application users. If in addition to activating the expensive statements trace.1 Expensive Statements Trace Expensive statements are individual SQL statements whose execution time exceeded a configured threshold. the expensive statements trace will also show the peak memory size used to execute the expensive statements.2 Expensive Statements Trace Options Several options are available for configuring the expensive statements trace.000) User.2. The expensive statements trace records information about these statements for further analysis. Related Information Expensive Statements Trace Options [page 144] Set a Statement Memory Limit [page 32] 4. For each SQL statement.2. application filters. as well as to certain applications and tables 144 PUBLIC © 2015 SAP SE or an SAP affiliate company. Table 41: Option Description Threshold duration Threshold execution time in microseconds (default 1. you enable per-statement memory tracking. You activate and configure the expensive statements trace in the Administration editor on either the Trace Configuration tab or the Performance tab. All rights reserved. Note Process tracing is possible only for components in the ABAP and Business Objects stacks.ini file. Additional Parameters You can configure the expensive statement trace further using the following properties in the expensive_statement section of global. You can do this by configuring the corresponding parameters in the resource_tracking section of the global. You can do this by configuring the correspond­ ing parameters in the resource_tracking section of the global. threshold_memory Threshold memory usage of statement execution in bytes Note Resource tracking and memory tracking must also be en­ abled.Option Description Passport trace level If you are activating the expensive statements trace as part of an end-to-end trace scenario with the Process Monitoring Infrastructure (PMI). Trace parameter values In SQL statements. For more information see. you can deselect the Trace parameter values checkbox to reduce the amount of data traced.ini configuration file: Table 42: Property Description trace_flush_interval Number of records after which trace file is flushed threshold_cpu_time Threshold CPU time of statement execution in microsec­ onds Note Resource tracking and CPU time tracking must also be enabled. All rights reserved. SAP Library for NetWeaver on SAP Help Portal.ini file. Related Information Process Monitoring with PMI (Process Monitoring Infrastructure) SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. 145 . If these parameter values are not required. field values may be specified as parameters (using a "?" in the syntax). This means that only requests that are marked with a passport of the specified level are traced. you can specify the passport trace level as an additional filter. see the documentation for the system views M_SQL_PLAN_CACHE_OVERVIEW and M_SQL_PLAN_CACHE in the SAP HANA SQL and System Views Reference. it can help you to find slow queries. it is an important tool for understanding and analyzing SQL processing. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . As the SQL plan cache collects statistics on the preparation and execution of SQL statements. the cached plan is reused. Filtering according to the following columns can help you identify statements that are more likely to be causing problems and/or could be optimized: Table 43: Column Description TOTAL_EXECUTION_TIME The total time spent for all executions of a plan This helps to identify which statements are dominant in terms of time.4. the SQL plan cache. The SAP HANA database provides an object. The two system views associated with the SQL plan cache are M_SQL_PLAN_CACHE_OVERVIEW and M_SQL_PLAN_CACHE. For a full list of all SQL cache columns including descriptions. The SQL plan cache contains a lot of information. Whenever the execution of a statement is requested. it is better to reuse it the next time the same statement is executed. If not. a SQL procedure checks the SQL plan cache to see if there is a plan already compiled.3 Analyzing SQL Execution with the SQL Plan Cache The SQL plan cache is a valuable tool for understanding and analyzing SQL processing. the statement is compiled and the generated plan is cached. EXECUTION_COUNT The number of times a plan has been executed This can help you identify SQL statements that are executed more frequently than ex­ pected. every SQL statement is compiled to a plan. that stores plans generated from previous executions. instead of compiling a new plan every time. TO­ TAL_LOCK_WAIT_COUNT The total number of waiting locks USER_NAME The name of the user who prepared the plan and therefore where the SQL originated (ABAP/index server/statistics server) This can help you identify SQL statements with high lock contention. You can access the SQL plan cache in the Administration editor on the Performance tab.2. For example. Related Information SAP HANA SQL and System Views Reference 146 PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. If a match is found. AVG_EXECUTION_TIME The average time it takes to execute a plan execution This can help you identify long-running SQL statements. as well as measure the overall performance of your system. Before it is executed. Once a plan has been compiled. You can use this to evaluate the execution plan that the SAP HANA database follows to execute an SQL statement. The plan explanation is displayed on the Result tab. you can also analyze query plans from a logical perspective and consider questions such as: ● Does SAP HANA read data from multiple tables when only one is required? ● Does SAP HANA read all records when only one is required? ● Does SAP HANA start with one table even though another table has a much smaller result set? To gain the insight you need to answer such questions. the SAP HANA studio provides the following features for query plan analysis: ● Plan explanation ● Plan visualization 4. Procedure 1.1 Analyzing SQL Execution with the Plan Explanation You can generate a plan explanation for any SQL statement in the SQL console. understand how much data is processed by the operators. The plan explanation is displayed on the Result tab. From a technical perspective. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. Note You can enter multiple statements. 2. However. Results Plan explanations are also stored in the EXPLAIN_PLAN_TABLE view of the SYS schema for later examination. analyzing query plans allows you to identify long running steps. to identify queries that are inefficiently processed. if you understand the idea and purpose behind the query.3 Query Plan Analysis In SAP HANA. you can both technically and logically analyze the steps SAP HANA took to process those queries. Enter a query into the SQL console and choose Explain Plan in the context menu. to generate several plan explanations at once. All rights reserved. and see whether data is processed in parallel. separated by the configured separator character (usually a semicolon).4.3. That is. 147 . Optional: Run the same statement on different systems/users by changing the SQL connection. assuming that the tables and views exist in the other systems and you have authorization to access them. you can generate a graphical view of the plan. Figure 37: Visualized Plan 148 PUBLIC © 2015 SAP SE or an SAP affiliate company. with estimated performance. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . is displayed.2 Analyzing SQL Execution with the Plan Visualizer To help you understand and analyze the execution plan of an SQL statement. All rights reserved. Procedure 1. Visualize the plan of the SQL statement in one of the following ways: a.3.4. A graphical representation of the query. right-click the statement and choose Visualize Plan. Enter the statement in the SQL console and choose Visualize Plan in the context menu. On the SQL Plan Cache tab or the Expensive Statements Trace tab of the Performance tab. b. Note: You can change the colors for each operator type under PlanViz Graph Window Preferences SAP HANA Appearance . meaning the operator simply exists to give a more structured graph display. by selecting 'None' for the 'Node Grouping' option provided on the left of the graph. Or use the shortcut key F8 Another similar high-level graphic is generated with execution time information for each of the parts. Almost all of these non-physical operators can be removed from the graph if you prefer. a gray color indicates that the operator is not physical. All rights reserved. Validate the estimated performance by choosing Execute Plan in the context menu. Figure 38: Executed Plan SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company.2. 149 . by default. As of SPS 9. 150 PUBLIC © 2015 SAP SE or an SAP affiliate company. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . All rights reserved. visualization of execution plans will automatically display an 'Overview' page as you can see in the screenshot below.Note Execution time is given as a pair of values: "Exclusive" (the execution time of the node). and "Inclusive" (the execution time including the descendent nodes. 151 . for example. You can explore the graphic further.1 Overview Page Visualization of execution plans will automatically display an 'Overview' page Starting from SPS 09 (client version). All rights reserved. so you can submit it as part of a support query. or rearrange nodes on the screen. Results This graphic is a very powerful tool for studying performance of queries on SAP HANA databases. 4. you can expand. Figure 39: Overview Page The following table describes the nature of each KPI: Table 44: KPI Description Compilation Initial compilation time SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing Comment PUBLIC © 2015 SAP SE or an SAP affiliate company. Some important KPIs required to begin a performance analysis are provided so that you can first get a big picture of what is going on before going into the complex details. for example. You can also save the graphic as an image or XML file.3.2. collapse. 2. where final results are fetched) Memory Allocated Total memory allocated for executing the statement Number of Tables Used Total number of tables touched upon by any operator during execution Maximum Rows Processed The largest number of rows processed by any single operator Result Record Count The final result record count 4.2 You can click on the value to open the 'Tables Used View' for more detail re­ garding each table Statement Statistics Visualization of execution plans for procedures displays a set of statistics for each SQL statement involved in a procedure. SQL Query The statement that was executed System The system where the execution occur­ red (that is. Basic information such as execution count.KPI Description Comment Execution Total duration of the query excluding compilation time Dominant Operators Operators sorted by their execution time (top 3) You can click on any operator name to move to the corresponding visualized operator in the graph Number of Nodes Number of servers involved in the exe­ cution You can click on the value to see how much time was spent in each node Number of Network Transfers Total number of network transfers that have occurred during execution You can click on the value to open 'Net­ work View' which visualizes the trans­ fers in more detail Dominant Network Transfers Network transfers sorted by their exe­ cution time (top 3) You can click on any network transfer to move to the corresponding visual­ ized transfer in the graph.3. All rights reserved. Note: This section only appears if the query was executed in a distributed en­ vironment. execution/compile time. Context This set of statistics is automatically visualized when a procedure is executed and it provides a good starting point for analyzing performance of procedures as it lets you easily drill-down into the most expensive SQL statements. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . allocated memory size and 152 PUBLIC © 2015 SAP SE or an SAP affiliate company. ● Show Plan in New Window: Displays the execution plan corresponding to the selected statement ONLY in a separate pop-up window. Explain Plan Result. Execution Times. Line Number. Execution Count. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. Open an SQL Console 2. ● Execute Statement: Enables users to reproduce the execution of a single statement Note To use this feature. Compilation Times. the 'Keep Temp Tables' button ( ) located on the top-right corner of a graph editor (Plan Visualizer) must be toggled before obtaining the executed plan of the procedure under analysis. Enter any procedure into the console and choose Visualize Plan from the context menu. The following pieces of information are available: SQL Statement. All rights reserved. Procedure 1. you will have access to these menus: ● Show Plan in Graph: Displays the execution plan corresponding to the selected statement in the context of entire plan. Memory Allocated. 153 . Result Record Count.so on are provided for each statement so that you can sort the column (criterion) of interest to find the most expensive statement. and Procedure Comment Figure 40: Statement Statistics view By right-clicking on any element of the SQL statement. and so on. close.3 Timeline View The Timeline view provides a complete overview of the execution plan based on visualization of sequential time-stamps. fetch. open. open the Timeline view.) The view supports: ● Synchronized selection with operators in graphical plan ● Hierarchical display of operators based on parent-child relationship ● Re-configuration of scales via a toolbar option Procedure 1.3. Choose F8 to see executed plan from the visualized plan. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . To see a temporal breakdown of the individual operations processed in the execution of the query. Figure 41: Timeline View 154 PUBLIC © 2015 SAP SE or an SAP affiliate company.2. Context The following pieces of information are available in the view: ● X-axis: time elapsed since query execution. ● Y-axis: list of operators ● Duration of each operator execution represented by rectangular bars ● The nature of each time-stamp (for example. All rights reserved.3. From the main menu choose Window Show View Timeline . You can use this view to drill down to the potential root-cause of a long running procedure. For example you might detect a single statement that takes up most of the time and check whether this is indeed expected or notice that an identical statement is repeatedly executed with varying statistics (for example. 4. big gap between minimum execution time and maximum execution time). Statement Statistics will be shown automatically and it will list all the statements that were involved in the procedure. 155 . ● Logical grouping operators for the column engine (those colored in gray) simply display the aggregated time-stamps of their constituent operators. sum. meaning some operators may not be actually running in parallel but simply calling one another.4 Operator List for Plan Visualizer The Operator List view is used within the context of the Plan Visualizer perspective. and so on) regarding the same KPIs on the filterd operator set and the remaining set (not within the filter) SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. min. so you might check the system environment Next Steps However. both visualized and executed. 'fetch'. 4. All rights reserved.3. isPhysical (meaning whether an operator is a real. users should take into account the following point and limitations when analyzing the Timeline view: ● Any gaps or multiple occurrence of 'open'. for example. offset. execution time. The Operator List can be used to dynamically explore the operator set along user defined filters in order to pinpoint specific operators of interest. physically executed one).2. Use this view to drill down to the potential root-cause of a long running query: For example you might ○ Detect a single operator that takes up most of the time and check whether this is expected ○ Notice that some operators are executed sequentially instead of running in parallel. The view supports: ● Display of various KPIs. ● The adjustment for exclusivity of time-stamps as described above is not applicable for column engine operators. It lists detailed characteristics of all operators within a current plan. CPU time ● Setting of filters along all the columns and KPIs ● Display of the number of operators within the filtered set ● Immediate aggregated information (max.. and so on in the Column Search operator invoked by the row engine indicates that there was another operation called by the column engine in the meantime.2. 3. double click on an operator you are interested in to check its positioning within a visualized plan. Press Ctrl+F when displaying an executed plan and specify search parameters. Display the executed plan. Finally. even before looking into the visualized plan. Further restrict the filter to physical operators (using "x" as the search criteria in filter column "Physical") 3. you might 1. 4. All rights reserved. When you carry out a search the operator list is updated accordingly Figure 42: Operator List You can use the Operator List view to analyze the set of operators within a plan for the occurrence of specific conditions. Filter all operators that process a certain number of (input) rows 2. Context The following pieces of information are available in the view: ● X-axis: servers involved.5 Network View The Network View can visualize sequential network data transfers between different servers based on their given timestamps when an execution plan is based on a distributed query.● Detailed display of all operators within the filtered set (which can be further sorted) ● Export of the (filtered) operator list to a CSV file ● Forwards and backwards browsing through the history of applied filters. press Ctrl+F and set "Execution Time . SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . As of SPS 09 the following features are supported: ● Show or hide columns using the preferences dialog (preferences icon next to save button) ● Change the order of columns using drag and drop ● Order and column visibility are stored in users workspace ● Remote search functionality. For example. ● Y-axis: time elapsed since query execution ● Duration of each operator represented by rectangular bars ● Size of data transferred in bytes The view supports: 156 PUBLIC © 2015 SAP SE or an SAP affiliate company.At least" to 50 4.2. All rights reserved. Procedure 1. 157 . so you might check the data size and/or network condition and see that network data transfers occur too frequently between a particular set of servers.● Synchronized selection with network operators in graphical plan ● Hover information for network bars ● Zoom in and out You may use this view to discover any issues related to a distributed environment. 3. Check the results For example you might notice that an unusually long time is spent on a particular network data transfer. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. Optimize your distributed landscape based on the results. so you might improve the distribution set-up. Open the view 2. The view displays 3 metrics per table: ● Maximum number of entries processed. double-clicking on any row (table) leads to a corresponding operator list filtered for physical operators accessing the selected table. that is the overall output cardinality of any processing step on that table in the statement execution.2. meaning how often a table has been accessed during statement execution ● Maximum processing time. All rights reserved. You may detect a table that has been processed which should not be needed from a business perspective.6 Tables Used The table usage view provides an overview on which tables have been used during the processing of a statement. Review the number of times a table is accessed. that is the maximum processing time across the possibly multiple table accesses The view content can be sorted along any column.3. ● Number of accesses. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . Context The Tables Used view can be used to cross-check your business understanding about which tables are needed to fulfill a given SQL statement. Procedure 1. so you might modify you SQL in a way that this table is no more used. Review the tables used in your SQL. 158 PUBLIC © 2015 SAP SE or an SAP affiliate company. 2. You may use this view to understand whether the data processing of your statement matches your business expectations.4. 159 .2. 4. All rights reserved. The data can be viewed by selecting any of the operator shown in the list. Context The view consists of two main sections: ● The left panel shows list of operators you can select to view performance trace data ● The right panel shows the performance trace data for the operator you have selected The view supports: ● synchronized selection between operators visualized in the graph ● double-clicking on a particular data to view its full-text Note This view is mainly targeted towards advanced users who are well acquainted with the core details of SAP HANA.3. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. so you can check if it is possible to improve the filtering in your SQL statement.7 Performance Trace The Performance Trace view displays the list of operators that have performance trace data.You may see that the number of processed entries is far higher than expected. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . 160 PUBLIC © 2015 SAP SE or an SAP affiliate company. You can double-click a particular statement or open "Visualize Plan" from the context menu to get the execution plan. Look for any additional properties relevant to a particular operator that you cannot find from other views. executed in a given time frame for a particular application session.8 Plan Trace Plan trace enables you to collect SQL queries and their execution plans.3. only 'SELECT' statements are traced with Plan trace. 3. Fill in missing gaps in the Timeline view by deducing from the sequential list of activities. All rights reserved. As of SPS 09.Procedure 1. 4. 2. you will be able to visualize the execution plan for performance analysis. Further break-down a specific operation into core engine-level details.2. Context You can access the plan trace configuration wizard in SAP HANA studio in two ways: ● By right-clicking on a particular session you wish to trace in the Administration editor -> Performance -> Sessions ● Going to Administration editor -> Trace Configuration tab For each SQL query that has been traced. a Plan Trace Editor will be automatically opened displaying the queries that have been traced. Once you stop an active trace. From the context menu. Each item (query) can be double-clicked for visualization of its execution plan SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. After execution. Open the SQL Console 2. 7. 161 . Execute some SELECT queries in the SQL Console Only SELECT queries are supported with SPS 09. choose Configure Trace Plan Trace 5. An editor (Plan Trace editor) will open automatically showing the list of queries that you have executed in 'SQL Console' 9. 'Press 'Finish' in the pop-up configuration wizard 6. go back to the sessions list and deactivate 'Plan Trace' via configuration wizard (as in steps 4 to 6) 8. All rights reserved. The column Application Source will contain the keyword 'SQLExecuteFormEditor' 4. Right-click on the session which corresponds to the SQL Console that you opened in step 1.Procedure 1. Open Administration editor Performance tab Sessions tab 3. All rights reserved. which will visualize the calculation scenario associated with the view. find potential tweaking points. and a source tab which shows the raw JSON data of the selected node The visualizer supports: ● link with 'Properties' view for each selected node ● double-clicking on a particular data source to drill down to its calculation scenario You may use this visualizer to check whether the scenario is designed according to your understanding. and. 162 PUBLIC © 2015 SAP SE or an SAP affiliate company. an output tab which provides attributes of a selected node.3.4. if not.9 Calculation Scenario Visualization By navigating to a particular calculation view in the Systems view.2. you can access the Visualize View context menu. The calculation scenario visualizer consists of following three main sections: ● left: an overview which provides a list of calculation nodes involved in the scenario with their main properties ● middle: the visualized graph of calculation scenario ● right: an input tab which provides mapping information of the selected node. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . The context name appears as part of the trace file name and should be easy for you to recognize and later find.4 Advanced Analysis If have you an advanced knowledge of SAP HANA and SQL databases and you suspect that automated processes are making poor decisions that have negative impacts on query performance. 4. you can use specific SQL commands to produce trace information to help analyze the following processes: ● Table joins ● Column searches Recommendation Perform these types of analysis only if analyses of query plans and SQL statements were not enough to find the root cause of slow query performance. choose Trace Configuration and create a new user-specific trace. it does provide some useful information for performance analysis. From the information within the trace files. A query-optimizer (qo) trace of a single SAP HANA table search provides the details you need for such an analysis. Specify a context name. In the Administration editor. 2.1 Analyzing Column Searches In SAP HANA. Context The qo trace provides a lot of information that is hard to consume if you are not an SAP HANA query-optimizer expert. however. you can analyze how the query-optimizer performed the column search. you can perform advanced analyses to better understand how those decisions were made. Specify your database user or application user. if a column search takes a long time.4. Procedure 1. proceed as follows.4. To start a user-specific qo trace and analyze the relevant trace information. The Trace Configuration dialog box opens. In SAP HANA. 163 . you can see which column the query-optimizer decided to use as the first column in the column search and you can determine whether that decision negatively impacted the performance of the column search. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. 3. and select Show Entire File. In the trace section. Switch off the trace by deleting the user-specific trace configuration.cpp.4. you can analyze how the tables are joined and in what order. Enter qo as filter text and search for the trex_qo component. For the trex_qo component. 5. then this may represent the source of poor performance. Find the sections detailing the estimated results for the terms. Run the query you want to trace. Choose Finish. b. 11. 8. you can send the trace file to SAP Support. Context The je trace provides a lot of information that is hard to consume if you are not an SAP HANA join engine expert. From the information within the trace files. You can use this information to determine whether query performance is negatively impacted by the table join. Compare the estimated results with the actual search results. These sections are marked with Evaluate. it does provide some useful information for performance analysis. select DEBUG as the system trace level. For more detailed analysis. however. 4. you can see which table is used as the first table when processing a join and how the order of tables in the join is defined. 6. The query-optimizer selects which column to use as the first column of the search based on the term with the lowest estimated number of results. A join evaluation (je) trace of joined SAP HANA tables provides the details you need for such an analysis. 10. open the file. 9.4. Results If the actual results indicate that a different term should have been used to start the column search. a. if a query on joined tables takes a long time. Select the Show All Components checkbox. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . 164 PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. 7. These sections are marked with GetEstimation. On the Diagnosis Files tab. c. analyze the trace information for each term (WHERE condition).cpp. search for the trace file. Find the sections detailing the actual search results for the terms.2 Analyzing Table Joins In SAP HANA. The trace section starts with i TraceContext TraceContext. 2. choose Trace Configuration and create a new user-specific trace. Specify your database user or application user. On the Diagnosis Files tab. 165 . Switch off the trace by deleting the user-specific trace configuration.cpp. 3. Select the Show All Components checkbox. 9. proceed as follows: Procedure 1. 5. these tools are intended only for use when requested by SAP Support: ● Performance trace This tool records performance indicators for individual query processing steps in database kernel. open the file. In the Administration editor. For the join_eval component.5 Additional Analysis Tools for Support To complement the standard tools for performance analysis. and select Show Entire File. select DEBUG as the system trace level. 7. Run the query you want to trace. 4. 12. search backwards for the beginning of the trace section. 11. however. In the trace section. analyze the following trace information: ○ Estimations for the WHERE conditions ○ Table size and join conditions ○ Join decision 4. The Trace Configuration dialog box opens. SAP HANA provides additional analysis tools that SAP Support can use to help determine the cause of performance issues. The following analysis tools are available in SAP HANA. Enter join as filter text and search for the join_eval component. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company.To start a je trace and analyze the relevant trace information. search for the indexserver trace file. Specify a context name. 6. The context name appears as part of the trace file name and should be easy for you to recognize and later find. 10. From the end of the file. 8. ● Kernel profiler This tool provides information about hotspots and expensive execution paths during query processing. All rights reserved. Choose Finish. You activate and configure the performance trace in the Administration editor on the Trace Configuration tab.2 Performance Trace Options Several options are available for configuring the performance trace. the results are saved to trace files that you can access on the Diagnosis Files tab of the Administration editor. It is recommended that you start performance tracing immediately before running the command(s) that you want to analyze and stop it immediately after they have finished.1 Performance Trace The performance trace is a performance tracing tool built into the SAP HANA database. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . the data size read and written. Information collected includes the processing time required in a particular step. a single specific application user. and information specific to the operator or processing-step-specific (for example.5. network communication. Related Information Performance Trace Options [page 166] 4. It is inactive by default. but instead must use a tool capable of reading the output format (*. All rights reserved. and a single specific application 166 PUBLIC © 2015 SAP SE or an SAP affiliate company. SAP Support has tools for evaluating performance traces. number of records used as input and output).5.4. Standard Trace Mode Table 45: Option Description Trace file The file to which the trace data is automatically saved after the performance trace is stop­ ped User and application filters Filters to restrict the trace to a single specific database user. You cannot analyze these files meaningfully in the SAP HANA studio. When you stop tracing. It records performance indicators for individual query processing steps in the database kernel.tpt). Related Information Process Monitoring with PMI (Process Monitoring Infrastructure) 4. you can specify the passport trace level as an ad­ ditional filter. ensure that you enter a value greater than the time it takes the scenario to run. Note Process tracing is possible only for components in the ABAP and Business Objects stacks. SAP Library for NetWeaver on SAP Help Portal. It is inactive by default. This means that only requests that are marked with a passport of the speci­ fied level are traced. the trace stops automatically. All rights reserved. It is recommended that you start kernel profiler tracing immediately before you execute the statements you want to analyze and stop it immediately after they have finished. 167 . for example.5. This role is intended only for SAP HANA development support. It can be used to analyze performance issues with systems on which third-party software cannot be installed. When you stop tracing. Caution To be able to use the kernel profile. It complements the performance trace by providing even more detailed information about the individual processing steps that are done in the database kernel. Duration How long you want tracing to run If a certain scenario is to be traced. information about frequent and/or expensive execution paths during query processing. After the specified duration. It is also advisable as this kind of tracing can negatively impact performance. Trace execution plans You can trace execution plans in addition to the default trace data. Function profiler The function profiler is a very fine-grained performance tracing tool based on source code instrumentation. or parts of the database that are not accessible by the performance trace.3 Kernel Profiler The kernel profiler is a sampling profiler built into the SAP HANA database. the results are saved to trace files that you can access on the Diagnosis Files tab of the SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. then enter a reasonable value.Option Description Passport trace level If you are activating the performance trace as part of an end-to-end trace scenario with the Process Monitoring Infrastructure (PMI). you must have the SAP_INTERNAL_HANA_SUPPORT role. This avoids the unnecessary recording of irrelevant statements. The kernel profile collects. For more information see. If there is no specific scenario to trace but instead general system performance. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . 168 PUBLIC © 2015 SAP SE or an SAP affiliate company. All rights reserved. it retrieves the call stacks of relevant threads several times. Table 46: Option Description Service(s) to profile The service(s) that you want to profile Wait time The amount of time the kernel profiler is to wait between call stack retrievals When you activate the kernel profiler. Memory limit Memory limit that will stop tracing The kernel profiler can potentially use a lot a memory.4 Kernel Profiler Options Several options are available for configuring the kernel profiler.5. that is KCacheGrind or DOT (default format). navigate to Diagnosis Information Diagnosis Files and use the Collect function. The SQL variant can be used when SAP HANA is online.Administration editor. You cannot analyze these files meaningfully in the SAP HANA studio. it must wait the specified time minus the time the previous retrieval took. To collect this information. application user The database user and/or application user you want to profile Use KCachegrind format to write trace files Output format of trace files (configurable when you stop tracing) 4. To prevent the SAP HANA database from running out of memory due to profiling.5 Diagnosis Information You can collect diagnosis information in the SAP HANA studio and using command line scripts. You activate and configure the kernel profile in the Administration editor on the Trace Configuration tab.5. Database user. Related Information Kernel Profiler Options [page 168] 4. you can specify a memory limit that cannot be exceeded. otherwise choose the Python script. If a wait time is specified. use the SAP HANA studio Administration Editor. but instead must use a tool capable of reading the configured output format. Additional information to judge the measurement quality like table locks and system alerts is also provided. and UI layers. Detailed performance analysis supports the repeated execution of statements thus allows you to check whether performance characteristics are stable. Implicit performance measurement can be triggered via F9 run command and provides execution time information for the database.4. XS. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. You can use this feature to quickly check SQL performance and to determine whether the observed poor performance is caused by system load or variability. SQL Console The extended SQL Console (part of the Catalog perspective) supports implicit and explicit performance measurement while executing a SQL statement. It allows you to easily acquire sound measurement data and to assess whether a SQL statement is problematic.6 Analysis Tools in SAP HANA Web-based Developer Workbench There are a number of tools available for application performance analysis in the SAP HANA Web-based Developer Workbench. 169 . All rights reserved. for each unique statement summary statistics are shown that detail the frequency of their usage as well as aggregated duration metrics. It allows you to analyze if the SAP HANA database is used effectively and efficiently for a given application. Furthermore. thus helps application developers to immediately understand the number and performance of their DB-related statements. 170 PUBLIC © 2015 SAP SE or an SAP affiliate company.Expensive Statements Trace The expensive statements trace in the Trace perspective allows you to quickly analyze how the SQL layer of your SAP HANA system is used (globally or within a specific session/application).if the trace is configured accordingly by the system administrator and you run your application with SAP passport. You can use the expensive statement trace to analyze: ● the overall SQL load in your current system ● the SQL interaction of a specific application. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing . All rights reserved. For each statement you can see metrics such as start time. to figure out if an application that is not currently performing well due to an issue on the application layer or on the database layer. a statistical summary is provided that gives insight on the overall SQL load. Immediate Database Performance Feedback This feature is part of the Immediate Feedback context in WebIDE. It details metrics such as: ● Number of statements executed ● Number of unique statements executed ● Number of result records Last. The expensive statement view supports filtering for passport-based application sessions . number of processed records and many more. for example.It provides a performance overview of all SQL statements which are executed during an Immediate Feedback session. statement and execution. All rights reserved.The Database Performance overview provides 3 metrics for each SQL statement: step. 171 . ● Step: the actual immediate feedback step in which a SQL statement was executed ● Statement: the SQL statement ● Execution time: the execution time of the statement (in milliseconds) You can use this feature to understand the performance behavior of the SQL statements (for example. SAP HANA Troubleshooting and Performance Analysis Guide Tools and Tracing PUBLIC © 2015 SAP SE or an SAP affiliate company. like subsequently taking a dedicated measurement using the SQL Console. finding expensive statements that belong to a particular function) and exploit this understanding for possible further performance investigations. Investigate why the service is inac­ tive. or adding additional storage. For more in­ formation. Resolve the prob­ lem. by checking the service's trace files. Availability 172 PUBLIC © 2015 SAP SE or an SAP affiliate company. log. You may need to acti­ vate tracing first. by checking the service's trace files. in­ cluding non-SAP HANA processes.5 Alerts This section lets you look up an Alert and get specific information on how to handle it and where to find any additional information. Investigate disk Disk usage of proc­ esses. Increase disk space. and trace files is used. Investigate why the service had to restart or be re­ started. Memory Further Informa­ tion Memory Problems Disk Related SAP HANA Troubleshooting and Performance Analysis Guide Alerts . for ex­ ample by shrinking volumes. Availability 4 Restarted Serv­ ices Identifies services that have re­ started since the last time the check was performed. 2 Disk Usage Determines what percentage of each disk contain­ ing data. Availability 1 Host physical memory usage Investigate mem­ Determines what ory usage of proc­ percentage of to­ tal physical mem­ esses. see the trace files. ory available on the host is used. deleting diagnosis files. for exam­ ple. This includes space used by non-SAP HANA files. All processes con­ suming memory are considered. 3 Inactive services Identifies inactive services. Table 47: Alerts Check ID Check Name Description User Action Category 0 Internal statistics server problem Identifies internal statistics server problem. for example. All rights reserved. 200.000 and 7.ini file.ini file is be­ tween 100. In the 'transaction' Configuration section of the in­ dexserver. 173 . Transactional Problems SAP HANA Troubleshooting and Performance Analysis Guide Alerts User Action Category Further Informa­ tion CPU CPU Related Configuration Delta Merge PUBLIC © 2015 SAP SE or an SAP affiliate company.Check ID Check Name Description 5 Host CPU Usage Investigate CPU Determines the usage percentage CPU idle time on the host and therefore whether or not CPU resources are running low. increase the value of the 'size' parameter.000 and 7. Change in SYS­ TEM layer the pa­ rameter active in section(s) merge­ dog to yes 12 Memory usage of name server Determines what percentage of allo­ cated shared memory is being used by the name server on a host.200.ini file. set the 'lock_wait_time­ out' parameter to a value between 100. Increase the Memory shared memory size of the name server. In the 'top­ ology' section of the nameserver.000. 10 Delta merge (mergdog) config­ uration Determines whether or not the 'active' parameter in the 'mergedog' section of system configuration file(s) is 'yes'. mer­ gedog is the sys­ tem process that periodically checks column ta­ bles to determine whether or not a delta merge oper­ ation needs to be executed.000 for the System layer. All rights reserved. Memory Problems 16 Lock wait timeout configuration Determines whether the 'lock_waittimeout' parameter in the 'transaction' sec­ tion of the index­ server. 174 PUBLIC © 2015 SAP SE or an SAP affiliate company. Memory Prob­ lems. sends a sum­ mary e-mail to specified recipi­ ents. Resolve the event and then mark it as resolved by ex­ ecuting the SQL statement ALTER SYSTEM SET EVENT HANDLED '<host>:<port>' <id>. SAP HANA Administration Guide > Table Par­ titioning 21 Internal Event Identifies internal database events. there have been any alerts since the last check and if so.Check ID Check Name Description User Action Category Further Informa­ tion 17 Record count of non-partitioned column-store ta­ bles Determines the number of records in non-partitioned column-store ta­ bles.000. SAP HANA Administration Guide > Table Par­ titioning 20 Table growth of non-partitioned column-store ta­ ble Determines the growth rate of non-partitioned columns tables.000 (2 billion) rows). Consider partition­ Memory ing the table. Partitioning need only be considered if tables are ex­ pected to grow rapidly (a non-par­ titioned table can­ not contain more than 2. Current table size is not critical. Consider partition­ Memory ing the table only if you expect it to grow rapidly. Memory Prob­ lems. Availability 22 Notification of all Determines Investigate the Availability alerts whether or not alerts. All rights reserved. SAP HANA Troubleshooting and Performance Analysis Guide Alerts .000. The maxi­ mum number of permitted connec­ tions is configured in the "session" section of the in­ dexserver. All rights reserved. For example. sends a summary e-mail to specified recipients. SAP HANA Troubleshooting and Performance Analysis Guide Alerts Sessions PUBLIC © 2015 SAP SE or an SAP affiliate company. Investigate why the maximum number of permit­ ted open connec­ tions is being ap­ proached. Availability there have been any high priority alerts since the last check and if so. the assigned service is not active. 25 Open connections Determines what percentage of the maximum number of permitted SQL connections are open. sends a sum­ mary e-mail to specified recipi­ ents. the re­ moval of a host failed. 175 .ini file.Check ID Check Name 23 Description User Action Category Notification of me­ Determines Investigate the Availability dium and high pri­ whether or not alerts. or the re­ moval of a service was performed in­ correctly. 26 Unassigned vol­ umes Identifies volumes that are not as­ signed a service. ority alerts there have been Further Informa­ tion any medium and high priority alerts since the last check and if so. 24 Notification of Determines Investigate the high priority alerts whether or not alerts. Configuration Investigate why the volume is not assigned a service. A table parti­ tion cannot con­ tain more than 2. 28 Most recent save­ point operation Determines how long ago the last savepoint was de­ fined.000 (2 billion) rows. 29 Size of delta stor­ age of columnstore tables Determines the size of the delta storage of column tables. consistent image of the data­ base was per­ sisted to disk.000. how long ago a com­ plete. in the partitions of column-store ta­ bles. choose the \"Disk Full Events \" link and mark the event as han­ dled. All rights reserved. User Action Category Further Informa­ tion Memory Memory Prob­ lems. Check Internal Disk Full Event (Alert 30) 176 PUBLIC © 2015 SAP SE or an SAP affiliate company. that is. Investigate the Memory delta merge his­ tory in the moni­ toring view M_DELTA_MERGE _STATISTICS. SAP HANA Administration Guide > Table Par­ titioning Backup SAP HANA Troubleshooting and Performance Analysis Guide Alerts .000. A disk-full event causes your data­ base to stop and must be resolved. Investigate why there was a delay defining the last savepoint and consider trigger­ ing the operation manually by exe­ cuting the SQL statement ALTER SYSTEM SAVE­ POINT.Check ID Check Name Description 27 Record count of column-store ta­ ble partitions Consider reparti­ Determines the number of records tioning the table. Resolve the diskDisk full event as fol­ lows: In the Ad­ ministration Editor on the Overview tab. Delta Merge 30 Check internal disk full event Determines whether or not the disks to which data and log files are written are full. Consider merging the table delta manually. execute the SQL statements ALTER SYSTEM SET EVENT AC­ KNOWLEDGED '<host>:<port>' <id> and ALTER SYSTEM SET EVENT HANDLED '<host>:<port>' <id>. Alternatively. It is also recommended that you perform a full data backup. Obtain a valid li­ Availability cense and install it. reconfigure the log mode of your sys­ tem to "normal". 177 . you must restart the database system to activate the changes. In the "persis­ tence" section of the global. Log mode "legacy" does not support point-in-recovery and is not recom­ mended for pro­ ductive systems.Backup in-time recovery. For the exact expiration date. A disk-full event causes your data­ base to stop and must be resolved. Authori­ zation and Licens­ ing 32 Log mode LEG­ ACY Determines whether or not the database is run­ ning in log mode "legacy".ini con­ figuration file. Configuration Pa­ rameter Issues SAP HANA Troubleshooting and Performance Analysis Guide Alerts Category Further Informa­ tion PUBLIC © 2015 SAP SE or an SAP affiliate company. see the monitoring view M_LICENSE.Check ID Check Name Description User Action 31 License expiry Determines whether or not the disks to which data and log files are written are full. Security. set the parameter "log_mode" to "normal" for the System layer. When you change the log mode. All rights reserved. If you need point. reconfigure the log mode of your sys­ tem to "normal". If you need point. and perform a new data backup as soon as possible. you must restart the database system to activate the changes. Investigate why the volume is not available. SAP HANA Admin­ istration Guide > Backing up and Recovering the SAP HANA Data­ base 34 Unavailable vol­ umes Determines whether or not all volumes are avail­ able. Without a data backup. SAP HANA Admin­ istration Guide > Backing up and Recovering the SAP HANA Data­ base 37 Age of most re­ cent data backup Perform a data Determines the age of the most re­ backup as soon as possible. Log mode "overwrite" does not support point-in-recovery (only recovery to a data backup) and is not recom­ mended for pro­ ductive systems. Perform a data backup as soon as possible. Investigate why Backup the last data backup failed.Check ID Check Name Description User Action 33 Log mode OVER­ WRITE Determines whether or not the database is run­ ning in log mode "overwrite". re­ solve the problem.Backup in-time recovery. Backup SAP HANA Admin­ istration Guide > Backing up and Recovering the SAP HANA Data­ base 36 Status of most re­ cent data backup Determines whether or not the most recent data backup was suc­ cessful.ini con­ figuration file. your database cannot be recovered. 178 PUBLIC © 2015 SAP SE or an SAP affiliate company. It is also recommended that you perform a full data backup. cent successful data backup. Configuration SAP HANA Admin­ istration Guide > Backing up and Recovering the SAP HANA Data­ base 35 Existence of data backup Determines whether or not a data backup ex­ ists. When you change the log mode. In the "persis­ tence" section of the global. set the parameter "log_mode" to "normal" for the System layer. All rights reserved. Category Backup Further Informa­ tion SAP HANA Admin­ istration Guide > Backing up and Recovering the SAP HANA Data­ base SAP HANA Troubleshooting and Performance Analysis Guide Alerts . Backup SAP HANA Admin­ istration Guide > Backing up and Recovering the SAP HANA Data­ base 39 Long-running statements Sessions/Trans­ Identifies longInvestigate the actions running SQL state­ statement.Check ID Check Name Description User Action Category Further Informa­ tion 38 Status of most re­ cent log backups Determines whether or not the most recent log backups for serv­ ices and volumes were successful. Transactional Problems 40 Total memory us­ age of columnstore tables Consider partition­ Memory Determines what percentage of the ing or repartition­ effective allocation ing the table. Close the cursor in Sessions/Trans­ the application. see the table HOST_LONG_IDL E_CURSOR (_SYS_STATIS­ TICS). 42 Long-idling cur­ sors Identifies longidling cursors. For more informa­ tion. see the table _SYS_STATIS­ TICS. All rights reserved. more information. 179 . see the table _SYS_STATIS­ TICS. or actions kill the connection by executing the SQL statement ALTER SYSTEM DISCONNECT SESSION <LOGI­ §ok„¨§±â…v@i˛flÇQòŁM¼º˜ TION_ID>. SAP HANA Troubleshooting and Performance Analysis Guide Alerts Availability Transactional Problems PUBLIC © 2015 SAP SE or an SAP affiliate company. the cumu­ lative size of all of a table's columns and internal struc­ tures) Memory Problems 41 In-memory Data­ Store activation Determines whether or not there is a problem with the activation of an in-memory DataStore object. For more information. Investigate why the log backup failed and resolve the problem.HOST_LONG §ok’¨´±û™V §ok†¨¯±àłv\i
fl˜Ç^ò—Mºº˜ MENTS.GLOBAL_DE §ok„¨¹±ë—vWifl Ç\ÒÀ TOR_STATUS and SAP Note 1665553. limit is being con­ sumed by individ­ ual column-store tables as a whole (that is. For ments. time dump files (*rtedump*) have been generated in the trace directory of the system. User Action Category Further Informa­ tion Diagnosis Files SAP HANA Troubleshooting and Performance Analysis Guide Alerts . Memory Problems 46 RTEdump files Identifies new run­ Check contents of the dump files. These contain in­ formation about. limit is being con­ sumed by the main storage of in­ dividual columnstore tables. 180 PUBLIC © 2015 SAP SE or an SAP affiliate company. build. CPU. limit a service is using. Memory Problems 44 Licensed memory usage Determines what percentage of li­ censed memory is used. You can see the peak memory allocation since installation in the system view M_LICENSE (col­ umn PROD­ UCT_USAGE).Check ID Check Name Description 43 Memory usage of services Check for services Memory Determines what that consume a lot percentage of its effective allocation of memory. Memory Problems 45 Memory usage of main storage of column-store ta­ bles Determines what Consider partition­ Memory percentage of the ing or repartition­ effective allocation ing the table. running threads. for example. and so on. Increase licensed Memory amount of main memory. All rights reserved. loaded modules. ]0˜Si˚‡�ˇ?j8MÒn.Check ID Check Name Description User Action Category Further Informa­ tion 47 Long-running seri­ alizable transac­ tions Identifies longrunning serializa­ ble transactions. Transactional Problems 49 Long-running Identifies longblocking situations running blocking situations. see the table HOST_LONG_SE­ . Sessions/Trans­ actions Transactional Problems SAP HANA Troubleshooting and Performance Analysis Guide Alerts Investigate the blocking and blocked transac­ tions and if appro­ priate cancel one of them. PUBLIC © 2015 SAP SE or an SAP affiliate company. 181 .]0Si˚Š“E TED_WRITE_TRA NSACTION (_SYS_STATIS­ TICS). Sessions/Trans­ actions Transactional Problems 48 Long-running un­ committed write transactions Identifies longrunning uncom­ mitted write trans­ actions.]0Sl˚ƒ� ?w8EÒNsJª˚|<Ù‚å TION (_SYS_STA­ TISTICS). For more information.]0Sa˚‘� ?`8XÒˇNsJ¼˚~˝− TION_ID>. All rights reserved. For more information. see the table HOST_UNCOM­ . Close the serializ­ able transaction in the application or kill the connection by executing the SQL statement ALTER SYSTEM DISCONNECT SESSION <LOGI­ .]0Sa˚‘� ?`8XÒˇNsJ¼˚~˝− TION_ID>. Close the uncom­ Sessions/Trans­ mitted transaction actions in the application or kill the connec­ tion by executing the SQL statement ALTER SYSTEM DISCONNECT SESSION <LOGI­ . Diagnosis Files See KBA 1977208 Crashdump files Identifies new crashdump files that have been generated in the trace directory of the system. columnstore ta­ bles have been un­ loaded from mem­ ory. Diagnosis Files 54 Savepoint dura­ tion Identifies longrunning savepoint operations. User Action SAP HANA Troubleshooting and Performance Analysis Guide Alerts . All rights reserved. Check the con­ tents of the dump files. Check disk I/O performance. problem with trace file rota­ tion or a high num­ ber of crashes). Memory Memory Problems 182 Description PUBLIC © 2015 SAP SE or an SAP affiliate company. Backup CPU Related Root Causes and Solu­ tions. An unusually large number of files can indicate a problem with the database (for ex­ ample. This can indi­ cate performance issues. Diagnosis Files 53 Pagedump files Identifies new pa­ gedump files that have been gener­ ated in the trace directory of the system. sis files written by the system (ex­ cluding zip-files). Check the diagno­ sis files in the SAP HANA studio for details. I/O Related Root Causes and Solutions 55 Columnstore un­ loads Check sizing with Determines how respect to data many columns in distribution.Check ID Check Name 50 51 Size of diagnosis files 52 Category Further Informa­ tion Investigate the di­ Number of diagno­ Determines the sis files number of diagno­ agnosis files. Diagnosis Files See KBA 1977162 Identifies large di­ agnosis files. Un­ usually large file sizes can indicate a problem with the database. Check the con­ tents of the dump files. The python trace affects sys­ tem performance. 58 Plan cache size Determines whether or not the plan cache is too small.Check ID Check Name Description User Action 56 Python trace ac­ tivity Determines whether or not the python trace is ac­ tive and for how long. deactivate the python trace in the relevant con­ figuration file. Investigate block­ Sessions/Trans­ ing and blocked actions transactions and if appropriate cancel some of them. in­ crease the value of the 'plan_cache_size' parameter. In the 'sql' section of the index­ server. This means that asynchronous reads are blocking and behave al­ most like synchro­ nous reads. Please refer to SAP Note 1930979. Increase the size Memory of the plan cache. I/O Related Root Causes and Solu­ tions SAP HANA Troubleshooting and Performance Analysis Guide Alerts Category Further Informa­ tion Security Disk PUBLIC © 2015 SAP SE or an SAP affiliate company. 59 Percentage of transactions Determines the percentage of transactions that are blocked. If no longer re­ Diagnosis Files quired. This might have nega­ tive impact on SAP HANA I/O performance in certain scenarios. 57 Secure store file system (SSFS) consistency Determines if the secure storage file system (SSFS) is consistent regard­ ing the database Check and make sure that the se­ cure storage file system (SSFS) is accessible and consistent regard­ ing the database. 183 .ini file. Transactional Problems 60 Sync/Async read ratio Identifies a bad trigger asynchro­ nous read ratio. All rights reserved. Security PUBLIC © 2015 SAP SE or an SAP affiliate company. Security Granting of SAP_INTER­ Äa*“ÀI˙JÄ1ç?
ÕðÀzptŁU�2ùæ'o PORT role Determines if the internal support role (SAP_INTER­ Äa*“ÀI˙JÄ1ç?
ÕðÀzptŁU�2ùæ'o PORT) is currently granted to any da­ tabase users. It is recom­ mended that you disable the pass­ word lifetime check of technical users so that their password never expires (ALTER USER <user­ name> DISABLE PASSWORD LIFE­ TIME). This means that asynchronous writes are blocking and behave al­ most like synchro­ nous writes. SAP HANA Troubleshooting and Performance Analysis Guide Alerts .Check ID Check Name Description 61 Sync/Async write ratio 62 63 184 Category Further Informa­ tion Please refer to Identifies a bad SAP Note trigger asynchro­ 1930979 nous write ratio. the user will be locked. expire in line with the configured password policy. revoke the role from them. This might have nega­ tive impact on SAP HANA I/O performance in certain scenarios. Disk I/O Related Root Causes and Solu­ tions Expiration of data­ base user pass­ words Identifies data­ Change the pass­ base users whose word of the data­ password is due to base user. If not. User Action Check if the corre­ sponding users still need the role. All rights reserved. If the user in question is a technical user. If the password ex­ pires. this may impact application availa­ bility. and resolve the is­ sue. exceeds a given threshold. Availability 71: Row store frag­ mentation Check for frag­ mentation of row store. Investigate why the log backup runs for too long. For more details please see SAP HANA Administra­ tion Guide. Enable automatic Backup log backup. Backup SAP HANA Admin­ istration Guide 66 Storage snapshot is prepared Determines whether or not the period. Security 65 Runtime of the log backups currently running Determines whether or not the most recent log backup terminates in the given time. All rights reserved. Investigate why the storage snap­ shot was not con­ firmed or aban­ doned. 185 . the repo­ sitory) after a sys­ tem upgrade. 70 Consistency of in­ ternal system components after system upgrade Verifies the con­ Contact SAP sup­ sistency of sche­ port.Check ID Check Name 64 Description User Action Category Total memory us­ Determines what age of table-based percentage of the audit log effective allocation limit is being con­ sumed by the da­ tabase table used for table-based audit logging. Memory SAP HANA Troubleshooting and Performance Analysis Guide Alerts Implement SAP Note 1813245. and re­ solve the issue. Further Informa­ tion SAP HANA Admin­ istration Guide PUBLIC © 2015 SAP SE or an SAP affiliate company. Backup SAP HANA Admin­ istration Guide 67 Table growth of rowstore tables Determines the growth rate of rowstore tables Try to reduce the size of row store table by removing unused data Memory 68 Total memory us­ age of row store Determines the current memory size of a row store used by a service Investigate mem­ ory usage by row store tables and consider cleanup of unused data Memory 69 Enablement of au­ tomatic log backup Determines whether auto­ matic log backup is enabled. during which the data­ base is prepared for a storage snap­ shot. mas and tables in internal system components (for example. Consider export­ ing the content of the table and then truncating the ta­ ble. 73 Overflow of row­ store version space Determines the overflow ratio of the rowstore ver­ sion space. All rights reserved. Check whether the sys­ tem has been fre­ quently and un­ usually restarting services. If it has. 186 PUBLIC © 2015 SAP SE or an SAP affiliate company. Make sure that log Backup backups are being automatically cre­ ated and that there is enough space available for them. Identify the con­ Memory nection or trans­ action that is blocking version garbage collec­ tion. then resolve the root cause of this issue and create log backups as soon as possible.Check ID Check Name Description User Action 72 Number of log segments Determines the number of log seg­ ments in the log volume of each serviceCheck for number of log seg­ ments. kill the blocking connection or transaction. Category Further Informa­ tion Transactional Problems SAP HANA Troubleshooting and Performance Analysis Guide Alerts . You can do this in the SAP HANA studio by executing the "MVCC Blocker Connection" and "MVCC Blocker Transaction" statements availa­ ble on the System Information tab of the Administration editor. If possible. If possible. Identify the con­ Memory nection or trans­ action that is blocking version garbage collec­ tion. 187 . Transactional Problems 75 Rowstore version space skew Determines whether the row­ store version chain is too long. If possible. Transactional Problems 76: Discrepancy be­ tween host server times Identifies discrep­ Check operating system time set­ ancies between the server times of tings. hosts in a scaleout system. SAP HANA Troubleshooting and Performance Analysis Guide Alerts User Action Category Further Informa­ tion Configuration PUBLIC © 2015 SAP SE or an SAP affiliate company. Identify the con­ Memory nection or trans­ action that is blocking version garbage collec­ tion.Check ID Check Name Description 74 Overflow of meta­ Determines the data version space overflow ratio of the metadata ver­ sion space. kill the blocking connection or transaction. kill the blocking connection or transaction. You can do this in the SAP HANA studio by executing the "MVCC Blocker Connection" and "MVCC Blocker Transaction" statements availa­ ble on the System Information tab of the Administration editor. You can do this in the SAP HANA studio by executing the "MVCC Blocker Connection" and "MVCC Blocker Transaction" statements availa­ ble on the System Information tab of the Administration editor. All rights reserved. and then check the corresponding indexserver alert traces.ini/ [inifile_checker]. add the parame­ ter(s) as an excep­ tion in global. Determine which tables encoun­ tered the table replication error using system view M_ASYNCHRO­ � E×ý¢ôñ(Ù·0p
¥ûÿ BLE_REPLICAS. Investigate the Disk disk usage of the database. 80 Availability of asynchronous ta­ ble replication Monitors error messages related to asynchronous table replication. 78 Connection be­ tween systems in system replication setup Identifies closed connections be­ tween the primary system and a sec­ ondary system. 79 Configuration con­ sistency of sys­ tems in system replication setup Identifies configu­ ration parameters that do not have the same value on the primary sys­ tem and a secon­ dary system. adjust the configuration. traces and backups are con­ sidered. network prob­ lem) and resolve the issue. If connections are closed. If different values are acceptable. Most configuration pa­ rameters should have the same value on both sys­ tems because the secondary system has to take over in the event of a dis­ aster. Category Availability Further Informa­ tion SAP HANA Admin­ istration Guide SAP HANA Admin­ istration Guide Availability SAP HANA Troubleshooting and Performance Analysis Guide Alerts . 188 PUBLIC © 2015 SAP SE or an SAP affiliate company. logs. See sys­ tem view M_DISK_USAGE for more details.Check ID Check Name Description User Action 77 Database disk us­ age Determines the to­ tal used disk space of the data­ base. the pri­ mary system is no longer being repli­ cated. All data. If the identified Configuration configuration pa­ rameter(s) should have the same value in both sys­ tems. All rights reserved. Investigate why connections are closed (for exam­ ple. ini file. 85 Insecure SSFS en­ If the encryption Change the SSFS cryption configu­ keys of your SAP master key of the ration HANA system are SystemPKI as not changed after soon as possible Security SAP HANA Secur­ ity Guide installation it is not granted that the initial keys are unique. in­ crease the value of the "total_size" parameter. In the cached view "view_cache" sec­ Further Informa­ tion tion of the index­ server. All rights reserved. sistency errors and the affected tables 84 Insecure SSFS en­ If the encryption Change the SSFS cryption configu­ keys of your SAP master key as ration HANA system are soon as possible Security SAP HANA Secur­ ity Guide not changed after installation it is not granted that the initial keys are unique. 82 Timezone conver­ Compares SAP Update SAP HANA Configuration sion HANA internal internal timezone timezone conver­ tables (refer to sion with Operat­ SAP note ing System time­ 1932132).Check ID Check Name Description User Action Category 81 Cached view size Determines how Increase the size Memory much memory is of the cached occupied by view. 189 . 83 Table consistency Identifies the num­ Contact SAP sup­ ber of table con­ Availability port. zone conversion. SAP HANA Troubleshooting and Performance Analysis Guide Alerts PUBLIC © 2015 SAP SE or an SAP affiliate company. local (for sin­ tion to a secure ports gle host HANA) or setting. see to . SAP HANA Troubleshooting and Performance Analysis Guide Alerts . 190 PUBLIC © 2015 SAP SE or an SAP affiliate company.Check ID Check Name Description User Action Category Further Informa­ tion 86 Insecure configu­ Other parameters Change configura­ Security SAP HANA Secur­ ration of TREXNet than .ini -> com­ munication -> lis­ teninterface will open internal com­ munication of HANA for external exploits. ity Guide HANA) for global. For more parameters equal information.all or .global SAP HANA Secur­ (for distributed ity Guide. All rights reserved. Based on this. Although various features included in SAP HANA options and capabilities are cited in the SAP HANA platform documentation. Depending on your SAP HANA installation license type. including the type and version of the back-end systems the SAP HANA administration and development tools are connected to. or wish to discuss licensing features available in SAP HANA options. All rights reserved. SAP HANA Troubleshooting and Performance Analysis Guide HANA Options: Performance Analysis PUBLIC © 2015 SAP SE or an SAP affiliate company. customers do not necessarily have the right to use features included in SAP HANA options and capabilities.1 Tools and Tracing This section gives you an overview of the tools and tracing options available for SAP HANA dynamic tiering. For customers to whom these license restrictions apply.sap. some of the features and tools described in the SAP HANA platform documentation may only be available in the SAP HANA options and capabilities.1 SAP HANA Dynamic Tiering Identify and resolve specific performance issues and enhance the performance of SAP HANA dynamic tiering. 6. The documentation for the SAP HANA optional components is available in SAP Help Portal at http://help. the use of features included in SAP HANA options and capabilities in a production system requires purchasing the corresponding software license(s) from SAP.1. Note Troubleshooting information for SAP HANA dynamic tiering is in the SAP HANA Dynamic Tiering: Administration Guide. Caution SAP HANA server software and tools can be used for several SAP HANA platform and options scenarios as well as the respective capabilities used in these scenarios. There are several types of licenses available for SAP HANA.com/hana_options. 6. If you have additional questions about what your particular license provides. 191 .6 HANA Options: Performance Analysis SAP HANA options provide additional functions you can use together with the SAP HANA base edition. each SAP HANA edition governs the options and capabilities available. which may be released independently of an SAP HANA Platform Support Package Stack (SPS). These topics describe what steps you can take to identify and resolve specific performance issues and what you can do to enhance the performance of your SAP HANA options. The availability of these is based on the available SAP HANA licenses and the SAP HANA landscape. please contact your SAP account team representative. 'federationexecution') = 'debug' WITH RECONFIGURE. If the timing shown is small.ini'.ini'. Alternatively.2 Query Plan Analysis The query plan shows the various operators involved in the query execution. you should review the query plan. To enable federation trace: ALTER SYSTEM ALTER CONFIGURATION ('indexserver. then SAP HANA dynamic tiering is probably not the cause of the performance problem. 'execution_strategies') = 'remote' WITH RECONFIGURE. Queries referencing both SAP HANA tables and SAP HANA dynamic tiering tables are either: ● Executed in SAP HANA by pulling data from SAP HANA dynamic tiering ● Relocated to SAP HANA dynamic tiering. All rights reserved. 2. where the data is pulled from SAP HANA Generally. 192 PUBLIC © 2015 SAP SE or an SAP affiliate company. Generally. and you think the optimizer is shipping the upper limit of query fragments.1. 6. relocating the query yields better performance. since SAP HANA dynamic tiering involves tables with large amounts of data.1 Federation Trace Federation trace can be turned ON to diagnose most issues with SAP HANA dynamic tiering. SAP HANA Troubleshooting and Performance Analysis Guide HANA Options: Performance Analysis .6. Review the visual query plan that shows the timings for various sub-trees. Change the execution strategy from 'auto' to 'remote': ALTER SYSTEM ALTER CONFIGURATION ('esserver. ALTER SYSTEM CLEAR SQL PLAN CACHE. Changing the Execution Strategy to Remote If the execution strategy 'auto' mode is not yielding the best plan.1. 'SYSTEM') SET ('row_engine'. If this operator is directly under the top-level Project node. 'fedtrace') = 'debug' WITH RECONFIGURE. then the entire query has been either pushed down or relocated to an SAP HANA dynamic tiering node. Another reason to use the latter strategy is when the SAP HANA dynamic tiering table is too large to fit in SAP HANA. it may be preferable to use the latter strategy. 'SYSTEM') SET ('trace'.1. Federation trace generates tracing information in the indexserver. If your query involves both SAP HANA tables and SAP HANA dynamic tiering tables and you are experiencing poor performance.ini'. try the following procedure to change the execution strategy from 'auto' to 'remote': 1. The Remote Row Scan operator deals with a query fragment executed by an SAP HANA dynamic tiering node.ini trace file. 'SYSTEM') SET ('trace'. you can query M_REMOTE_STATEMENTS to show timing results for query fragments executed on SAP HANA dynamic tiering. ALTER SYSTEM ALTER CONFIGURATION ('indexserver. Clear the query plan cache. If neither execution strategy improves performance. total count of values. We recommend that simple statistics. If the actual table has significantly different number of rows. you may be missing statistics on the tables that may prevent the optimizer from choosing an optimal query plan. Review your query to see if there are any unsupported operators.Changing the Execution Strategy to Auto If the execution strategy is set to 'auto'. The query optimizer decides what to push down to SAP HANA dynamic tiering based on the capability supported by the option. then it lets the SAP HANA execution engine compensate for it. then the query plan chosen may not be optimal. 193 . Related Information Statistics [page 193] 6. Simple Creates simple statistics for each column in the table. maximum value. or builtins and see if you can rewrite the query without them. we recommend that you create statistics on the extended tables. then the optimizer chooses the best strategy for executing the query: either relocating the query to SAP HANA dynamic tiering. Finally. there may be a capability issue. builtin. or other item that SAP HANA dynamic tiering does not understand.2.1 Statistics Statistics help the query optimizer in choosing the right query plan. To create simple statistics on column c1: CREATE STATISTICS ON TAB1 (c1) TYPE SIMPLE To create simple statistics on all columns: CREATE STATISTICS ON TAB1 TYPE SIMPLE SAP HANA Troubleshooting and Performance Analysis Guide HANA Options: Performance Analysis PUBLIC © 2015 SAP SE or an SAP affiliate company. If the query deals with some operator. the query optimizer assumes the table size to be 1 million rows. 'auto' provides best performance. are present on key columns of extended tables. or executing the query in SAP HANA. at the very least. and null counts. In most cases. Missing statistics may prevent the query optimizer from selecting the optimal query plan. distinct count of values. 1. SAP HANA currently supports two types of statistics: Histogram Creates a multi-bucket histogram.ini'.1. 'execution_strategies') = 'auto' WITH RECONFIGURE. Histogram statistics provides much better estimates for range queries. When no statistics for an extended table are present. Statistics are: minimum value. 'SYSTEM') SET ('row_engine'. All rights reserved. Change the execution strategy from 'remote' to 'auto': ALTER SYSTEM ALTER CONFIGURATION ('esserver. To ensure that optimizer has the correct information. use multiple files in the same IMPORT FROM statement. note that this will put heavy demand on the delta memory. Currently. like SAP HANA. Breaking into multiple files for IMPORT FROM yields better performance than a single file import. IMPORT FROM does not support the THREADS parameter on the import command and hence it is better to break the files into multiple files if you are planning on loading hundreds of millions of rows. although singleton writes are supported. 6.1. When importing very large amounts of data into an extended table. and recommends the optimal loading mechanisms. you can do this load serially. Singleton writes are not the best use case for SAP HANA dynamic tiering.3.3. SAP HANA Troubleshooting and Performance Analysis Guide HANA Options: Performance Analysis . SAP HANA dynamic tiering is optimized for batch writes.1 IMPORT FROM Statement If you have a CSV file for the data to be loaded.1. then INSERT-SELECT is a better loading method than the IMPORT FROM statement. Alternatively.1.To create histogram-type statistics on column c1: CREATE STATISTICS ON TAB1 (c1) TYPE HISTOGRAM 6. The IMPORT FROM statement is sent directly to the SAP HANA dynamic tiering node for the data load. When dealing with a delta-enabled extended table.3 Data Loading Performance SAP HANA dynamic tiering supports all data loading methods for extended tables. 194 PUBLIC © 2015 SAP SE or an SAP affiliate company.2 INSERT Statement with SELECT Statement If the data is present in another SAP HANA table. 6. the IMPORT FROM statement is by far the best method for loading data into an SAP HANA dynamic tiering table. However. All rights reserved. In general. The SAP HANA dynamic tiering node does a parallel fetch from SAP HANA. you can run these IMPORT FROM statements in parallel if required. Make sure delta memory is appropriately configured. This section explores the various mechanisms for inserting data into extended tables. The query optimizer tries to relocate the INSERT-SELECT on an extended table to the SAP HANA dynamic tiering node. thereby speeding up the INSERT-SELECT. Re-enable binary load (instead of the ASCII load): ALTER SYSTEM ALTER CONFIGURATION ('esserver. SAP HANA dynamic tiering converts an array-insert into a LOAD statement on the SAP HANA dynamic tiering node. All rights reserved. Re-enable the bulk load mechanism for optimizing array inserts: ALTER SYSTEM ALTER CONFIGURATION ('esserver.3 Parameterized Array Inserts Array insert is by far the most optimal mechanism to load data into an extended table. 'bulk_inserts_as_load') = 'true' WITH RECONFIGURE. 'SYSTEM') SET ('row_engine'. Both parameters are 'true' by default. 'bulk_load_as_binary') = 'true' WITH RECONFIGURE.3. If you need to re-enable the defaults: 1. Bulk load is controlled by the bulk_inserts_as_load and bulk_load_as_binary parameters. 2.1. 195 . SAP HANA Troubleshooting and Performance Analysis Guide HANA Options: Performance Analysis PUBLIC © 2015 SAP SE or an SAP affiliate company. 'SYSTEM') SET ('row_engine'.ini'.ini'.6. sap. SAP HANA Troubleshooting and Performance Analysis Guide Important Disclaimer for Features in SAP HANA Platform. some of the features and tools described in the SAP HANA platform documentation may only be available in the SAP HANA options and capabilities. customers do not necessarily have the right to use features included in SAP HANA options and capabilities. There are several types of licenses available for SAP HANA. The availability of these is based on the available SAP HANA licenses and the SAP HANA landscape.com/hana_options. including the type and version of the back-end systems the SAP HANA administration and development tools are connected to.Important Disclaimer for Features in SAP HANA Platform. Options and Capabilities . Options and Capabilities SAP HANA server software and tools can be used for several SAP HANA platform and options scenarios as well as the respective capabilities used in these scenarios. please contact your SAP account team representative. The documentation for the SAP HANA optional components is available in SAP Help Portal at http:// help. Although various features included in SAP HANA options and capabilities are cited in the SAP HANA platform documentation. each SAP HANA edition governs the options and capabilities available. the use of features included in SAP HANA options and capabilities in a production system requires purchasing the corresponding software license(s) from SAP. Based on this. 196 PUBLIC © 2015 SAP SE or an SAP affiliate company. Depending on your SAP HANA installation license type. which may be released independently of an SAP HANA Platform Support Package Stack (SPS). For customers to whom these license restrictions apply. All rights reserved. If you have additional questions about what your particular license provides. or wish to discuss licensing features available in SAP HANA options. Accessibility The information contained in the SAP documentation represents SAP's current view of accessibility criteria as of the date of publication. SAP does not warrant the correctness and completeness of the Code given herein. SAP reserves the right to use the masculine form of the noun and pronoun. Internet Hyperlinks The SAP documentation may contain hyperlinks to the Internet. These hyperlinks are intended to serve as a hint about where to find related information. Gender-Neutral Language As far as possible. it is in no way intended to be a binding guideline on how to ensure accessibility of software products.sap. SAP shall not be liable for any damages caused by the use of related information unless damages have been caused by SAP's gross negligence or willful misconduct. this document does not result in any direct or indirect contractual obligations of SAP.Important Disclaimers and Legal Information Coding Samples Any software coding and/or code lines / strings ("Code") included in this documentation are only examples and are not intended to be used in a productive system environment. SAP HANA Troubleshooting and Performance Analysis Guide Important Disclaimers and Legal Information PUBLIC © 2015 SAP SE or an SAP affiliate company. SAP does not warrant the availability and correctness of this related information or the ability of this information to serve a particular purpose. Furthermore. however. This is to ensure that the documentation remains comprehensible. unless damages were caused by SAP intentionally or by SAP's gross negligence. 197 . All links are categorized for transparency (see: http://help. or a gender-neutral noun (such as "sales person" or "working days") is used. The Code is only intended to better explain and visualize the syntax and phrasing rules of certain coding. SAP documentation is gender neutral. the reader is addressed directly with "you". the third-person singular cannot be avoided or a gender-neutral noun does not exist. If when referring to members of both sexes.com/disclaimer). however. does not apply in cases of wilful misconduct or gross negligence of SAP. SAP in particular disclaims any liability in relation to this document. and SAP shall not be liable for errors or damages caused by the usage of the Code. All rights reserved. This disclaimer. Depending on the context. com/corporate-en/legal/copyright/ index. The information contained herein may be changed without prior notice. Please see http://www. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. if any.sap. National product specifications may vary. Nothing herein should be construed as constituting an additional warranty. The only warranties for SAP or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services. All other product and service names mentioned are the trademarks of their respective companies. All rights reserved. These materials are provided by SAP SE or an SAP affiliate company for informational purposes only. and SAP or its affiliated companies shall not be liable for errors or omissions with respect to the materials. . SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries.www. without representation or warranty of any kind.epx for additional trademark information and notices. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company.com/contactsap © 2015 SAP SE or an SAP affiliate company.sap.