Understanding Oracle RAC 12.1.0.2 Internals the Cache Fusion Edition

Understanding Oracle RAC (12.1.0.2) Internals: The Cache Fusion Edition Markus Michalewicz Director of Product Management Oracle Real Application Clusters (RAC) October 1st, 2014 @OracleRACpm http://www.linkedin.com/in/markusmichalewicz http://www.slideshare.net/MarkusMichalewicz Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 3 All rights reserved.The Secret to the Scalability of Any System – A straight vertical line Simplification 1 Copyright © 2014. Oracle and/or its affiliates. Simplification 2 4 . Oracle and/or its affiliates. 5 .Three Alternative Architectures – The Same Idea Applies Copyright © 2014. All rights reserved. Oracle and/or its affiliates. All rights reserved. 6 .Three Alternative Architectures – The Same Idea Applies Copyright © 2014. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 7 . Oracle and/or its affiliates.Program Agenda 1 Cache Fusion Overview 2 (Dynamic Re-) Mastering 3 Handling Contentions 4 RAC Meets Multitenant 5 RAC Meets In-Memory Copyright © 2014. Oracle and/or its affiliates. Oracle Confidential – Internal/Restricted/Highly Restricted 8 .Program Agenda 1 Cache Fusion Overview 2 (Dynamic Re-) Mastering 3 Handling Contentions 4 RAC Meets Multitenant 5 RAC Meets In-Memory Copyright © 2014. All rights reserved. Instance • Master • Session Copyright © 2014. All rights reserved. Oracle and/or its affiliates.Oracle RAC Combines it All & Adds Services • Requester • Listener • Service • Holder Block. unless otherwise stated. PQ/PX: 9 . • Once a session is established within an instance. SCAN and SCAN Listener 2. listeners establish connections with an instance. Oracle and/or its affiliates. Local Listeners per node • Services are used to load balance work. All rights reserved. • In an Oracle RAC environment two types of listeners are used (for this purpose): 1. the sessions stays until closed. Copyright © 2014. 10 .Listeners and Services Direct Work to an Instance • In an Oracle environment. • Local listeners will establish connections on instances where the service is offered. • Spinning disk access time: milliseconds 11 . cache hits are always faster than reading from spinning disks Copyright © 2014. Oracle and/or its affiliates. All rights reserved. Whether local or remote.Getting Access to Data • Local Cache Hit • Remote Cache Hit • Access time: nanoseconds • Data is on disk • Access time: microseconds • Flash cache access time: microseconds • Disk controller cache: access time: micros. 12 .Maximum Three Way Communication write write write Message Message Block Case 1 (all local) Case 2 (local / remote) Case 3 (all distributed) • All entities in one instance • Some entities in one instance • All entities are dispersed • Data holding instance •Data holding instance • Session “holding” instance • Session “holding” instance • Mastering Instance • Three way communication to obtain a given block of data • Two way communication Copyright © 2014. All rights reserved. • Message operations have been subject to improvements since Oracle RAC 10g times. Oracle and/or its affiliates. Once data has been shipped to an instance. it resides in the cache for further (local) access. Updates will be communicated as required (via messages). Copyright © 2014.Dynamic Retrieval of Data + 1 2 3 + • In order to fulfill a given request.g. Oracle and/or its affiliates. All rights reserved. 13 . Oracle RAC can decide to: • Assemble data spread across instances and disks • Perform a (full) disk read for (parts of) the data • Use assembly and (full) disk reads dynamically • The decision which access path to use is dependent on various factors (e. IO capacity. network utilization) • Respective parameters are monitored • The access path can change accordingly and dynamically. Oracle and/or its affiliates.Program Agenda 1 Cache Fusion Overview 2 (Dynamic Re-) Mastering 3 Handling Contentions 4 RAC Meets Multitenant 5 RAC Meets In-Memory Copyright © 2014. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 14 . Copyright © 2014. Oracle RAC Oracle GI | HUB argentina Oracle RAC Oracle GI | HUB brazil • If a session requests write access to a block that is not mastered by the instance hosting the session. • Relocating the master to the instance where data is requested most reduces the messaging. • Illustration above is symbolic for this matter. every object is mastered (as part of Global Cache Services / GCS) by an instance. the master instance needs to be contacted (via message). Oracle and/or its affiliates. changing the master for an object from one instance to another.Mastering and Dynamic Re-Mastering write write Oracle RAC Oracle RAC Oracle RAC Oracle GI | HUB Oracle GI | HUB germany argentina Oracle GI | HUB brazil Oracle RAC Oracle GI | HUB germany • In Oracle RAC. Oracle RAC Oracle GI | HUB germany Oracle RAC Oracle GI | HUB argentina Oracle RAC Oracle GI | HUB brazil • Based on user access patterns. All rights reserved. a Dynamic Re-Mastering (DRM) can be triggered. 15 . • The Global Resource Directory (GRD) is used to “keep track” of mastering. • See MOS note 1619155. 16 . All rights reserved. write • Externally means access patterns can influence the mastering of user data. • DRM activity is also performed upon instances leaving or joining the cluster. Oracle and/or its affiliates. Copyright © 2014.Dynamic Re-Mastering (DRM) – Recommendations • DRM is used internally and externally.1 – “Best Practices and Recommendations for RAC databases using SGA larger than 300GB” as applicable • “dbms_cacheutil” can be used to manually set and release affinity under well understood circumstances. • Consider using smaller SGA sizes. • Internally DRM is used for the management of UNDO data across instances for example. • Do not turn DRM off (_gc_policy_time=0)! Oracle RAC Oracle RAC Oracle GI | HUB Oracle GI | HUB germany argentina Oracle RAC Oracle GI | HUB brazil • Optimize _gc_policy_minimum to run / trigger DRM at a convenient moment under normal operations. 0. • The lock state is saved in memory shared by the database and the kernel. • CFA saves user/kernel context switches. Oracle and/or its affiliates. All rights reserved.1. including the Oracle Database Appliance.New in 12. Lock grant • CFA contributes to a better. frees up CPU cycles in LMS. Oracle RAC Oracle RAC Oracle RAC Oracle GI Oracle GI Oracle GI germany argentina brazil • CFA will be activated on Engineered Systems over time. and “speeds up” messages. linear scaling.2: Cache Fusion Accelerator • The Cache Fusion Accelerator (CFA) is an write • OS kernel (Linux & Solaris only) module • which can respond directly to certain lock requests via RDSv3. 17 . LMS process Accelerator Copyright © 2014. • CFA is one of a long list of improvements. 18 . frees up CPU cycles in LMS. • The lock state is saved in memory shared by the database and the kernel. and “speeds up” messages. LMS process Accelerator Copyright © 2014.1. • CFA contributes to a better. • CFA is one of a long list of improvements.0. • CFA saves user/kernel context switches. Oracle RAC Oracle RAC Oracle RAC Oracle GI Oracle GI Oracle GI germany argentina brazil • CFA will be activated on Engineered Systems over time. All rights reserved.2: Cache Fusion Accelerator • The Cache Fusion Accelerator (CFA) is an write • OS kernel (Linux & Solaris only) module • which can respond directly to certain lock requests via RDSv3. linear scaling. Oracle and/or its affiliates. including the Oracle Database Appliance.New in 12. Oracle and/or its affiliates.Program Agenda 1 Cache Fusion Overview 2 (Dynamic Re-) Mastering 3 Handling Contentions 4 RAC Meets Multitenant 5 RAC Meets In-Memory Copyright © 2014. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 19 . g. • From a contention perspective. • User data as well as “metadata” (e. contention can occur between instances (not only within an instance). Copyright © 2014. All rights reserved. Oracle and/or its affiliates. Note: for scalability. a 2-node setup is basically the same as a 3 or more nodes setup. • The difference is in the required messagebased communication to obtain a block considering the mastering. an index) can be subject to contention • As soon as you scale out.How To Handle Contention – Basics write write write write write write write write write write write write write write • Contention can occur in any multiuser system (even in SI databases) • You can still guide users to only one instance in an Oracle RAC environment. only write/write contention needs to be considered. 20 . All rights reserved. write hot spots occur on indexes. • Pending redo must be written to log before the block can be transferred • Latency for a deferred block transfer becomes dependent on delay for log IO • Contention can affect related data as much as it can affect the actual “user data”.How To Handle Contention – Considerations write write write write write write write write write write write write Sequence REDO • Frequent transactional changes to the same data blocks in all instances may result in “write hot spots”. Oracle and/or its affiliates. • Block with pending changes may be “pinged” by other instances. Copyright © 2014. • In 99% of OLTP performance issues. 21 . • Right growing indexes and index contention are common. 4 of Exadata and Oracle Database Appliance by default (Smart Logs and SSDs. respectively) •Separate disks for logs from other IO busy disks • Drop unused indexes Get the best out of Oracle Partitioning . Guide users via services accordingly.A practical guide and reference https://www.slideshare.2. e. Oracle and/or its affiliates. All rights reserved.g.How To Handle Contention – General Solutions write write write write write write write write write write write write Sequence REDO • Avoid frequent transactional changes to the same data blocks in all instances using partitioning and services.2. SSDs • Use either for better cache locality: • Global hash partitioned indexes • Locally partitioned indexes •Implemented in 11.net/secret/nHJV1CjWWOlGOl Copyright © 2014. • Logically partition data so that subdata is handled in one instance only. 22 . • Place redo logs on fast storage if performance critical. How To Handle Contention – Use Connection Pools Connection Pool • UCP supports “Connection Affinity”: • Transaction-Based Affinity • Web Session Affinity http://docs.oracle. 23 . Copyright © 2014. of open session. Connection Pool busy idle • Fast Application Notification (FAN) – enabled connection pools receive Load Balancing information based on the Workload Repository and on a per-service basis.htm#JJUCP8197 write write write write Connectionwrite Pool write • Connection Pools limit the number of connections to the database – example: • Oracle Universal Connection Pool (UCP) • Use “Database Resident Connection Pooling” when no.com/database/121/JJUCP/rac. Oracle and/or its affiliates. All rights reserved. of active sessions much smaller then no. Program Agenda 1 Cache Fusion Overview 2 (Dynamic Re-) Mastering 3 Handling Contentions 4 RAC Meets Multitenant 5 RAC Meets In-Memory Copyright © 2014. All rights reserved. Oracle and/or its affiliates. Oracle Confidential – Internal/Restricted/Highly Restricted 24 . • Multithreaded Redo Log writer already implemented. • Future improvements will ensure greater efficiency. Oracle and/or its affiliates. providing higher consolidation benefit and agility. Copyright © 2014. • PDBs can be used to conveniently align users to instances with all the benefits. • Oracle RAC Multitenant Databases help to consolidate. ensuring extremely fast failover. 25 .Oracle Multitenant and Oracle RAC – a Symbiosis cons_1 cons_2 Oracle RAC Oracle GI | HUB Oracle RAC Oracle GI | HUB germany argentina REDO •Pluggable Databases (PDBs) represent themselves as services in an Oracle RAC Multitenant Database. All rights reserved. • PDBshttp://www. MoreRAC information. Oracle and/or its affiliates. see: Oracle Multitenant meetsfast Oracle RAC - • Oracle RAC Multitenant Databases help to consolidate. • Multithreaded Redo Log writer already implemented. providing higher consolidation benefit and agility.net/MarkusMichalewicz/oraclecan be used to conveniently align users to instances with multitenant-meets-oracle-rac-ioug-2014-version all the benefits.slideshare. • Future improvements will ensure greater efficiency.Oracle Multitenant and Oracle RAC – a Symbiosis cons_1 cons_2 Oracle RAC Oracle GI | HUB Oracle RAC Oracle GI | HUB germany argentina REDO •Pluggable Databases (PDBs) represent themselves as services in an Oracle Multitenant Database. Copyright © 2014. All rights reserved. 26 . ensuring extremely failover. • Per-PDB/service optimized GCS operations coming in future. Oracle and/or its affiliates. All rights reserved. Oracle Confidential – Internal/Restricted/Highly Restricted 27 .Program Agenda 1 Cache Fusion Overview 2 (Dynamic Re-) Mastering 3 Handling Contentions 4 RAC Meets Multitenant 5 RAC Meets In-Memory Copyright © 2014. Oracle In-Memory and Oracle RAC Breakthrough: Dual Format Database Generate Reports Instantly In-Memory Column Store Column Store Replaces Analytic Indexes Full HA & Integration with Industry Standards Copyright © 2014. 28 . All rights reserved. Oracle and/or its affiliates. 29 . Oracle and/or its affiliates.Oracle In-Memory and Oracle RAC – A Dream-Team Breakthrough: Dual Format Database Addresses In-Memory Column Store Column Store Replaces Analytic Indexes Addresses Copyright © 2014. All rights reserved. 30 . Senior Oracle DBA Yahoo Inc.“Full support for RAC scale-out means Oracle Database In-Memory can be used on our largest Data Warehouse. enabling more near real-time analytics. All rights reserved.” – Sudhi Vijayakumar. Copyright © 2014. Oracle and/or its affiliates. All rights reserved.Oracle Database In-Memory: Unique Fault Tolerance • Similar to storage mirroring • Duplicate in-memory columns on another node • Enabled per table/partition • Application transparent • Downtime eliminated by using duplicate after failure Only Available on Engineered Systems Copyright © 2014. Oracle and/or its affiliates. 31 . ” – Jens-Christian Pokolm Analyst IT-DB Architecture & Engineering Postbank Systems AG Copyright © 2014. Oracle’s In-Memory architecture takes the right approach to balancing real-time speed with continuous availability.“Downtime is extremely costly for our business. All rights reserved. 32 . Oracle and/or its affiliates. All rights reserved. • Memory allocated for In-Memory is subject to simplified lock mgmt. Copyright © 2014. • PCI Flash works mostly locally. simplified In-Memory Format In-Memory Format Normal Buffer Cache Normal Buffer Cache ? Oracle RAC ? Oracle RAC ? Oracle RAC Oracle GI | HUB Oracle GI | HUB Oracle GI | HUB germany argentina brazil Coming soon… • In-Memory compresses data.Optimized Used of Memory in Every Layer • Compressed • Lock mgmt. • Coming soon on ODA: a fully integrated shared flash cache. optimizing the use of memory. 33 . • In-Memory Speed and Capacity of Low Cost Disk • Shared solutions require RAC external synchronization. Oracle and/or its affiliates. Oracle and/or its affiliates.Oracle Learning Streams: http://education.A User Guide – Markus Michalewicz – Next Generation Oracle Automatic Storage Management .com/streams/ • Database and Middleware streams available • Videos – Feature different groups and presenters – Cover a broad range of topics and products • FREE OF CHARGE Oracle RAC PM / Dev contributions: – Oracle Flex Cluster: Optimized Resource Management for the Cloud .Ian Cookson – Oracle Grid Infrastructure 12c Bundled Agents – Shankar Iyer – ACFS Product Overview and Use Cases .Mark Scardina – Oracle RAC Practical Performance Management and Tuning – Markus Michalewicz Copyright © 2014.Ara Shakian – The Oracle Real Application Clusters (RAC) Family of Solutions . All rights reserved.oracle.Jim Williams – Implementing DBaaS with Oracle RAC 12c and Quality of Service Management . . Copyright © 2014. Oracle and/or its affiliates. All rights reserved. 36 .