Description
Oracle GoldenGate 11gFundamentals for Oracle Electronic Presentation D66519GC10 Edition 1.0 April 2011 Author Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Karen Kehn Disclaimer Technical Contributors and Reviewers Glenn Goodrich Eric Siglin Sue Jang Loren Penton Randy Richeson This document contains proprietary information and is protected by copyright and other intellectual property laws. You may copy and print this document solely for your own use in an Oracle training course. The document may not be modified or altered in any way. Except where your use constitutes "fair use" under copyright law, you may not use, share, download, upload, copy, print, display, perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express authorization of Oracle. The information contained in this document is subject to change without notice. If you find any problems in the document, please report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not warranted to be error-free. Restricted Rights Notice Heike Hundt Viktor Tchemodanov Joe Greenwald Deirdre Matishak Editors If this documentation is delivered to the United States Government or anyone using the documentation on behalf of the United States Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS The U.S. Government’s rights to use, modify, reproduce, release, perform, display, or disclose these training materials are restricted by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract. Daniel Milne Trademark Notice Richard Wallis Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Publishers Nita Brozowski Joseph Fernandez Introduction to Oracle GoldenGate 11g: Fundamentals for Oracle Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Objectives After completing this lesson, you should be able to: • Describe Oracle GoldenGate features and functionality • Identify key capabilities and differentiators • Describe Oracle GoldenGate high-availability and disaster tolerance solutions • Describe Oracle GoldenGate real-time data integration solutions 1-2 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Oracle GoldenGate 11g 1-3 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Key Capabilities and Technology Differentiators 1-4 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Value Propositions for Oracle GoldenGate Oracle GoldenGate 11gR1: Delivers continuous operations for mission-critical applications to eliminate unplanned and planned downtime and the related costs Lowers IT costs through heterogeneous support for multiple platforms to leverage lower-cost infrastructure for query offloading Improves efficiencies through improved performance, scalability of real-time feeds, and data distribution Reduces risk by ensuring data integrity and reliability between source and target systems Reduces barriers to sharing data because it has no application impact for real-time data acquisition. Allows improved visibility and business insight. 1-5 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Oracle and/or its affiliates.Oracle GoldenGate Topologies Unidirectional Query Offloading Broadcast Data Distribution 1-6 Bidirectional Standby DB or Active-Active for HA Integration/Consolidation Data Warehouse Peer-to-Peer Load Balancing. . All rights reserved. Multimaster Cascading Data Marts Copyright © 2011. 0 Copyright © 2011. Web and Event Services.Oracle Middleware for Business Intelligence Oracle Applications Custom Applications MDM Applications Business Intelligence Activity Monitoring SOA Platforms Comprehensive Data Integration Solution SOA Abstraction Layer Process Manager Data Services Data Federation Oracle Data Integrator Oracle GoldenGate Oracle Data Quality ELT/ETL Real-Time Data Data Profiling Data Transformation Log-Based CDC Data Parsing Bulk Data Movement Replication Data Cleansing Data Lineage Data Verification Match and Merge Storage 1-7 Service Bus Data Warehouse/ Data Mart OLTP System OLAP Cube Flat Files Web 2. All rights reserved. SOA . Oracle and/or its affiliates. Eliminates downtime for migration and upgrades • Least intrusive to source systems Oracle GoldenGate 1-8 Heterogeneous Targets Bulk Data Movement and Transformation • Fast ELT solution • Optimized SET-based transformation for high-volume transformations • Data lineage for improved manageability • Integrates to Data Quality Oracle Data Integrator Enterprise Edition Copyright © 2011.Oracle Data Integrator EE and Oracle GoldenGate Oracle Data Integrator Enterprise Edition ELT Transformation Oracle GoldenGate Real-Time Data Heterogeneous Sources Real-Time Data Integration and Replication • • • • Fast real-time solution Sub-second latency for real-time feeds Guaranteed delivery eliminates data loss. . Oracle and/or its affiliates. All rights reserved. in-database transformations • Read-consistent changed data with referential integrity • Complete data recoverability via Trail files BI Application Real-Time Streaming of Transactions tx6 Production OLTP Databases 1-9 Capture tx5 Source Trail tx4 LAN/WAN/ Internet tx3 tx2 tx1 Target Trail Copyright © 2011. All rights reserved. Deliver ODI Data Warehouse .Oracle GoldenGate for Real-Time Data Warehousing • Sub-second data latency • Minimal overhead and no batch windows • High-performance. Oracle and/or its affiliates. and maintenance – Migrate from non-Oracle databases to Oracle 11gR2. – Upgrade Oracle Database 8i. migrations. . Oracle and/or its affiliates. – Perform database maintenance. All rights reserved. 9i. – Perform application upgrades (Siebel CRM). • Offloading queries from legacy systems to Oracle databases • Global Data Synchronization for distributed systems 1 . – Upgrade/migrate the database server or OS. 10g to 11gR2.10 Copyright © 2011.Oracle GoldenGate Solutions for Oracle Database • Continuous availability via active-active databases • Zero downtime upgrades. • Minimizes risk with fall-back option • Improves success with phased user migration 1 . hardware. All rights reserved.11 Siebel CRM v6 or 7 Siebel CRM v6 or 7 Capture Delivery Route LAN/WAN/Web/IP Non Oracle DB Delivery Fallback Data Flow Copyright © 2011. Oracle and/or its affiliates. OS. Capture Oracle DB . and/or application.Oracle GoldenGate for Oracle Database Eliminate Downtime for Migrations and Application Upgrades Zero Downtime Siebel Upgrade and DB Migration • Upgrade/migrate/ maintain database. or 10g to 11g • Leverage new features of Oracle Database 11g without impacting business operations. • Minimize risks by using the failback option. 9i. All rights reserved.12 Zero Downtime Database Upgrades Switchover Application Real-Time Updates Capture Delivery Route LAN/WAN/Web/IP Oracle 8i/9i/10g Delivery Post-Switchover Data Flow Compare & Verify Copyright © 2011.Oracle GoldenGate for Oracle Database Eliminate Downtime During Oracle Database Upgrades • Zero database downtime for upgrades from 8i. 1 . Capture Oracle 11g . Oracle and/or its affiliates. All rights reserved. etc. • Utilize Oracle GoldenGate for: Disaster Recovery and Data Protection Switchover Application – Non-Oracle platforms Delivery Route LAN/WAN/Web/IP Source – Cross-OS and Oracle database version requirements 1 . Oracle and/or its affiliates. Delivery Post-Switchover Data Flow Copyright © 2011.Oracle GoldenGate for Oracle Database Eliminate Unplanned Downtime with Active Data Guard and Oracle GoldenGate • Utilize Active Data Guard for OracleOracle databases. testing.13 Real-Time Updates Capture – Active-active configurations Can be used for reporting. Capture Standby . database version.Oracle GoldenGate for Oracle Database Improve Production System Performance and Lower Costs Query Offloading Off-load queries from production systems in: Transaction Processing • Heterogeneous Activity configurations – Different OS. Oracle and/or its affiliates. Delivery Oracle Replica . or different type of database – Legacy system Capture query off-load Legacy • Active-active Production environments OLTP 1 .14 Read-only activity Application Real-Time Data Route LAN/WAN/Web/IP Copyright © 2011. All rights reserved. Oracle and/or its affiliates.15 Trail File Trail File Delivery Route LAN/WAN/Web/IP Copyright © 2011. Reporting Instance . All rights reserved.Oracle GoldenGate for Operational Reporting • Sub-second data latency • No performance degradation for the source system • Read-consistent changed data with referential integrity • Complete data recoverability via Trail files Operational Reporting OLTP Application Reporting Application Real-Time Data Capture Production Database 1 . Oracle and/or its affiliates.Oracle GoldenGate for Oracle Database Increase Return on Investment (ROI) on Existing Servers and Synchronize Global Data Active-Active • Utilize secondary systems for transactions.16 Source & Target DB Copyright © 2011. Capture . Application Application Capture Source & Target DB Delivery Route LAN/WAN/Web/IP Delivery 1 . • Synchronize data across data centers around the globe. All rights reserved. • Enable continuous availability during unplanned and planned outages. .17 Copyright © 2011.Quiz Which statements are true about Oracle GoldenGate? (Select more than one. Oracle and/or its affiliates. GoldenGate is a middleware product that does not require an Oracle database. d. All rights reserved.) a. GoldenGate is an Oracle Database product that supports other Oracle products. 1 . GoldenGate can support high availability. b. c. GoldenGate captures changes from the Oracle Redo logs or non-Oracle transaction logs and moves them to another database. All rights reserved. a. Oracle and/or its affiliates. and it is designed to support a heterogeneous database environment.18 Copyright © 2011. . False 1 . True b.Quiz Oracle GoldenGate is middleware software for business intelligence. 19 Copyright © 2011. you should have learned how to: • Describe Oracle GoldenGate features and functionality • Describe Oracle GoldenGate solutions for real-time business intelligence • Describe Oracle GoldenGate for continuous availability 1 . .Summary In this lesson. Oracle and/or its affiliates. All rights reserved. All rights reserved. .Technology Overview Copyright © 2011. Oracle and/or its affiliates. Objectives After completing this lesson. . All rights reserved. you should be able to: • Describe Oracle GoldenGate functionality and components • List supported databases and platforms • Describe the Oracle GoldenGate product line – – – – 2-2 Oracle GoldenGate Veridata Management Pack for Oracle GoldenGate Oracle GoldenGate for Flat File Oracle GoldenGate for Java Copyright © 2011. Oracle and/or its affiliates. Oracle GoldenGate: Modular Building Blocks Extract: Committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. Oracle and/or its affiliates. All rights reserved. Target Oracle and non-Oracle Database(s) . Extract Source Oracle and non-Oracle Database(s) 2-3 LAN/WAN Internet TCP/IP Copyright © 2011. Oracle and/or its affiliates. Target Oracle and non-Oracle Database(s) .Oracle GoldenGate: Modular Building Blocks Extract: Committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. All rights reserved. Trail: GoldenGate stages and queues data for routing. Extract Source Oracle and non-Oracle Database(s) 2-4 Trail LAN/WAN Internet TCP/IP Copyright © 2011. Oracle GoldenGate: Modular Building Blocks Extract: Committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. Extract Source Oracle and non-Oracle Database(s) 2-5 Trail Pump LAN/WAN Internet TCP/IP Copyright © 2011. Oracle and/or its affiliates. Trail: GoldenGate stages and queues data for routing. Target Oracle and non-Oracle Database(s) . All rights reserved. Pump: GoldenGate distributes data for routing to targets. Oracle GoldenGate: Modular Building Blocks Extract: Committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. Extract Source Oracle and non-Oracle Database(s) 2-6 Trail Trail Pump LAN/WAN Internet TCP/IP Copyright © 2011. Oracle and/or its affiliates. All rights reserved. Route: Data is compressed and encrypted for routing to targets. Target Oracle and non-Oracle Database(s) . Pump: GoldenGate distributes data for routing to targets. Trail: GoldenGate stages and queues data for routing. transforming the data as required.Oracle GoldenGate: Modular Building Blocks Extract: Committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. Route: Data is compressed and encrypted for routing to targets. Trail: GoldenGate stages and queues data for routing. Replicat: Replicat applies data with transaction integrity. Oracle and/or its affiliates. Pump: GoldenGate distributes data for routing to targets. All rights reserved. Replicat Target Oracle and non-Oracle Database(s) . Extract Source Oracle and non-Oracle Database(s) 2-7 Trail Trail Pump LAN/WAN Internet TCP/IP Copyright © 2011. Route: Data is compressed and encrypted for routing to targets. Replicat Target Oracle and non-Oracle Database(s) . Replicat: Replicat applies data with transaction integrity. All rights reserved. transforming the data as required Extract Source Oracle and non-Oracle Database(s) 2-8 Trail Trail Pump LAN/WAN Internet TCP/IP Bidirectional Copyright © 2011. Trail: GoldenGate stages and queues data for routing Pump: GoldenGate distributes data for routing to targets.Oracle GoldenGate: Modular Building Blocks Extract: Committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. Oracle and/or its affiliates. HP Neoview Flat File products 2-9 • Windows • Linux • Sun Solaris • HP NonStop • HP-UX • HP TRU64 • HP OpenVMS • IBM AIX • IBM z/OS • z/linux Copyright © 2011.Supported Databases and Operating Systems Databases Operating Systems and Platforms Oracle GoldenGate Capture: c-tree Oracle MySQL DB2 Microsoft SQL Server Sybase ASE Teradata Enscribe SQL/MP SQL/MX JMS message queues Oracle GoldenGate Delivery: All listed above. plus: TimesTen. . All rights reserved. Oracle and/or its affiliates. All rights reserved. Oracle and/or its affiliates.Oracle GoldenGate Product Line Product Description Oracle GoldenGate Veridata Add-on capability to validate data in replicated systems Management Pack for GoldenGate Add-on management pack (includes Oracle GoldenGate Director) Oracle GoldenGate Application Adapters Prepackaged application content. such as for Flat File and Java adapters 2 .10 Copyright © 2011. . 11 Copyright © 2011.Oracle GoldenGate Veridata • • Oracle GoldenGate Veridata performs highspeed. All rights reserved. Oracle and/or its affiliates. Veridata identifies and reports data discrepancies without interrupting those systems or the business processes that they support. . 2 . low-impact data comparisons between homogeneous and heterogeneous databases. 12 Synchronize Synchronize GoldenGate Copyright © 2011. configure. GoldenGate GoldenGate Director Synchronize Synchronize GoldenGate 2 . and report on all GoldenGate transactional data synchronization processes.GoldenGate Director: Overview GoldenGate Director is a graphical enterprise application that offers a visual and intuitive way to define. All rights reserved. GoldenGate . manage. Oracle and/or its affiliates. Oracle and/or its affiliates. 2 .Adapter Integration Options for Oracle GoldenGate • Oracle GoldenGate adapters integrate with installations of the core product to do the following: – Read an Oracle GoldenGate trail and write transactions to a flat file that can be used by other applications. All rights reserved. . – Read JMS messages and deliver them as an Oracle GoldenGate trail.13 Copyright © 2011. – Read an Oracle GoldenGate trail and deliver transactions to a JMS provider or other messaging system or custom application. All rights reserved. Log Primary Extract Data Integration Server Trail Data Pump Extract Extract Flat File User Exit Control File Trail Manager Manager Data Files 2 . . Oracle and/or its affiliates.Oracle GoldenGate for Flat File Source Database Server Database Source Oracle GoldenGate for Flat File is used to output transactional data captured by Oracle GoldenGate to rolling flat files that are then consumed by thirdparty products.14 Copyright © 2011. Source Database Server Data Integration Server Manager Manager Network VAM Message Extract Capture Source Application JMS Messages Collector Trail Source definitions Trail Gendef Replicat Target Database 2 . All rights reserved. transactional data captured by Oracle GoldenGate can be delivered to targets other than a relational database (such as a JMS). writing files to disk or integrating with a custom application’s Java API.15 Copyright © 2011.Oracle GoldenGate Application Adapter for Java Through the Oracle GoldenGate Java API. Oracle and/or its affiliates. . that uses the message capture properties file and parser-specific data definitions to create the source definitions file 2 .16 Copyright © 2011. Gendef. . All rights reserved.JMS Messaging Capture The Oracle GoldenGate JMS message capture requires two components: • The dynamically linked shared VAM library that is attached to the Oracle GoldenGate Extract process • A separate utility. Oracle and/or its affiliates. Quiz Which statements are true? a.17 Copyright © 2011. b. d. . 2 . The Extract process can capture committed changes from the transaction logs and write them to a trail file. The Data Pump process can capture committed changes from the transaction logs and write them to a trail file. c. The Data Pump process can move data from a local trail file to a remote trail file. All rights reserved. Oracle and/or its affiliates. The Replicat process can move changes from a local trail file to the database. 18 Copyright © 2011. Oracle and/or its affiliates. Oracle GoldenGate Veridata b. Oracle GoldenGate Director c.) a.Quiz What products make-up the Oracle GoldenGate product line? (Select all that apply. All rights reserved. Flat File and Java adapters) 2 . Oracle GoldenGate Application Adapters (that is. . All rights reserved. .Summary In this lesson.19 Oracle GoldenGate Veridata Management Pack for Oracle GoldenGate Oracle GoldenGate for Flat File Oracle GoldenGate for Java Copyright © 2011. you should have learned how to: • List the building blocks that make up GoldenGate functionality • List the supported platforms and infrastructure • Describe the Oracle GoldenGate product line – – – – 2 . Oracle and/or its affiliates. All rights reserved. Oracle and/or its affiliates. .Oracle GoldenGate Architecture Copyright © 2011. All rights reserved. .Objectives After completing this lesson. you should be able to: • Describe the logical Oracle GoldenGate architecture and components • Describe Oracle GoldenGate process groups • Explain change capture and delivery • Explain initial data load • Contrast batch and online operations • Explain Oracle GoldenGate checkpointing • Describe the commit sequence number (CSN) • Describe Oracle GoldenGate files 3-2 Copyright © 2011. Oracle and/or its affiliates. All rights reserved. Oracle and/or its affiliates. 3-3 Copyright © 2011.Oracle GoldenGate Uses • • Primarily used for change data capture and delivery from database transaction logs Can optionally be used for initial load directly from database tables – Especially useful for synchronizing heterogeneous databases – Database-specific methods may be preferable for homogeneous configurations. . Oracle GoldenGate Components Oracle GoldenGate has the following components: • Extract • Data Pump • Replicat • Trails or extract files • Checkpoints • Manager • Collector 3-4 Copyright © 2011. All rights reserved. Oracle and/or its affiliates. . Oracle and/or its affiliates. All rights reserved. Target DB .Oracle GoldenGate Logical Architecture Data source for initial load: source tables Network Initial Load Replicat Extract Manager Collector Extract Data source for change synchronization: Transaction log or vendor access module Replicat Trail or File (Optional) Data Pump 3-5 Manager Change Synchronization Copyright © 2011. Oracle GoldenGate Process Groups • A process group consists of the following: – – – – A process (either Extract or Replicat) Parameter file Checkpoint file Any other files associated with the process • Groups can be defined by using the ADD EXTRACT and ADD REPLICAT commands. 3-6 Copyright © 2011. . All rights reserved. Oracle and/or its affiliates. • Each process group must have a unique name. Copyright © 2011.GGSCI • • 3-7 Processes are added and started using the GoldenGate Software Command Interface (GGSCI) with the group name. and monitor Oracle GoldenGate. All rights reserved. control. . Oracle and/or its affiliates. The GGSCI is used to issue a complete range of commands that configure. Oracle and/or its affiliates. Oracle GoldenGate uses two types of parameter files: – GLOBALS file: Stores parameters that relate to the Oracle GoldenGate instance as a whole – Run-time parameter file: Coupled with a specific process (such as Extract) • 3-8 By default. parameter files are saved to the dirprm subdirectory of the Oracle GoldenGate directory.Parameter Files • • Most Oracle GoldenGate functionality is controlled by means of parameters specified in ASCII parameter files. Copyright © 2011. . All rights reserved. Oracle and/or its affiliates. All rights reserved.Change Data Capture and Delivery Source Database Network (TCP/IP) Extract Transaction Log Replicat Trail Manager 3-9 Server Collector Manager Copyright © 2011. Target DB . Oracle and/or its affiliates.10 Manager Copyright © 2011. All rights reserved.Change Data Capture and Delivery Using a Data Pump Source Database Network (TCP/IP) Server Collector Extract Transaction Log (Optional) Replicat Remote Trail Data Pump Manager 3 . Target DB . All rights reserved. transforms it if necessary.11 Change data capture and delivery can be run either continuously (online) or as a special run (batch run) to capture changes for a specific period of time. The initial load is always a special run (batch run). Copyright © 2011. .Online Versus Batch Operation • • • 3 . and applies it to the target tables so that the movement of transaction data begins from a synchronized state. An initial load takes a copy of the entire source data set. Oracle and/or its affiliates. • File to Replicat – Extract writes to a file that Replicat applies using SQL. All rights reserved.12 Copyright © 2011. 3 . • File to database utility – Extract writes to a file formatted for a DB bulk load utility.Running an Initial Load GoldenGate can be used to load data in the following ways: • Direct load – Extract sends data directly to Replicat to apply using SQL. Oracle and/or its affiliates. • Direct bulk load – Replicat uses the Oracle SQL*Loader API. . Oracle and/or its affiliates. All rights reserved.13 Manager Copyright © 2011. .Initial Load Network (TCP/IP) Extract Replicat Source Database Tables Target DB Server Collector Or DB bulk load utility Files Manager 3 . Oracle and/or its affiliates. TX 4 Current Read Position Source Database Capture Capture (Extract). Pump (Extract). TX 3 Insert. TX 2 Commit. TX 3 Current Write Position Commit. TX 3 Begin. TX 1 Start of Oldest Open (Uncommitted) Transaction Insert.14 Copyright © 2011. TX 1 Begin. TX 3 Insert. TX 2 Begin. Commit Ordered Source Trail Capture Checkpoint 3 . TX 4 Commit. TX 1 Insert. All rights reserved. TX 2 Insert. TX 3 Begin. TX 2 Commit. TX 3 Delete. . TX 2 Update. and Delivery (Replicat) save positions to a checkpoint file so they can recover in case of failure. TX 2 Begin.Checkpoints Begin. TX 2 Commit.15 Commit Ordered Source Trail Pump Commit Ordered Target Trail Pump Checkpoint Copyright © 2011. TX 1 Insert. TX 2 Begin. TX 2 Current Write Position Current Read Position Commit. TX 2 Insert. . TX 3 Delete. Oracle and/or its affiliates. TX 3 Current Write Position Pump Checkpoint Begin. TX 1 Begin. TX 2 Commit.Checkpoints Begin. TX 2 Update. TX 4 Commit. TX 3 Insert. TX 3 Begin. TX 3 Begin. All rights reserved. TX 3 Insert. TX 1 Start of Oldest Open (Uncommitted) Transaction Insert. TX 2 Insert. TX 2 Commit. TX 2 Capture Checkpoint Begin. TX 4 Current Read Position Source Database Capture Capture Checkpoint 3 . TX 2 Current Write Position Current Read Position Current Read Position Commit.Checkpointing Begin. TX 2 Commit. TX 2 Begin. TX 3 Begin. Oracle and/or its affiliates. TX 2 Insert. TX 2 Update. TX 2 Capture Checkpoint Begin. TX 4 Current Read Position Source Database Commit Ordered Source Trail Capture Capture Checkpoint 3 . TX 2 Insert. TX 3 Commit. TX 3 Current Write Position Delivery Checkpoint Begin. All rights reserved. TX 3 Delete. TX 1 Insert.16 Pump Commit Ordered Target Trail Pump Checkpoint Copyright © 2011. Delivery Target Database Delivery Checkpoint . TX 2 Insert. TX 3 Begin. TX 1 Start of Oldest Open (Uncommitted) Transaction Insert. TX 4 Pump Checkpoint Commit. TX 1 Begin. TX 2 Commit. TX 3 Insert. .17 A CSN is an identifier that Oracle GoldenGate constructs to identify a transaction for the purpose of maintaining transactional consistency and data integrity. A CSN uniquely identifies a particular point in time at which a transaction commits to the database. to reposition Replicat in the trail. or to fulfill other purposes. All rights reserved. Oracle and/or its affiliates.Commit Sequence Number (CSN) • • • 3 . Copyright © 2011. The CSN can be required to position Extract in the transaction log. . All rights reserved. How is Oracle GoldenGate different from simply replicating database operations? 2. What is the purpose of checkpointing? 3 .18 Copyright © 2011.Discussion Questions 1. Oracle and/or its affiliates. 3 .19 Copyright © 2011. c. GoldenGate checkpoints are stored in an Oracle control file. . All rights reserved. d. Oracle and/or its affiliates. b.Quiz Which of the following statements is true? a. The CKPT background process records the GoldenGate checkpoint. There is likely to be only one GoldenGate checkpoint file. The GoldenGate checkpoint supports GoldenGate recovery. Oracle and/or its affiliates. and GGSCI commands 3 .Summary In this lesson. All rights reserved.20 Copyright © 2011. process groups. you should have learned how to: • Describe the uses of Oracle GoldenGate • Explain change capture and delivery (with and without a data pump) • Explain initial data load • Contrast batch and online operation • Explain Oracle GoldenGate checkpointing • Describe Oracle GoldenGate parameters. . Oracle and/or its affiliates. All rights reserved. .Installing Oracle GoldenGate Copyright © 2011. All rights reserved. you should be able to: • Download the required Oracle GoldenGate Media Pack • Understand how to install Oracle GoldenGate on Linux. .Objectives After completing this lesson. and Windows • Run Oracle GoldenGate commands from the Oracle GoldenGate Software Command Interface (GGSCI) • Identify the types of GGSCI commands available • Use an OBEY file 4-2 Copyright © 2011. UNIX. Oracle and/or its affiliates. All rights reserved. . Check the documentation for certification requirements.1 and 10. Oracle and/or its affiliates. 4-3 Copyright © 2011.2 (DML and DDL support) – Oracle 11g (DML and DDL) • Operating system – A number of combinations are certified for the database version and operating system.System Requirements • Supported platforms for Oracle databases: – Oracle 9.2 (DML and DDL support) – Oracle 10. – Swap space must be sufficient for each Oracle GoldenGate Extract and Replicat process. 4-4 Copyright © 2011. All rights reserved. This can be determined after setting up one Extract or Replicat. – Each Extract and Replicat process needs approximately 25 MB to 55 MB of memory.Operating System Requirements • Memory requirements – The amount of memory that GoldenGate requires depends on the number of concurrent processes that will be running. . Oracle and/or its affiliates. All rights reserved. – Configure one port for communication between the Manager process and other Oracle GoldenGate processes. 4-5 Copyright © 2011.Operating System Requirements • Disk requirements – 50 MB to 150 MB free disk space depending on the database and platform – Trail files: You can assign a separate disk for trial files. – Configure a range of ports for local Oracle GoldenGate communications. The default size is 10 MB. . including DNS. • TCP/IP – Configure the system to use TCP/IP services. but to optimize space you can use the PURGEOLDEXTRACTS parameter. Oracle and/or its affiliates. . Select and download the Oracle GoldenGate Media Pack. Oracle and/or its affiliates. In the Media Pack search: – – Select a Product Pack: Oracle Fusion Middleware.oracle. In a browser. 4-6 Copyright © 2011. 2. All rights reserved. 3. 4. Enter your personal information.com.Downloading Oracle GoldenGate 1. Select your platform from the drop-down list. go to http://edelivery. create the working directories: GGSCI> CREATE SUBDIRS 4-7 Copyright © 2011.tar file to the current directory: tar -xvof <filename>. Oracle and/or its affiliates. Linux. .tar • • From the Oracle GoldenGate directory. Extract the . run the GGSCI program. In GGSCI. All rights reserved.zip file to the directory where you want GoldenGate to be installed. or z/OS • • Extract the Oracle GoldenGate .Installation on UNIX. Configure a Windows Service Name for Manager process in a GLOBALS parameter file (required only if there are multiple Managers on the server): C:\GGS> • INSTALL ADDSERVICE ADDEVENTS Run the command shell and change to the new GoldenGate directory. Oracle and/or its affiliates.Installation on Windows • • • Download the Oracle GoldenGate Media Pack .zip file to C:\GGS. . All rights reserved. Then create subdirectories: GGSCI> CREATE SUBDIRS 4-8 Copyright © 2011.zip file to the C:\GGS folder (GGS is an example). Unzip the . All rights reserved.GoldenGate Directories Directory Contents dirchk GoldenGate checkpoint files dirdat GoldenGate trail and Extract files dirdef Data definitions produced by DEFGEN and used to translate heterogeneous data dirpcs Process status files dirprm Parameter files dirrpt Process report files dirsql SQL scripts dirtmp Temporary storage for transactions that exceed allocated memory 4-9 Copyright © 2011. . Oracle and/or its affiliates. oracle. 4 . All rights reserved.Oracle GoldenGate Documentation • • • • • Quick Install Guide Installation and Setup Guides (by database) Administration Guide Reference Guide Troubleshooting and Tuning Guide Note: You can download the documentation from http://www. Oracle and/or its affiliates.11 Copyright © 2011.com/technology/documentation/goldengate. .html. All rights reserved.12 Copyright © 2011. Start GGSCI from the GoldenGate install directory: Shell> cd <GoldenGate install location> Shell> GGSCI • For the Help Summary page: GGSCI> HELP • For Help on a specific command: GGSCI> HELP <command> <object> GGSCI> HELP ADD EXTRACT • To re-execute a command: GGSCI> ! 4 .GGSCI Command Interface • • GGSCI is the command interface that executes GoldenGate commands. Oracle and/or its affiliates. . .GGSCI Commands MANAGER EXTRACT REPLICAT ADD X ALTER EXTTRAIL RMTTRAIL TRANDATA CHECKPOINT TABLE TRACE TABLE X X X X X X X X X X CLEANUP X X DELETE X X X X X X X X X X X X X X X X KILL X X X LAG X X X INFO X ER X REFRESH X SEND X X X X START X X X X X X X STATS STATUS X X X X STOP X X X X 4 .13 Copyright © 2011. Oracle and/or its affiliates. All rights reserved. 15 Copyright © 2011. LIST TABLES DDL DUMPDDL [SHOW] Miscellaneous !command. VIEW GGSEVT. Oracle and/or its affiliates. . VERSIONS. SHOW. FC. All rights reserved. VIEW PARAMS Database DBLOGIN. VIEW REPORT 4 . EDIT PARAMS. OBEY. SHELL. ENCRYPT PASSWORD. CREATE SUBDIRS. HISTORY.GGSCI Commands Commands Parameters SET EDITOR. INFO ALL. HELP. . EXTRACT myext • Start an Extract group: GGSCI> START EXTRACT myext 4 . All rights reserved. Oracle and/or its affiliates.GGSCI Examples • Start a Manager process: GGSCI> START MGR • Add an Extract group: GGSCI> ADD EXTRACT myext. TRANLOG. BEGIN NOW • Add a local trail: GGSCI> ADD EXTTRAIL /ggs/dirdat/lt.17 Copyright © 2011. Copyright © 2011. Oracle and/or its affiliates. BEGIN NOW ADD EXTTRAIL /ggs/dirdat/lt.18 The ALLOWNESTED command enables the use of nested OBEY files. TRANLOG. with one command per line: START MGR ADD EXTRACT myext. All rights reserved. .OBEY Files • • OBEY files are used to automate a series of frequently used commands. EXTRACT myext START EXTRACT myext • Then use the GGSCI OBEY command to run the file: GGSCI> OBEY <filename>.oby • 4 . Create and save a text file that contains the commands. The maximum number of nested files is 16.19 Copyright © 2011. All rights reserved. Oracle and/or its affiliates.New Oracle GoldenGate 11gR1 GGSCI Miscellaneous Commands • • • The new ALLOWNESTED command enables the use of nested OBEY files. An attempt to run a nested OBEY file in the default mode of NOALLOWNESTED causes an error. ALLOWNESTED | NOALLOWNESTED 4 . . All rights reserved. as in this example: Shell> cd <GoldenGate install location> Shell> extract paramfile <filepath> reportfile <filepath> [-p <port>] Shell> replicat paramfile <filepath> reportfile <filepath> • 4 .Running GoldenGate from the OS Shell • You can also start GoldenGate processes from the OS command shell when running a batch job or initial load. Oracle and/or its affiliates.20 This is especially useful in scheduling GoldenGate batch jobs to run during off-peak hours using a command-line capable scheduler. Copyright © 2011. . Discussion Questions 1. 4 . All rights reserved.21 Where can you download the GoldenGate software? What is GGSCI? Where can you view the GoldenGate command syntax? What is an OBEY file. 3. why would you use one? Copyright © 2011. . 2. Oracle and/or its affiliates. 4. 22 Copyright © 2011. Oracle and/or its affiliates. you should have learned how to: • Download the required Oracle GoldenGate Media Pack • Install Oracle GoldenGate on Linux. and Windows • Locate and use Oracle GoldenGate documentation • Run GoldenGate commands from the GGSCI prompt. . All rights reserved. UNIX. from OBEY files. and from the OS shell 4 .Summary In this lesson. All rights reserved. Oracle and/or its affiliates.Practices The practices for this lesson cover the following topics: • Preparing for the course labs • Installing Oracle GoldenGate • Introducing the GGSCI command-line interface 4 . .23 Copyright © 2011. Configuration Overview and Preparing the Environment Copyright © 2011. All rights reserved. Oracle and/or its affiliates. . All rights reserved.Objectives After completing this lesson. . Oracle and/or its affiliates. you should be able to: • Describe the high-level configuration steps • Configure and start the Manager process • Configure supplemental logging on the database and at the table level • Generate a source definitions file • Prepare a source database for transaction capture 5-2 Copyright © 2011. Start the Manager process. All rights reserved. Oracle and/or its affiliates. Enable supplemental logging. create source definitions.Configuring Oracle GoldenGate Oracle GoldenGate can be deployed in four steps: 1. Preparing the environment a) b) c) d) Set up a database user to access Oracle GoldenGate. Change capture 3. . Initial load 4. 2. If you are using a heterogeneous source and/or target. Change delivery 5-3 Copyright © 2011. . All rights reserved.Configuring Oracle GoldenGate 1. Preparing the environment Source Database 3. Change Capture 5-4 Replicat Remote Trail 4. Oracle and/or its affiliates. Change Delivery Copyright © 2011. Initial Load (various methods) Target DB Data Pump Extract Transaction Log Local Trail 2. ALTER SESSION X X RESOURCE X x CONNECT X X SELECT ANY DICTIONARY X X FLASHBACK ANY TABLE or FLASHBACK ON <owner. UPDATE. All rights reserved. create a database user with the following privileges: User Privilege Extract (Source Side) Replicat (Target Side) CREATE SESSION.Preparing the Environment: Oracle Database To ensure access by GoldenGate processes. .table> X X INSERT.table> X SELECT ANY TABLE or SELECT ON <owner. DELETE ON <target tables> X CREATE TABLE X EXECUTE on DBMS_FLASHBACK package 5-5 X Copyright © 2011. Oracle and/or its affiliates. Oracle and/or its affiliates. This is required for GoldenGate so that it can locate the correct row on the target for update and delete operations. so they are available in the redo logs. Before GoldenGate can start capturing real-time data. All rights reserved. . This task is accomplished via the ADD TRANDATA GGSCI command. the database logs only those column values that change. Copyright © 2011.TRANDATA Command • • • • 5-6 By default. the Oracle database must be set to log the table key values whenever it logs a row change. • On the target database: – Ensure access by GoldenGate processes. SQL> ALTER SYSTEM SWITCH LOGFILE. Oracle and/or its affiliates. 5-7 Copyright © 2011.<table2> – Enable archive logging as a secondary data source in case the online logs recycle before Extract is finished with them. All rights reserved. PASSWORD <pw> GGSCI> ADD TRANDATA <owner>. .Preparing the Environment: Oracle Database • On the source database: – Enable minimal supplemental logging at the database level to allow GoldenGate to properly capture updates to primary keys and chained rows: SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA.<table1> GGSCI> ADD TRANDATA <owner>. – Enable supplemental logging at the table level: GGSCI> DBLOGIN USERID <login>. All rights reserved.Preparing the Environment: Manager Overview • The Manager module provides a command-line interface to perform a variety of tasks: – – – – – • • 5-8 Starting. stopping. Oracle and/or its affiliates. Copyright © 2011. Manager parameters are entered in the mgr. and monitoring GoldenGate processes Setting parameters to configure GoldenGate processes Error and lag reporting Resource management Trail file management The Manager process must be running on each system before Extract or Replicat can be started.prm file under the dirprm directory. . Preparing the Environment: Configuring Manager • Create the parameter file by using GGSCI commands: GGSCI> EDIT PARAMS MGR • Start the Manager by using GGSCI: GGSCI> START MGR • To determine which port Manager is using: GGSCI> INFO MGR 5-9 Copyright © 2011. Oracle and/or its affiliates. All rights reserved. . All rights reserved. . 8002. USECHECKPOINTS PURGEOLDEXTRACTS /ggs/dirdat/bb*. USECHECKPOINTS.10 Copyright © 2011. MINKEEPDAYS 5 AUTOSTART ER * AUTORESTART EXTRACT *. 9500–9520 PURGEOLDEXTRACTS /ggs/dirdat/aa*. RETRIES 5 LAGREPORTHOURS 1 LAGINFOMINUTES 3 LAGCRITICALMINUTES 5 5 . Oracle and/or its affiliates.Preparing the Environment: Sample Manager Parameter File PORT 7809 DYNAMICPORTLIST 8001. WAITMINUTES 2. All rights reserved. At startup. Copyright © 2011. . This source definition file is used to interpret layouts for data stored in GoldenGate trails. GoldenGate must understand both the source and target layouts. Oracle and/or its affiliates. transforming.11 When capturing.Preparing the Environment: Overview of Source Definitions • • • • 5 . Replicat reads the definition file specified with the SOURCEDEFS parameter. The DEFGEN utility produces a file containing a definition of the layouts of the source files and tables. and delivering data across disparate systems and databases. This file must be transferred to the target system as a text file.Preparing the Environment: Running DEFGEN • DEFGEN is initiated from the command prompt: defgen paramfile <paramfile> [ reportfile <reportfile> ] • UNIX example: defgen paramfile /ggs/dirprm/defgen. .12 Definitions are saved to the file specified in the parameter file. Copyright © 2011.rpt • 5 .rpt • Windows example: defgen paramfile c:\ggs\dirprm\defgen. All rights reserved.prm reportfile /ggs/dirrpt/defgen. Oracle and/or its affiliates.prm reportfile c:\ggs\dirrpt\defgen. .Quiz ADD TRANDATA enables supplemental logging of key values in the transaction logs whenever a row change occurs.13 Copyright © 2011. All rights reserved. True b. a. False 5 . Oracle and/or its affiliates. 14 Copyright © 2011. False 5 . All rights reserved. True b. Oracle and/or its affiliates. .Quiz Both the extract and replicat users require the DBA role for the database. a. All rights reserved. Oracle and/or its affiliates.Summary In this lesson. you should have learned how to: • Describe the high-level configuration steps • Configure and start the Manager process • Generate a source definitions file • Prepare a source database for transaction capture 5 . .15 Copyright © 2011. .16 Copyright © 2011. All rights reserved. Oracle and/or its affiliates.Practices This practices for this lesson cover the following topics: • Preparing your environment • Creating the GLOBALS parameter file • Configuring and starting the GoldenGate Manager • Using the TRANDATA option 5 . . Oracle and/or its affiliates. All rights reserved.Configuring Change Capture Copyright © 2011. .Objectives After completing this lesson. you should be able to: • Describe what the Extract process does • Configure and start an Extract process • Configure and start an Extract data pump • Add local and remote trails • Configure Extract to access logs on Oracle Automatic Storage Management (ASM) 6-2 Copyright © 2011. Oracle and/or its affiliates. All rights reserved. All rights reserved. .Step 2: Change Capture 1. Change capture 6-3 Replicat Remote Trail 4. Preparing the environment Source Database 3. Oracle and/or its affiliates. Change delivery Copyright © 2011. Initial load (various methods) Target DB Data Pump Extract Transaction Log Local Trail 2. can be configured to distribute data from local trails to remote systems.Extract Overview • • • Extract captures all the changes that are made to objects that you configure for synchronization. Copyright © 2011. A primary Extract can be configured to: – – – – • 6-4 Capture changed data from database logs or archive logs Capture changed data from JMS Capture data directly from source tables for initial data load Send the data to be written to a local or remote trail or file A secondary Extract. called a data pump. . Oracle and/or its affiliates. Extract sends the data for that transaction to the trail for propagation to the target system. All rights reserved. When a transaction is committed. All rights reserved. A data pump: – Reads the local trail – Manipulates the data or passes it through without change – Sends the data to one or more targets • A data pump is useful: – As a safeguard against network and target failures – To break complex data filtering and transformation into phases – To consolidate data from many sources – To synchronize one source with multiple targets 6-5 Copyright © 2011.Data Pump Overview • • An Extract data pump can write to one or multiple remote trails and/or servers (not to be confused with an Oracle Data Pump). Oracle and/or its affiliates. . . Oracle and/or its affiliates. All rights reserved.Data Pumps: One-to-Many Trails Trail Primary Extract Data Pump Trail Trail Trail 6-7 Copyright © 2011. All rights reserved.Data Pumps: One-to-Many Target Systems Data Pump Trail Primary Extract Data Pump Trail Trail Data Pump Trail 6-8 Copyright © 2011. . Oracle and/or its affiliates. All rights reserved. Oracle and/or its affiliates. 3. 6-9 Copyright © 2011. Start the Extract processes.Setting Up Change Capture On the source system: 1. 4. Add a remote trail. (Optional) Add a local trail and a data pump Extract that reads from the transaction logs located on the source and has an associated parameter file. This Extract reads from the transaction logs located on the source and has an associated parameter file. 2. . Add a primary Extract. or data pump Extract. use the GGSCI ADD EXTRACT command (this automatically creates a checkpoint): ADD EXTRACT <group name> . RMTHOST {<host name> | <IP address>} .ADD EXTRACT Command • For a regular. . <processing options>] • For an alias Extract: ADD EXTRACT <group name> .10 Copyright © 2011. {MGRPORT <port>} | {PORT <port>} [. Oracle and/or its affiliates. RMTNAME <name>] [. <data source options> . All rights reserved. DESC “<description>”] 6 . passive. <starting point> [. ADD EXTRACT finance.” ADD EXTRACT load. EXTTRAILSOURCE c:\ggs\dirdat\lt • Create an initial-load Extract named “load. Start extracting with records generated at 8:00 AM on January 31. 2009.11 Copyright © 2011. BEGIN NOW • Create an Extract group named “finance” that extracts database changes from the transaction logs. BEGIN 2009-01-31 08:00 • Create a data pump Extract group named “pump” that reads from the GoldenGate trail c:\ggs\dirdat\lt. SOURCEISTABLE 6 . All rights reserved. .ADD EXTRACT Examples • Create an Extract group named “finance” that extracts database changes from the transaction logs. TRANLOG. TRANLOG. Start extracting with records generated at the time when you add the Extract group: ADD EXTRACT finance. Oracle and/or its affiliates. ADD EXTRACT pump. Oracle and/or its affiliates. issue the following command: GGSCI> EDIT PARAMS <group name> • Then modify the following parameters in the file: EXTRACT finance USERID login.Editing Extract Parameters • To edit a parameter file for an online Extract group. All rights reserved. PASSWORD pw RMTHOST serverx. . MGRPORT 7809 RMTTRAIL /ggs/dirdat/lt TABLE SALES. 6 .INVENTORY.ORDERS. TABLE SALES.12 Copyright © 2011. as additional files are needed.Overview of Trails • • • • Trails are used in GoldenGate to support the continuous extraction and replication of database changes. or on an intermediary system. Copyright © 2011. all file names in a trail begin with the same two characters. Trails are stored in the dirdat sub-directory of the Oracle GoldenGate directory. All rights reserved. Only one primary Extract process writes to a trail. A trail can exist on the source or target system.13 When created. . Oracle and/or its affiliates. Processes that read the trail include: – Data pump Extract – Replicat • • 6 . Then. each name is appended with a unique six-digit serial number. All rights reserved.Adding a Local or Remote Trail • Add a local or remote trail with the GGSCI command: ADD EXTTRAIL | RMTTRAIL <trail name> . . EXTRACT finance.14 Copyright © 2011. MEGABYTES 5 6 . Oracle and/or its affiliates. EXTRACT parts. EXTRACT <group name> [. MEGABYTES <n>] • If you are using a data pump: – The primary Extract needs a local trail (EXTTRAIL) – The data pump Extract needs a remote trail (RMTTRAIL) ADD EXTTRAIL c:\ggs\dirdat\aa. MEGABYTES 10 ADD RMTTRAIL c:\ggs\dirdat\bb. Starting the Extract • Start an Extract process with the GGSCI command: START EXTRACT <group name> • • 6 . All rights reserved.15 If the output trail is remote. Copyright © 2011. this normally triggers the target Manager process to start a Server Collector process with default parameters. Users can start a Server Collector statically and modify the parameters (but this option is rarely used). . Oracle and/or its affiliates. INVENTORY. Transaction Log Extract /ggs/dirdat/lt000000 GGSCI> ADD EXTRACT finance. TABLE SALES.ORDERS. BEGIN NOW GGSCI> ADD EXTTRAIL /ggs/dirdat/lt.Primary Extract Configuration for Oracle GGSCI> EDIT PARAMS finance Source Database EXTRACT finance USERID login. Oracle and/or its affiliates. PASSWORD pw EXTTRAIL /ggs/dirdat/lt TABLE SALES. All rights reserved. .16 Copyright © 2011. EXTRACT finance GGSCI> START EXTRACT finance /ggs/dirdat/lt000001 Trail 6 . TRANLOG. . MGRPORT <port> RMTTRAIL .INVENTORY.ORDERS./dirdat/lt GGSCI> ADD RMTTRAIL .17 EXTRACT pump PASSTHRU RMTHOST <target>. Oracle and/or its affiliates./dirdat/rt TABLE SALES./dirdat/rt. GGSCI> ADD EXTRACT pump. TABLE SALES. EXTRACT pump GGSCI> START EXTRACT pump Copyright © 2011.Data Pump Configuration for Oracle GGSCI> EDIT PARAMS pump Local Trail Data Pump Remote Trail /ggs/dirdat/rt000000 /ggs/dirdat/rt000001 6 . EXTTRAILSOURCE . All rights reserved. Oracle and/or its affiliates. All rights reserved. . and backup files. 6 .Automatic Storage Management (ASM) Automatic Storage Management (ASM) enables a disk group to be designated for Oracle database files. control files.18 ASM Instance DB Instance Meta and Data Management Data Access Copyright © 2011. Data mapping information is provided. • Confirm that the ASM instance is listed in the tnsnames. All rights reserved. ASMPASSWORD <password> 6 .19 Copyright © 2011. Oracle and/or its affiliates.ora file. verify the following: • A user with SYSDBA privileges in the ASM instance must be used. • • Confirm that the Oracle listener is listening for new connections to the ASM instance. Use the TRANLOGOPTIONS parameter with the ASMUSER and ASMPASSWORD options for ASM: TRANLOGOPTIONS ASMUSER SYS@<ASM_instance>. .Ensuring ASM Connectivity To ensure that Oracle GoldenGate can connect to an ASM instance. What does Extract do? Where does Extract capture transactional changes from? What parameters tell Extract where to send data? What commands are used to create and start an Extract group? 5. What command option is used to set the maximum size of a GoldenGate trail file before it rolls to the next file? 6 .Discussion Questions 1. . 2. Oracle and/or its affiliates. 3. All rights reserved. 4.20 Copyright © 2011. Oracle and/or its affiliates. What is a data pump? 7. What other parameter is commonly used on data pumps? 6 . .Discussion Questions 6. All rights reserved. Why might you use multiple data pumps for one source trail? 9. What is the advantage of using a data pump? 8.21 Copyright © 2011. What parameter is used to identify the remote target system? 10. 22 Copyright © 2011. All rights reserved. . you should have learned how to: • Configure and start an Extract process • Add local and remote trails • Configure and start a data pump • Understand the parameters to connect to an ASM instance 6 . Oracle and/or its affiliates.Summary In this lesson. 23 Setting up an Extract data pump and starting the two Extracts Copyright © 2011. Oracle and/or its affiliates.Practice 6-1 Overview: The practices for this lesson cover the following topics: • Setting up the Extract and the EXTTRAIL • 6 . . All rights reserved. Configuring Initial Load Copyright © 2011. . All rights reserved. Oracle and/or its affiliates. Oracle and/or its affiliates. All rights reserved.Objectives After completing this lesson. you should be able to: • Describe Oracle GoldenGate initial load methods • Understand the prerequisites required before initiating an initial load • Explain the advantages of Oracle GoldenGate methods • Configure an initial load by using Oracle GoldenGate 7-2 Copyright © 2011. . Preparing the environment Source Database 3. Oracle and/or its affiliates. All rights reserved. Change capture 7-3 Replicat Remote Trail 4.Step 3. . Initial Load 1. Initial load (various methods) Target DB Data Pump Extract Transaction Log Local Trail 2. Change delivery Copyright © 2011. including native bulk load utilities GoldenGate change delivery can handle collisions with initial load. without locking tables Fetch data in arrays to speed performance Parallel processing using WHERE clauses or RANGE function Distribute data over multiple network controllers Flexible load alternatives.Initial Load: Advantages of GoldenGate Methods • • • • • • • • 7-4 Work across heterogeneous database types and platforms No application downtime required Read directly from source tables. All rights reserved. Oracle and/or its affiliates. . Copyright © 2011. Initial Load: Resource Limitations • • • • 7-5 How close are your systems? How large are your tables? What are the outage time constraints? How much disk space do you have to store changes? Copyright © 2011. . All rights reserved. Oracle and/or its affiliates. Oracle and/or its affiliates.Prerequisites for Initial Load • • • • • 7-6 Disable DDL processing. All rights reserved. Prepare the target tables. Create a data definitions file (if the source and target databases have dissimilar definitions). . Copyright © 2011. Configure the Manager process. Create change-synchronization groups (for capture and replication transactional changes during the initial load). Initial Load: Oracle GoldenGate Methods GoldenGate Method Extract writes to: Load Method File to Replicat Trail (canonical format) Replicat via SQL File to database utility Formatted text file Database utility Direct load Replicat (directly) Replicat via SQL Direct bulk load Replicat (directly) Replicat via SQL*Loader API 7-7 Copyright © 2011. All rights reserved. Oracle and/or its affiliates. . Oracle and/or its affiliates. Target DB . All rights reserved.Initial Load: File to Replicat Manager Extract Replicat Files Source Database ADD EXTRACT <name> Extract parameters: SOURCEISTABLE RMTTRAIL<name> 7-8 Copyright © 2011. Target DB . Oracle and/or its affiliates.Initial Load: File to Database Utility File SQL* Loader File BCP File SSIS Manager Extract Source Database ADD EXTRACT <name> Extract parameters: SOURCEISTABLE RMTFILE <name> FORMATASCII BCP or SQLLOADER 7-9 Copyright © 2011. All rights reserved. GROUP <name> 7 . Oracle and/or its affiliates. .10 ADD REPLICAT <name>. All rights reserved. SOURCEISTABLE Extract parameters: RMTTASK REPLICAT.Initial Load: Direct Load Manager Manager Extract Replicat Target DB Source Database ADD EXTRACT <name>. SPECIALRUN Copyright © 2011. 12 Copyright © 2011.Initial Load: Direct Bulk Load (to Oracle) Manager Extract Manager Replicat SQL* Loader API Oracle Target Source Database ADD EXTRACT <name>. . Oracle and/or its affiliates. GROUP <name> Replicat parameters: BULKLOAD 7 . SPECIALRUN Extract parameters: RMTTASK REPLICAT. All rights reserved. SOURCEISTABLE ADD REPLICAT <name>. Discussion Questions 1. Oracle and/or its affiliates. All rights reserved. . for initial load)? 7 . What GoldenGate command arguments specify that Extract and Replicat run as batch tasks (for example. What are the GoldenGate methods for initial load? 2.13 Copyright © 2011. you should have learned how to: • Describe Oracle GoldenGate initial load methods • Explain the advantages of Oracle GoldenGate methods • Configure an initial load by using Oracle GoldenGate 7 .14 Copyright © 2011. Oracle and/or its affiliates. All rights reserved. .Summary In this lesson. All rights reserved.Practices This practices for this lesson cover the following topics: • Setting up the initial load by using the File to Replicat method • Setting up the initial data load by using the Direct Load method 7 .15 Copyright © 2011. . Oracle and/or its affiliates. Configuring Change Delivery Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Objectives After completing this lesson, you should be able to: • Explain what the Replicat process does • Configure and start a Replicat process • Configure Replicat to handle collisions between changes and the initial-load data 8-2 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Step 4. Change Delivery 1. Preparing the environment Source Database 3. Initial load (various methods) Target DB Data Pump Extract Transaction Log Local Trail Replicat Remote Trail 2. Change capture 8-3 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 4. Change delivery Replicat Overview • • • The Replicat process runs on the target system. Multiple Replicat processes can be used with multiple Extract processes in parallel to increase throughput. Replicat can: – – – – 8-4 Read data out of GoldenGate trails Perform data filtering by table, row, or operation Perform data transformation Perform database operations just as your application performed them Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Change Delivery Tasks On the target system: • Create a checkpoint table in the target database (best practice): DBLOGIN ADD CHECKPOINTTABLE • Create a parameter file for Replicat: EDIT PARAMS • Create a Replicat group: ADD REPLICAT • Start the Replicat process: START REPLICAT 8-5 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Sample Configuration Trail Replicat Target DB 8-6 GGSCI> DBLOGIN USERID login PASSWORD pw GGSCI> ADD CHECKPOINTTABLE checkpt GGSCO> EDIT PARAMS REPORD REPLICAT REPORD USERID ggsuser PASSWORD ggspass ASSUMETARGETDEFS DISCARDFILE /ggs/dirrpt/REPORD.dsc, APPEND MAP SALES.ORDERS, TARGET USSALES.USORDERS; MAP SALES.INVENTORY, TARGET USSALES.USINVENTORY; GGSCI> ADD REPLICAT REPORD, EXTTRAIL /ggs/dirdat/rt GGSCI> START REPLICAT REPORD Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Avoiding Collisions with Initial Load • • If the source database remains active during an initial load, you must either avoid or handle any collisions when updating the target with interim changes. If you can back up, restore, or clone the database at a point in time, you can avoid collisions by starting Replicat to read trail records from a specific transaction commit sequence number (CSN): GGSCI> START REPLICAT <group> ATCSN | AFTERCSN <csn> 8-7 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Handling Collisions with Initial Load • • If you cannot avoid collisions. All rights reserved. – When Replicat encounters a missing-record error for an update or delete. it writes the change record over the initial data load record. • After all of the change data generated during the load has been replicated. the change record is discarded. . The Replicat HANDLECOLLISIONS parameter can be used. you must handle them. Oracle and/or its affiliates. – When Replicat encounters a duplicate-record error on an insert. turn off HANDLECOLLISIONS: GGSCI> SEND REPLICAT <group> NOHANDLECOLLISIONS GGSCI> EDIT PARAMS <group> (to remove parameter) 8-8 Copyright © 2011. What does Replicat do? 2. What GGSCI command creates a GoldenGate checkpoint table on the target database? 6. What commands are used to create and start a Replicat group? 5. . How does Replicat know the layout of the source tables when source and target schemas differ? 4. What parameter manages conflicts between initial load and change replication? Where is it specified? 8-9 Copyright © 2011. When is ASSUMETARGETDEFS valid? 3.Discussion Questions 1. All rights reserved. Oracle and/or its affiliates. What is the purpose of the DISCARDFILE? 7. you should have learned how to: • Explain what the Replicat process does • Configure and start a Replicat process • Configure Replicat to handle collisions between changes and the initial load data 8 .10 Copyright © 2011. All rights reserved. . Oracle and/or its affiliates.Summary In this lesson. Practices The practices for this lesson cover the following topics: • Setting up the checkpoint table on the target system • Setting up GoldenGate delivery • Generating data and starting GoldenGate processes 8 .11 Copyright © 2011. . All rights reserved. Oracle and/or its affiliates. Extract Trail and Files Copyright © 2011. Oracle and/or its affiliates. All rights reserved. . .Objectives After completing this lesson. you should be able to: • Describe and contrast Extract trails and files • Describe the formats that Extract trails and files may have • View Extract trails and files with Logdump • Reverse the sequence of operations in an Extract trail or file (to back out changes) 9-2 Copyright © 2011. Oracle and/or its affiliates. All rights reserved. with variable length records. – I/O is performed using large block writes. . All rights reserved.Overview of Extract Trails and Files • Extract writes data to any of the following: – – – – • Local trail (EXTTRAIL) on the local system Local file (EXTFILE) on the local system Remote trail (RMTTRAIL) on a remote system Remote file (RMTFILE) on a remote system Extract trails and files are unstructured. Copyright © 2011. – Multiple Replicat processes may process the same trail. Oracle and/or its affiliates. • Extract writes checkpoints for trails during change capture: – This guarantees that no data is lost during restart. • 9-3 Extract does not write checkpoints for files. then distribute over IP with a data pump to remote trails – To multiple trails: — For distribution to multiple systems/disk storage devices — For parallel processing by downstream processes • 9-4 Trails and files can be transported online using TCP/IP or sent in batch using any file transfer method. Oracle and/or its affiliates. Copyright © 2011. . All rights reserved.Extract Trails and Files Distribution • Extract can write: – To local trails. only the primary key and changed columns are recorded. By default. Committed transactional order is preserved.Extract Trails and Files Contents • • • • • 9-5 Each record in the trail contains an operation that has been committed in the source database. Oracle and/or its affiliates. All rights reserved. Copyright © 2011. Flags indicate the first and last records in each transaction. . Operations in a transaction are grouped together in the order in which they were applied. All rights reserved. • Configure Manager to purge used trail data (best practice). 9-6 Copyright © 2011. Oracle and/or its affiliates. .Extract Trails and Files Cleanup Trail files can be purged after they are consumed: • The temporary storage requirement is small if processes keep pace. allowing them to be exchanged rapidly and accurately among heterogeneous databases. . which contains a header area as well as a data area 9-7 Copyright © 2011.Trail Format • • By default. Each trail file contains the following: – Record header area. which is stored at the beginning of the file and contains information about the trail file itself – Record data area. trails are formatted in the canonical format. All rights reserved. Oracle and/or its affiliates. Character set .DB type.Compatibility level . version.Hostname . All rights reserved.OS type and version .Creation time .Timestamp .File sequence number . Oracle and/or its affiliates.Record Header Area Each trail file has a file header that contains: Trail File Information .Group name .Hardware type .Commit sequence number (CSN) Extract Information .GoldenGate version . .File size 9-8 First and Last Record Information . and character set Copyright © 2011. .Record Data Area The trail file header and the data area of the GoldenGate trail record contains the following: • Trail header file • The time that the change was written to the Oracle GoldenGate file • The type of database operation • The length of the record • The relative byte address within the train file • The table name • The data changes in hex format • Optional user token area 9-9 Copyright © 2011. All rights reserved. Oracle and/or its affiliates. Setting the Compatibility Level • • • The <major>. or RMTTRAIL parameter: RMTTRAIL /ggs/dirdat/extdat. Copyright © 2011. This allows customers to use different versions of GoldenGate Extract. Set in the Extract EXTFILE.4 • 9 . trail files. Oracle and/or its affiliates. .<minor> setting identifies the trail file format version numbers used by GoldenGate. All rights reserved. RMTFILE. EXTTRAIL. FORMAT RELEASE 10.10 The input and output trails of a data pump must have the same compatibility level. and Replicat together. All rights reserved. This is beneficial if database load utilities or other programs are used that require different input.11 Copyright © 2011. . These alternative formats include: – FORMATASCII – FORMATSQL – FORMATXML 9 . Oracle and/or its affiliates.Alternative Trail Formats • • • Instead of the default canonical format. alternative formats can be used to output data. This parameter is required by the file-to-database-utility initial-load method. Oracle and/or its affiliates. The FORMATASCII statement must be before the extract files or trails statements that are listed in the parameter file.FORMATASCII • • • • Output is in external ASCII format. FORMATASCII can format data for popular database load utilities. – FORMATASCII.12 Copyright © 2011. . 9 . All rights reserved. BCP – FORMATASCII. SQLLOADER • Data cannot be processed by GoldenGate Replicat because Replicat expects the default canonical format. UPDATE CUSTOMER SET BALANCE = 100 WHERE CUSTNAME = "Eric".A.CUSTOMER. FORMATASCII without options produces: B. C. Oracle and/or its affiliates. FORMATASCII.CUSTNAME. DELIMITER '|' produces: B|1997-02-17:14:09:46. "San Fran".BALANCE. 550). • Example 1.CUSTOMER.1997-02-17:14:09:46.13 Copyright © 2011.A.1873474.'Eric'. I.8. NONAMES. • Example 2.CUSTNAME.BALANCE.'San • Fran'. All rights reserved.TEST. COMMIT. V.TEST.'Eric'. .550.421335.100.421335|8|1873474| I|A|CUSTOMER|'Eric'|'San Fran'|550| V|A|CUSTOMER|CUSTNAME|'Eric'|BALANCE|100| C| 9 .FORMATASCII Sample Output Sample transaction INSERT INTO CUSTOMER VALUES ("Eric".LOCATION. Oracle and/or its affiliates. Data cannot be processed by GoldenGate Replicat because Replicat expects the default canonical format.FORMATSQL • • • 9 . Copyright © 2011. UPDATE. All rights reserved.14 Output is in external SQL DML format. FORMATSQL generates SQL statements (INSERT. DELETE) that can be applied to both SQL and Enscribe tables. . TRANSACTION_ID) VALUES ('WILL'.STATE) VALUES ('WILL'.PRODUCT_PRICE. 9 . .'WA').'100').ORDER_DATE.'SEATTLE'. INSERT INTO TEST.PRO DUCT_AMOUNT.ORDER_ID.3.1.TRANSACTION_ID) VALUES ('JANE'.TCUSTMER WHERE CUST_CODE='WILL'.'100').PRODUCT_PRICE. DELETE FROM TEST.15 Copyright © 2011.'. Oracle and/or its affiliates.TCUSTORD WHERE CUST_CODE='JANE' AND ORDER_DATE='1995-11-11:13:52:00' AND PRODUCT_CODE='PLANE' AND ORDER_ID='256'.'.'DENVER'.133300.PRODUCT_CODE. DELETE FROM TEST. C.'BG SOFTWARE CO.'144'.NAME.TCUSTORD WHERE CUST_CODE='WILL' AND ORDER_DATE='1994-09-30:15:33:00' AND PRODUCT_CODE='CAR' AND ORDER_ID='144'.'CAR'.'CO').CITY.STATE) VALUES ('JANE'.CITY.'PLANE'.155.PRODUCT_CODE.'ROCKY FLYER INC. All rights reserved. INSERT INTO TEST.00.TCUSTMER WHERE CUST_CODE='JANE'. DELETE FROM TEST.17520.TCUSTMER (CUST_CODE.FORMATSQL Sample Output B.'1994-0930:15:33:00'.'1995-1111:13:52:00'.00.1226440129. INSERT INTO TEST. INSERT INTO TEST.000000.TCUSTMER (CUST_CODE.NAME.ORDER_DATE.ORDER_ID. DELETE FROM TEST.2008-11-11:13:48:49.TCUSTORD (CUST_CODE.'256'.PRO DUCT_AMOUNT.TCUSTORD (CUST_CODE. FORMATXML • • • 9 . All rights reserved. The NOBINARYCHARS parameter is used with FORMATXML.16 Output is in XML format. . Oracle and/or its affiliates. Copyright © 2011.) Data cannot be processed by GoldenGate Replicat because Replicat expects the default canonical format. (Contact Oracle Support for additional guidance for using this parameter. 17 Copyright © 2011. Oracle and/or its affiliates.TCUSTMER" type="insert"> <columns> <column name="CUST_CODE" key="true">ZOE</column> <column name="NAME">ZOE'S USED BICYCLES</column> <column name="CITY">ABERDEEN</column> <column name="STATE">WA</column> </columns> </dbupdate> <dbupdate table="TEST. All rights reserved. .000000"> <dbupdate table="TEST.FORMATXML Sample Output <transaction timestamp="2008-11-11:14:33:12.TCUSTMER" type="insert"> <columns> <column name="CUST_CODE" key="true">ZEKE</column> <column name="NAME">ZEKE'S MOTION INC.TCUSTMER" type="insert"> <columns> <column name="CUST_CODE" key="true">VAN</column> <column name="NAME">VAN'S BICYCLES</column> <column name="CITY">ABERDEEN</column> <column name="STATE">WA</column> </columns> </dbupdate> </transaction> 9 .</column> <column name="CITY">ABERDEEN</column> <column name="STATE">WA</column> </columns> </dbupdate> <dbupdate table="TEST. All rights reserved.18 Copyright © 2011.Logdump Utility • The Logdump utility allows you to: – Display or search for information that is stored in GoldenGate trails or extract files – Save a portion of a GoldenGate trail to a separate trail file • To start Logdump from the GoldenGate installation directory: Shell> logdump • To access help: Logdump 1> help 9 . Oracle and/or its affiliates. . .Opening a Trail Logdump> open dirdat/rt000000 Logdump responds with: Current LogTrail is /ggs/dirdat/rt000000 9 .19 Copyright © 2011. All rights reserved. Oracle and/or its affiliates. .Setting up a View • To view the trail file header: Logdump 1> fileheader on • To view the record header with the data: Logdump 2> ghdr on • To add column information: Logdump 3> detail on • To add hex and ASCII data values to the column list: Logdump 4> detail data • To control how much record data is displayed: Logdump 5> reclen 280 9 . All rights reserved. Oracle and/or its affiliates.20 Copyright © 2011. .@9.. Length 303 3000 012f 3000 0008 660d 0a71 3100 0006 0001 3200 | 0.?4...034.../0.1uri:tellurian::h ome:mccargar:ggs:ggs Oracle:source6..631 FileHeader Name: *FileHeader* 3000 0008 0037 6f6d 4f72 2f64 0138 012f 0000 0031 653a 6163 6972 0000 3000 0016 7572 6d63 6c65 6461 0800 0008 3300 693a 6361 3a73 742f 01e2 660d 000c 7465 7267 6f75 6572 4039 0a71 02f1 6c6c 6172 7263 3030 0000 GroupID x30 '0' TrailInfo 3100 7834 7572 3a67 6536 3030 0c00 0006 eac7 6961 6773 0000 3030 0000 0001 7f3f 6e3a 3a67 1700 3700 0000 Length 587 Length 303 Length 103 Length 88 Length 85 Length 4 Length 587 Len 587 RBA 0 3200 3400 3a68 6773 112e 0005 001d Info x00 | | | | | | | 0........2.. All rights reserved.7..q1.8.... etc..3..2...... . /dirdat/er0000007....q1.../0....... .f..f. ....21 Copyright © 2011.Viewing the Trail File Header Logdump Logdump Reading Logdump 14662 >fileheader detail 14663 >pos 0 forward from RBA 0 14664 >n TokenID x46 'F' Record Header Info x00 TokenID x30 '0' TrailInfo Info x00 TokenID x31 '1' MachineInfo Info x00 TokenID x32 '2' DatabaseInfo Info x00 TokenID x33 '3' ProducerInfo Info x00 TokenID x34 '4' ContinunityInfo Info x00 TokenID x5a 'Z' Record Trailer Info x00 2008/07/18 13:40:26..... ... 9 .x4. Oracle and/or its affiliates.. and to move from one record to another in sequence: Logdump 6 > pos 0 Logdump 7 > next (or just type n) • To position at an approximate starting point and locate the next good header record: Logdump 8 > pos <approximate RBA> Logdump 9 > scanforheader (or just type sfh) 9 .Viewing Trail Records • To go to the first record. Oracle and/or its affiliates. All rights reserved.22 Copyright © 2011. . in hex Length of record and its RBA position in the trail file Copyright © 2011.23 Column information Record data.Viewing Trail Records I/O type Record header: contains transaction information. Below the header is the data area. Record data. Oracle and/or its affiliates. in ASCII . All rights reserved. Operation type and the time that the record was written Source table Image type: could be a before or after image 9 . 8 9 .... Oracle and/or its affiliates. 22661 Records/Trans ... . 193 Files/Trans ..Counting Records in the Trail Logdump> count LogTrail /ggs/dirdat/rt000000 has 4828 records Total Data Bytes 334802 Avg Bytes/Record 69 Delete 900 Insert 3902 FieldComp 26 Before Images 900 After Images 3928 Average of 25 Transactions Bytes/Trans .24 Copyright © 2011.. All rights reserved.... .Counting Records in the Trail TCUSTMER Total Data Bytes Avg Bytes/Record Delete Insert FieldComp Before Images After Images TCUSTORD Total Data Bytes Avg Bytes/Record Delete Insert Field Comp Before Images After Images 9 .25 10562 55 300 1578 12 300 1590 229178 78 600 2324 14 600 23388 Copyright © 2011. All rights reserved. Oracle and/or its affiliates. (x00) BeforeAfter: A (x41) RecLength : 56 (x0038) IO Time : 2002/04/30 15:56:40. . All rights reserved. (x00) AuditRBA : 105974056 2002/04/30 15:56:40.814 IOType : 5 (x05) OrigNode : 108 (x6c) TransInd : . Oracle and/or its affiliates. (x01) FormatType : F (x46) SyskeyLen : 0 (x00) Incomplete : .26 18 records Copyright © 2011.Filtering by a Filename Logdump 7 > filter include filename TCUST* Logdump 8 > filter match all Logdump 9 > n _______________________________________________________________ Hdr-Ind : E (x45) Partition : . (x00) UndoFlag : .814 Insert Len 56 Log RBA 1230 File: TCUSTMER Partition 0 After Image: 3220 2020 4A61 6D65 7320 2020 2020 4A6F 686E 736F| 2 James Johnso 6E20 2020 2020 2020 2020 2020 2020 4368 6F75 6472| n Choudr 616E 7420 2020 2020 2020 2020 2020 4C41 | LA Filtering suppressed 9 . (x00) UndoFlag : .27 Copyright © 2011. (x00) AuditRBA : 109406324 2002/04/30 16:22:14. (x00) BeforeAfter: B (x42) RecLength : 56 (x0038) IO Time : 2002/04/30 16:22:14.205 IOType : 3 (x03) OrigNode : 108 (x6c) TransInd : . . All rights reserved.205 Delete Len 56 Log RBA 64424 File: TCUSTMER Partition 0 Before Image: 3620 2020 4A61 6D65 7320 2020 2020 4A6F 686E 736F | 6 James Johnso 6E20 2020 2020 2020 2020 2020 2020 4574 6865 6C20 | n Ethel 2020 2020 2020 2020 2020 2020 2020 4C41 | LA Filtering suppressed 545 records 9 .Locating a Hex Data Value Logdump 27 > filter inc hex /68656C20/ Logdump 28 > pos 0 Current position set to RBA Logdump 29 > n __________________________________________________________ Hdr-Ind : E (x45) Partition : . (x01) FormatType : F (x46) SyskeyLen : 0 (x00) Incomplete : . Oracle and/or its affiliates. All rights reserved. Oracle and/or its affiliates. . issue the following command: Logdump> save newtrail 10 records 9 .29 Copyright © 2011.Saving Records to a New Trail To save the 10 records from the current position in the file. 30 Copyright © 2011. Oracle and/or its affiliates. use the LOG option: Logdump> log to MySession. All rights reserved. logging remains in effect for all sessions of Logdump until it is disable with the Log STOP command: Logdump> log stop 9 .txt • When enabled.Keeping a Log of Your Session • To start and stop the logging of a Logdump session. . Oracle and/or its affiliates. such as corrupt data or accidental delete operations while keeping the rest of the application alive – Is used to restore a database to a specific point in time.31 Copyright © 2011. All rights reserved.Overview of the Reverse Utility • • The Reverse utility reorders operations within GoldenGate trails in reverse sequence. allowing to back out all operations during regression testing to restore the original test baseline 9 . . The Reverse utility: – Provides selectively backing out of certain operations. Inserts become deletes. it reverses the file sequence.Overall Process of the Reverse Utility Source: Transaction log or GoldenGate trails Source Database Extract ext1 File Reverse input SPECIALRUN TRANLOG BEGIN END GETUPDATEBEFORES NOCOMPRESSDELETES Filter criteria (if any) EXTFILE or RMTFILE (Table statements) 9 . All rights reserved. Update before images become after images. . Deletes become inserts. Oracle and/or its affiliates. If the input is a series of files. Begin and end transaction indicators are reversed. Copyright © 2011.32 File Replicat rep1 output Operation sequence is reversed. All rights reserved.33 Copyright © 2011.dat • Run Replicat to back out operations: $ ggs/> replicat paramfile dirprm/rep1. .Reverse: Overall Process • Run Extract from either GoldenGate trails or the source database transaction logs: $ ggs/> extract paramfile dirprm/ext1.prm 9 . Oracle and/or its affiliates.dat dirdat/output.prm • Run Reverse to produce the reordered output: $ ggs/> reverse dirdat/input. ACCOUNTS.Sample Parameter Files • Extract Parameters (dirprm/ext1.SALES. USERID user.dat ASSUMETARGETDEFS MAP HR.34 Copyright © 2011. All rights reserved. MGRPORT 7809 RMTFILE /ggs/dirdat/input. TARGET HR. Oracle and/or its affiliates. 9 . PASSWORD password EXTFILE /ggs/dirdat/output. TABLE HR.dat TABLE HR. PASSWORD password BEGIN 2004-05-30 17:00 END 2004-05-30 18:00 GETUPDATEBEFORES NOCOMPRESSDELETES RMTHOST target. USERID user.SALES.ACCOUNTS. • Replicat Parameters (dirprm/rep1. TRANLOG SOURCDB dsn. .prm) SPECIALRUN END RUNTIME TARGETDB dsn.SALES.prm) SPECIALRUN. MAP HR. TARGET HR.ACCOUNTS. 35 Copyright © 2011. All rights reserved. In what formats are Extract trails and files written? 3.Discussion Questions 1. . What is a trail? 2. What GoldenGate utility enables you to view trail contents? 9 . Oracle and/or its affiliates. 36 Copyright © 2011. Oracle and/or its affiliates. you should have learned how to: • Describe and contrast Extract trails and files • Describe the formats that Extract trails and files may have • View Extract trails and files with Logdump • Reverse the sequence of operations in an Extract trail or file (to back out changes) 9 . All rights reserved. .Summary In this lesson. 9 . All rights reserved. Oracle and/or its affiliates.37 Copyright © 2011. .Practice This practice for this lesson covers using the Logdump utility. Oracle and/or its affiliates. . All rights reserved.Oracle GoldenGate Parameters Copyright © 2011. and Replicat 10 . you should be able to: • Edit parameter files • Contrast GLOBALS with process parameters • Describe commonly used parameters for GLOBALS. All rights reserved.Objectives After completing this lesson. Extract. . Manager.2 Copyright © 2011. Oracle and/or its affiliates. Oracle and/or its affiliates. All rights reserved.Oracle GoldenGate Parameter Files • There are two types of parameter files: – The GLOBALS file.3 Copyright © 2011. such as Extract 10 . which are coupled with a specific process. . which stores parameters that relate to the Oracle GoldenGate instance as a whole – Run-time parameter files. 4 Copyright © 2011. run GGSCI using the EDIT PARAMS option./GLOBALS • Manager and utility parameter files are identified by keywords: GGSCI> EDIT PARAMS MGR GGSCI> EDIT PARAMS DEFGEN • Extract and Replicat parameter files are identified by the process group name: GGSCI> EDIT PARAMS <group name> 10 .Using Parameter Files • To create a parameter file. . The GLOBALS parameter file is identified by its file path: • GGSCI> EDIT PARAMS . All rights reserved. Oracle and/or its affiliates. prm – Most apply to all tables processed.5 Copyright © 2011. Replicat. Oracle and/or its affiliates. Server Collector.GLOBALS Versus Process Parameters • GLOBALS parameters apply to all processes. – Set when the process starts – Override GLOBALS settings – Reside by default in the dirprm directory in files named <processname>. but some can be specified at the table level. All rights reserved. – Set when Manager starts – Reside in <GoldenGate install directory>/GLOBALS • Process parameters apply to a specific process (Manager. Extract. . 10 . Utilities). • Some of the most common parameters include: – MGRSERVNAME ggsmanager1: Defines a unique Manager service name on Windows – CHECKPOINTTABLE dbo.ggschkpt: Defines the default table name used for Replicat’s checkpoint table 10 . All rights reserved. Once set. Oracle and/or its affiliates. GLOBALS parameters are rarely changed. .6 Copyright © 2011.GLOBALS Parameters • GLOBALS parameters: – Control things common to all processes in a GoldenGate instance – Can be overridden by parameters at the process level • • You must exit GGSCI to save. Copyright © 2011. user interface. and reporting of thresholds and errors. Manager controls other GoldenGate processes. resources. Oracle and/or its affiliates.Manager Parameters: Overview • • • 10 .7 Manager is GoldenGate’s parent process. All rights reserved. The default parameter settings usually suffice. . 8 Copyright © 2011. USECHECKPOINTS 10 . Oracle and/or its affiliates. RETRIES 5 LAGREPORTHOURS 1 LAGINFOMINUTES 3 LAGCRITICALMINUTES 5 PURGEOLDEXTRACTS /ggs/dirdat/rt*. All rights reserved.Sample Manager Parameter File PORT 7809 DYNAMICPORTLIST 9001–9100 AUTOSTART ER * AUTORESTART EXTRACT *. WAITMINUTES 2. . Oracle and/or its affiliates. All rights reserved.Manager Parameter Categories Category Parameter Summary General Allows comments in the parameter file. . filters messages that are written to the system logs Port Management Establishes the TCP/IP port.9 Copyright © 2011. specifies a time to wait before assigning a port number Process Management Determines what processes and how long after a failure they are restarted Event Management Reports processes that stop abnormally and what information is reported to the error log Database Login Provide login information Maintenance Trail maintenance. including trail data that is no longer needed 10 . Extract Parameter Overview Extract parameters specify: • Group name (associated with a checkpoint file) • Where to send the data: – Local system – Multiple remote systems – One-to-many GoldenGate trails • What is being captured: – Which tables – Which rows and columns – Which operations • • 10 . Oracle and/or its affiliates. . All rights reserved.10 Which column mapping to apply Which data transformations to apply Copyright © 2011. data can be captured with the following specifications: – – – – – • • Committed data only Full image for inserts Only primary key and changed columns for updates Only primary key for deletes Only after-image of update Send data without transformation Buffer transactions – Until a block is full or – Until time elapses • 10 .11 Based on average transaction volumes Copyright © 2011.Extract Parameter Defaults Extract parameters can be modified or assume a default value: • For insert. Oracle and/or its affiliates. . and delete operations. All rights reserved. update. PASSWORD password RMTHOST manhattan. 10 . MGRPORT 7809 RMTTRAIL /ggs/dirdat/rt TABLE SALES. Oracle and/or its affiliates.Sample Extract Parameter File EXTRACT ODS USERID GoldenGateUser. TABLE SALES. .12 Copyright © 2011.ORDERS. All rights reserved.INVENTORY. All rights reserved. Oracle and/or its affiliates.13 Copyright © 2011.Extract Parameter Categories Category Parameter Summary General Verifies parameter file syntax and retrieves variables that were set by other parameters Processing Method Determines when a processing run begins and ends Database Login Provide login information Selecting and Mapping Data What information is extracted and in what format Routing Data Provides the location where the data is written Formatting Data Formats data in another format besides the default Oracle GoldenGate format 10 . . 14 Copyright © 2011. and memory allocations Maintenance Specifies how often trail files are created or purged Security Data decryption in a trail or extract file 10 . All rights reserved.Extract Parameter Categories Category Parameter Summary Custom Processing Whether to invoke a user exit routine or a macro Reporting Displays what information is included in statistical displays Error Handling Contains records that cannot be processed and error handling for DDL extraction Tuning Controls how long data is buffered before writing to a trail. Oracle and/or its affiliates. . All rights reserved. Oracle and/or its affiliates.Extract Example: TABLE Parameter • • Use the TABLE parameter in an Extract parameter file to specify objects for extraction. .15 Select and filter records Select and map columns Transform data Designate key columns Define user tokens Trim trailing spaces Pass a parameter to a user exit Execute stored procedures and queries Copyright © 2011. Some of the TABLE options do the following: – – – – – – – – 10 . A number of options control the archive log: • – To specify an alternative log format: TRANLOGOPTIONS ALTARCHIVEDLOGFORMAT log_%t_%s_%r.16 Copyright © 2011.arc – To specify an alternative archive log location: TRANLOGOPTIONS ALTARCHIVELOGDEST /oradata/archive/log2 – To cause Extract to read from the archived logs exclusively: TRANLOGOPTIONS ARCHIVELOGONLY 10 . Oracle and/or its affiliates. . All rights reserved.Extract Example: TRANLOGOPTIONS Parameter • Use the TRANLOGOPTIONS parameter to control databasespecific aspects of log-based extraction. Extract Example: TRANLOGOPTIONS Parameter • Additionally. . All rights reserved. Oracle and/or its affiliates.17 Copyright © 2011. there are a number of options for loop prevention: – To specify the name of the Replicat database user so that those transactions are not captured by Extract: TRANLOGOPTIONS EXCLUDEUSER ggsrep – To specify the transaction name of the Replicat database user so that those transactions are not captured by Extract: TRANLOGOPTIONS EXCLUDETRANS “ggs_repl” 10 . 18 Optional row-level selection criteria Optional column mapping facilities Optional transformation services Optional stored procedure or SQL query execution Error handling Various optional parameter settings Copyright © 2011. Oracle and/or its affiliates.Replicat Parameter Overview Replicat parameters specify: • A group name that is also associated with a checkpoint file • List of source to target relationships: – – – – • • 10 . . All rights reserved. or delete operations • Smart transactional grouping – 1.Replicat Parameter Defaults Replicat parameters can be modified or assume a default value: • Apply all inserts. Oracle and/or its affiliates. • Process ABENDs on any operational failure: – Roll back transactions to the last good checkpoint – Optional error handling – Optional mapping to secondary table for exceptions 10 .000 source operations are grouped into a single target transaction.19 Copyright © 2011. . All rights reserved. update. APPEND MAP HR. TARGET HR. Oracle and/or its affiliates.CODES.Sample Replicat Parameter File REPLICAT SALESRPT USERID ggsuser. MAP HR.ORDERS. All rights reserved.20 Copyright © 2011.CODES. PASSWORD ggspass ASSUMETARGETDEFS DISCARDFILE /ggs/dirrpt/SALESRPT. WHERE (STATE = “CA” AND OFFICE = “LA”).ORDERS. TARGET HR.dsc. . TARGET SALES.STUDENT WHERE (STUDENT_NUMBER < 400000).STUDENT. 10 . MAP SALES. Oracle and/or its affiliates. and Mapping Data What information is replicated and in what format Routing Data Defines the name of the Extract file or trail that contains data to be replicated 10 .21 Copyright © 2011. .Replicat Parameter Categories Category Parameter Summary General Verifies parameter file syntax and retrieves variables that were set by other parameters Processing Method Determines when a processing run begins and ends Database Login Provides login information Selecting. All rights reserved. Converting. .Replicat Parameter Categories Category Parameter Summary Custom Processing Whether to invoke a user exit routine or a macro Reporting Displays what information is included in statistical displays and the number of records processed Error Handling Determines how Replicat manages errors for duplicate or missing records Tuning Controls the parameters of how fast Replicat can process the data and memory allocations Maintenance Specifies how often discard files are created or obsolete trail files are purged Security Decrypt data in a trail or extract file 10 . All rights reserved.22 Copyright © 2011. Oracle and/or its affiliates. Copyright © 2011.Replicat Example: MAP Parameter • • • 10 . particular subsets of the data can be replicated to the target table. Oracle and/or its affiliates. With the MAP parameter. MAP also enables the user to map certain fields or columns from the source record into the start record format (“column mapping”). All rights reserved.23 The MAP parameter establishes a relationship between one source and one target table. . 3. 2. All rights reserved. . Oracle and/or its affiliates. 10 .25 What are some typical Manager parameters? What are some typical Extract parameters? What are some typical Replicat parameters? Where are GoldenGate parameters documented? Copyright © 2011.Discussion Questions 1. 4. 26 Copyright © 2011. Oracle and/or its affiliates.Summary In this lesson. Extract. Manager. you should have learned how to: • Edit parameter files • Contrast GLOBALS with process parameters • Describe commonly used parameters for GLOBALS. . All rights reserved. and Replicat 10 . 27 Copyright © 2011. Oracle and/or its affiliates. All rights reserved. .Practices This practices for this lesson cover the following topics: • Modifying source Manager parameters • Modifying the target Manager parameters • Modifying the Extract parameters on the source database 10 . . All rights reserved.Data Selection and Filtering Copyright © 2011. Oracle and/or its affiliates. .2 Copyright © 2011.Objectives After completing this lesson. you should be able to: • Select and filter data for replication • Map columns between different schemas • Use built-in functions • Use SQLEXEC to interact directly with a database 11 . All rights reserved. Oracle and/or its affiliates. All rights reserved.Overview of Data Mapping and Manipulation • • All data selection. Oracle and/or its affiliates. . mapping. and manipulation is done by using options of the TABLE (Extract) and MAP (Replicat) parameters.3 Selecting records and columns Selecting and converting operations Mapping dissimilar columns Using transaction history Testing and transforming data Using tokens Copyright © 2011. Data can be integrated between different source and target tables by: – – – – – – 11 . Types of Definition Files • • • 11 - 4 When you configure column mapping or transformation on the target system, a source-definitions file is required. When you configure column mapping or transformation on the source system, a target-definitions file is required. When you configure column mapping or transformation on a intermediary system that contains neither a source database nor a target database, a source-definitions file and a target-definitions file must be provided on that system. Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Data Selection: Overview GoldenGate provides the ability to select or filter out data based on a variety of levels and conditions: Parameter / Clause Selects TABLE or MAP Table TABLE 11 - 5 WHERE Row FILTER Row, Operation, Range COLS | COLSEXCEPT Columns Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Data Selection: WHERE Clause • • • The WHERE clause is the simplest form of selection. The WHERE clause appears on either the MAP or TABLE parameter and must be surrounded by parentheses. The WHERE clause cannot: – Perform arithmetic operations – Refer to trail header and user token values • 11 - 6 Use the FILTER clause for more complex selections with built-in functions. Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Data Selection: WHERE Clause WHERE can perform an evaluation for: Element Description Columns Example PRODUCT_AMT Comparison operators Numeric values Literal strings Field tests Conjunctive operators =, <>, >, <, >=, <= -123, 5500.123 "AUTO", "Ca" @NULL,@PRESENT,@ABSENT AND, OR 11 - 7 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Data Selection: WHERE Clause Examples • Only rows where the state column has a value of CA are returned: WHERE (STATE = “CA”); • Only rows where the amount column has a value of NULL (note that if the amount was not part of the update, the result is false): WHERE (AMOUNT = @NULL); • Only rows where the amount was part of the operation and has a value that is not NULL: WHERE (AMOUNT @PRESENT AND AMOUNT <> @NULL); • Only rows where the account identifier is greater than CORP-ABC: WHERE (ACCOUNT_ID > “CORP-ABC”); 11 - 8 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Data Selection: FILTER Clause • • • The FILTER clause provides complex evaluations to include or exclude data selection. The FILTER clause appears on either the MAP or TABLE parameter and must be surrounded by parentheses. With FILTER you can: – Deploy other GoldenGate built-in functions – Use multiple FILTERs on one statement — If any filter fails, the entire filter clause fails. – Include multiple option clauses (for example, on insert/ update) – Raise a user-defined error for exception processing 11 - 9 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Data Selection: FILTER Clause Examples • The following example includes rows where the price multiplied by the amount exceeds 10,000: FILTER ((PRODUCT_PRICE*PRODUCT_AMOUNT)>10000); • The following example includes rows containing the string “JOE”: FILTER (@STRFIND(NAME, "JOE")>0); • The following example executes the filter for both updates and deletes, but not inserts: FILTER (ON UPDATE, ON DELETE, @COMPUTE (PRODUCT_PRICE*PRODUCT_AMOUNT)>10000); 11 - 10 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Oracle and/or its affiliates. All rights reserved.11 The RANGE function divides a workload into multiple. It guarantees that the same row will always be processed by the same process. It determines which group that range falls in.Data Selection: RANGE Function • • • 11 . randomly distributed groups of data. by computing a hash against the primary key or user-defined columns. Copyright © 2011. . Replicat #1 MAP SALES. Replicat #2 MAP SALES. .ACCOUNT.Data Selection: RANGE Function Examples • • For transaction volume beyond the capacity of a single Replicat. TARGET SALES. FILTER (@RANGE (1. All rights reserved.3)). 11 . the example below shows three Replicat groups.3)).12 Copyright © 2011.3)).ACCOUNT.ACCOUNT.ACCOUNT. FILTER (@RANGE (3. FILTER (@RANGE (2. Oracle and/or its affiliates. each processing one-third of the data. Replicat #3 MAP SALES. TARGET SALES.ACCOUNT.ACCOUNT. Hashing each operation by primary key to a particular Replicat guarantees the original sequence of operations. TARGET SALES. 3. FILTER (@RANGE (3.ACCOUNT. By hashing the REP_ID column.3.3)). FILTER (@RANGE (2. TABLE SALES.13 Copyright © 2011. All rights reserved.ACCOUNT.REP_ID)). related by REP_ID.3)).REP_ID)). RMTTRAIL /ggs/dirdat/cc TABLE SALES.Data Selection: RANGE Function Examples • • Two tables.REP. TABLE SALES. REP and ACCOUNT.3. FILTER (@RANGE (2. TABLE SALES.ACCOUNT. . 11 . related rows will always be processed to the same Replicat: RMTTRAIL /ggs/dirdat/aa TABLE SALES. FILTER (@RANGE (3.REP. Oracle and/or its affiliates. RMTTRAIL /ggs/dirdat/bb TABLE SALES. require three Replicats to handle the transaction volumes.REP_ID)).REP. FILTER (@RANGE (1.3)). FILTER (@RANGE (1. – Use COLMAP to map target columns from your source columns.Column Mapping: Overview • • GoldenGate provides the capability to map columns from one table to another. 11 .14 Copyright © 2011. All rights reserved. Data can be transformed between dissimilar database tables. • GoldenGate automatically matches source to target column names with USEDEFAULTS. Oracle and/or its affiliates. . • Mapping can be applied when either extracting or replicating data. PH_PREFIX. PH_NUMBER ) ). NAME = CUST_NAME.CONTACT. TARGET HR. PHONE_NUMBER = @STRCAT( “(”. . AREA_CODE. Oracle and/or its affiliates. COLMAP (USEDEFAULTS.Column Mapping: Example MAP HR.15 Copyright © 2011. “-”. “)”.PHONE. 11 . All rights reserved. .ACCTHISTORY. “OPTYPE”). 11 . TARGET REPORT.ACCOUNT. OP_TYPE = @GETENV(“GGHEADER”. BEFORE_AFTER_IND = @GETENV(“GGHEADER”. All rights reserved.Column Mapping: Building History This example uses special values to build a history of operations data: INSERTALLRECORDS MAP SALES. ).16 Copyright © 2011. TRAN_TIME = @GETENV(“GGHEADER”.”COMMITTIMESTAMP”). “BEFOREAFTERINDICATOR”). COLMAP (USEDEFAULTS. Oracle and/or its affiliates. Additional functions can be used by the ability to call your own logic through user exits. All rights reserved.17 GoldenGate provides the capability to transform columns by using a set of built-in functions. Copyright © 2011. .Data Transformation Using Functions • • • 11 . Transformation functions can be applied for either Extract or Replicat. Oracle and/or its affiliates. and other column values to determine the appropriate value and format for target columns Functions are identified with the @ prefix. Oracle and/or its affiliates.18 Perform string and number conversion Extract portions of strings or concatenate columns Compare strings or numbers Perform a variety of date mappings Use single or nested IF statements to evaluate numbers. you can: – – – – – • 11 . strings. All rights reserved. Copyright © 2011.Functions: Overview • Using column conversion functions. . Oracle and/or its affiliates. NULL. or invalid COLTEST Tests whether a column value is present. All rights reserved. NULL. or invalid VALONEOF Returns TRUE if a column contains one of a list of values 11 . . missing.19 Copyright © 2011.Functions: Performing Tests on Column Values Function Description CASE Allows user to select a value depending on a series of value tests EVAL Allows a user to select a value depending on a series of independent tests IF Selects one of two values depending on whether a conditional statement returns TRUE or FALSE COLSTAT Returns whether a column value is missing. "EAST") 11 . All rights reserved. . AZ. 0) – The following returns WEST if the STATE column is CA. @IF functions examples: – The following returns an amount only if the AMT column is greater than zero. "NV"). otherwise. it returns EAST: REGION = @IF (@VALONEOF (STATE. or NV.@IF Function • • • The @IF function is used to return one or two values based on a condition. "CA". zero is returned. Otherwise. "AZ". AMOUNT_COL = @IF (AMT > 0. "WEST". This function can be used with other Oracle GoldenGate functions to begin a conditional argument that tests for one or more exception conditions.20 Copyright © 2011. Oracle and/or its affiliates. AMT. 21 Copyright © 2011. Oracle and/or its affiliates. .Functions: Working with Dates Function Description DATE Returns a date from a variety of sources in a variety of output formats DATEDIFF Returns the difference between two dates or times DATENOW Returns the current date and time 11 . All rights reserved. date1_mm. date1_yy. "YYYYMMDDHHMISS". date1_dd) – The following converts a numeric column stored as YYYMMDDHHMISS to a Julian timestamp: julian_ts_col = @DATE ("JTS". and day columns into a date: date_col = @DATE ("YYYY-MM-DD". month. "YY". numeric_date) 11 .22 Copyright © 2011. Oracle and/or its affiliates.@DATE Function • • The @DATE function is used to return dates and times in a variety of formats to the target column based on the format passed into the source column. "DD". Examples of @DATE functions: – The following converts year. All rights reserved. . "MM". Returns 1 for equal and 0 if not equal. . Oracle and/or its affiliates.23 Copyright © 2011. STREXT Extracts selected characters from a string STRFIND Finds the occurrence of a string within a string STRLEN Returns the length of a string 11 .Functions: Working with Strings and Numbers Function Description COMPUTE Returns the result of an arithmetic expression NUMBIN Converts a binary string into a number NUMSTR Converts a string into a number STRCAT Concatenates two or more strings STRCMP Compares two strings to determine whether they are equal. or whether the first is less or greater than the second STREQ Tests to see whether two strings are equal. All rights reserved. Functions: Working with Strings and Numbers Function Description STRLTRIM Trims leading spaces in a column STRNCAT Concatenates one or more strings up to a specified number of characters per string STRNCMP Compares two strings up to a certain number of characters STRNUM Converts a number into a string. All rights reserved. with justification and zerofill options STRRTRIM Trims trailing spaces in a column STRSUB Substitutes one string for another within a column STRTRIM Trims both leading and trailing spaces in a column STRUP Changes a string to uppercase 11 .24 Copyright © 2011. . Oracle and/or its affiliates. area code.@STRCAT Function • The @STRCAT function is used to concatenate one or more strings or string (character) columns. "-". "." . and local phone number into an international phone number with hyphens between the components: INTL_PHONE = @STRCAT (COUNTRY_CODE. separated by a semicolon: NAME = @STRCAT (LASTNAME. AREA_CODE. Oracle and/or its affiliates. LOCAL_PHONE) 11 .25 Copyright © 2011. "-". .FIRSTNAME) – The following concatenates a country code. All rights reserved. @STRCAT function examples: • – The following concatenates the LASTNAME and FIRSTNAME columns. 10) 11 . 1. All rights reserved. 4.@STREXT Function • The @STREXT function is used to extract a portion of a string.26 Copyright © 2011. 7. . Oracle and/or its affiliates. PHONE_NO = @STREXT (PHONE. @STREXT function example: • – The following example uses three @STREXT functions to extract a phone number into three different columns: AREA_CODE = @STREXT (PHONE. PREFIX = @STREXT (PHONE. 6). 3). All rights reserved. last replicated operation. GETVAL Extracts parameters from a stored procedure as input to a FILTER or COLMAP clause HEXTOBIN Converts a hexadecimal string to a binary string HIGHVAL. Can retrieve the commit timestamp in local time or GMT. LOWVAL Emulate COBOL functions that allow you to set a numeric limit on string or binary data types RANGE Divides a workload into multiple groups of data. RANGE uses a hash against primary key or user-defined columns.27 Copyright © 2011. Oracle and/or its affiliates. trail file header. and lag. while ensuring the same row will always be sent to the same process.Other Functions Function Description BINARY Keeps source data in its original binary format in the target when source column is defined as character BINTOHEX Converts a binary string to a hexadecimal string GETENV Returns information on the GoldenGate environment. trail record header. TOKEN Maps environmental values that are stored in the user token area to the target column 11 . . account. the primary key columns are used. RMTTRAIL /ggs/dirdat/bb TABLE fin. @RANGE preserves data integrity by guaranteeing that the same row will always be processed by the same process group. FILTER (@RANGE (2. Note: 11 . Oracle and/or its affiliates. 2)). All rights reserved. Copyright © 2011. @RANGE function example: – The following example uses one Extract process and processes the load into two trails: RMTTRAIL /ggs/dirdat/aa TABLE fin. FILTER (@RANGE (1.@RANGE Function • • • The @RANGE function is used to divide the rows of any table across two or more Oracle GoldenGate processes. 2)).28 Because no columns were defined on which to base the range calculation. .account. 29 The SQLEXEC parameter extends GoldenGate capabilities by enabling Extract and Replicat to communicate with the database through SQL queries or run stored procedures.Overview of SQLEXEC • • 11 . All rights reserved. . SQLEXEC also extends data integration beyond what can be done with GoldenGate functions. Copyright © 2011. Oracle and/or its affiliates. Oracle and/or its affiliates. Use SQLEXEC at the root level (without input/output parameters) to call a stored procedure. All rights reserved. • 11 .SQLEXEC: Basic Functionality • Execute a stored procedure or SQL by query using the SQLEXEC clause of the TABLE or MAP parameter.31 Copyright © 2011. • (Optional) Extract output parameters from the stored procedure or SQL query as input to a FILTER or COLMAP clause by using the @GETVAL function. run a SQL query. or issue a database command. . . Oracle and/or its affiliates.32 SQL Server Sybase Ingres The stored procedure interface supports the following data types for input and output parameters: Oracle DB2 SQL Server/Sybase/Teradata CHAR VARCHAR2 DATE All numeric types LOBS up to 200 bytes CHAR VARCHAR2 DATE All numeric types BLOB data types CHAR VARCHAR DATETIME All numeric types Copyright © 2011.SQLEXEC: DBMS and Data Type Support • SQLEXEC is available for the following databases: Oracle Teradata DB2 • 11 . All rights reserved. SQLEXEC: Using with a LOOKUP Stored Procedure The following example uses SQLEXEC to run a stored procedure named LOOKUP that performs a query to return a description based on a code: CREATE OR REPLACE PROCEDURE LOOKUP (CODE_PARAM IN VARCHAR2. .34 Copyright © 2011. Oracle and/or its affiliates. All rights reserved. DESC_PARAM OUT VARCHAR2) BEGIN SELECT DESC_COL INTO DESC_PARAM FROM LOOKUP_TABLE WHERE CODE_COL = CODE_PARAM END. 11 . ACCOUNT. newacct_id = account_id.NEWACCT.SQLEXEC: Using with a Lookup Stored Procedure Contents of the MAP statement: • Maps data from the ACCOUNT table to the NEWACCT table • When processing any rows from ACCOUNT. . PARAMS (code_param = account_code)). Extract performs the LOOKUP stored procedure prior to executing the column map • Maps values returned in desc_param to the newacct_val column using the @GETVAL function: MAP HR.35 Copyright © 2011. TARGET HR.desc_param)). & SQLEXEC (SPNAME lookup. newacct_val = @GETVAL(lookup. Oracle and/or its affiliates. & COLMAP (USEDEFAULTS. 11 . All rights reserved. @GETVAL is used to retrieve the return parameter: MAP HR.SQLEXEC: Using with a SQL Query For an Oracle database.36 Copyright © 2011. 11 . & SQLEXEC (id lookup. All rights reserved. the following example performs a SQL query directly to return the description. TARGET HR. Oracle and/or its affiliates. & PARAMS (code_param = account_code)).ACCOUNT.desc_param)). newacct_val = @GETVAL(lookup. & query “select desc_param from lookup_table where code_col = :code_param”. & COLMAP (USEDEFAULTS. newacct_id = account_id. .NEWACCT. ] {PARAMS <param spec> | NOPARAMS}) • To execute a query within a TABLE or MAP statement: SQLEXEC (ID <logical name>. All rights reserved. [ID <logical name. {PARAMS <param spec> | NOPARAMS}) 11 .37 Copyright © 2011. SQLEXEC can pass and accept parameters. QUERY “ <sql query> ”. . To execute a procedure within a TABLE or MAP statement: SQLEXEC (SPNAME <sp name>. but not database commands. Oracle and/or its affiliates.SQLEXEC: Using Within a TABLE or MAP Statement • • • When used within a TABLE or MAP statement. It can be used for procedures and queries. it can execute a stored procedure. SQLEXEC does not need to be tied to a specific table and can be used to perform general SQL operations.SQLEXEC: Using as a Stand-Alone Statement • When SQLEXEC is used as a stand-alone parameter statement in the Extract or Replicat parameter file. . • Parameter Syntax Purpose SQLEXEC “exec <procedure name>()” Executes a stored procedure SQLEXEC “<sql query>” Executes a query SQLEXEC “<database command>” Executes a database command 11 .38 Copyright © 2011. or database command. For these situations. Oracle and/or its affiliates. query. All rights reserved. True b. False 11 . .Quiz SQLEXEC can communicate only via a SQL query. Oracle and/or its affiliates.39 Copyright © 2011. All rights reserved. a. Target-definitions file c. Source-definitions file b. Oracle and/or its affiliates. because transformations never occur on the target 11 . Neither.40 Copyright © 2011.) a. All rights reserved. Both a source-definitions file and a target-definitions file d.Quiz What is required if you are configuring column mapping or transformation on the target system ? (Select one. . Oracle and/or its affiliates.41 Copyright © 2011. you should have learned how to: • Select and filter data for replication • Map columns between different schemas • Use built-in functions • Use SQLEXEC to interact directly with a database 11 .Summary In this lesson. . All rights reserved. . All rights reserved.Additional Transformation Topics Copyright © 2011. Oracle and/or its affiliates. 2 Copyright © 2011. Oracle and/or its affiliates.Objectives After completing this lesson. . All rights reserved. you should be able to: • Create and invoke macros • Set and retrieve user tokens • Run user exits in GoldenGate processes • Replicate Oracle sequences 12 . Overview of Macros • • Macros enable easier and more efficient building of parameters (not related to your parameter files. – Invoke other macros. Oracle and/or its affiliates. – Create macro libraries and share across parameter files. but commands that can be used across all parameter files). The following are some ways you can use macros: – Write once and use many times (such as the userid/password line). – Eliminate the need for redundant column specifications. All rights reserved.3 Copyright © 2011. – Consolidate multiple statements. 12 . . • Macro statements include the following: – Macro name – Optional parameter list – Macro body MACRO #<macro name> PARAMS (#<param1>. …) BEGIN <macro body> END. Oracle and/or its affiliates. 12 .Creating Macros • • Macros can be defined in any parameter file or library. .4 Copyright © 2011. All rights reserved. #<param2>. Macro and parameter names must begin with a macro character (the default is the pound (#) character). “DD”. to invoke a macro. Oracle and/or its affiliates.ACCOUNT.5 Copyright © 2011.Invoking a Macro First. Ship_Date = #make_date(Ship_YR. #month.Order_DAY). #year. All rights reserved. place an invocation statement in the parameter file wherever you want the macro to occur: MAP SALES. Then. “MM”.Order_MO. 20.Ship_MO. . define the macro: EXTRACT EXSALES MACRO #make_date PARAMS (#year. TARGET REPORT. 19). @IF(#year < 50. #month. “CC”. Order_Date = #make_date(Order_YR. 12 . #day) BEGIN @DATE(“YYYY-MM-DD”. COLMAP ( TARGETCOL1 = SOURCECOL1.ACCT. “YY”. #day) END.Ship_DAY) ). . All rights reserved. define the macro: MACRO #option_defaults BEGIN GETINSERTS GETUPDATES GETDELETES INSERTDELETES END. 12 .6 Copyright © 2011. Oracle and/or its affiliates.Reusing Parameter Sets First.TARGTAB. TARGET SALES. #option_defaults () MAP SALES. TARGET SALES.TARGTAB2.SRCTAB. Then invoke the macro: #option_defaults () IGNOREUPDATES MAP SALES.SRCTAB2. Reusing Parameter Sets The macro expands to the following: GETINSERTS GETUPDATES GETDELETES INSERTDELETES IGNOREUPDATES MAP SALES. . TARGET SALES. TARGET SALES. GETINSERTS GETUPDATES GETDELETES INSERTDELETES MAP SALES.TARGTAB. 12 .SRCTAB.7 Copyright © 2011. All rights reserved.SRCTAB2. Oracle and/or its affiliates.TARGTAB2. PASSWORD ggs123. Oracle and/or its affiliates. you can define a macro once and then use it within many parameter files. TARGET fin.Creating Macro Libraries • • • You can create a macro library that contains one or more macros.mac REPLICAT rep ASSUMETARGETDEFS USERID ggs.acct_tab.account. In this Replicat parameter file. All rights reserved. . • 12 . By using a macro library. use the INCLUDE parameter at the beginning of the parameter file: INCLUDE /ggs/dirprm/mdatelib.8 Macros listing can be turned on or off by using the LIST and NOLIST parameters. MAP fin. Copyright © 2011. CMDTRACE ON MAP test. TARGET test.Tracing Macro Expansion • • • You can trace macro expansion with the CMDTRACE parameter. OFF (default value). All rights reserved. . CMDTRACE OFF 12 . For this example.. tracing is enabled before #testmac is invoked. Oracle and/or its affiliates. COLMAP (#testmac).table2. COL3 = COL4. and DETAIL. END.. and then disabled after the macro’s execution: REPLICAT REP MACRO #testmac BEGIN COL1 = COL2.table1. . Options for the CMDTRACE parameter include ON.9 Copyright © 2011. TARGET SALES.PRODUCT.Overview of User Tokens • • User tokens are environment values that are captured and stored in the trail record for replication to target columns or other purposes. TRANSTIME = @TOKEN(“TKN2") ). • Use token values to populate target columns through a MAP COLMAP clause and @TOKEN functions.PRODUCT. . TKN2 = @GETENV(“GGHEADER".“COMMITTIMESTAMP") ). COLMAP (USEDEFAULTS. All rights reserved. For example: MAP SALES. OSUSER = @TOKEN(“TKN1"). TOKENS (TKN1 = @GETENV(“GGENVIRONMENT". Set token values through a TABLE TOKENS clause and @GETENV functions.“OSUSERNAME").10 Copyright © 2011. Oracle and/or its affiliates.PRODUCT_HISTORY. For example (set for each transaction): TABLE SALES. 12 . Oracle and/or its affiliates. . All rights reserved.11 Copyright © 2011.Environmental Values Available to @GETENV Source Option Returns values for/from General “LAG” “LASTERR” “JULIANTIMESTAMP” “RECSOUTPUT” Lag (in unit specified) Last failed operation Julian timestamp Number of records written to trail GoldenGate “GGENVIRONMENT” “GGFILEHEADER” “GGHEADER” “RECORD” GoldenGate environment Trail file header Trail record header Trail record location Database “DBENVIRONMENT” “TRANSACTION” Database environment Source transaction Operating System “OSVARIABLE” OS environmental variable 12 . “RECORDLENGTH"). TKN-DB-VER = @GETENV (“DBENVIRONMENT". TKN-COMMIT-TS = @GETENV (“GGHEADER". “COMMITTIMESTAMP"). “OSUSERNAME"). TKN-DOMAIN = @GETENV (“GGENVIRONMENT". All rights reserved. TKN-LENGTH = @GETENV (“GGHEADER". “BEFOREAFTERINDICATOR).PRODUCT. TKN-OP-TYPE = @GETENV (“GGHEADER". “DOMAINNAME"). TOKENS ( (Token name) TKN-OSUSER = @GETENV (“GGENVIRONMENT". “DBVERSION") ). . “TABLENAME"). Oracle and/or its affiliates. 12 .12 Copyright © 2011. “OPTYPE").User Tokens Display Values are stored in the GoldenGate record header using a TOKENS clause and @GETENV functions: EXTRACT EXTDEMO TABLE SALES. TKN-BA-IND = @GETENV (“GGHEADER". TKN-TABLE = @GETENV (“GGHEADER". 13 Copyright © 2011.ORDER.Using User Tokens Tokens are retrieved through a MAP COLMAP clause and @TOKEN functions: MAP SALES. 12 . TKN_ROWID = @TOKEN ("TKN-ROWID")).CUSTOMER_HISTORY. MAP SALES. TARGET REPORT. TRAN_TIME = @TOKEN ("TKN-COMMIT-TS"). TARGET REPORT. TKN_NUMRECS = @TOKEN ("TKN-NUMRECS").ORDER_HISTORY. COLMAP (USEDEFAULTS.CUSTOMER. COLMAP (USEDEFAULTS. All rights reserved. Oracle and/or its affiliates. . BEFORE_AFTER_IND = @TOKEN (“TKN-BA-IND”). OP_TYPE = @TOKEN (“TKN-OP-TYPE”). .000000 3604496 4058 SOURCE.Viewing User Exits in Logdump logdump 2> usertoken on logdump 3> usertoken detail logdump 4> next User tokens: TKN-HOST TKN-GROUP TKN-BA_IND TKN-COMMIT_TS TKN-POS TKN-RBA TKN-TABLE TKN-OPTYPE TKN-LENGTH TKN-TRAN_IND TKN-LAG_SEC TKN-LAG_MIN TKN-LAG_MSEC TKN-NUMRECS TKN-DBNAME TKN-DB_USER TKN-DB_VER TKN-INAME TKN-ROWID 12 . Oracle and/or its affiliates.14 : : : : : : : : : : : : : : : : : : : jemhadar EXTORA AFTER 2003-03-24 17:08:59.0 ora901 AAABBAAABAAAB0BAAF Copyright © 2011.0. All rights reserved.1.0.CUSTOMER INSERT 57 BEGIN 1 0 1229 8 ORA901 GGOODRIC 9. Oracle and/or its affiliates. All rights reserved. or SQLEXEC • Can perform an unlimited number of functions 12 . .Overview of User Exits User exits: • Are custom logic written in C or C++ by the customer • Are invoked at different points in Extract or Replicat processing (through CUSEREXIT parameter) • Allow you to extend or customize the functionality of data movement and integration beyond what is supported through mapping. functions.15 Copyright © 2011. Oracle and/or its affiliates. by sending a formatted email message or paging a supervisor based on some field value) • Accumulate totals and gather statistics • Perform conflict detection or custom handling of errors or discards • Determine the net difference in a record before and after an update (conflict resolution technique) 12 .Uses for User Exits User exits: • Perform arithmetic operations or data transformations beyond those provided with GoldenGate built-in functions • Perform additional table lookups or clean up invalid data • Respond to events in custom ways (for example. All rights reserved.16 Copyright © 2011. . .User Exits: High-Level Processing Logic A user exit: • Accepts different events and information from Extract or Replicat • Passes the information to the appropriate paragraph or routine for processing • Returns a response and information to the caller 12 .17 Copyright © 2011. All rights reserved. Oracle and/or its affiliates. . Copyright © 2011.Implementing User Exits • • • On Windows: Create a DLL in C and create a routine to be called from Extract or Replicat. On UNIX: Create a shared object in C and create a routine to be called from Extract or Replicat. Call the ERCALLBACK function from the shared object to retrieve record and application context information. All rights reserved. Oracle and/or its affiliates. include the usrdecs.h file (in the GoldenGate install directory).18 In the source for the DLL/shared object. The routine must accept the following parameters: EXIT_CALL_TYPE EXIT_CALL_RESULT EXIT_PARAMS • • 12 . end transaction. the Extract or Replicat process calls the user exit: at start processing. discard record. process record. EXIT_CALL_RESULT provides a response to the routine: OK. fatal error.User Exit Parameters • • • • 12 . function parameter. Callback routines retrieve record and GoldenGate context information and modify the contents of data records. “more records” indicator. or skip record. or call result. Copyright © 2011.19 EXIT_CALL_TYPE indicates when. stop processing. EXIT_PARAMS supplies information to the routine: calling program path and name. ERCALLBACK implements a callback routine. . ignore. process marker. Oracle and/or its affiliates. All rights reserved. begin transaction. ABEND. during processing. stop. Copyright © 2011. Each directory contains the .20 Sample user exit files are located in <GoldenGate installation directory>/ UserExitExamples.Sample User Exits • • 12 . All rights reserved. Oracle and/or its affiliates.c file as well as makefiles and a readme.txt file. . 21 Copyright © 2011. Oracle and/or its affiliates. • Example parameter file syntax on UNIX systems: CUSEREXIT eruserexit.so MyUserExit • Example parameter file syntax on Windows systems: CUSEREXIT eruserexit.dll MyUserExit 12 .Calling User Exits • You can call a user exit from Extract or Replicat by using the CUSEREXIT parameter. All rights reserved. . employees_seq. All rights reserved.employees_seq. or generate a primary key value. For example: MAP hr. 12 . Oracle and/or its affiliates. For example: SEQUENCE hr. GoldenGate supports the replication of Oracle sequence values. such as to maintain a count.22 Copyright © 2011. • Use the Replicat MAP parameter to apply sequence values to the target. determine order of entry. Use the Extract SEQUENCE parameter to extract sequence values from the transaction log for propagation to a trail and delivery to another databases.employees_seq.Overview of Oracle Sequences • • • Oracle sequence numbers are used for a variety of purposes. . TARGET payroll. All rights reserved.23 The default Replicat CHECKSEQUENCEVALUE parameter ensures that target sequence values are: – Higher than the source values (if the increment interval is positive) or – Lower than the source values (if the increment interval is negative) You can change this default behavior if there are no gaps in the sequence updates (for example. . Oracle and/or its affiliates. Copyright © 2011.Overview of Oracle Sequences • • 12 . from a trail corruption or process failure) and you want to improve the performance of GoldenGate. Procedures 12 . Oracle and/or its affiliates. a. Macros b.Quiz User _______ are a set of variables.24 Copyright © 2011. All rights reserved. Tokens c. . Exits d. Quiz User exits can be used for tables being processed by a data pump Extract in pass-through mode. . All rights reserved.25 Copyright © 2011. False 12 . Oracle and/or its affiliates. a. True b. Summary In this lesson. you should have learned how to: • Create and invoke macros • Set and retrieve user tokens • Run user exits in GoldenGate processes • Replicate Oracle sequences 12 . All rights reserved. Oracle and/or its affiliates.26 Copyright © 2011. . Oracle and/or its affiliates. 12 .27 Copyright © 2011.Practices The practice for this lesson covers modifying an existing set of macros and user tokens. All rights reserved. . Configuration Options Copyright © 2011. Oracle and/or its affiliates. All rights reserved. . and password encryption Event records to automatically trigger actions Copyright © 2011.Objectives After completing this lesson. All rights reserved. . you should be able to use the following: • BATCHSQL to speed delivery • • • 13 . trail.2 Data compression across the network Message. Oracle and/or its affiliates. Copyright © 2011. Referential integrity is preserved. and executed many times with different variables.Overview of BATCHSQL • • • • 13 .3 The BATCHSQL parameter enhances the performance of Replicat by organizing SQL statements into arrays. D). as opposed to individual operations. . Each statement type is prepared once. and applying them at an accelerated rate. Oracle and/or its affiliates. Operations containing the same table. operation type (I. cached. All rights reserved. and column list are grouped into a batch. U. BATCHSQL can be used with change capture or initial loads. Oracle and/or its affiliates.4 Copyright © 2011. . All rights reserved.BATCHSQL Syntax BATCHSQL is implemented with the Replicat BATCHSQL parameter: BATCHSQL [BATCHERRORMODE | NOBATCHERRORMODE] [BATCHESPERQUEUE <n>] [BATCHTRANSOPS <n>] [BYTESPERQUEUE <n>] [OPSPERBATCH <n>] [OPSPERQUEUE <n>] [TRACE] 13 . 000 bytes of data per row change. BATCHSQL benefits diminish. At 100 bytes of data per row change.BATCHSQL Results • • • • 13 .6 Smaller row changes will show a higher gain in performance than larger row changes. At around 5. Oracle and/or its affiliates. Copyright © 2011. All rights reserved. . Actual performance benefits will vary depending on the mix of operations. BATCHSQL has been known to improve Replicat’s performance by as much as 400 to 500 percent. Copyright © 2011. More information can be found at www. All rights reserved.zlib. GoldenGate uses the zlib compression. . Oracle and/or its affiliates.7 GoldenGate provides optional data compression when sending data over TCP/IP. Automatic decompression is performed by the Server Collector on a remote system. The compression threshold allows the user to set the minimum block size to compress.Compression Options • • • • 13 .net. Example of Compression • Compression is specified on the Extract RMTHOST parameter: RMTHOST <host> | <ip address>. COMPRESSTHRESHOLD <byte size> ] – • COMPRESS specifies that outgoing blocks of captured changes are compressed. COMPRESS ] [. MGRPORT <port> [.000 bytes). All rights reserved. Example: RMTHOST newyork. Oracle and/or its affiliates. COMPRESS. – COMPRESSTHRESHOLD sets the minimum byte size for which compression will occur (the default is 1. COMPRESSTHRESHOLD 750 13 . .8 Copyright © 2011. MGRPORT 7809. Oracle and/or its affiliates. 13 . a symmetric 64-bit block cipher from CounterPane Internet Security – The data is automatically decrypted by Server Collector before saving the data to the trail. . All rights reserved.9 Copyright © 2011. – Encrypts only the record data in a trail or extract file – The data is decrypted by a downstream data pump or Replicat.Encryption Overview Encrypting and decrypting data can be done by using the following methods: • Message encryption – Encrypts the messages sent over TCP/IP – Uses Blowfish. • Database password encryption – An encrypted password can be generated using a default key or user-defined key. • Trail or extract file encryption – GoldenGate uses 256-key byte substitution. All rights reserved.Encryption Overview Extract Network (TCP/IP) Server Collector Replicat Trail Message Encryption (Blowfish) Trail or Extract File Encryption (GoldenGate) Parameters Parameters Database Password Encryption 13 .10 Copyright © 2011. . Oracle and/or its affiliates. Message Encryption 1. 13 . Copy the ENCKEYS file to the source and target GoldenGate install directory.11 Copyright © 2011. . Run the GoldenGate KEYGEN utility to generate random hex keys: C:\GGS> RUN KEYGEN <key length> <number of keys> 2. Oracle and/or its affiliates. All rights reserved. no file extension) in the GoldenGate install directory: ##Key name Key value superkey 0x420E61BE7002D63560929CCA17A4E1FB secretkey 0x027742185BBF232D7C664A5E1A76B040 3. Enter key names and values in an ASCII text file named ENCKEYS (uppercase. Oracle and/or its affiliates.Message Encryption 4. use the RMTHOST ENCRYPT and KEYNAME parameters: RMTHOST West. All rights reserved. . MGRPORT 7809. ENCRYPT BLOWFISH. In the Extract parameter files. Configure a static Server Collector and start it manually with the ENCRYPT and KEYNAME parameters: server -p <port> -ENCRYPT BLOWFISH -KEYNAME <keyname> 13 . KEYNAME superkey 5.12 Copyright © 2011. KEYNAME <keyname> Startup command: server -p <port> -ENCRYPT BLOWFISH -KEYNAME <keyname> Message Encryption (Blowfish) 13 . ENCRYPT BLOWFISH. All rights reserved.13 Copyright © 2011. Oracle and/or its affiliates..Options: Message Encryption Keygen ENCKEYS Extract ENCKEYS Network (TCP/IP) Server Collector Replicat Trail Extract Parameters: RMTHOST… MGRPORT. .. Downstream data pumps can also decrypt the trail for transformation and pass it on either encrypted or decrypted. GoldenGate uses 256-key byte substitution. All rights reserved.Trail or Extract File Encryption • • • • Only the data records are encrypted in the trail. . Extract Network (TCP/IP) Server Collector Replicat Trail Extract Parameters: ENCRYPTTRAIL <table statements> Replicat Parameters: DECRYPTTRAIL <map statements> Trail or Extract File Encryption 13 . Oracle and/or its affiliates. Set ENCRYPTTRAIL before tables that you want to be encrypted and NOENCRYPTTRAIL before other tables.14 Copyright © 2011. . Paste the encrypted password in the Extract or Replicat PASSWORD parameter. Oracle and/or its affiliates.. ENCRYPTKEY DEFAULT 13 . Encrypted password: AACAAAAAAAAAAAOARAQIDGEEXAFAQJ 2. using default key. Generate an encrypted password with a GoldenGate default key code: GGSCI> ENCRYPT PASSWORD <password> For example: GGSCI> ENCRYPT PASSWORD goldenpassword No key specified. USERID joe.15 Copyright © 2011. PASSWORD AACAAAAAAAAAAAOARAQIDGEEXAFAQJ. For example: SOURCEDB MySource.Password Encryption: Method 1 1.. All rights reserved. .Password Encryption: Method 2 1. USERID joe. ENCRYPTKEY DRKEY Encrypted password: AACAAAAAAAAAAAIAJFGBNEYGTGSBSHVB 2. Generate an encrypted password with a user-defined key: GGSCI> ENCRYPT PASSWORD <password>. ENCRYPTKEY drkey 13 . For example : ##Key name Key value drkey 0x11DF0E2C2BC20EB335CB98F05471A737 3. Paste the encrypted password in the Extract or Replicat PASSWORD parameter. Oracle and/or its affiliates.16 Copyright © 2011. ENCRYPTKEY <keyname> For example: GGSCI> ENCRYPT PASSWORD MyPass. For example: SOURCEDB MySource. PASSWORD AACAAAAAAAAAAAIAJFGBNEYGTGSBSHVB. Enter the key name and value in the ENCKEYS file. All rights reserved. USERID . PASSWORD <encrypted password> . All rights reserved. USERID . Oracle and/or its affiliates.Summary of Password Encryption ENCKEYS Extract (user-defined key) (user-defined key) Server Collector Network (TCP/IP) Extract Parameters: [ SOURCEDB ] . .17 ENCKEYS Copyright © 2011. ENCRYPTKEY DEFAULT | <keyname> Password Encryption 13 . PASSWORD <encrypted password> . ENCRYPTKEY DEFAULT | <keyname> Replicat Replicat Parameters: [ TARGETDB ] . The event record is one of the following: – A record in a data table that satisfies a filter condition for which you want an action to occur – A record that you write to a dedicated event table when you want an action to occur 13 .18 Copyright © 2011.Event Marker System • • GoldenGate provides an event marker system that enables GoldenGate processes to take a defined action based on an event record in the transaction log or trail. All rights reserved. Oracle and/or its affiliates. . start batch processes. • Run a shell command (for example. • Write a checkpoint before or after writing the record to the trail.Uses for Event Actions Examples of actions that you might take upon detecting an event record: • Start or stop a process. to switch an application. and system event log. • Generate a report file. GoldenGate error log. • Roll over the trail file. 13 .19 Copyright © 2011. Oracle and/or its affiliates. . • Ignore or discard the current record. or start end-of-day reporting). • Log an informational or warning message to the report file. All rights reserved. • Activate tracing. Oracle and/or its affiliates. .Event Actions Flowchart INSERT / UPDATE / DELETE Values(…) in an event table INSERT / UPDATE / DELETE Values(…) in a data table Reports Logs Discards Chkpts Reports Logs EVENT PROCESSING EVENT PROCESSING Extract Transaction Log 13 .20 Discards Chkpts Replicat Network (TCP/IP) Target Trail Copyright © 2011. All rights reserved. Implementing Event Actions 1.21 Copyright © 2011. 3. include the EVENTACTIONS parameter. FILTER (account_no = 100). Oracle and/or its affiliates. This example uses a separate event table to manage events (whenever a record is written to the event table. the trail file is rolled over): TABLE source.event_table. 13 . All rights reserved. EVENTACTIONS (DISCARD.account. Add an EVENTACTION option to a TABLE or MAP statement. LOG). Another example uses data values to trigger events (any record where account_no = 100 is discarded and a log message is written): MAP source. . In the same TABLE or MAP statement where you specified the event record. TARGET target. 2.account. EVENTACTION (ROLLOVER). Event Actions: Heartbeat Example MAP source. EVENTACTIONS (LOG).heartbeat.heartbeat. Info EVENT PROCESSING Log Warning TX1 TX2 Replicat Heartbeat TX4 13 . @DATENOW() > 60 AND @DATEDIFF (“SS”. EVENTACTIONS (LOG WARNING). . FILTER ((@DATEDIFF (“SS”.heartbeat. @DATENOW() > 120). TARGET target. hb_timestamp. MAP source. All rights reserved.heartbeat. hb_timestamp.22 Target Trail Copyright © 2011. @DATENOW() < 120). TARGET target. FILTER (@DATEDIFF (“SS”. Oracle and/or its affiliates. HBTIMESTAMP. Extract Source Trans Log Network (TCP/IP) Replicat Replicat Trail Target Extract Trail 13 . All rights reserved.23 2. Copyright © 2011. Oracle and/or its affiliates. 3. Trans Log .Event Actions: Automated Switchover Example Switchover Application Application 1. Trans Log . Oracle and/or its affiliates. Extract Source Trans Log Replicat 2. Replicat Trail Target Extract Trail 13 . 2a.Event Actions: Automated Synchronization Example Application Application ETL ETL 2a. Network (TCP/IP) 3. All rights reserved.24 Copyright © 2011. 1. Oracle and/or its affiliates. GLOBALS b. SOURCEDEFS d. All rights reserved.Quiz Key names and values can be stored in: a.25 Copyright © 2011. ENCKEYS c. . DEFSFILE 13 . All rights reserved.26 Copyright © 2011. .Quiz Only the ENCRYPTTRAIL extract parameter should be set to encrypt and decrypt the records in the trail file. False 13 . Oracle and/or its affiliates. a. True b. you should have learned how to use the following: • BATCHSQL to speed delivery • • • 13 . trail. All rights reserved. and password encryption Event records to automatically trigger actions Copyright © 2011. Oracle and/or its affiliates.27 Data compression across the network Message. .Summary In this lesson. Practices The practices for this lesson cover the following topics: • Setting up the database and source definitions file • GoldenGate encryption 13 . All rights reserved. Oracle and/or its affiliates.28 Copyright © 2011. . . Oracle and/or its affiliates. All rights reserved.Bidirectional Replication Copyright © 2011. All rights reserved. . you should be able to: • Describe bidirectional configuration • Detect and avoid loops • Handle bidirectional issues such as loop detection and handling conflicts • Handle identity types 14 .Objectives After completing this lesson. Oracle and/or its affiliates.2 Copyright © 2011. All rights reserved. Transaction Log .Bidirectional Flowchart Network (TCP/IP) Extract Source Replicat Target Trail Transaction Log Source Target Network (TCP/IP) Replicat Extract Trail 14 .3 Copyright © 2011. Oracle and/or its affiliates. GoldenGate provides loop detection. Oracle and/or its affiliates. . GoldenGate’s low latency reduces the risk of conflicts. All rights reserved. Copyright © 2011.4 Available for both homogeneous and heterogeneous configuration Distributed processing Both sides are active.Capabilities of a Bidirectional Configuration • • • • • 14 . . Conflict avoidance. All rights reserved. Oracle and/or its affiliates.5 Loop detection – Detect whether GoldenGate or the application performed the operation. Sequence numbers and identity data types Truncate table operations Copyright © 2011.Issues to Consider for a Bidirectional Configuration • • • • 14 . detection. – Determine the business rules on how to handle collisions. and resolution – Detect whether an update occurred on both the source and target before the changes were applied by GoldenGate. this loop continues endlessly. . The row is updated on System A. The update operation is captured and sent to System B. All rights reserved. 3. 2. The update operation is captured and sent to System A. Without loop detection. • 14 .Preventing Data Looping • The following example sequence demonstrates the problem that occurs without loop detection: 1. 5. Oracle and/or its affiliates. Copyright © 2011. The row is updated on System B. 4.6 A row is updated on System A. The insert is captured and sent to System A. . The row is inserted on System B. A row is inserted on System A. 2. The insert is captured and sent to System B. 4. Oracle and/or its affiliates.7 Another example involves the following sequence: 1. All rights reserved. The insert is attempted on System A. but the operation fails with a conflict on the primary key causing synchronization services to HALT. 5. 3. Copyright © 2011.Preventing Data Looping • 14 . 8 Copyright © 2011.Loop Detection Techniques Loop detection technique depends on the source database: • Oracle – Use EXCLUDEUSER or EXCLUDEUSERID (Oracle 10g and later). Oracle and/or its affiliates. All rights reserved. Suppress capture of Replicat transactions with one of the Extract parameters: TRANLOGOPTIONS EXCLUDEUSER <username> TRANLOGOPTIONS EXCLUDEUSERID <Oracle user ID> 14 . – – Execute Replicat with a unique database username or Oracle user ID. . . Oracle and/or its affiliates. – Add a trace table to detect GoldenGate operations with the GGSCI command: ADD TRACETABLE <owner>.<table name> – If not using the default table name.<table name> 14 .Loop Detection Techniques • Use an Oracle trace table. add a parameter in both Extract and Replicat: TRACETABLE <owner>.9 Copyright © 2011. All rights reserved. Copyright © 2011. Conflicts can be avoided at the application level by assuring that records are always updated on only one system. All rights reserved. GoldenGate can capture both the before and after images of updates so that you can compare these before applying updates. – You can write compare logic in a filter or user exit. Oracle and/or its affiliates.10 Conflicts can be minimized by low latency.Conflict Avoidance and Detection • • • 14 . . EXCEPTIONSONLY. EXCEPTION MAP SRCTAB. QUERY “SELECT COUNTER FROM TARGTAB” “WHERE PKCOL = :P1”.Conflict Detection by Filter REPERROR 9999. RAISEERROR 9999). . TARGET TARGTAB. Oracle and/or its affiliates.COUNTER <> CHECK. FILTER (ON UPDATE. ERRTYPE = “Conflict Detected”). All rights reserved. TARGET TARGET_EXCEPT. BEFORE. 14 . INSERTALLRECORDS MAP SRCTAB. SQLEXEC (ID CHECK. COLMAP (USEDEFAULTS. PARAMS (P1 = PKCOL)).11 Copyright © 2011. BEFOREFILTER.COUNTER. ON UPDATE. • Map the data to an exception table for manual resolution. 14 . For example: • Apply the net difference instead of the after value.12 Copyright © 2011. Oracle and/or its affiliates. . All rights reserved. • Accept or ignore the change based on date and time.Conflict Resolution Addressing conflict resolution depends on your business rules. All rights reserved.Conflict Resolution: Example Initial balance: $500 Trans A: Mr. correct balance should be $500 + $75 – $20 = $555! 14 . Oracle and/or its affiliates. Smith withdraws $20 in NY: balance = $480 Trans A replicated to NY: balance = $575 Trans B replicated to LA: balance = $480 Time At end of day. Smith deposits $75 in LA: balance = $575 Trans B: Mrs.13 Copyright © 2011. . Smith withdraws $20 in NY: end balance = $480 Trans A replicated to NY: begin/end balance = $500 / $575 Conflict detected between trans A begin balance (500) and current NY balance (480). so apply net difference: Trans B end balance (480) – Trans B begin balance (500) = –20 resulting in LA end balance of $555. Trans B replicated to LA: begin/end balance = $500 / $480 Conflict detected between trans B begin balance (500) and current LA balance (575). .14 Copyright © 2011. so apply net difference: Trans A end balance (575) – Trans A begin balance (500) = 75 resulting in NY end balance of $555. Smith deposits $75 in LA: end balance = $575 Trans B: Mrs. All rights reserved.Conflict Resolution by Applying Net Differences Initial balance: $500 Trans A: Mr. Oracle and/or its affiliates. Time 14 . assign odd and even values to ensure that the source and target sequence numbers are unique. All rights reserved.15 Oracle GoldenGate does not support the replication of sequence values in a bidirectional configuration. each system must use a starting value and increment based on the number of systems. In a multidirectional configuration. Oracle and/or its affiliates.Oracle Sequence Numbers • • • 14 . Copyright © 2011. . In a bidirectional configuration. Use IGNORETRUNCATES (default) for the other direction. All rights reserved. Ensure that GETTRUNCATES is ON for only one direction.Truncate Table Operations • • • Truncate table operations cannot be detected for loops. .16 Copyright © 2011. • Change database security so truncates can be issued on only one system. Oracle and/or its affiliates. 14 . False 14 . All rights reserved. Oracle and/or its affiliates. . a. True b.17 Copyright © 2011.Quiz Oracle GoldenGate supports bidirectional synchronization only for homogeneous environments. USEREXCLUDE 14 . Oracle and/or its affiliates. .Quiz Which parameters in Oracle can be used instead of a trace table to allow loop detection? (Select all that apply. EXCLUDEUSERID d.) a.18 Copyright © 2011. All rights reserved. EXCLUDEUSER b. IGNORETRUNCATES c. .19 Copyright © 2011. All rights reserved. Oracle and/or its affiliates. you should have learned how to: • Handle loop detection in a bidirectional environment • Understand conflict resolution by applying net differences • Understand the limitations of using Oracle sequence numbers and truncate operations in a bidirectional configuration 14 .Summary In this lesson. . All rights reserved.Practices The practices for this lesson cover the following topics: • Two-way active-active data replication • Conflict detection 14 .20 Copyright © 2011. Oracle and/or its affiliates. DDL Replication Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Objectives After completing this lesson, you should be able to: • Understand the concept of DDL replication • Explain the requirements for Oracle DDL replication • Describe GoldenGate options for DDL replication • Explain how to activate DDL capture 15 - 2 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Overview of DDL Replication • • • 15 - 3 DDL replication is available for all Oracle versions supported by GoldenGate DML synchronization. DDL in relation to DML: – DDL can be active with or without DML synchronization. – The same Extract/Replicat should be processing both DML and DDL to avoid timing issues. DDL operations are recognized differently by Extract and Replicat: – Source: DDL disabled by default; Extract must be configured to enable. – Target: DDL enabled by default to maintain data integrity – Replicat must be configured to ignore or filter DDL. Copyright © 2011, Oracle and/or its affiliates. All rights reserved. DDL Replication Requirements and Restrictions • • • • • • 15 - 4 Identical source and target data def: ASSUMETARGETDEFS Data pumps must be configured in PASSTHRU mode. WILDCARDRESOLVE must remain set to DYNAMIC (default). Data manipulation is not supported. GoldenGate user exits are not supported for DDL activity. Restrictions exist for DDL operations that involve userdefined types and LOB data. Copyright © 2011, Oracle and/or its affiliates. All rights reserved. DDL Replication Requirements and Restrictions • • • 15 - 5 DDL on objects in TABLE or MAP statements inherit the limitations in allowed characters of those parameters. Restrictions exist for DDL operations that involve stored procedures. DDL statements larger than 2 MB require special handling. Copyright © 2011, Oracle and/or its affiliates. All rights reserved. DDL Replication Requirements and Restrictions • • Schema names using Oracle reserved names are ignored. The GETTRUNCATES parameter should not be used with full DDL support. • Table name cannot be longer than 16 characters plus quotation marks for ALTER TABLE RENAME. ALTER TABLE..MOVE TABLESPACE • – Is supported when tablespace is all SMALLFILE or BIGFILE – Stop Extract before issuing a MOVE TABLESPACE. • • 15 - 6 The recycle bin must be turned off. ALTER DATABASE and ALTER SYSTEM are not captured. Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Characteristics for DDL Replication • New names must be specified in TABLE/MAP statements. • Extract sends all DDL to each trail when writing to multiple trails. Oracle GoldenGate supports all DDL operations up to 2 MB in size on the following objects: • Clusters Functions Indexes Packages Procedure 15 - 7 tables tablespaces roles sequences synonyms triggers types views materialized views users Copyright © 2011, Oracle and/or its affiliates. All rights reserved. TRIGGER. SEQUENCE*. All rights reserved. – DDL operations other than those listed above – Examples: CREATE USER. DROP.8 Copyright © 2011.DDL Scopes • • • MAPPED – Is specified in a TABLE or MAP statement – Operations: CREATE. Oracle and/or its affiliates. RENAME. MATERIALIZED VIEW* UNMAPPED – Does not have a TABLE or MAP statement OTHER – TABLE or MAP statements do not apply. ALTER TABLESPACE 15 . INDEX. ALTER. REVOKE* – Objects: TABLE*. . CREATE ROLE. GRANT*. All rights reserved.Using Unqualified Object Names • If the unqualified object is of MAPPED scope.Sets the schema that is specified in the TARGET clause of the MAP statement (if the actual schema of the object is the same as the source session schema): MAP accounts. TARGET accounts2.9 Sets the schema to the source session schema (if the actual schema of the object is different than the source session schema) If the unqualified object is of UNMAPPED or OTHER scope. Copyright © 2011. - • 15 . Replicat does the following: . .*. Oracle and/or its affiliates.*. Replicat sets the schema to the source session schema. . valid for Extract and Replicat.] 15 . MAPPED | UNMAPPED | OTHER | ALL] [.10 Copyright © 2011. Oracle and/or its affiliates. INSTRCOMMENTS ‘<comment_string>’] ] [. INSTR ‘<string>’] [. OBJNAME “<name>”] [. enables DDL support and filters the operations: DDL [ {INCLUDE | EXCLUDE} [.DDL Parameter The DDL parameter.. OBJTYPE ‘<type>’] [. All rights reserved.. OPTYPE <type>] [. the string ‘cust’ is replaced with the string ‘customers’ for tables owned by “fin:”. Oracle and/or its affiliates. DDLSUBST ‘cust’ WITH ‘customers’ INCLUDE ALL OBJTYPE ‘table’ OBJNAME “fin.*” 15 . In the following example. Multiple statements can be used. .12 Copyright © 2011. All rights reserved.DDL String Substitution • • • DDLSUBST parameter substitutes strings in a DDL operation. Oracle and/or its affiliates. ABEND. All rights reserved. or DISCARD 15 .Error Handling • • The DDLERROR parameter contains specific error handling rules to handle a full range of anticipated errors.13 Copyright © 2011. RESTARTCOLLISIONS | NORESTARTCOLLISIONS] Where <response> can be IGNORE. Extract syntax: DDLERROR [RESTARTSKIP <num skips>] [SKIPTRIGGERERROR <num errors>] • Replicat syntax: DDLERROR {<error> | DEFAULT} {<response>} [RETRYOP MAXRETRIES <n> [RETRYDELAY <delay>]] {INCLUDE <clause> | EXCLUDE <clause>} [. . IGNOREMISSINGTABLES | ABENDONMISSINGTABLES] [. DDLOPTIONS for Oracle DDLOPTIONS [.14 Copyright © 2011. PASSWORD ENCRYPTKEY [DEFAULT | ENCRYPTKEY <keyname>] [. UPDATEMETADATA] [. MAPSESSIONSCHEMA] <source_schema> TARGET <target_schema> [. USEOWNERFORSESSION] 15 . REMOVECOMMENTS {BEFORE | AFTER}] [. DEFAULTUSERPASSWORDPASSWORD <password> [ENCRYPTKEY DEFAULT | ENCRYPTKEY <keyname>]] [. REPLICATEPASSWORD | NOREPLICATEPASSWORD] [. . NOCROSSRENAME] [. GETREPLICATES | IGNOREREPLICATES] [. Oracle and/or its affiliates. MAPDERIVED | NOMAPDERIVED] [. All rights reserved. REPORT | NOREPORT] [. GETAPPLOPS | IGNOREAPPLOPS] [. ADDTRANDATA [ABEND | RETRYOP <RETRYDELAY <seconds> MAXRETRIES <retries>] [. *. TABLE SRC1.*.*.*.16 Extract DDL INCLUDE OBJNAME “SRC. TARGET DST1. TARGET DST.*. DDL INCLUDE OBJNAME “DST. Oracle and/or its affiliates. . MAP SRC1.Mapping Schemas MAPSESSIONSCHEMA allows the mapping of a source session schema to another schema on the target: 15 . All rights reserved. Replicat DDLOPTIONS MAPSESSIONSCHEMA SRC TARGET DST DDLOPTIONS MAPSESSIONSCHEMA SRC1 TARGET DST1 MAP SRC.*.*” Copyright © 2011.*” & INCLUDE OBJNAME “DST1.*” TABLE SRC.*” & INCLUDE OBJNAME “SRC1. All rights reserved.Supporting DDL in an Active-Active Bidirectional Configuration • • The UPDATEMETADATA parameter notifies Replicat on the system where DDL originated that this DDL was propagated to the other system. . Oracle and/or its affiliates. This functionality keeps Replicat metadata cache synchronized with the current metadata of the local database: DDLOPTIONS UPDATEMETADATA • 15 .17 Valid for Replicat (Oracle only) Copyright © 2011. Oracle and/or its affiliates. All rights reserved.Activating Oracle DDL Capture • DDL marker table: GGS_MARKER – Stores DDL information • DDL history table: GGS_DDL_HIST – Stores object metadata history • DDL trigger: GGS_DDL_TRIGGER_BEFORE – Activates when there is a DDL operation – Writes to the marker and history tables • User role: GGS_GGSUSER_ROLE – Establishes the role needed for DDL replication – Should be the user that executes Extract 15 . .18 Copyright © 2011. Oracle GoldenGate supports the synchronization of only DDL changes b. Oracle GoldenGate supports the synchronization of only DML changes c. . All rights reserved. Oracle GoldenGate supports the synchronization of both DDL and DML changes 15 .Quiz Which of the following statements are true? (Select all that apply. Oracle and/or its affiliates.) a.20 Copyright © 2011. All rights reserved. . Oracle and/or its affiliates. b.Quiz What parameter in DDLOPTIONS supports DDL replication in an active-active bidirectional configuration? a.21 MAPPSESSIONSCHEMA IGNOREAPPLOPS UPDATEMETADATA NOCROSSRENAME Copyright © 2011. d. c. 15 . . you should have learned how to: • Understand how DDL operations are categorized into scopes • Use the DDL and DDLOPTIONS parameters • 15 . All rights reserved.Summary In this lesson. Oracle and/or its affiliates.22 Run the necessary scripts to set up the database for DDL capture Copyright © 2011. 15 . All rights reserved. Oracle and/or its affiliates. .Practices The practice for this lesson covers DDL replication.23 Copyright © 2011. Tuning and Troubleshooting Copyright © 2011. All rights reserved. Oracle and/or its affiliates. . lags. Oracle and/or its affiliates. All rights reserved. and error messages 16 . you should be able to: • Use command permissions • Manage trail files • Automate the process setup • Handle TCP/IP errors • Use reporting and statistics • Monitor processes.Objectives After completing this lesson.2 Copyright © 2011. . you create a CMDSEC file in the Oracle GoldenGate directory.3 Command security can be established for Oracle GoldenGate to control which users have access to which GoldenGate functions.Using Command Security • • • 16 . . Security levels are defined by the operating system’s user groups. All rights reserved. Oracle and/or its affiliates. Copyright © 2011. To implement security for GoldenGate commands. All rights reserved. STATUS * dpt1 * YES Except for the preceding rule. START REPLICAT * * NO Except for the preceding rule.4 Copyright © 2011. START REPLICAT is denied to all users. Oracle and/or its affiliates. * * root root YES Grants the root user any command 16 . all users in dpt1 are granted all STATUS commands. .Sample CMDSEC Statements File Contents Explanation #GG command security Comment line STATUS REPLICAT * Smith NO STATUS REPLICAT is denied to user Smith. * EXTRACT 200 * NO All EXTRACT commands are denied to all groups with ID of 200. if not managed properly. is preferred because it allows the trail files to be managed in a more centralized fashion. Oracle and/or its affiliates. as opposed to the Extract or Replicat version of PURGEOLDEXTRACTS. All rights reserved.Managing Trail Files • • • 16 . . Use the PURGEOLDEXTRACTS parameter in the Manager parameter file to purge trail files when Oracle GoldenGate has finished processing them. can consume a significant amount of disk space. Copyright © 2011.5 Trail files. The Manager parameter. Copyright © 2011. located in the Oracle GoldenGate directory. All rights reserved. . Oracle and/or its affiliates.Handling TCP/IP Errors • A number of issues can pertain specifically to the TCP/IP connection. can help troubleshoot TCP/IP errors.6 The TCPERRS file. including: – Bottlenecks – Connection refused error – Not enough bandwidth • 16 . 7 Copyright © 2011. Oracle and/or its affiliates. .TCPERRS File # TCP/IP error handling parameters # Default error response is abend # # Error Response Delay(csecs) Max Retries ECONNABORTED RETRY 1000 10 ECONNREFUSED RETRY 1000 12 ECONNRESET RETRY 500 10 ENETDOWN RETRY 3000 50 ENETRESET RETRY 1000 10 ENOBUFS RETRY 100 60 ENOTCONN RETRY 100 10 EPIPE RETRY 500 10 ESHUTDOWN RETRY 1000 10 ETIMEDOUT RETRY 1000 10 NODYNPORTS RETRY 100 10 16 . All rights reserved. Troubleshooting Oracle GoldenGate Typically. Oracle and/or its affiliates. All rights reserved. one of the challenges of solving problems in a replication environment is deciding which component is at the root of the problem: • The system or network? • The database or applications? • The Oracle GoldenGate installation? • A specific Oracle GoldenGate process? • The way that Oracle GoldenGate is configured? • SQL or procedures? 16 .8 Copyright © 2011. . 9 Copyright © 2011.log file (known as the error log) – Process reports – The discard file – The Event Viewer on Windows systems or the syslog on UNIX systems to view errors at the operating-system level 16 . Oracle and/or its affiliates. .Monitoring Oracle GoldenGate Processing • • You can monitor Oracle GoldenGate processing to view process status. All rights reserved. statistics. These processes can be view through: – GGSCI information commands – The ggserr. and events. stopped.Obtaining Process Information Through GGSCI Command What It Shows INFO {EXTRACT | REPLICAT} <group> [DETAIL] Run status.10 Copyright © 2011. Oracle and/or its affiliates. and environmental information INFO MANAGER Run status and port number INFO ALL INFO output for all Oracle GoldenGate processes on the system STATS {EXTRACT | REPLICAT} <group> Statistics for operations processed STATUS {EXTRACT | REPLICAT} <group> Run status (starting. . approximate lag. running. All rights reserved. checkpoints. abended) STATUS MANAGER Run status LAG {EXTRACT | REPLICAT} <group> Latency between last record processed and timestamp in the data source 16 . Obtaining Process Information Through GGSCI Command What It Shows INFO {EXTTRAIL | RMTTRAIL} <path name> Name of associated process, position of last data processed, maximum file size SEND MANAGER Run status, information about child processes, port information, trail purge settings SEND {EXTRACT | REPLICAT} Depending on the process, returns information about memory pool, lag, TCP statistics, longrunning transactions, process status, recovery progress, and more. VIEW REPORT <group> Contents of the process report VIEW GGSEVT Contents of the Oracle GoldenGate error log <command> ER <wildcard> Information dependent on the <command> type: INFO LAG SEND STATS STATUS 16 - 11 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Using the Error Log 2006-01-08 11:20:56 GGS INFO 301 GoldenGate Manager for Oracle, mgr.prm: Command received from GUI (START GGSCI ). 2006-01-08 11:20:56 GGS INFO 302 GoldenGate Manager for Oracle, mgr.prm: Manager started GGSCI process on port 7840. 2006-01-08 11:21:31 GGS INFO 301 GoldenGate Manager for Oracle, mgr.prm: Command received from GUI (START GGSCI ). 2006-01-08 11:21:31 GGS INFO 302 GoldenGate Manager for Oracle, mgr.prm: Manager started GGSCI process on port 7841. 2006-01-08 11:24:15 GGS INFO 301 GoldenGate Manager for Oracle, mgr.prm: Command received from GUI (START GGSCI ). 2006-01-08 11:24:15 GGS INFO 302 GoldenGate Manager for Oracle, mgr.prm: Manager started GGSCI process on port 7842. 2006-01-08 11:24:16 GGS INFO 399 GoldenGate Command Interpreter for Oracle: GGSCI command (ggs): add extract extcust tranlog, begin now. 2006-01-08 11:30:19 GGS INFO 399 GoldenGate Command Interpreter for Oracle: GGSCI command (ggs): add rmttrail /home/ggs, extract ggs 16 - 12 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Process Reports Process reports (depending on the process) allow you to view the following: • Parameters in use • Table and column mapping • Database information • Run-time messages and errors • Run-time statistics for the number of operations processed 16 - 13 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Sample Extract Process Report ******************************************************************** ** Running with the following parameters ** ******************************************************************** sourceisfile userid ggs, password ******** rmthost sys1, mgrport 8040 rmtfile /home/ggsora/dirdat/tcustord.dat, purge table tcustord; Processing table TCUSTORD ******************************************************************** ** Run Time Statistics ** ******************************************************************** Report at 2006-08-13 11:07:36 (activity since 2006-08-13 11:07:31) Output to /home/ggsora/dirdat/tcustord.dat: From Table TCUSTORD: # inserts: 2 # updates:0 # deletes:0 # discards:0 16 - 14 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Discard Files • • Discard files are used to capture information about Oracle GoldenGate operations that have failed. The Discard file reports information such as: – The database error message – The sequence number of the data source or trail file – The relative byte address of the record in the data source or trail file – The details of the discarded operation, such as column values of a DML statement or the text of a DDL statement • • 16 - 15 A Discard file can be used for both Extract and Replicat. To use a Discard file, include the DISCARDFILE parameter in the Extract or Replicat parameter file. Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Sample Discard File ORA-20017: asta0009 6144935 ORA-06512: at "LON.STARTASTA0009_INSERT", line 31 ORA-04088: error during execution of trigger 'LON.STARTASTA0009_INSERT' Operation failed at seqno 45 rba 12483311 Problem replicating PRODTAB.ASTA0009 to ASTA0009 Error occurred with insert record (target format)... * A_TIMESTAMP = 2006-05-15 13:18:32 RELA_PERSON_NR = 3618047 RELA_BEZART = 1 RELA_BEZCODE = 01 RELA_AZ_BAFL = 2819220 RELA_STEMPEL = 0 AKTION = I OK = 1.0000 NOTOK = -1.0000 * 16 - 16 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Using the System Logs • Oracle GoldenGate writes errors that are generated at the level of the operating system: – Event Viewer on Windows – syslog on UNIX and Linux • • Use the SYSLOG parameter to control the types of messages that Oracle GoldenGate sends to the system logs on a Windows or UNIX system. By using the SYSLOG parameter, messages can be filtered to: – Include all Oracle GoldenGate messages – Suppress all Oracle GoldenGate messages – Include information, warning, or error messages, or any combination of those types 16 - 17 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Process report 16 . TCPERRS b.18 Copyright © 2011. GGSCI d. All rights reserved.) a. CMDSEC c. Oracle and/or its affiliates. .Quiz What tool would you use to determine whether all of the GoldenGate processes are up and running? (Select all that apply. All rights reserved.Quiz What files are used to capture information about Oracle GoldenGate operations that have failed? a. Discard files b. TCPERRS files 16 . Oracle and/or its affiliates.19 Copyright © 2011. Purge trail files c. . Summary In this lesson. Oracle and/or its affiliates. and the Discard file Copyright © 2011. All rights reserved. you should have learned how to: • Purge trail files • Handle TCP/IP errors with the TCPERRS file • 16 . process reports. .20 Monitor Oracle GoldenGate processes by using GGSCI.
Copyright © 2024 DOKUMEN.SITE Inc.