How to Build High Performance Flowcharts Faster FINAL

EMM Regional User GroupBreakout Session: How to Build High-Performing IBM (Unica) Campaign Flowcharts Faster © 2012 IBM Corporation Agenda
Flowchart Performance Tuning Approaches
Unica Campaign and Databases
Campaign Configuration Parameters
Leveraging in-database processing
Custom Macros 2 © 2012 IBM Corporation Target Audience
Organization Roles – Database marketer/flowchart developer – Application owner – Technical application lead – Application administrator
Responsibilities – Application Configuration – Advanced flowchart design and performance tuning – Creating reusable application objects (catalogs, custom macros, etc) – Support and training new users 3 © 2012 IBM Corporation Flowchart Performance Tuning Approaches © 2012 IBM Corporation Minus operation – Data sorts
Optimize Calculation Timing – Calculate later on smaller audiences – Calculate earlier as summary table or strategic segments
Partition or simplify the problem – Break complex logic into simpler components – Process one chunk at a time or in parallel
Reduce the number of processing steps – Combine multiple queries accessing the same data into one – Replace multiple levels of segmentation with derived fields 5 © 2012 IBM Corporation .Flowchart Performance Tuning Approaches in General
Optimize workload distribution between Campaign and the Database – Table joins – Merge process vs. Join. database Union. I/O – Virtual memory – Connections and query threads – Using DB bulk loaders – Using DB statistics collection commands
Identify techniques that work well in your environment and package them for reuse – Product configurations – Flowchart templates – Custom macros
Identify techniques that do not perform and develop alternatives – Frequent audience switches could be avoided with base-dimension table mappings 6 © 2012 IBM Corporation . Cont’d
Optimize the use of available resources and tools – Memory. CPU.Tuning Approaches. IBM Unica Campaign and Databases © 2012 IBM Corporation . Products. Zips.Terminology In Data Marts
“Dimension” tables contain normalized lists of values (States. etc)
“Fact” tables contain quantitative characteristics of business transactions with foreign key for dimensions
A table is either a fact or a dimension 8 © 2012 IBM Corporation . is associated with one or more key columns
“Base” tables contain audience keys
“Dimension” tables could be joined to base tables or other dimensions
A table could be mapped in multiple times in different roles 9 © 2012 IBM Corporation .Terminology Cont’d In IBM Unica Campaign
“Audience” represents a marketable entity. – Easy to understand and navigate for the user – Fewer table joins for performance
Propagate audience keys as widely as possible across the schema. © 2012 IBM Corporation .
Calendar based summary tables are not as useful for marketing as they are for other applications.Best Practices
Have a decision support schema. 5.3+ this behavior is configurable 11 © 2012 IBM Corporation .The Role of Table Mapping
To provide Unica Campaign with metadata about audience levels and table relationships
To control the structure of SQL generated by Unica Campaign: – Joins between base and dimension tables are executed in the database – Joins between base tables are performed on Unica Campaign server *) – By selecting appropriate table mapping the users can change the structure of SQL
Mapping tables at the right audience level(s) could simplify flowchart logic *) In UC 7. Table Mapping: Example 1 •Normalized Base Table •Dimension of Transactions Customer CustID (PK) Name Address … •De-normalized Base Table •Dimension of Customer Transactions Product ProdID Audience Level: Customer (PK) SKU Brand … TxnID (PK) CustID ProdID TimeID ChannelID Quantity Amount Discount (FK) (FK) (FK) (FK) Channel ChannelID (PK) Description … Dimension Time TimeID (PK) Date … Dimension Dimension 12 © 2012 IBM Corporation . Table Mapping: Example 2 Normalized Base Table Household HH ID (PK) Address Ciy State ZIP … Audience Level: Household Dimension of HH Denormalized Base Table Customer Dimension of Customer CustID (PK) First Name Last name HH ID … Product ProdID (PK) Brand SKU Brand ID … BrandID (PK) Brand Name Description … Dimension of Transaction Transactions TxnID (PK) CustID ProdID TimeID ChannelID Quantity Amount Discount (FK) (FK) (FK) (FK) Channel ChannelID (PK) Description … Time TimeID (PK) Date … Dimension of Transaction Dimension of Transaction Dimension of Product 13 © 2012 IBM Corporation © 2012 IBM Corporation . Flowchart Execution with In-db Optimization Campaign Server Database Universe and Suppression Queries Merge Query Process Insert into Select Queries Process Insert into Select Queries Query for Derived Field Data DF Calc Data Retrieval ID List Segment Process Select Query Temp Table Creation ID List Upload Query for Output Fields 14 © 2012 IBM Corporation Process Insert into Select Query © 2012 IBM Corporation .
What is a good logging level? © 2012 IBM Corporation .Know Your Flowchart Log
The flowchart log is the only way to know how a flowchart is actually performing. 089 (1752) [I] [PROCESS] [Active] Select: N_RECORDS = 89 04/20/2005 17:14:22.907 (1752) [I] [DB QUERY] [Active] Northwind (thread 000004B8): SELECT 04/20/2005 17:14:20.667 (1752) [I] [PROCESS] SESSION_RUN_START 04/20/2005 17:14:20.099 (1752) [I] [PROCESS] [Active] Select PROCESS_RUN_DONE © 2012 IBM Corporation 16 © 2012 IBM Corporation . W.797 (1752) [I] [PROCESS] [Active] Select PROCESS_RUN_START 04/20/2005 17:14:20.069 (1752) [I] [TABLE ACC] [Active] Northwind (thread 000004B8): Data retrieval 04/20/2005 17:14:22. E) Category Process Name Message Body 04/20/2005 17:14:20.957 (1752) [I] [TABLE ACC] [Active] Northwind (thread 000004B8): Query completed. 04/20/2005 17:14:22.Log File Structure Timestamp PID Level (I. City) [Output List] Northwind (thread 00000288): INSERT INTO dbo. 4 records affected. CompanyName.Customers.Customers ON UAC_27_1.CompanyName. [PROCESS] [Active] Select: N_RECORDS = 89 [PROCESS] [Active] Select PROCESS_RUN_DONE [PROCESS] [Universe] Merge PROCESS_RUN_START [sprocrun:757] [CELL ACC] [Universe] CellAccess: Data is ready.Orders ORDER BY CustomerID [TABLE ACC] [Active] Northwind (thread 000004B8): Data retrieval completed.City FROM (UAC_27_1 LEFT OUTER JOIN dbo.CustomerID = dbo.OUTPUT_TEST1(customer_ID.CustomerID) 17 [TABLE ACC] [Output List] Northwind (thread 00000288): Statement successful. dbo. [CELL ACC] [Value Segment] CellAccess: Derived field computation completed.Customers. Data is ready.CustomerID.Address. dbo. [sdbtacc:2269] [DB QUERY] [Output List] Northwind (thread 00000ED8): SELECT COUNT(*) FROM dbo.Customers. [CELL ACC] [Value Segment] Cell: Creating segments [PROCESS] [Value Segment] Segment: MAX_SEG_SIZE 3 [PROCESS] [Value Segment] Segment PROCESS_RUN_DONE [sprocrun:757] [PROCESS] [Output List] Snapshot PROCESS_RUN_START [sprocrun:757] [DB QUERY] Address. Derived Fields SQL SELECT UAC_27_1. 830 records retrieved and returned to caller.Understanding Log Files Flowchart Run Info Process Run Info [PROCESS] SESSION_RUN_START [PROCESS] [Active] Select PROCESS_RUN_START [DB QUERY] [Active] Northwind (thread 000004B8): SELECT CustomerID FROM dbo.OUTPUT_TEST1 [sdbtacc:2195] [PROCESS] [Output List] Snapshot: Records Written to Table "OUTPUT_TEST1": 9 [PROCESS] [Output List] Snapshot PROCESS_RUN_DONE [PROCESS] SESSION_RUN_DONE © 2012 IBM Corporation © 2012 IBM Corporation . dbo.Customers. [PROCESS] [Universe] Merge: N_RECORDS 74 [PROCESS] [Universe] Merge PROCESS_RUN_DONE [sprocrun:757] [PROCESS] [Value Segment] Segment PROCESS_RUN_START [sprocrun:757] [CELL ACC] [Value Segment] CellAccess: Pre-computing derived fields. IBM Unica Campaign Configuration Parameters © 2012 IBM Corporation . x – Edit affinium_config.Setting Configuration Parameters
In Campaign 7.1+ – Edit parameter settings in AM configuration tab – If a parameter not exposed in AM GUI • Export current AM settings to XML file by using configTool • Add parameters to XML file • Re-import XML file into AM by using configTool again
In Campaign 7.xml
In Campaign 7.x and AM 7.0.x and AM 7.5+ and UC 8+ the installer adds new parameters to configuration automatically 19 © 2012 IBM Corporation . Server Level Configuration
maxVirtualMemory – Default. can be modified for each flowchart – Maximum recommended settings • 1024 for 32-bit versions • 2048 for 64-bit versions
useInDbOptimization – Default. can be modified for each flowchart – Recommended for environments with • High data volumes • Powerful databases 20 © 2012 IBM Corporation .
Loaders are a shared resource and other applications can take priority – Managing parallelism can be a non – trivial exercise
Campaign invokes the loader through OS scripting and sets up data and control file for the DB loader to consume.
Teradata uses 2 loader utilities – FastLoad for entering data into an empty table – Mload for appending data to a table
Performance increases especially seen in Contact History and Response Processing writes 21 © 2012 IBM Corporation .Database Loaders
Database loaders improve the performance of moving data from the campaign server back to the Database. Database Loader Execution Flowchart Generates the Control File Flowchart Generates the Data File Flowchart Invokes the Loader Script Loader Script Get in the queue First In queue? Yes Load Data and Exit No Wait X Seconds 22 © 2012 IBM Corporation © 2012 IBM Corporation . Database Tuning Configurations* Many knobs and levers exist to tune Campaigns interactions with your database… Connections. I/O – MaxQueryThreads SQL Generation – EnableSelectDistinct DB Table Statistics Through Scripts – PostTempTableCreateRunScript DB Table Statistics Through SQL – TempTablePostExecutionSQL DB Tables Suffixes – SuffixOnTempTableCreation * Additional Settings included in Addendum 23 © 2012 IBM Corporation . Query Threads/Parallelism. Leveraging In Database Processing © 2012 IBM Corporation . Track • Features that work well – – – – – Select Segment Merge Snapshot Derived fields with raw SQL macros © 2012 IBM Corporation . Campaign evaluates every process box for “In-DB eligibility”
Features do not qualify – Cross data source table joins – Audience switching (*) – Output de-duping – Cell size limits (*) – Data sampling (*) – Campaign macros (*) – Persistent derived fields (*) – MailList.Eligibility for In-DB Processing • With In-DB processing turned on. Response. Map the work table at a desired audience Cell size limits .Create a raw SQL custom macro using database function rownum() .Use a combination of Select and Snapshot to create a work table .Design Recommendations Scenario Recommendation Audience switching .Create a raw SQL custom macro using database function random() .Use the custom macro in Select or Segment to generate a random sample of IDs in the cell DFs with Campaign macros .Use the custom macro in Select or Segment to limit the number of IDs in the cell Data sampling .Create a derived field referencing the custom macro Persistent Derived Fields Use Extract processes to store calculated data in the database © 2012 IBM Corporation .Create a raw SQL custom macro using equivalent database function . SQL custom macros work seamlessly with In-DB optimization – SQL custom macros in Selects and Segments are now eligible for In-DB processing – Both “ID” and “ID + Value” custom macros can be processed in the database – Logical and arithmetic expressions built out of custom macros can be processed in the database – Derived fields using SQL custom macros in Snapshots and MailLists are now eligible for In-DB processing 27 © 2012 IBM Corporation .SQL Macros + In-DB Processing
As of Campaign 7.3+. Unica Macros vs.5 ROW_NUMBER / COUNT = 0.5) ROW_NUMBER / COUNT = 0.5 ROW_NUMBER / COUNT = 0. DB Functions Unica GROUPBY SQL Server Oracle DB2 Netezza Teradata Aggregation Functions AvgOf AVG AVG AVG AVG AVG CountOf COUNT COUNT COUNT COUNT COUNT DistinctOf COUNT (DISTINCT) COUNT (DISTINCT) COUNT (DISTINCT) COUNT (DISTINCT) COUNT (DISTINCT) MaxOf MAX MAX MAX MAX MAX MinOf MIN MIN MIN MIN MIN SumOf SUM SUM SUM SUM SUM StdDevOf STDDEV STDDEV STDDEV STDDEV STDDEV_POP Analytic Functions 28 IndexOf ROW_NUMBER ROW_NUMBER ROW_NUMBER ROW_NUMBER ROW_NUMBER MedianOf ROW_NUMBER / COUNT = 0.5 RankOf DENSE_RANK DENSE_RANK DENSE_RANK DENSE_RANK RANK © 2012 IBM Corporation .5 PERCENTILE_DISC (0. Utilizing Custom Macros © 2012 IBM Corporation . Definition Tools  Custom Macros Name and parameter list Description/Help Expression Includes TEMPTABLE token Uses {} for conditional query generation Expression type Data source Return value type © 2012 IBM Corporation .Custom Macros . Discount) from dbo.CustomerID Mid stream SQL: select a.UnitPrice*a.Quantity – a.CustomerID.Quantity – a.CustomerID group by a.CustomerID=b. UAC_34 b where a.vOrderDetails a group by a. sum(a.UnitPrice*a.Usage Used in query definitions Listed in Query Helper… Syntax help is available Top level Select SQL: select a.vOrderDetails a.Discount) from dbo.CustomerID © 2012 IBM Corporation .Custom Macros .CustomerID. sum(a. Custom Macros – Data Mapping Columns of interest © 2012 IBM Corporation . Creation Name and parameters Description/Help Macro expression with parameterized SQL Note alias for the calculated column 33 © 2012 IBM Corporation .Custom Macros . Custom Macros .Queries 34 © 2012 IBM Corporation © 2012 IBM Corporation . CustomerID GROUP BY vOrderDetails. Unica Campaign evaluates selection criteria and creates output cell SELECT * INTO UAC_7_a FROM ( SELECT DISTINCT UAC_7_8. Unica Campaign creates temp table for the results of Order_Aggr(Amount.CustomerID.value > 10000) ) a © 2012 IBM Corporation .CustomerID FROM UAC_7_8 INNER JOIN UAC_7_0 ON (UAC_7_8.CustomerID=UAC_7_0.CustomerID ) a 2. sum) SELECT * INTO UAC_7_8 FROM ( SELECT vOrderDetails.CustomerID=UAC_7_0. UAC_7_0 where vOrderDetails. sum(Amount) value FROM vOrderDetails.CustomerID) WHERE (UAC_7_8.Custom Macros – Single Execution 1. CustomerID ) a UC evaluates expression and creates output cell SELECT * INTO UAC_7_f FROM ( SELECT DISTINCT CustomerID FROM ( UAC_7_d FULL OUTER JOIN UAC_7_e ON UAC_7_d.CustomerID) INNER JOIN UAC_7_0 ON (UAC_7_d.Custom Macros – Expression Execution 1.CustomerID=UAC_7_0.CustomerID=UAC_7_e.CustomerID) OR (UAC_7_e. sum(Amount) value FROM vOrderDetails.CustomerID ) a SELECT * INTO UAC_7_d FROM ( 2.CustomerID) WHERE ((UAC_7_d. UAC_7_0 where OrderDetails.value / UAC_7_e. sum(Discount) value FROM vOrderDetails.CustomerID=UAC_7_0.CustomerID group by vOrderDetails.value) > 0.CustomerID=UAC_7_0.CustomerID=UAC_7_0. UAC_7_0 where vOrderDetails.CustomerID GROUP BY vOrderDetails.CustomerID.CustomerID. SELECT vOrderDetails. UC creates temp table for each custom macro SELECT * INTO UAC_7_e FROM (SELECT vOrderDetails.1) ) a © 2012 IBM Corporation . Q&A Contact your Account Manager for a product demonstration: [email protected] Learn more at www.ibm.com/software/marketing-solutions/cafe 37 © 2012 IBM Corporation . Addendum
Campaign Configuration Settings
Loader Settings
Data Source Settings © 2012 IBM Corporation . I/O – BulkInsertBlockSize – BulkReaderBlockSize – ConnectionCacheSize – MaxQueryThreads – QueryThreadSleep – ShareConnection 39 © 2012 IBM Corporation . Query Threads/Parallelism.Datasource Configuration Level Connections. Datasource Configuration Level SQL Generation – AllowBaseJoinsInSelect – AllowSegmentUsingSQLCase – EnableBaseDimSelfJoin – EnableSelectDistinct – EnableSelectOrderBy – PrefixOnSelectSQL – SQLOnConnect – UseExceptForMerge – UseMergeForTrack – UseNonANSIJoin – UseNotInForMerge – UseSQLToProfile 40 © 2012 IBM Corporation . Datasource Configuration Level DB Loaders – LoaderCommand – LoaderCommandForAppend – LoaderControlFileTemplate – LoaderControlFileTemplateForAppend – LoaderDelimiter – LoaderDelimiterForAppend – LoaderDelimiterAtEnd – LoaderDelimiterAtEndForAppend – LoaderUseLocaleDP – MinReqForLoaderCommand – MinReqForLoaderCommandForAppend © 2012 IBM Corporation . <FLOWCHARTNAME> Name of the flowchart being run.Tokens Available Token Description <AMUSER> IBM Unica user name associated with the flowchart being run. <CAMPAIGNCODE> Code for the campaign associated with the flowchart. <CAMPAIGNNAME> Name of the campaign associated with the flowchart being run. <DBUSER> DB user name for the database. <USER> DB user from the current flowchart connection to the data source. <TABLENAME> DB table name that Campaign is loading data into. <DATABASE> Name of the data source that Campaign is loading data into. <DSN> Value of the DSN property.Datasource Configuration Level DB Loaders . <NUMFIELDS> Number of fields in the table. © 2012 IBM Corporation . <CONTROLFILE> Full path and file name to the temporary control file. <DATAFILE> Full path and filename to the temporary data file created by Campaign. <PASSWORD> DB password from the current flowchart connection to the data source. Tokens Available Token Description <DBCOLUMNNUMBER> Column ordinal in the database. except the last.Datasource Configuration Level DB Loaders . including the angle brackets < >. <~xyz> Places the specified characters only on the last repeated line. © 2012 IBM Corporation . <FIELDLENGTH> Length of the field being loaded into the database. <FIELDNUMBER> Number of the field being loaded into the database. * If your database happens to not understand the field type. you can manually specify the appropriate text for the field type and use the <FIELDLENGTH> token. A typical use is <. <!xyz> Places the specified character(s). <NATIVETYPE> Actual database type that this field is loaded into.> which repeats a comma all fields except the last. on all lines. The length of this field is specified between the (). <xyz> Places the specified character(s) on all fields being loaded into the database. <FIELDNAME> Name of the field being loaded into the database. <FIELDTYPE> Literal "CHAR( )". CHAR. Datasource Configuration Level DB Tables Statistics First Group – PostExtractTableCreateRunScript – PostSegmentTableCreateRunScript – PostSnapshotTableCreateRunScript – PostTempTableCreateRunScript – PostUserBaseTableCreateRunScript – PostUserTableCreateRunScript – TempTablePreTruncateRunScript (only Teradata) 44 © 2012 IBM Corporation . Datasource Configuration Level DB Tables Statistics Second Group – ExtractTablePostExecutionSQL – SegmentTablePostExecutionSQL – SnapshotTablePostExecutionSQL – TempTablePostExecutionSQL – UserBaseTablePostExecutionSQL – UserTablePostExecutionSQL – TempTablePreTruncateExecutionSQL (only Teradata) 45 © 2012 IBM Corporation . Datasource Configuration Level DB Tables Statistics Third Group – SuffixOnAllOtherSQL – SuffixOnCreateDateField (Teradata  FORMAT 'YYYY-MM-DD') – SuffixOnExtractTableCreation – SuffixOnInsertSQL – SuffixOnSegmentTableCreation – SuffixOnSelectSQL – SuffixOnSnapshotTableCreation – SuffixOnTempTableCreation – SuffixOnUserBaseTableCreation – SuffixOnUserTableCreation 46 © 2012 IBM Corporation . intention to extend to DB2) 47 © 2012 IBM Corporation .Datasource Configuration Level Temporary Tables – AllowTempTables – DeleteAsRecreate – DeleteAsTruncate – DisallowTempTableDirectCreate – DoNotCreateServerBinFile – MaxRowFetchRecords – MaxTempTableJoinPctSelectAll – MaxTempTableJoinPctWithCondition – UseTempTablePool (only Teradata. <FLOWCHARTNAME> Name of the flowchart associated with the temp/Extract/Segment/Snapshot table creation. © 2012 IBM Corporation . <CAMPAIGNNAME> Name of the campaign associated with the flowchart for which temp/Extract/Segment/Snapshot table was created.Tokens Available Token 48 Description <AMUSER> IBM Unica Marketing user name associated with the flowchart for which temp/Extract/Segment/Snapshot table was created. <USER> Campaign user name of the user running the flowchart. <TABLENAME> Temp/Extract/Segment/Snapshot table name. <DBUSER> Database user name for the database where the temp/Extract/Segment/Snapshot table was created. <CAMPAIGNCODE> Code for the campaign associated with the flowchart for which temp/Extract/Segment/Snapshot table was created.Datasource Configuration Level DB Tables Statistics . <PASSWORD> DB password from the current flowchart connection to the data source. <KEYCOLUMNS> Temp/Extract/Segment/Snapshot table column name(s). ibm.com Learn more at www.ibm.Q&A Contact your Account Manager for a product demonstration: [email protected]/software/marketing-solutions/cafe 49 © 2012 IBM Corporation .