78727688 Data Stage Admin Guide[1]

DataStage Admin ActivitiesMarch 2010 Wipro Technologies @ Office Depot January 19, 2012 © Wipro Ltd - Confidential 1 Content ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ Purpose & Scope Start/Stop DS Engine To Check the status of DS Engine To Release Job locks To Delete Jobs To Delete Datasets To Restore a Dataset System Monitoring Tools ± iostat ± vmstat To View Who Last Modified a Job DataStage Housekeeping Add Environment Variables Suggested Environment Variables For Reporting Import DataStage Jobs From Command Prompt Export DataStage Jobs From Command Prompt Compiling Jobs Message Handler To Configure/Test DataBase connectivity ± Oracle ± Other Databases Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd - Confidential 2 ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ January 19, 2012 Purpose & Scope ‡ Purpose Purpose of this Presentation is to give some knowledge on DataStage Admin Activities performed on Server side and DataStage Client Side. ‡ Scope  This document will educate you about DataStage repository database (UV).  This presentation doesn¶t cover all the properties of UV Database.  This document will educate you about Configuring and trouble shooting the database connectivity for DataStage. January 19, 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd - Confidential 3 A script ³uv´ is provided for these purposes. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd .Start/Stop DS Engine From time to time you may need to stop or restart the DataStage engine. use: # dshome/bin/uv -admin ±start January 19. To stop the server engine.Confidential 4 . su to dsadm (su ± dsadm) and then Issue appropriate stop or start command below. when you want to shut down the server. use: # dshome/bin/uv -admin ±stop To restart the server engine. for example. 2012 5 . $ uv -admin ±info The following data will be displayed $ uv -admin -info Details for DataStage Engine release 7.5.1 instance "ade" ================================================================ Install history : Installed by root (admin:dsadm) on: Feb 03 2005 14:18:21 : Upgraded by root (admin:dsadm) on: Sep 21 2006 17:20:25 : Upgraded by root (admin:dsadm) on: Sep 21 2006 19:53:45 : Upgraded by root (admin:dsadm) on: Sep 21 2006 20:57:20 : Upgraded by root (admin:dsadm) on: Sep 22 2006 10:17:17 : Upgraded by root (admin:dsadm) on: Sep 22 2006 11:53:46 : Upgraded by root (admin:dsadm) on: Sep 22 2006 14:32:55 Instance tag : ade Engine status : Running w/active nls Engine location : /ascential/Ascential/DataStage/DSEngine Binary location : /ascential/Ascential/DataStage/DSEngine/bin Impersonation : Enabled Administrator : dsadm Autostart mode : disabled Autostart link : none Startup script : /ascential/Ascential/DataStage/DSEngine/sample/ds.To Check the status of DS Engine The following command is used to check the status of the DS Engine.Confidential January 19.2.rc Cache Segments : 0 active User Segments Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . To Release Job locks There are Three methods to unlock the DataStage jobs ± Using DataStage Administrator Tool. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . ± Using UV Utility ± Using DataStage Director January 19.Confidential 6 . Execute UNLOCK INODE <inodenumber> ALL UNLOCK USER <user number> ALL January 19. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd .READU EVERY´ Identify the values for INODE and USER columns for the job for which the locks need to be released.Confidential 7 .Unlock jobs -Using DataStage Administrator Tool ‡ ‡ ‡ ‡ ‡ ‡ Log into Administrator (as dsadm) Open the Command line for the project Execute ³LIST. Clear locks held by a process) Give the PID value here(Enter pid#=) and press Enter.Confidential 8 . January 19. Then select the Option ³7´ (7. List all locks) Identify the values for PID columns for the job for which the locks need to be released. Administer processes/locks >>) Again Select the option ³4´ (4. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd .Unlock jobs -Using UV Utility ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ Logon to Unix Su to dsadm Go to a corresponding project Type ³UV´ Type ³DS.TOOLS´ Select the option ³4´ (4. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd .Confidential 9 .Unlock jobs -Using DataStage Director ± ± ± ± ± ± Open DataStage Director Go to the Job Cleanup Resources In the Job Resources window. Select ³Show All´ (Processes) Find your User Name and click on ³Logout´ January 19. Note: The deleted jobs can be restored only with the disk backup January 19. you could automate this via script./dsenv . . Also developers can delete the jobs through Designer Client.To Delete Jobs The DataStage jobs can be deleted either via DS Administrator or the TCL prompt. In Administrator select the project and run this command: DELETE DS_JOBS <job name> 2. 1. From the TCL prompt you will need to do this: $DSHOME/./bin/uvsh LOGTO <project name> DELETE DS_JOBS <job name> Using TCL.Confidential 10 . 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . 1. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . 2. Using orchadmin Using DataSet Management January 19. Using this we can delete the datasets.To Delete Datasets The Datasets in DataStage can be deleted using following two methods.Confidential 11 .ds Using DataSet Management The DataSet Management utility is available in DataStage Manager and Director. Using orchadmin The orchadmin command is used to delete the datasets $APT_ORCHHOME/bin/orchadmin delete <dataset name> For Example orchadmin delete /detld1/etl/Ascential//DataStage/Projects/Staging/EDPRO_20100225_129. To Restore a Dataset The dataset can be restored from a disk backup ‡ Restore the header of the dataset. which will be found in the directory where specified in the job. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd .Confidential 12 . ‡ January 19. Which will be found in the resource directory specified in the config file used to run the job. Restore the detail records of the dataset. iostat can also help a user to determine if there is excessive IO for a specific job.00 96. Here is an example from a Linux machine which shows a relatively light load: (The first set of output is cumulative data since the machine was booted) $ iostat 10 Device: dev8-0 tps 13.09 Blk_wrtn/s 122.33 Blk_read 346233038 Blk_wrtn 293951288 « every N seconds (10 in the command line example) iostat outputs: Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev8-0 4.00 0. understanding where that throughput is coming from is vital. If one or more disks have high throughput.50 Blk_read/s 144. The specifics of iostat output vary slightly from system to system.System Monitoring Tools iostat iostat is useful for examining the throughput of various disk resources.00 0 96 January 19. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd .Confidential 13 . once it begins running.Confidential 14 . Paging suggests EE is consuming too much total memory.System Monitoring Tools vmstat vmstat is useful for examining system paging. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . should never be paging to disk (si and so should be zero). $vmstat 1 procs r b w swpd 0 0 0 10692 free 24648 memory buff cache 51872 228836 swap si so 0 0 bi 0 io bo 1 system cpu in cs us sy id 2 2 1 1 0 vmstat produces the following every N seconds: 0 0 0 10692 24648 51872 228836 0 0 0 0 328 41 1 0 99 mpstat will produce a similar report based on each processor of an SMP January 19. Ideally. a EE flow. January 19. DTM.To View Who Last Modified a Job ‡ ‡ ‡ ‡ ‡ Logon to Unix Su to dsadm Go to a project folder Type ³UV´ Issue the following SQL SELECT INSTANCE. MODIFIER FROM DS_AUDIT WHERE CLASS = '2' AND INSTANCE LIKE '%MyJob'.Confidential 15 . 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . January 19. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . ± Clearing /tmp folder ± Clearing the &PH& directories ± Delete the temporary files.DataStage Housekeeping There is very little housekeeping that can be done from the OS level. It is advised to do this in the application side itself. Data Sets or File Sets.Confidential 16 . DataStage Housekeeping Before clearing the &PH& directories make sure ever one logged out of the datastage. ± ± ± ± ± ± ± To check the user connections use the command ³uv ±admin ±info´ .Confidential 17 . $DSHOME/dsenv $DSHOME/bin/uv LOGTO UV RUN ADMIN_BP REMOVE.FILE &PH& January 19. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . ± LOGTO (Project Name) ± CLEAR. From any project directory .SHM Or $DSHOME/bin/dsdlockd -p Run the following command for each project. Confidential 18 .Add Environment Variables The following are the steps to add the environment variables in DataStage ± ± ± ± ± Logon to DataStage Administrator Choose the Project Go to Environment Select User Defined tab to add the new used defined variables. January 19. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . Select the other options to change the values for the existing variables. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd .Confidential 19 .Suggested Environment Variables for Reporting The following environment variables are concerned with various aspects of WebSphere® DataStage® jobs reporting their progress. ± ± ± ± ± ± ± ± ± ± APT_DUMP_SCORE APT_ERROR_CONFIGURATION APT_MSG_FILELINE APT_PM_PLAYER_MEMORY APT_PM_PLAYER_TIMING APT_RECORD_COUNTS OSH_DUMP OSH_ECHO OSH_EXPLAIN OSH_PRINT_SCHEMAS January 19. Import DataStage Jobs from Command Prompt Using DS_IMPORTDSX Command This command runs within the DS engine using dssh. 2.\engine\bin\dssh At the dssh prompt enter the DS_IMPORTDSX command The DSXimport command is as follows.\.. 1.. It can import any job executables found within specified DSX files. DS_IMPORTDSX filename [[-OVERWRITE] -JOB[S] * | jobname «] | [-LIST] January 19. 3. CD to the project directory on the server: Run the dssh shell: . 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd .Confidential 20 . Run it as following. If you don¶t specify a JOB or a LIST argument. <OVERWRITE> Specify this to overwrite any existing executables of the same name.DSX file rather than import them. <JOB[S]> Specify one or more job executables to import. <LIST> Specify this to list the executables in a . January 19. the default is to import all job executables in the specified file. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . <ilename> The name of the DSX file containing the components to import.Import DataStage Jobs from Command Prompt The following are the arguments.Confidential 21 . Confidential 22 . <username> The user name to use for connecting to the DataStage Server.Export DataStage Jobs from Command Prompt Using dscmdexport command The dscmdexport command used to export the datastage jobs from Command prompt. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . <password> The user¶s password. dscmdexport /H hostname /U username /P password /O omitflag project pathname /V The arguments are as follows: <hostname> Specifies the DataStage Server from which the file will be exported. January 19. dsx dscmdexport /H R101 /O 1 dstage2 C:/scratch/dstage2.Confidential 23 .dsx /V > c:/scratch/exportlog You can simply type dscmdexport at the command prompt to get help on the command options January 19. <project> Specify the project to export the components from. For example. <V> Use this flag to switch the verbose option on.dsx Messages from the export are sent to the console by default.Export DataStage Jobs from Command Prompt <omitflag> Set this to 1 to omit the username and password (only possible if you are connected to the DataStage Server via LAN Manager). 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . <pathname> The file to which to export. the following command exports the project dstage2 from the R101 to the file dstage2. but can be redirected to a file using '>'. For example: dscmdexport /H R101 /O 1 dstage99 c:/scratch/project99. Compiling Jobs There is a program on the Client side called dscc. * = All Routines. * = All BuildOps. \folder\* = folder /qspa Provision all dependant QS objects /r (value) Routine to compile. January 19.exe. T = txt . * = All Jobs. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . \folder\* = folder /d (value) ASB Domain Name /f Use the Force Compile /h (value) Name of Host to attach to /j (value) Job to compile. \folder\* = folder /rd (value) Report Directory and Name /rt (value) Type of report to produce : X = xml.Confidential 24 . C:\IS81CL\Clients\Classic>dscc /? /? Show program usage /bo (value) BuildOp to compile. This resides in the Clients\Classic folder where the Designer and Director reside. This is used to Compile the jobs.default /u (value) User Name Also the jobs can be compiled from DataStage Designer Client or from Manager through ³Run Multiple Job Compile´ Option. \folder\* = folder /jt (value) Job Type To Compile -1 (Default) = All Types 0 = Server 1 = Mainframe 2 = Sequence 3 = Parallel /mfcgb (value) Mainframe Code Gen base directory location /mful (value) Mainframe Job Upload Profile /ouc Only Compile Uncompiled Objects /p (value) Password project Project To Attach to /qs (value) QS objects to provision * = All QS objects. Message Handler The Message Handler is used to suppress the Warnings. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd .Confidential 25 . ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ Log into the Director Select the job And then view the logs Select the warning Right click on the Warning message A popup will display Select ³ Add Rule to the Message Handler´ Now choose the ³Add rule to chosen message handler´ And click on the Button ³ Add Rule´ January 19. This file normally resides in the ORACLE HOME\NETWORK\ADMIN directory Add the following entry in your TNSNAMES. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd .Confidential 26 .ORA file and change the values shown in brackets to suit your environment: <addressname> = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(Host = <hostname>)(Port = <port>)) ) (CONNECT_DATA = (SERVICE_NAME = <sid>) ) ) This can also be done through LDAP January 19.ora file.To Configure/Test DataBase Connectivity For Oracle Configure Tnsnames. Which defines databases addresses for establishing connections. It cannot tell if the databases behind the listener is up or not For Example $ tnsping myDB TNS Ping Utility for Linux: Version 10.0. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . Oracle.Confidential 27 . 2005.2.To Configure/Test DataBase Connectivity Tnsping TNSPING is a utility in the ORACLE HOME/bin directory used to test if a SQL*Net connect string can connect to a remote listener (check if the socket is reachable).Production on 24-MAY-2007 08:55:13 Copyright (c) 1997.0 .2.ora Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = MyDB))) OK (10 msec) January 19. All rights reserved.0/db_1/network/admin/sqlnet. Used parameter files: /app/oracle/product/10.1. Note: This utility only tests if the listener is available. 0 .NAMES.com/wiki/Tnsping" Categories: Network services | Utilities January 19.7. All rights reserved. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd .2.DIRECTORY_PATH=(LDAP) in sqlnet.Confidential 28 .2.0.orafaq.Production on 10-DEC-2007 15:05:50 Copyright (c) 1997 Oracle Corporation.TO Configure/Test DataBase Connectivity When using an LDAP (OID) server . Used parameter files: /app/oracle/product/10.ora Used LDAP adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED)(SERVICE_NAME = MyDB))) OK (300 msec) Retrieved from "http://www.ora: $ tnsping myDB TNS Ping Utility for Solaris: Version 9.0/db_1/network/admin/sqlnet. Confidential 29 .To Configure/Test DataBase Connectivity The SQLPLUS is used to check the databases behind the listener is up or not The following is the syntax. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . sqlplus username/password@servicename January 19. To Configure/Test DataBase Connectivity Other Databases ‡ ‡ Add an entry in odbc.x ODBC Driver LogonID=user1 pwd=tigress4 Database=odbc_demo ServerName=my_server January 19. The following example shows the configuration for a DSN called Informix 9 [Informix9] Driver=/work/informix/lib/cli/iclis09b.so Description=Informix 9. 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd . Every DSN to which the application connects must have an entry in this file.ini file.Confidential 30 . Inc./example When prompted put in the odbc entry name and the credential similar to this below sample: $ . 2012 Confidential © Copyright 2007 Wipro Ltd © Wipro Ltd .Confidential 31 ./dsenv Then cd to the branded_odbc/example directory $cd /export/home/752NLS/Ascential/DataStage/branded_odbc/example and run the example program $.ini> Enter the user name : <yourusername> Enter the password : <youruserpassword> And then you should see the below (or otherwise some type of error) Enter SQL statements (Press ENTER to QUIT) SQL> Exiting from the Example ODBC program January 19.odbc. Enter the data source name : <yourdatasourcename from .TO Configure/Test DataBase Connectivity The datadirect example program is used to check where the database is in relation to the DS Server. The following are the steps used to check this. ./example . $cd /export/home/752NLS/Ascential/DataStage/DSEngine $. ODBC Example Application./example DataDirect Technologies. Thank You January 19. 2012 © Wipro Ltd .Confidential 32 .