DataStage Scribble SheetVarious Allow DataStage to see a UniData account Need to activate Uniserver on the UniData account (free licence). This allows Objectcall to work from DataStage. Useful hex numbers Character Tab Line Feed Carriage Return Escape Space Oracle null written to hash file Text mark Sub-multi-value mark Multi-value mark Attribute mark Oracle null Pick null – i.e. “” CHAR() 09 10 13 27 32 128 251 252 253 254 See below 0 Hex 09 0A 0D An Oracle null is defined by a bit being set, and as such can not be tested for directly from DataStage. However, you can use an NVL to set a null value in the Oracle stage to something else, and then use that same value in the lookup derivation: ORAOCI stage, user-defined SQL: NVL(SOURCE_SYSTEM_K4_VALUE,-999)=:4 Key field derivation in the look up stage: IF lu.ANIMAL_PRODUCTION_PK = "" THEN -999 ELSE lu.ANIMAL_PRODUCTION_PK ODBC’s Server details From DataStage Manager, to enable a new ODBC driver need to update the following DataStage config files: .odbc.ini uvodbc.config Note that there can also exist a local uvodbc.config file for a project. If there isn’t DataStage uses the default file. This default file is located in $DSHOME / DataStage / DSEngine Or possibly /dsadm/DataStage//DSEngine/ To get to $DSHOME type: The file is called .odbc.ini cd `cat /.dshome` Stop/Start DataStage script su to root cd to ./DSEngine/bin execute ./uv –admin –stop execute ./uv –admin –start to start Look in: /$DSHOME/sample/ds.rc 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818.doc Page 1 of 20 DataStage Scribble Sheet Command to find number of files that can be opened: To increase number of files that can be opened: Place this after the umask command ulimit –aH ulimit –n 512 DataStage environment variables on Unix Look in: .. /DSEngine/dsenv DataStage and Oracle etc. variables set in here. Can also set Unix variables and then use them in DataStage jobs (dsenv is read before every DataStage job. Accessed through Job Control => Parameters => Add Environmet Variable (which also allows creation of new environment variables) UniVerse on NT useful bits CREATE.FILE {filename} {type} {modulo} {separation} File types 1 and 19 are not hashed, good for programs. Type 30 is Dynamic. Types 2 to 18 are hashed. Dictionary level defaults to type 3, modulo 1, separation 2. To create a type 2 file, mod and sep of 1, dictionary uses defaults: CREATE.FILE {filename} 2 1 1 To create a dynamic file, which is type 30, a directory (folder) with a flat file for the dictionary: CREATE.FILE {filename} DYNAMIC or CREATE.FILE {filename} 30 1 2 or CREATE.FILE {filename} 19 At DOS, both type 19 and type 30 (directory) files appear as a <DIR>, the difference being that the type 19 (and type 1) directory is empty, whereas the type 30 directory contains files called DATA.30 and OVER.30 Type 19 files is a normal directory So use type 19 files to ftp programs back and forth. To create a file in a different directory or path name, use DataStage or refer to JCFILE CREATE.FL Distributive Files on UniVerse (Distributed? Unidata?) DF.DEFINE command DF.MODIFY command Usage: DF.MODIFY -d dist_file -o op -p part_file -n _part_number UniVerse on UNIX useful bits As above, plus: Running UNIX commands from within UniVerse paragraph or program: sh –c “UNIX command” Inverting the case in a UniVerse session At TCL type: Then to revert: PTERM -CASE NOINVERT PTERM -CASE INVERT 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818.doc Page 2 of 20 ST_HV2005 SP.CORE SAVING IA. RANGE.COUNT and CD.COUNT DET.ST<<I2.CORE 011: COPY FROM CORE. bytes: CD CD.EXP IA.SUP Output from RETRIEVE command to UNIX or DOS file Use the COMO command to spool screen output to a file in the {Project}/&COMO&/ directory UniData on UNIX useful bits This shows how to prompt for variables from within a paragraph (<<Variable>>). or n.HDG "SP.Iacs No>>" BY. 3000 etc.FILE DIR filename CREATE.VOC RANGE2 RANGE. i.RECDT" TO DELIM ". 2000. Then run these commands to show how many items in the file are greater than 1000.VOC from JCFILE into VOC.SYSTEST TO CORE OVERWRITING 012: DATA Y 003: LIST IACS.FILES 007: DATE 008: ! who am I 003: AE IACS.DataStage Scribble Sheet Look for large records within a hash file Copy RANGE2." <<Enter filename for output>> 019: COMO OFF 021: FINISH: * CREATE.HVDC.ST<<I2.Scheme Year>> "<<I3.FILE filename DYNAMIC creates UNIX directory creates a dynamic hashed file Universe debug tool – RAID / [string] [line] B B* C D D* [line]G L variable M variable=VALUE M Searches the source for [string] Sets break point at line [line] Lists break points Continue program execution Delete break point at current line Deletes all break points Continue program execution from [line] Display next line to be executed Continue execution until variable changes Continue execution until variable changes to VALUE Quit raid Steps through the code one. use of the DATA command to feed data into a prompt. and also how to output the result of an English statement to a file at UNIX level: 001: PA 003: DISPLAY If you really want to run this please type YES 004: IF <<CONFIRM>> NE 'YES' THEN GO FINISH 006: COMO APPEND CLEAR.Scheme Year>> <<I3.Iacs No>> 004: DATA 101 005: DATA R 343963 008: DATA FI 010: SELECT IACS.e.RECDT COL. subroutine becomes a single line Display call stack trace Displays 10 lines of source code around [line] Q [n] S [n] S* T [line] W 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818.doc Page 3 of 20 .ON RANGE2 TOTAL RANGE. lines at a time Steps around subroutine calls.COUNT LIST {File name} BY RANGE2 BREAK. DataStage Scribble Sheet [line] Z [n] variable / variable ! STRING Displays the next 10 lines of source code Subtracts n from the current line Displays the value of variable Changes the value of variable to STRING DataStage on NT useful bits Job Control priority Any code in the Job Control tab is executed before any stages in the diagram window are executed DataStage Admin menu At ECL type DS. DataStage shortcut <CTRL> E to edit the highlighted stage derivation DataStage system files C:\Ascential\DataStage\ServerEngine\UVTemp – this directory holds temporary UniVerse files.doc Page 4 of 20 . with ‘0’ as the starting value. TOTCOUNT NEXTID = Arg1 IF UNASSIGNED(OLDPARAM) THEN OLDPARAM = NEXTID TOTCOUNT = 0 END IF NEXTID = OLDPARAM THEN TOTCOUNT +=1 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818. or DS.TOOLS for main tools menu.PLADMIN for process and locks menu Starting a DataStage job without an input file Simply put a stage variable in the first Transform. for instance called ‘DummyCounter’. This example increments a counter while NextId remains the same (GeneralCounter routine): COMMON /COUNTER/ OLDPARAM. Capture files seem to hold ECL output. This pathname is configurable. Example of the use of Stage Variables Using COMMON instead of stage variables Another way to keep things in memory is by using a common area. This will run for 1 row. Then increment it in the stage variable box “DummyCounter + 1” and then in all the constraints put “DummyCounter < 2”. mainly sort and capture files. More than 2 runs for more rows. certainly in Unix anyway. doc Page 5 of 20 .RoutineName) Useful DataStage files • • • • • DS_ROUTINES {routine name} attribute 15 contains the code used in the routine.records contain the code used in the stages – this is for DataStage 5. • <2> called OLETYPE = CTrxOutput • <3> called NAME = {link name} • <6> = Constraint • <10> = Constraint • <14> = Output derivation column name • <16> = Data element • <17> = Type code.1 DS_JOB44 V0S12P2 (or in 5.records contain the code used in the stages DS_JOBOBJECTS J\{job number}\ transform id .RoutineName) DSLogFatal(MSG. that hash file can then be normalised Put messages into the log EQUATE RoutineName to “…whatever…” DSLogInfo(MSG.DataStage Scribble Sheet END ELSE OLDPARAM = NEXTID TOTCOUNT = 1 END Ans = TOTCOUNT DataStage Designer: Folder stage versus Sequential File stage • • • Folder stage is much quicker All rows read in from the source folders end with CHAR(254) (not C/R L/F) If these are converted to CHAR(253) and written to a hash file.RoutineName) DSLogWarn(MSG. where • 1 = Char • 2 = Numeric • 3 = Decimal • 4 = Integer • 5 = SmallInt • 9 = Date • 10 = Time • 11 = Timestamp • 12 = VarChar • -1 = LongVarChar • -2 = Binary • -4 = LongVarBinary • -5 = BigInt • -6 = TinyInt • -7 = Bit • -9 = NVarChar • <18> = Length • <19> = Scale 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818. See JCFILE JC_RTNS for routine to weed out the code and write it into a program DS_JOBS {jobname} attribute 5 contains the job number DS_JOB{job number}.1 DS_JOBOBJECTS J\44\V0S81P2) for example. rebuilt when transform OK’d in Designer • <45> = Stage variable? J\nn\V0S… item in file DS_JOBOBJECTS • <2> called OLETYPE = CCustomStage • <3> called NAME = {stage name} • <7> = Id of CCustomeOutput output link • <8> = stage type. multi-valued • <17> = default value. eg ORAOCI8 • <17> = parameters (with #’s). multi-valued • <31> = List of other jobs called – not used? • <94> to <96> = Job parameter details J\nn\V0S… item in file DS_JOBOBJECTS • <2> called OLETYPE = CTransformerStage • <3> called NAME = {transform name} • <37> = CStageVar/{n} • <38> = Stage variable names.doc Page 6 of 20 . options. multi-valued • <41> = Stage variable initial values. multi-valued • <16> = prompt. create/drop table • <16> = input column names 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818. multi-valued • <43> = Expanded copy of <40>. SQL code. if any. SQL code • <16> = input column names • <23> = key J\nn\V0S… item in file DS_JOBOBJECTS • <2> called OLETYPE = CCustomInput • <3> called NAME = {input link name} • <15> = multi-valued details of passive stage: table name. CR/LF delimited • <13> = {text}/number of parameters • <14> = parameter name. multi-valued • <15> = ??. options.DataStage Scribble Sheet • • • • • • <20> = Nullable • <21> = Key • <22> = Display length • <23> = actual derivation code • <25> = copy of <23> with any transforms applied • <26> = something to do with input (output?) columns • <45> = Constraint? • <100> = Constraint? J\nn\ROOT item in file DS_JOBOBJECTS • <2> called OLETYPE = CJobDefn • <3> called NAME = {job name} • <4> = Short job description • <7> = Full job description • <9> = Before job subroutine and argument list • <11> = Job control code. multi-valued • <18> = help text. multi-valued • <19> = type code. multi-valued • <39> = Stage variable description fields. multi-valued J\nn\V0S… item in file DS_JOBOBJECTS • <2> called OLETYPE = CCustomOutput • <3> called NAME = {output link name} • <14> = multi-valued details of passive stage: table name. multi-valued • <40> = Stage variable derivations. doc Page 7 of 20 .internal parameters for job running • <15> = invocation id expression • <16> = 4 if invoking multiple instance job • <17> = <15> • <19> to <24> . “Executed OK” in quotes. <LHS> = <RHS> DS_METADATA • <4> = Short description • <5> = Long description • <9> = ‘0’ for normal file. 2 Executed OK. 2 = Validate. calling job parameter name or “hard coded value” • <23> = <21> ? J\nn\V0S… item in file DS_JOBOBJECTS • <2> called OLETYPE = CJSSequencer • <3> called NAME = {blue sequencer name} • <8> = output mode: 0 for ALL. 5 User Status.DataStage Scribble Sheet • • • • • • • J\nn\V0S… item in file DS_JOBOBJECTS • <2> called OLETYPE = CHashedFileStage • <3> called NAME = {hash stage name} • <8> = directory path J\nn\V0S… item in file DS_JOBOBJECTS • <2> called OLETYPE = CHashedInput • <3> called NAME = {input link name} • <6> = Hash file name • <7> = Clear file option (1 or 0) • <8> = Backup file option (1 or 0) • <16> = Allow stage write cache option (1 or 0) • <17> = Create file options.parameters • <20> = parameter name in job being called • <21> = actual value fed through. 4 Warnings. if blank file not created • <18> = Delete file option (1 or 0) J\nn\V0S… item in file DS_JOBOBJECTS • <2> called OLETYPE = CHashedOutput • <3> called NAME = {link name} • <6> = Hash file name • <7> = Select criteria (input or look-up file to this job) • <9> = {Normalised field name} or the text Un-Normalised J\nn\V0S… item in file DS_JOBOBJECTS for Sequencer • <2> called OLETYPE = CJSJobActivity • <3> called NAME = {stage name} • <11> = Execution action: 0 = Run. 3 = Reset • <12> = Called Job name • <13> to <18> . = {user status}. ‘1’ for associated (mv’d) file • <11> = text / number of fields • <12> = Column name • <13> = Description with CR/LF’s embedded • <14> = Data element 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818. 1 = Reset if required then run. 1 for ANY J\nn\V0S… item in file DS_JOBOBJECTS • <2> called OLETYPE = CJSActivityOutput • <3> called NAME = {output link name} • <6> Trigger expression Type: 1 Otherwise. 6 Conditional • <8> Trigger expression Text: N/A. “Executed Failed” . “Executed finished with warnings”. 3 Failed. that appears in Director (for instance if can not clear status file normally) • 3 – Aborted • 2 – Finished • 0 or 1 – Running? See JCFILE JC_TRANS for routine to duplicate derivations and many other useful bits.5 • number) • ‘S’ingle or ‘M’ulti-valued • <98> = text / number of fields • valued zeros • "DISPLAY_NUMERIC" • zeros • • <104> = set to multi-valued <108> = ditto <110> = ditto <111> = text: "DISPLAY" v7.ERR) with both the DSU.DataStage Scribble Sheet • • • • • • • <15> = SQL type code. recs exist in DS_JOBOBJECTS as above RT_STATUSnnnn where nnnn = JOBNO • Job status file. v7.doc Page 8 of 20 . From ECL or within another routine need to do: CALL DSU.5 • • default=0" • • • <114> = set to multi-valued zeros • <115> = ditto • <116> = ditto DS_CONTAINERS • Similar to DS_JOBS but for containers • <5> = JOBNO. Also see JC_TRANSALL. at the front and the ERR argument – good and useful example of this is JCFILE CREATE.1 • In the .5: Level? <103> = text: v7.5 <113> = text: "zoned. defined as above <16> = Length <17> = Scale <18> = Nullable – 0 or 1 <19> = Key – 1 or 0 <20> = Display length <21> = Association Only displayed if <9> = 1 <22> = Position (attribute Only displayed if <9> = 1 <23> = Attribute type – Only displayed if <9> = 1 <102> = Dunno. set to multiv7.ExecDOS(TEXT.FL to create files in a remote path name 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818. JC_TABLES and JC_METADATA Calling DataStage routines from ECL or within another routine For instance from within a DataStage job the AfterJob routine would be ExecDOS and the parameter passed would be whatever.Instance and JobName.1 record can change status – attribute <2> . id is JobName.Instance. Here’s example Brian had to run jobs on two different servers: You can run dsjob from Command Line stage remotely and pass parameters as well (undocumented feature) in the following format: dsjob -run #MyParam# (I set this to -wait) DevScanDocsBuildTxtForBatch . Example to run a job locally then move a file: dsjob -run -param PATH=/radf/hfa/global/DataStage/Projects/IACS_SYSTEST/FlatFiles -param FNAME=MetricsBrm9 -wait -local IACS_SYSTEST ProduceMetricsFromLogs # mv .FindMatchingContracts" VAR = FindMatchingContracts(InFile.doc Page 9 of 20 . but dsjob also allows a login to a remote server .DBSOURCE) At top of routine: Deffun FindMatchingContracts(InFile. The –local is only really necessary if changing environment variables. OutFile. or define the function at the top of the routine and then use it in the same way as in a derivation to feed into a variable: Deffun FindMatchingContracts(InFile. Can also run a job remotely using this command.H These are included in the routine used to set / change a job’s parameters: $INCLUDE DSINCLUDE DSD_STAGE./FlatFiles/MetricsBrm9.] [-wait] ProjectName JobName 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818.csv .H $INCLUDE DSINCLUDE DSD.. on another DataStage server for instance.H See routine SetSequencerParameters currently in project DevERDP in category IACS_update_for_current_systems which picks up a date from VOC and feeds it into a parameter Calling Transform functions from within another routine Either use the CALL.FindMatchingContracts" Later in code: VAR = FindMatchingContracts(InFile./.H $INCLUDE DSINCLUDE DSD_RTSTATUS.OutFile.DSU… method above./HVDC/HVDC_XML The –wait waits for the job to finish. Other dsjob options are: dsjob [-server servname] [-user userid] [-password pwd] –run [-mode] [-param p1 –param p2 etc. DBSOURCE) Calling "DSU.H $INCLUDE DSINCLUDE JOBCONTROL. DBSOURCE) Calling "DSU.OutFile.dsjob -server servername -user username -password password Refer to DataStage Server Job Developer’s Guide manual for dsjob documentation.DataStage Scribble Sheet Calling DataStage functions from within a routine * Include this in program to correctly use datastage functions $INCLUDE DSINCLUDE JOBCONTROL.DBSOURCE) Calling a DataStage Job from within a routine Can run DataStage jobs from command line (UNIX prompt) by using dsjob.this works locally.. OutFile. dsx E:\DSProjectBKP\DMSPresentation\%DAYNUM%_%DSPROJECT%. InputArg. write a script and run it using ftp to move files around: OpenPath ".RoutineName) END * Now write the script to execute – note the “quote site…” bit to do a remote chmod on the file after putting it Script = "open worcs" 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818. SystemReturnCode) Alternatively call the DataStage routine ExecDOS which uses DSExecute Executing a Unix shell script from within a batch job Call DSExecute("UNIX".SystemReturnCode) For example (on Unix).FILE ORDER_CUSTOMER".txt %DSPROJECT% /V > Executing an ECL command from within a batch job Use the following type of command in the batch job: Call DSExecute("UV". "CLEAR. Output. SystemReturnCode) Alternatively call the DataStage routine ExecTCL which uses DSExecute Executing a DOS command from within a batch job Call DSExecute("DOS"." End * Check here for any errors so far IF TXT <> "" THEN TXT = CHANGE(TXT.2% dscmdexport /H=erdpds1 /U=DataStage /P=datateam E:\DSProjectBKP\DMSPresentation\%DAYNUM%_%DSPROJECT%.ExecSH(command.". Output. Output.@AM. SystemReturnCode) Alternatively call the DataStage routine ExecSH which uses DSExecute Call DSU. Output.doc Page 10 of 20 . ") Call DSLogFatal(TXT. InputArg." To Fv Else TXT<-1> = "Unable to write file to .DataStage Scribble Sheet Refer to JCFILE runMetrics for an example of this Calling a DataStage Job from within a routine Refer to JCFILE Call_DSJob_from_routine which shows the code that needs to go into the routine to do the calling Or just call the job from within Job Control Run a DataStage export from DOS in a batch job (can be scheduled) : Set Variables set BKDBASE=PRBK set DSPROJECT=ProdProbis set DAYNUM=%date:~4. RC) Executing a TCL command from within a batch job Call DSExecute("TCL". command. * Job failed a validation run 21 . For example: Equ DSJS.RUNFAILED DSJS.* Job has crashed 97 .ME.LinkName.RESET DSJS.TransformName.."LinkName".VALWARN DSJS.RoutineName) End command = "ftp -i -n < ":ScriptName Call DSU./erdp" "mdelete ":WILDCARD "put ":FILENAME:VERSION "mput ":WILDCARD "quote site chmod 646 ":FILENAME:VERSION "bye" ScriptName = "IUC090script" Write Script On Fv.ERRWARN) Ans = DSGetLinkInfo(JOBHANDLE. for instance with the output from an Oracle COUNT command Feed through the contents of UserStatus as a parameter Returns the number of rows processed in a link Returns the number of rows processed in a link in another job DSGetJobInfo(DSJ. ScriptName On Error Null Else Null DataStage Macros The following can be used within the derivations of a transform to return the information: Return the name of this job Sets the DataStage internal variable UserStatus.H which shows return values for DSJ.DSJ.".JOBSTATUS etc.ROW_COUNT) From a sequencer: click on Insert parameter value Find the jobname where user status was set Click on $UserStatus DSGetLinkInfo(DSJ. ScriptName Else Call DSLogFatal("Unable to write file ":ScriptName:" to .* Job finished a normal run with warnings 3 .VALOK DSJS.STOPPED To To To To To To To To To To 0 .DataStage Scribble Sheet or Script<-1> Script<-1> Script<-1> Script<-1> Script<-1> Script<-1> Script<-1> Script<-1> = = = = = = = = "user f803163 phant0m" "lcd /radf/sinay/uat/temp" "cd .* Job has not been compiled 99 .NOTRUNNABLE To Equ DSJS.JOBNAME) SetUserStatus(inputlinkname.* Job finished a validation run with warnings 13 .RUNNING Equ Equ Equ Equ Equ Equ Equ Equ Equ DSJS.* Job finished a normal run with a fatal error 11 . DSJ. SystemReturnCode) Delete Fv.CRASHED DSJS.DSJ.* Job was stopped by operator intervention (can't tell run type) 98 .* This is the only status that means the job is actually running 1 .doc Page 11 of 20 .* Job finished a reset run 96 ."TransformName".* Any other status Equ DSJS.RUNWARN DSJS.VALFAILED DSJS.ME.NOTRUNNING To 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818.* Job finished a validation run with no warnings 12 .* Job finished a normal run with no warnings 2 .RUNOK DSJS.ExecSH(command.DSJ.LINKROWCOUNT) COMMON command to include in DataStage routines: $INCLUDE DSINCLUDE JOBCONTROL.H DataStage warning statii These can be found in the directory \Ascential\DataStage\Projects\{project name}\dsinclude in a file called JOBCONTROL.LINKROWCOUNT) Call a DataStage routine which does the following: JOBHANDLE = DSAttachJob(JobName. DSJ. DSJ.JOBSTATUS) IF Status = DSJS.JOBNAME) IF JOBNAME = -1 THEN * Ignore jobs that are not compiled END ELSE BEGIN CASE CASE STATUS = DSJS.CRASHED THEN ErrCode = DSRunJob(hJob1. In the Before [SQL] tab put the following: CALL {PROCEDURE NAME}(#Parameter#) CALL EXTRACT_ANIMAL_DATA(#LOAD_BATCH_ID#) Even if there is no parameter must have the brackets at the end: CALL PR_TEMP_FIX_FROM_TO_DATES() 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818.STOPPED TXT = "Job ":JOBNAME:" status Stopped" CASE STATUS = DSJS.doc Page 12 of 20 . DSJ. DSJ.VALFAILED TXT = "Job ":JOBNAME:" status Validation Failed" CASE STATUS = DSJS.ERRNONE) STATUS = DSGetJobInfo(HJOB.RUNFAILED OR Status = DSJS.RoutineName) END Ensure job is in a runnable state from within Job Control (Batch job) * Ensure job is in a runnable state NOTRUNNABLE = 98 RUNFAILED (Aborted) = 3 CRASHED = 96 Finished = 1 Compiled = 99 Reset = 21 * Ensure job is in a runnable state Status = DSGetJobInfo(hJob1.JOBSTATUS) JOBNAME = DSGetJobInfo(HJOB.RUNFAILED TXT = "Job ":JOBNAME:" status Run Failed (Aborted)" CASE STATUS = DSJS.RUNRESET) ErrCode = DSWaitForJob(hJob1) hJob1 = DSAttachJob("JOBNAME".":STATUS * Call DSLogInfo(TXT. "JobControl") Abort End END DataStage calling an Oracle Stored Procedure Needs to have an ORAOCI stage which does something (eg.NOTRUNNABLE TXT = "Job ":JOBNAME:" status Not Runnable" CASE 1 TXT = "Job ":JOBNAME:" okay . User defined SQL to do “SELECT dummy FROM dual”. DSJ. DSJ.RoutineName) TXT = "" END CASE IF TXT <> "" THEN Call DSLogWarn(TXT.ERRFATAL) If NOT(hJob1) Then Call DSLogFatal("Job Attach Failed:JOBNAME".CRASHED TXT = "Job ":JOBNAME:" status Crashed" CASE STATUS = DSJS.DataStage Scribble Sheet Accessing job statii from within Job Control HJOB = DSAttachJob(ID. remove prefixies in column derivations …can then do selects on the file. so STH_Customers becomes STH_Customers. 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818. put the name of both tables separated by a comma.Account_Number = B. have an ORAOCI stage for one table. Accessing local machine from a UniVerse stage In the box called Data source name put: localuv Creating hash files in another path Creating File DUPLICATES. B and C in lieu of the full file names. OUTPUT tab => GENERAL => Table names field.Insurance_Indicator FROM STH_Customers C.Lessee_Name. where the 2 tables have a common field (like a key field): • In the ORAOCI stage.…etc… FROM STH_Customers SELECT C.…etc….FL Incorporating field from another file in the file/table stage For example.DataStage Scribble Sheet Merging data from sequential files • • There is a merge transform within DataStage which allows 2 sequential files to be merged and output fields from both files.csv or .STH_Bookings B WHERE C. but want to also import a related field from another table. or lookups etc.STH_Bookings • In => COLUMNS load the relevant fields from both files • In => GENERAL => Query Type change to User-defined SQL query • In => SQL => User-defined change the query to include the bold bits below: • • • SELECT Account_Number.TAB o define format o tick Column Name Header o select OEM box o click on GUESS box (if change name click Modify) o doing this will generate a Schema. look-up on a txt file (on NT) This allows sql statements on a sequential (.doc Page 13 of 20 .Account_Number Note the addition of the second file name.Account_Number. Can also use an ORAOCI file stage with a user-defined SQL statement that joins the relevant files (can have more than 2) and outputs the necessary fields. select.ini file first • DataStage Manager: o Import table definitions from the driver just created • DataStage Job o Use ODBC stage o Quote = 000 o Load columns. Executing Command = C:\Ascential\DataStage\ServerEngine\bin\mkdbfile d:\FTPData\MigrationDev\DUPLICATES 30 1 4 20 50 80 1628 See JCFILE CREATE.B.C.txt) file! • Save the csv file as a tab delimited text file • Use Control Panel to set up an ODBC driver on the (NT) server: o System DSN o Microsoft text driver o Select the directory o .ini file o If ever need to modify anything delete this Schema. and the where clause Using an ODBC stage to access.Lessee_Name. re-import the metadata in Manager. Warning Messages Warning message in log regarding Phantom processes For example: DataStage Job 270 Phantom 1364 Program "JOB. If there are too many try just deleting the ones where it goes out of step and re-entering them manually.doc Page 14 of 20 . Might need to feed the relevant data into a hash file first. recreate it and load the columns from the re-imported metadata (having removed ‘I’-types). Other log files &PH& at Universe level. Can set up pointer to it called PH: 1. i. Warning message doing a ‘View data’ on an Oracle file stage Message: 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818. put the table name into the ‘build’ section. &PH& 3. stage can’t find the table name. Read on HASH file when correct key doesn’t find existing record And other similar problems where the data looks like it’s in the wrong column when viewed in the DataStage hash file stage. the hash file is built once at the start of the process. In this case you have a transformer called TRANS1.1362629138. The Job number is 270 meaning that under that project directory there will be a subdirectory called RT_BP270. Under this directory will be the source code for JOB.DT.e.DataStage Scribble Sheet View or Read on HASH file shows no data in one or more columns Also related to the problem below. Zero length string used. Change the user-defined SQL to be built normally.TRANS1.1362629138.TRANS1": Line 301. the DS_METADATA record might have attribute 22.1215067440. F Ardent DataStage Repository file 2. then return it to user-defined SQL Warning message accessing data from an Oracle file stage Message: Variables not bound Due to trying to accumulate data in the SQL in the Oracle stage. Each transformer will have a program. This should avoid having to enter the columns manually as recommended below. To fix this preferably correct the DS_METADATA record after saving it. then in the job delete the file stage. the position indicator. D_VOC . This happens where columns in a hash file have been moved around./DataStage/DSEngine/errlog Warning message accessing data from an Oracle file stage Message: table name not in link??? Normally happens if using user-defined SQL. using the GROUP BY command. for instance when creating a new xref hash file. The resolution is to delete all the columns in the hash file stage and enter them manually.1215067440. or change the Position column in the hash file stage where the metadata has been loaded. It should be possible to work out line 301 by looking at the transformer. Alternatively. Amazingly this also works in a container. set incorrectly. When this metadata is loaded into another stage. This is often caused by having loaded the column definitions from DataStage Manager Table Definitions. If the columns of the new file are saved. Variable previously undefined. column data might appear to be missing. save the stage.DT. For instance.NSR_CLAIM. the stage might say TIMESTAMP whereas the Oracle table says DATE.N_CUSTOMER: [IBM][CLI Driver][AS] SQL0407N Assignment of a NULL value to a NOT NULL column "*N " is not allowed.COMMON_LAND. This can happen if a job does “Attempting to Cleanup after ABORT raised in stage” but there was no reason why it aborted.ORA_RefVendor: ORA-00932: inconsistent datatypes This is due to the Oracle table having a datatype different to that defined in the stage Columns. To fix pass the relevant file fields through an IsNull test after every look-up NOTE: All fields in a Primary Key in Oracle are mandatory – i. Warning message when running DataStage job Message: CustConvert. maybe lock left at Universe level (LIST. To do with locking. then available from any account Message: DA1522_26000_ETL2TransformJob.SourceFile: ds_udtopen() .Operation not permitted.UDTOpen GCI $DS. 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818.Client and server product identifiers don't match: received The udt counter and returned counter M0094NSRClaimPaddr.DA1522_Prepare_Trf.UDTOpen error 24. can not have a NULL primary key field Message: CustConvertRepeatRows.Error in mutex_unlock() .O_CIO2: [IBM][CLI Driver][AS] SQL0913N Unsuccessful execution caused by deadlock or timeout.Unable to connect to server UniData Client error: call to UniOpenPos returned 20 . The fields used later in the process then can contain an SQL null.Data: ds_ipcflush() .. Reason code "TESTCONV1. SQLSTATE=23502 This is caused by DataStage reading a file and the record is not on file.CUSTO00004".SourceFile: DSD..1522.. SQLSTATE=57033 This is caused by: their being a problem with the Journal Receiver on the AS400 – the Journal Receiver is full or the file has reached its maximum number of allowed records (need to set file to NOMAX) or there is a lock on the file on the AS400 Message: Buffer overflow Normally caused by a field being too long for its target field Message: M0094NSRClaimPaddr.NSR_CLAIM.READU) but no longer there at UNIX level (not in ps –ef).e..DataStage Scribble Sheet DSBrowser. Caused by running out of licences on the Unidata server Message: Unable to Compile and Catalog DSHELPER subroutine aaPopulateXrefFileCM.in To do with UniData – DSHELPER has to be globally catalogued in the UniData system.doc Page 15 of 20 . one item per line ls –l | fgrep –f {filename} 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818. cw etc. h.DataStage Scribble Sheet Message: Data has been truncated To identify field that is too long change output to fixed-length flat file and run the job. k.column is the sort key Script to remove a list of files or directories (called H_…) for i in `ls|grep H_` note apostrophe is top left hand key do rm -rf $I -rf. to revert to the previous directory to revert to home directory Get Unix to finish typing for you (it will fill the unique bit of the file name): <ESC>\ and file name is completed automatically Changing the UNIX prompt export PS1='$PWD > ' export PS1='$ ' Sort output from ls –al by the fifth column: ls –al | sort –k5 sort {-o output filename} {-t~} –k1.READU Then kill it using LOGOUT –{pid} LISTU – shows users logged on UNLOCK ALL or other option – from UV account. type in: stty erase <BACKSPACE> and in the . possibly by doing a LIST. -k field.READU) DOS – useful bits rmdir {directory name} /s del D_{file name} UNIX – useful bits To ‘kill’ a process running udt: Find PID. j. l. using ‘who’ ps –ef | grep pts/nn or grep PID kill –2 PID Unix backspace key gives ^H.profile put: stty erase ‘^H’ Repeat commands: set –o vi cd cd {space} then use normal vi commands. then the log identifies the column! Killing a process Find the pid of the process.doc Page 16 of 20 . all contents and don’t prompt done Display files that match ‘a’ OR ‘b’: Create a file with the text to search for.1 {–k3} {input filename} where –t defines the delimiter. clears all locks UNLOCK USER 61234 ALL for one specific user (Userno column in LIST. / to find. then reverse the process: tar –cvf {newname}. or pick out relevant columns: select * from v$sqlarea order by cpu_time desc Usefull columns include SQL_TEXT. tables. All and DBA are as above. views. BUFFER_GETS. DISK_READS. packages.tar Useful ORACLE views There are a number of views you can look at (in the SYS schema).DataStage Scribble Sheet Display only directories in the current directory: ls –l | grep ‘^d’ ^d means first character is a d Displays lines in a file that do NOT have 0 to 9 as the first character: grep –v ‘^[0-9]’ filename Displays directories excluding DataStage ones: ls -l |grep -v 'RT_' |grep -v 'DS_'|grep ^d|pg Search for a particular text string in a mass of files: find .. –exec grep “string to find” ‘{}’ \. ELAPSED_TIME./HashFiles/ | grep –v ‘_D’ | grep –v ‘LOG’ | pg Displays lines containing abc OR xyz: grep -E "abc|xyz" file1 Count number of rows in INPUTFILE_FF starting with ‘20’: grep ^20 /usr/dstage/data/Basel2/work/DA1842/INPUTFILE_FF | wc -l df –k . functions. USER_OBJECTS .. -print The argument '{}' inserts each found file into the grep command line. All and DBA as above.tar.g. ORACLE performance For stats re processes that are running use. sequences etc) in your schema. The \.tar … uncompress {newname}. EXECUTIONS. ROWS_PROCESSED 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818. USER_TAB_COLUMNS . ALL_TABLES shows the tables that you can access.tar *{sel criteria}* compress {newname}. Display all hash files excluding dictionary levels and log files: ls –l . argument indicates the exec command line has ended.shows all the tables owned by the schema you are logged in as.shows all the columns on all the tables in your schema.Z tar –xvf {newname}.shows all database objects (e. CPU_TIME. show used and free space in kilobytes Save and compress a bunch of files. USER_TABLES .doc Page 17 of 20 . DBA_TABLES shows all tables. <=.B. SELECT COL1.projectid and p.00 AND ORDER_NUM > 200200 ORDER BY AMOUNT. 22). 13. AMOUNT FROM ORDERS WHERE AMOUNT > 2500. AMOUNT FROM ORDERS WHERE REP_OFFICE IN (11. <.doc Page 18 of 20 . AMOUNT FROM ORDERS WHERE AMOUNT BETWEEN 20000. p. C. – show colums in the table SELECT (*) FROM {tablename}. <>. SELECT NAME FROM SALESPREPS WHERE REP_OFFICE IS {NOT} NULL.00 AND 29999.regionname.projectid = a.regiontype = 'L' and p.projectver = r.regioncode left outer join projectpartyaddress a on p. DESC {tablename}. (SALES – QUOTA) FROM SALESREPS WHERE SALES < QUOTA. COL2.projectver and role = 'appl' left outer join address d on a.projectver = a. CREDIT_LIMIT FROM CUSTOMERS WHERE COMPANY_NAME LIKE ‘Smiths_n %’.C.…etc….cphnumber from project p left outer join projectregions r on p. >.STH_Bookings B WHERE C. Example using left outer join /*ECS applications*/ select g.regioncode = g.projectid = r.addressid = d. SELECT ORDER_NUM. SELECT ORDER_NUM. Comparisons allowed are: =. SELECT NAME. I've also added a TRUNC around the TIMECREATED to disregard the time element of the column when doing the evaluation.projectid. SELECT TABLEID. ORDER_DATE. Underscore ‘_’ matches 1 character.99 ORDER BY ORDER_NUM.projectid and p.Account_Number. TIMECREATED.projectver left outer join refgoregion g on r. QUOTA.Account_Number = B.Account_Number.addressid where schemeid = 'ECS' and r. Allocating table names a letter and using that: SELECT C. Select customer called Smithson … or Smithsen … SELECT COMPANY.Lessee_Name. d. JOBID FROM data_control_table WHERE scheme = 'Aesis' AND TRUNC(TIMECREATED) = TO_DATE ('08-DEC-03'. SELECT A.projectid not in (select projectid from projectevent where eventcode in ('APP'. Also have the NOT operand.'RJC')) Example using TO_DATE and TRUNC You need the TO_DATE to tell oracle that it's a date value and what the format is to use.'ARI'. >= /* used to include comments on SQL query page */ Simple SQL commands: SELECT TABLE_NAME FROM USER_TABLES.Insurance_Indicator FROM STH_Customers C. tableid 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818. AMOUNT. B. COL3 FROM {tablename}.projectver. 'DD-MON-YY') ORDER BY timecreated.DataStage Scribble Sheet SQL Wildcard character is ‘%’ (matches zero or more characters). p. SALES. 00 WHERE CUST_NUM = 2125. SUM(AMOUNT) FROM ORDERS GROUP BY REP. CREDIT_LIMIT) VALUES (‘Acme Industries’. CREATE TABLE HCLJEAN AS SELECT * FROM HUB_CODE_LOOKUPS DROP TABLE PRODUCTS.doc Page 19 of 20 . grant select on dcspr_request to public REVOKE INSERT Revoke permission to insert ON CUSTOMERS FROM {Userid}. REVOKE ALL Revoke all privileges ON CUSTOMERS FROM {Userid}.8.29) GROUP BY SUBSTR(APDATE. PRODUCT_ID CHAR(5). SELECT COUNT(AMOUNT) FROM ORDERS WHERE AMOUNT > 25000. COUNT(SUBSTR(APDATE. Other commands (briefly): LOWER(‘example OF-TEXT’) = example of-text UPPER(‘example OF-TEXT’) = EXAMPLE OF-TEXT INITCAP(‘example OF-TEXT’) = Example Of-Text GRANT INSERT Grant permission to insert ON CUSTOMERS TO {Userid}.2)) FROM COMPOSITE_FWPS_AGREEMENT WHERE SIND IN (8. CUST. Clear table Type of Join Operation Description 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818. Total amount of orders by customer (group orders by customer and total): SELECT CUST. 2125. UPDATE CUSTOMERS SET CREDIT_LIMIT = 75000.2). Average size of an order in the database: SELECT AVG(AMOUNT) FROM ORDERS WHERE CUST = 213423.00). SELECT REP. SELECT Grant permission to update or select ON CUSTOMERS TO {Userid}.8. MAX(QUOTA) FROM SALESREPS. DESC VARCHAR(30). Delete table TRUNCATE TABLE PRODUCTS. PRICE MONEY. GRANT UPDATE. CUST.2). 109.12.DataStage Scribble Sheet Summarising data: SELECT AVG(100 * (SALES / QUOTA)) FROM SALESREPS. CUST_REP. QTY_ON_HAND INTEGER). CUST_NUM. CREATE TABLE PRODUCTS Define a new table (MFR_ID CHAR(3).00. SUM(AMOUNT) FROM ORDERS GROUP BY CUST. Updating data in the database: Inserting data into the database: INSERT INTO CUSTOMERS (COMPANY. For 3 particular sinds count and group by year (APDATE=DD^MMM^YY): SELECT SUBSTR(APDATE. 25000. Delting data from the database: DELETE FROM CUSTOMERS WHERE CUST_NUM = 2125. Minimum and maximum size of an order in the database: SELECT MIN(QUOTA).8. Count how many orders in the database (counts number of rows): SELECT COUNT(*) FROM ORDERS. Count distinct number of titles: SELECT COUNT(DISTINCT TITLE) FROM SALESREPS. g. the GROUP BY function does include a grouping by the NULL value for any term. value_to_replace_NULL_with) e. In 1.canberra. If you want to include nulls in a different way you can also use the function NVL(field_which_could_be_null.DataStage Scribble Sheet Pure Iner Join Complete Set Right and Left Only Left Outer Join Right Outer Join Left Only Right Only A AND B A OR B A NOR B A B A NOT B B NOT A Merges only those rows with the same key values in both input files Merges all rows from both files Merges all rows from both files except those rows with the same key values Merges all rows from the first file (A) with rows from the second file (B) with the same key Merges all rows from the second file (B) with rows from the first file (A) with the same key Merges all rows from the first file (A) except rows with the same key in the second file (B) Merges all rows from the second file (B) except rows with the same key in the first file (A) NULL handling In a WHERE clause any condition that does not explicitly mention NULL values automatically fails if one of the values is NULL.au/~sam/whp/sql-guide. If you want to cater for nulls then you should put:where SALMONELLA_STATUTORY_SPP_FLAG IS NULL (it's quicker to write too). so the NULL fields are included and counted. 'X') = 'X' Odds and sods • SQL guide web site: http://www. WHERE NVL(SALMONELLA_STATUTORY_SPP_FLAG.doc Page 20 of 20 . Hence SALMONELLA_STATUTORY_SPP_FLAG <> 'T' / 'F' will automatically yield FALSE if SALMONELLA_STATUTORY_SPP_FLAG is a null.edu.html 16/05/11 /opt/scribd/conversion/tmp/scratch6111/58361818.