Professional Documents
Culture Documents
1. Select Start > Programs > IBM InfoSphere Information Server > IBM InfoSphere DataStage and QualityStage Designer. 2. In the Attach window, type your user name and password. 3. Select the tutorial project from the Project list, and then click OK. The Designer client opens and displays the New window. 4. Click Cancel to close the New window because you are opening an existing job and not creating a new job or other object. 5. In the repository tree, open the Tutorial folder double-click the samplejob job. All of the objects that you need for the tutorial are in this folder. The job opens in the Designer client display area. The following figure shows the Designer client with the samplejob job open. The Tutorial folder is shown in the repository tree. Figure 1. Designer client
Lesson checkpoint
In this lesson, you opened your first job. You learned the following tasks: y How to start the Designer client y How to open a job y Where to find the tutorial objects in the repository tree
In this lesson, you view the sample job to understand its components. You compile the job to prepare it to run on your system. About this task The sample job has a Sequential File stage to read data from the flat file and a Data Set stage to write data to the staging area. The two stages are joined by a link. The data that will flow between the two stages on the link was defined when the job was designed. When the job is run, the data will flow down this link.
Lesson checkpoint
In this lesson, you explored a simple data extraction job that reads data from a file and writes it to a staging area. You learned the following tasks: y How to open stage editors y How to view the data that a stage represents y How to compile a job so that it is ready to run
You run the job from the Director client. The Director client is the operating console. You use the Director client to run and troubleshoot jobs that you are developing in the Designer client. You also use the Director client to run fully developed jobs in the production environment. You use the job log to debug any errors you receive when you run the job.
2. Select the sample job in the right pane of the Director client, and select Job > Run Now. 3. In the Job Run Options window, specify the path of the project folder (for example, C:\IBM\InformationServer\Server\Projects\Tutorial and clickRun. The job status changes to Running. 4. When the job status changes to Finished, select View > Log. 5. Examine the job log to see the type of information that the Director client reports as it runs a job. The messages that you see are either control or information type. Jobs can also have Fatal and Warning messages. The following figure shows the log view of the job. Figure 2. Job log
Lesson checkpoint
In this lesson you ran the sample job and looked at the results. You learned the following tasks: y How to start the Director client from the Designer client y How to run a job and look at the log file y How to view the data written by the job
Before you begin If you closed the Designer client after completing module 1, you will need to start the Designer client again. About this task You create a parallel job and save it to a new folder in the Tutorial folder in the repository tree. To create a job: Procedure 1. In the Designer client, select File > New. 2. In the New window, select the Jobs folder in the left pane and then select the parallel job icon in the right pane. 3. Click OK. A new empty job design window opens in the design area. 4. Select File > Save. 5. In the Save Parallel Job As window, right-click on the Tutorial folder and select New > Folder from the shortcut menu. 6. Type in a name for the folder, for example, My Jobs then move to the Item name field. 7. Type in the name of the job in the Item name field. Call the job populate_cc_spechand_lookupfiles. 8. Check that the Folder path field contains the path \Tutorial\My Jobs, then click Save. You have created a new parallel job named populate_cc_spechand_lookupfiles and saved it in the folder Tutorial\My Jobs in the repository.
Lesson checkpoint
In this lesson you created a job and saved it to a specified place in the repository. You learned the following tasks: y How to create a job in the Designer client. y How to name the job and save it to a folder in the repository tree.
specifies the customer's language. You are designing a job that reads the comma-separated file and writes the contents to a lookup table in a lookup file set. This table will be used by a subsequent job when it populates a dimension table.
Stage or link
country_codes
country_code_lookup
Link
country_codes_data
8. Always use specific names for your stages and links rather than the default names assigned by the Designer client. Using specific names make your job designs easier to document and easier to maintain. 9. Select File > Save to save the job. Your job design should now look something like the one shown in this figure: Figure 1. Job design
Specifying properties and column metadata for the Sequential File stage
You will now edit the first of the stages that you added to specify what the stage does when you run the job. You will also specify the column metadata for the data that will flow down the link that joins the two stages. About this task To edit the stages and add properties and metadata: Procedure 1. Double-click the country_codes Sequential File stage to open the stage editor. The editor opens in the Properties tab of the Output page. 2. Select the File property under the Source category. 3. In the File field, type the path name for your project folder (where the data files were copied when the tutorial was set up) and add the name CustomerCountry.csv (for example C:\IBM\InformationServer\Server\Projects\Tutorial\CustomerCountry .csv), and then press enter. (You can browse for the path name if you prefer, click the browse button on the right of the File field.) You specified the name of the comma-separated file that the stage reads when the job runs. 4. Select the First Line is Column Name property under the Options category. 5. Click the down arrow next to the First Line is Columns Names field and select True from the list. The row that contains the column names is dropped when the job reads the file. 6. Click the Format tab. 7. In the record-level category, select the Record delimiter string property from the Available properties to add.
8. Select DOS format from the Record delimiter string list. This setting ensures that the file can be read when the engine tier is installed on a UNIX or Linux computer. 9. Click the Columns tab. Because the CustomerCountry.csv file contains only three columns, type the column definitions into the Columns tab. (If a file contains many columns, it is less time consuming and more accurate to import the column definitions directly from the data source.) Note that column names are case-sensitive, so use the case in the instructions. 10. Double-click the first line of the table. Fill in the fields as follows:
Table 1. Column definition Column Name CUSTOMER_ NUMBER Key Yes SQL Type Char 7 Length Description Key column for the look up - the customer identifier
11. You will use the default values for the remaining fields. 12. Add two more rows to the table to specify the remaining two columns and fill them in as follows:
Table 2. Additional column definitions Column Name COUNTRY LANGUAGE Key No No SQL Type Char Char 2 2 Length Description The code that identifies the customer's country The code that identifies the customer's language
13. Your Columns tab should look like the one in the following figure (if you have National Language Support installed, there is an additional field named Extended): 14. Figure 2. Columns tab
15. 16. Click the Save button to save the column definitions that you specified as a table definition object in the repository. The definitions can then be reused in other jobs. 17. In the Save Table Definition window, enter the following information:
Option Description
Saved
CustomerCountry.csv
Table/file name
country_codes_data
Short description
Long description
18. Click OK to specify the locator for the table definition. The locator identifies the table definition. 19. In the Save Table Definition As window, save the table definition in the Tutorial folder and name it country_codes_data.
20. Click the View Data button and click OK in the Data Browser window to use the default settings. The data browser shows you the data that the CustomerCountry.csv file contains. Since you specified the column definitions, the Designer client can read the file and show you the results. 21. Close the Data Browser window. 22. Click OK to close the stage editor. 23. Save the job. Notice that a small table icon has appeared on the Country_codes_data link. This icon shows that the link now has metadata. You have designed the first part of your job.
Specifying properties for the Lookup File Set stage and running the job
In this part of the lesson, you configure the next stage in your job. You already specified the column metadata for data that will flow down the link between the two stages, so there are fewer properties to specify in this task. About this task To configure the Lookup File Set stage: Procedure 1. Double-click the country_code_lookup Lookup File Set stage to open the stage editor. The editor opens in the Properties tab of the Input page. 2. Select the Lookup Keys category; then double-click the Key property in the Available Properties to add area. 3. In the Key field, click the down arrow and select CUSTOMER_NUMBER from the list and press enter. You specified that the CUSTOMER_NUMBER column will be the lookup key for the lookup table that you are creating. 4. Select the Lookup File Set property under the Target category. 5. In the Lookup File Set field, type the path name for the lookup file set that the stage will create, (for example,C:\IBM\InformationServer\Server\Projects\Tutorial\countrylookup.f s) and press enter. 6. Click OK to save your property settings and close the Lookup File Set stage editor. 7. Save the job and then compile and run the job by using the techniques that you learned in Lesson 1. You have now written a lookup table that can be used by another job later on in the tutorial.
Lesson checkpoint
You have now designed and run your very first job. You learned the following tasks: y How to add stages and links to a job y How to set the stage properties that determine what the stage will do when you run the job y How to specify column metadata for the job and to save the column metadata to the repository for use in other jobs
Ensure that your job named populate_cc_spechand_lookupfiles is open and active. About this task In this part of the lesson, you are consolidating the job design skills that you learned and loading the column metadata from the table definition that you imported earlier. Procedure 1. Add a Sequential file stage and a Lookup File Set stage to your job and link them together. Position them under the stages and link that you added earlier in this lesson. 2. Rename the stages and link as follows:
Stage or Link Name
Sequential File
special_handling
Lookup File
special_handling_lookup
Link
special_handling_data
3. Your job design should now look like the one shown in this figure: 4. Figure 1. Job design
5. 6. Open the stage editor for the special_handling Sequential File stage and specify that it will read the file SpecialHandling.csv and that the first line of this file contains column names. 7. Click the Format tab.
8. In the record-level category, select the Record delimiter string property from the Available properties to add. 9. Select DOS format from the Record delimiter string list. This setting ensures that the file can be read when the engine tier is installed on a UNIX or Linux computer. 10. Click the Columns tab. 11. Click Load. You load the column metadata from the table definition that you previously saved as an object in the repository. 12. In the Table Definitions window, browse the repository tree to the folder where you stored the SpecialHandling.csv column definitions. 13. Select the SpecialHandling.csv table definition and click OK. 14. In the Selected Columns window, ensure that all of the columns appear in the Selected columns list and click OK. The column definitions appear in the Columns tab of the stage editor. 15. Close the Sequential File stage editor. 16. Open the stage editor for the special_handling_lookup stage. 17. Specify a path name for the destination file set and specify that the lookup key is the SPECIAL_HANDLING_CODE column then close the stage editor. 18. Save, compile, and run the job.
Lesson checkpoint
You have now added to your job design and learned how to import the metadata that the job uses. You learned the following tasks: y How to import column metadata directly from a data source y How to load column metadata from a definition that you saved in the repository
Job parameters
Sometimes, you want to specify information when you run the job rather than when you design it. In your job design, you can specify a job parameter to represent this information. When you run the job, you are then prompted to supply a value for the job parameter. You specified the location of four files in the job that you designed in Lesson 2.3. In each part of the job, you specified a file that contains the source data and a file to write the lookup data set to. In this lesson, you will replace all four file names with job parameters. You will then supply the actual path names of the files when you run the job. You will save the definitions of these job parameters in a parameter set in the repository. When you want use the same job parameters in a job later on in this tutorial, you can load them into the job design from the parameter set. Parameter sets enable the same job parameters to be used by different jobs.
Ensure that the job named populate_cc_spechand_lookupfile that you designed in Lesson 2.3 is open and active. Procedure 1. Select Edit > Job Properties. 2. In the Job Properties window, click the Parameters tab. 3. Double-click the first line of the grid to add a new row. 4. In the Parameter name field, type country_codes_source. 5. In the Prompt field, type path name for the country codes file. 6. In the Type field, select a data type of Pathname. 7. In the Help Text field, type Enter the path name for the comma-separated file that contains the country code definitions. 8. Repeat steps 3-7 to define three more job parameters containing the following entries:
Table 1. Job parameters Parameter Name country_codes_lookup Prompt path name for the country codes lookup file set Type path name Help text
Enter the path na the file set for country code look table
special_handling_source
path name
Enter the path na the comma-separat file that contain special handling definitions
special_handling_lookup
path name
Enter the path na the file set for special handling table
9. The Parameters tab of the Job Properties window should now look like the one in the following figure: 10. Figure 1. Parameters tab
11. 12. Click OK to close the Job Properties window. 13. Click File > Save to save the job.
Table 2. Job parameters to be added to job Stage country_codes_lookup stage special_handling stage special_handling_lookup stage Property Lookup file set File Lookup file set Parameter name country_codes_lookup special_handling_source special_handling_lookup
Lesson checkpoint
You defined job parameters to represent the file names in your job and specified values for these parameters when you ran the job. You learned the following tasks: y How to define job parameters y How to add job parameters in your job design y How to specify values for the job parameters when you run the job
Parameter sets
You use parameter sets to define job parameters that you are likely to reuse in other jobs. Whenever you need this set of parameters in a job design, you can insert them into the job properties from the parameter set. You can also define different sets of values for each parameter set. These parameter sets are stored as files in the InfoSphere DataStage server installation directory and are available to use in your job designs or when you run jobs that use these parameter sets. If you make any changes to a parameter set object, these changes are reflected in job designs that use this object until the job is compiled. The parameters that a job is compiled with are available when the job is run. However, if you change the design after the job is compiled, the job will link to the current version of the parameter set. You can create parameter sets from existing job parameters, or you can specify the job parameters as part of the task of creating a new parameter set.
10. Click OK, specify a repository folder in which to store the parameter set, and then click Save. 11. The Designer client asks if you want to replace the selected parameters with the parameter set that you have just created. Click No. 12. Click OK to close the Job Parameters window. 13. Save the job. You created a parameter set that is available for another job that you will create later in this tutorial. The current job continues to use the individual parameters rather than the parameter set.
Lesson checkpoint
You have now created a parameter set. You learned the following tasks: y How to create a parameter set from a set of existing job parameters y How to specify a set of default values for the parameters in the parameter set
The job will also specify some stricter data typing for the remaining columns. Stricter data typing helps to impose quality controls on the data that you are processing. Finally, the job applies a function to one of the data columns to delete space characters that the column contains. This transformation job prepares the data in that column for a later operation. The transformation job that you are designing uses a Transformer stage, but there are also several other types of processing stages available in the Designer client that can transform data. For example, you can use the Modify stage in your job, if you want to change only the data types in a data set. Several of the processing stages can drop data columns as part of their processing. In the current job, you use the Transformer stage because you require a transformation function that you can customize. Several functions are available to use in the Transformer stage.
o o o o o o o o o o
CUST_NAME ADDR_1 ADDR_2 CITY REGION_CODE ZIP TEL_NUM REVIEW_MONTH SETUP_DATE STATUS_CODE
3. Drag these columns from the upper left pane to the stripped_bill_to link in the upper right pane of the stage editor. You are specifying that only these columns will flow through the Transformer stage when the job is run. The remaining columns will be dropped. 4. In the stripped_bill_to column definitions at the bottom of the right pane, edit the SQL type and length fields for your columns as specified in the following table:
Table 1. Column definitions Column CUSTOMER_NUMBER CUST_NAME ADDR_1 ADDR_2 CITY REGION_CODE ZIP TEL_NUM REVIEW_MONTH SETUP_DATE STATUS_CODE Char VarChar VarChar VarChar VarChar Char VarChar VarChar VarChar VarChar Char SQL Type 7 30 30 30 30 2 10 10 2 12 1
Length
5. By specifying stricter data typing for your data, you will be able to better diagnose inconsistencies in your source data when you run the job. 6. Double-click the Derivation field for the CUSTOMER_NUMBER column in the stripped_bill_to link. The expression editor opens. 7. In the expression editor, type the following text: trim(full_bill_to.CUSTOMER_NUMBER,' ','A'). The text specifies a function that deletes all the space characters from the CUSTOMER_NUMBER column on the full_bill_to link before writing it to the CUSTOMER_NUMBER column on the stripped_bill_to link. Your Transformer stage editor should look like the one in the following figure:
8. Click OK to close the Transformer stage editor. 9. Open the stage editor for the int_GlobalCoBillTo Data Set stage and go to the Columns tab of the Input page. Notice that the stage editor has acquired the metadata from the stripped_bill_to link.
Lesson checkpoint
In this lesson you learned how to design and configure a transformation job. You learned the following tasks: y How to configure a Transformer stage y How to link stages by using a different method for drawing links. y How to load column metadata into a link, by using a drag-and-drop operation. y How to run a job from within the Designer client and monitor the performance of the job.
You can configure Lookup stages to search for data in a Lookup file set, or they can search for data in a relational database. The job will look up the data in a reference table in a Lookup File Set stage that was created in Lesson 2.2 of this tutorial. When you use lookup file sets, you must specify the lookup key column when you define the file set. You defined the key columns for the lookup tables that you used in this lesson when you created the file sets in Module 2.
1. Double-click the Lookup_Country Lookup stage to open the Lookup stage editor. The Lookup stage editor is similar in appearance to the Transformer stage editor. 2. Click the title bar of the stripped_bill_to link in the left pane and drag it over to the Column Name column of the country_code link in the right pane. When the cursor changes shape, release the mouse button. All of the columns from the stripped_bill_to link appear in the country_code link. 3. Select the Country column in the Country_Reference link and drag it to the country_code link. The result of copying the columns from the Country_Reference link to the country_code link is that whenever the value of the incoming CUSTOMER_NUMBER column matches the value of the CUSTOMER_NUMBER column of the lookup table, the corresponding Country column will be added to that row of data. The stage editor looks like the one in the following figure: Figure 2. Lookup stage editor
4. Double-click the Condition bar in the Country_Reference link. The Lookup Stage Conditions window opens. Select the Lookup Failure field and select Continue from the list. You are specifying that, if a CUSTOMER_NUMBER value from the stripped_bill_to link does not match any CUSTOMER_NUMBER column values in the reference table, the job continues to the next CUSTOMER_NUMBER column. 5. Close the Lookup stage editor. 6. Open the temp_dataset Data Set stage and specify a file name for the data set. 7. Save, compile and run the job. The Job Run Options window displays all the parameters in the parameter set. 8. In the Job Run Options window, select lookupvalues1 from the list next to the parameter set name. The parameters values are filled in with the path names that you specified when you created the parameter set. 9. Click Run to run the job and then click View Data in the temp_dataset stage to examine the results.
Lesson checkpoint
With this lesson, you started to design more complex and sophisticated jobs. You learned the following tasks: y How to copy stages, links, and associated configuration data between jobs. y How to combine data in a job by using a Lookup stage
3. Double-click the Lookup_Country Lookup stage to open the Lookup stage editor. 4. Double-Click the Condition bar in the country_reference link to open the Lookup Stage Conditions window. 5. In the Lookup Stage Conditions window, select the Lookup Failure field and select Reject from the list. Close the Lookup stage editor. This step specifies that, whenever a row from the stripped_bill_to link has no matching entry in the country code lookup table, the row is rejected and written to the Rejected_Rows Sequential File stage. 6. Edit the Rejected_Rows Sequential File stage and specify a path name for the file that the stage will write to (for example,c:\tutorial\rejects.txt). This stage derives the column metadata from the Lookup stage, and you cannot alter it. 7. Save, compile the CleansePrepare job, and run the job. 8. Open the Rejected_Rows Sequential File stage editor and click View Data to look at the rows that were rejected.
Lesson checkpoint
You learned the following tasks: y How to add a reject link to your job y How to configure the Lookups stage so that it rejects data where a lookup fails
About this task In this lesson, you will further transform your data to apply some business rules and perform another lookup of a reference table. In the sample bill_to data, one of the columns is overloaded. The SET_UP data column can contain a special handling code as well as the date that the account was set up. The transformation logic that is being added to the job extracts this special handling code into a separate column. The job then looks up the text description corresponding to the code from the lookup table that you populated in Lesson 2 and adds the description to the output data. The transformation logic also adds a row count to the output data.
d. Place the Data Set stage immediately to the right of the Lookup_Spec_Handling Lookup stage and name the Data Set stage Target. 2. Link the stages: a. Link the Business_Rules Transformer stage to the Lookup_Spec_Handling Lookup stage and name the link with_business_rules. b. Link the Special_Handling_Lookup Lookup File Set stage to the Lookup_Spec_Handling Lookup stage and name the linkspecial_handling. c. Link the Lookup_Spec_Handling Lookup stage to the Target Data Set stage and name the link finished_data.
3. Drag the arrowhead end of the country_code link and attach it to the Business_Rules Transformer stage. The Temp_Dataset stage is not required for this job, therefore you can remove it. 4. Delete the Temp_Dataset Data Set stage. 5. Drag the Business_Rules Transformer stage down so that it aligns horizontally with the Lookup stages. Your CleansePrepare job should now resemble the one in the following figure: Figure 1. Job design
1. Open the Business_Rules Transformer stage editor, and click the Show/Hide Stage Variables
icon to display the stage variable grid in the right pane. You will define some stage variables later in this procedure.
2. Select the following columns in the country_code input link and drag them to the with_business_rules output link: o o o o o o o o CUSTOMER_NUMBER CUST_NAME ADDR_1 ADDR_2 CITY REGION_CODE ZIP TEL_NUM
3. In the metadata area for the with_business_rules output link, add the following new columns:
Table 1. Column definitions
Column name SOURCE RECNUM SETUP_DATE SPECIAL_HANDLING_ CODE Char Char Char Integer
SQL Type 10 10 10 10
Nullable
4. The new columns appear in the graphical representation of the link, but are highlighted in red because they do not yet have valid derivations. 5. In the graphical area, double-click the Derivation field of the SOURCE column. 6. In the expression editor, type 'GlobalCo':. Position your mouse pointer immediately to the right of this text, right-click and select Input Column from the menu. Then select the COUNTRY column from the list. When you run the job, the SOURCE column for each row will contain the two-letter country code prefixed with the text GlobalCo, for example, GlobalCoUS. 7. In the Transformer stage editor toolbar, click the Stage Properties tool on the far left. The Transformer Stage Properties window opens. 8. Click the Variables tab and, by using the techniques that you learned for defining table definitions, add the following stage variables to the grid:
Table 2. Stage variables Name xtractSpecialHandling TrimDate Char VarChar SQL Type 1 10 Precision
9. When you close the Properties window, these stage variables appear in the Stage Variables area above the with_business_rules link. 10. Double-click the Derivation fields of each of the stage variables in turn and type the following expressions in the expression editor:
Table 3. Derivations Stage variable xtractSpecialHandling Expression if Len (country_code.SETUP_DATE) < 2 Then country_code.SETUP_DATE Else Field (country_code.SETUP_DATE,' ',2) Description
This expression checks that the SETUP_DATE column a special handling code. If it does, the value of xtractSpecialHandling is set to that code. If the colum contains a date and a code, the code is extracted and the of xtractSpecialHandling is set to that code.
TrimDate
This expression checks that the SETUP_DATE column a date. If the SETUP_DATE column does not contain a then the expression sets the value of the TrimDate varia string 01/01/0001. If the SETUP_DATE column contai then the date is extracted and the value of the TrimDate
Table 3. Derivations Stage variable Expression Description is set to a string that contains the date.
11. Select the xtractSpecialHandling stage variable and drag it to the Derivation field of the SPECIAL_HANDLING_CODE column and drop it on the with_business_rules link. A line is drawn between the stage variable and the column, and the name xtractSpecialHandling appears in theDerivation field. For each row that is processed, the SPECIAL_HANDLING_CODE column writes the current value of the xtractSpecialHandlingvariable. 12. Select the TrimDate stage variable and drag it to the Derivation field of the SETUP_DATE column and drop it on the with_business_rules link.A line is drawn between the stage variable and the column, and the name TrimDate appears in the Derivation field. For each row processed, the SETUP_DATE column writes the current value of the TrimDate variable. 13. Double-click the Derivation field of the RECNUM column and type 'GC': in the expression editor. Right-click and select System Variable from the menu. Then select @OUTROWNUM. You added row numbers to your output. Your transformer editor should look like the one in the following picture: Figure 2. Transformer stage editor
7. Select the DESCRIPTION column in the special_handling reference link and drag it to the finished_data output link (the LANGUAGE column is not used). 8. Double-click the Condition bar in the special_handling reference link to open the Lookup Stage Conditions window. Specify that the processing will continue if the lookup fails for a data row. You do not need to specify a reject link for this stage. Only a minority of the rows in the bill_to data contain a special handling code, so if the rows that do not contain a code are rejected, most of the data is rejected. 9. Specify a job parameter to represent the file that the Target Sequential File stage will write to, and add this job parameter to the stage. 10. Save, compile and run the CleansePrepare job.
Lesson checkpoint
In this lesson, you consolidated your existing skills in defining transformation jobs and added some new skills. You learned the following tasks: y How to define and use stage variables in a Transformer stage y How to use system variables to generate output column values
In this lesson, you use data connection objects to provide the information that is needed to connect to a database.
7. Click OK. 8. In the Save Data Connection As window, select the tutorial folder and click Save.
Lesson checkpoint
You learned how to create a data connection object and store the object in the repository.
The table definition is imported and appears in the tutorial folder. The table definition has a different icon from the table definitions that you used previously. This icon identifies that the table definition was imported by using a connector and is available to other projects and to other suite components.
Lesson checkpoint
You learned how to import column metadata from a database using a connector.
Connectors
Connectors are stages that you use to connect to data sources and data targets to read or write data. In the Database section of the palette in the Designer are many types of stages that connect to the same types of data sources or targets. For example, if you click the down arrow next to the ODBC icon in the palette, you can choose to add either an ODBC connector stage or an ODBC Enterprise stage to your job. If your database type supports connector stages, use them because they provide the following advantages over other types of stages: y Creates job parameters from the connector stage (without first defining the job parameters in the job properties). y Saves any connection information that you specify in the stage as a data connection object. y Reconciles data types between source and target to avoid runtime errors. y Generates detailed error information if a connector encounters problems when the job runs.
a. In the Select tables area, open the tutorial folder, browse the icons that represent your database and your schema, and select the table definition that you imported in Lesson 4.2. b. Drag the table definition to the area to the right of the repository tree. c. In the table definition, click Select All and drag all the columns to the Insert Columns area. The SQL builder page should look like the one in the following figure: Figure 2. SQL builder editor
d. Click the SQL tab to view the SQL statement; then click OK to close the SQL builder. The SQL statement is displayed in the Insert statement field, and your ODBC connector should look like the one in the following figure:
9. Click OK to close the ODBC Connector. 10. Save, compile, and run the job. You wrote the BillTo data to the tutorial database table. This table forms the bill_to dimension of the star schema that is being implemented for the GlobalCo delivery data in the business scenario that the tutorial is based on.
Lesson checkpoint
You learned how to use a connector stage to connect to and write to a relational database table. You learned the following tasks: y How to configure a connector stage y How to use a data connection object to supply database connection details y How to use the SQL builder to define the SQL statement by accessing the database.
The default configuration file is created when InfoSphere DataStage is installed. Although the system has four processors, the configuration file specifies two processing nodes. Specify fewer processing nodes than there are physical processors to ensure that your computer has processing resources available for other tasks while it runs InfoSphere DataStage jobs. This file contains the following fields: node The name of the processing node that this entry defines. fastname The name of the node as it is referred to on the fastest network in the system. For an SMP system, all processors share a single connection to the network, so the fastname node is the same for all the nodes that you are defining in the configuration file. pools Specifies that nodes belong to a particular pool of processing nodes. A pool of nodes typically has access to the same resource, for example, access to a high-speed network link or to a mainframe computer. The pools string is empty for both nodes, specifying that both nodes belong to the default pool. resource disk Specifies the name of the directory where the processing node will write data set files. When you create a data set or file set, you specify where the controlling file is called and where it is stored, but the controlling file points to other files that store the data. These files are written to the directory that is specified by the resource disk field. resource scratchdisk Specifies the name of a directory where intermediate, temporary data is stored. Configuration files can be more complex and sophisticated than the example file and can be used to tune your system to get the best possible performance from the parallel jobs that you design.
Lesson checkpoint
In this lesson, you learned how the configuration file is used to control parallel processing. You learned the following concepts and tasks: y About configuration files y How to open the default configuration file y What the default configuration file contains
In the simplest scenario, do not worry about how your data is partitioned. InfoSphere DataStage can partition your data and implement the most efficient partitioning method. Most partitioning operations result in a set of partitions that are as near to equal size as possible, ensuring an even load across your processors. As you perform other operations, you need to control partitioning to ensure that you get consistent results. For example, you are using an aggregator stage to summarize your data to get the answers that you need. You must ensure that related data is grouped together in the same partition before the summary operation is performed on that partition. In this lesson, you will run the sample job that you ran in Lesson 1. By default, the data that is read from the file is not partitioned when it is written to the data set. You change the job so that it has the same number of partitions as there are nodes defined in your system's default configuration file.
4. Click the disk icon in the toolbar to open the Data Set viewer
and click OK. 5. View the data in the data set to see its structure. 6. Close the window.
To specify round-robin partitioning: Procedure 1. Open the sample job that you used in Module 1. 2. Open the GlobalCoBillTo_ds Data Set stage editor. 3. Open the Partitioning tab of the Input page. 4. In the Partition type field, select the round-robin partitioning method. 5. Compile and run the job. 6. Return to the data set management tool and open the GlobalCo_BillTo.ds data set. You can see that the data set now has multiple data partitions. The following figure shows the data set partitions on the system. Figure 2. Data Set Management window showing multiple partitions
Lesson checkpoint
In this lesson, you learned some basics about data partitioning.
You learned the following tasks: y How to use the data set management tool to view data sets y How to set a partitioning method for a stage
7. Select the APT_CONFIG_FILE environment variable, and edit the file name in the path name under the Value column heading to point to your new configuration file. The Environment variables window should resemble the one in the following picture: Figure 2. Environment variables window
You deployed your new configuration file. Keep the Administrator client open, because you will use it to restore the default configuration file at the end of this lesson.
4. In the Designer client, open the data set management tool and open the GlobalCo_BillTo.ds data set. You see that the data is in a single partition because the new configuration file specifies only one processing node. 5. Reopen the Administrator client to restore the default configuration file by editing the path for the APT_CONFIG_FILE environment variable to point to the default.apt file.
Lesson checkpoint
You learned how to create a configuration file and use it to alter the operation of parallel jobs. You learned the following tasks: y How to create a configuration file based on the default file. y How to edit the configuration file. y How to deploy the configuration file