IDQ91 HF1 Analyst LG 201107 WM

Informatica Data Quality
Analyst 9.X
Lab Guide
Version IDQ91_HF1_AT_201107
Informatica Data Quality, Analyst Lab Guide
Version 9.1 HF1
July 2011
Copyright (c) 20012010 Informatica Corporation.
All rights reserved.
This software and documentation contain proprietary information of Informatica Corporation and are provided under a license
agreement containing restrictions on use and disclosure and are also protected by copyright law. Reverse engineering of the
software is prohibited. No part of this document may be reproduced or transmitted in any form, by any means (electronic,
photocopying, recording or otherwise) without prior consent of Informatica Corporation. This Software may be protected by U.S.
and/or international Patents and other Patents Pending.
Use, duplication, or disclosure of the Software by the U.S. Government is subject to the restrictions set forth in the applicable
software license agreement and as provided in DFARS 227.7202-1(a) and 227.7702-3(a) (1995), DFARS 252.227-7013(c)(1)(ii)
(OCT 1988), FAR 12.212(a) (1995), FAR 52.227-19, or FAR 52.227-14 (ALT III), as applicable.
The information in this product or documentation is subject to change without notice. If you find any problems in this product or
documentation, please report them to us in writing.
Informatica, PowerCenter, PowerCenterRT, PowerCenter Connect, PowerCenter Data Analyzer, PowerExchange, PowerMart,
Metadata Manager, Informatica Data Quality, Informatica Data Explorer, Informatica Complex Data Exchange and Informatica On
Demand Data Replicator are trademarks or registered trademarks of Informatica Corporation in the United States and in jurisdictions
throughout the world. All other company and product names may be trade names or trademarks of their respective owners.
Portions of this software and/or documentation are subject to copyright held by third parties, including without limitation: Copyright
Sun Microsystems. All rights reserved. Copyright Platon Data Technology GmbH. All rights reserved. Copyright Melissa Data
Corporation. All rights reserved. Copyright 1995-2006 MySQL AB. All rights reserved
This product includes software developed by the Apache Software Foundation (http://www.apache.org/). The Apache Software is
Copyright 1999-2006 The Apache Software Foundation. All rights reserved.
ICU is Copyright (c) 1995-2003 International Business Machines Corporation and others. All rights reserved. Permission is hereby
granted, free of charge, to any person obtaining a copy of the ICU software and associated documentation files (the Software), to
deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, and/or
sell copies of the Software, and to permit persons to whom the Software is furnished to do so.
ACE(TM)and TAO(TM), are copyrighted by Douglas C. Schmidt and his research group at Washington University, University of
California, Irvine, and Vanderbilt University, Copyright (c) 1993-2006, all rights reserved.
Tcl is copyrighted by the Regents of the University of California, Sun Microsystems, Inc., Scriptics Corporation and other parties. The
authors hereby grant permission to use, copy, modify, distribute, and license this software and its documentation for any purpose.
InstallAnywhere is Copyright Macrovision (Copyright 2005 Zero G Software, Inc.) All Rights Reserved.
Portions of this software use the Swede product developed by Seaview Software (www.seaviewsoft.com).
This product includes software developed by the JDOM Project (http://www.jdom.org/). Copyright 2000-2004 Jason Hunter and
Brett McLaughlin. All rights reserved.
This product includes software developed by the JFreeChart project (http://www.jfree.org/freechart/). Your right to use such
materials is set forth in the GNU Lesser General Public License Agreement, which may be found at
http://www.gnu.org/copyleft/lgpl.html. These materials are provided free of charge by Informatica, as is, without warranty of any
kind, either express or implied, including but not limited to the implied warranties of merchantability and fitness for a particular
purpose.
This product includes software developed by the JDIC project (https://jdic.dev.java.net/). Your right to use such materials is set forth
in the GNU Lesser General Public License Agreement, which may be found at http://www.gnu.org/copyleft/lgpl.html. These materials
are provided free of charge by Informatica, as is, without warranty of any kind, either express or implied, including but not limited to
the implied warranties of merchantability and fitness for a particular purpose.
This product includes software developed by lf2prod.com (http://common.l2fprod.com/). Your right to use such materials is set forth
in the Apache License Agreement, which may be found at http://www.apache.org/licenses/LICENSE-2.0.html.
DISCLAIMER: Informatica Corporation provides this documentation as is without warranty of any kind, either express or implied,
including, but not limited to, the implied warranties of non-infringement, merchantability, or use for a particular purpose. Informatica
Corporation does not warrant that this software or documentation is error free. The information provided in this software or
documentation may include technical inaccuracies or typographical errors. The information in this software and documentation is
subject to change at any time without notice.
Part Number: IDQ-LBG-90100-000
Informatica Data Quality Analyst 9.1 HF1 iii

TABLE OF CONTENTS
INTRODUCTION AND OVERVIEW .................................................................................................................. 1
THE BUSINESS CASE .................................................................................................................................................. 1

DATA QUALITY PROJECTS - WHERE TO START? ................................................................................................... 1
UNIT 1: IMPORTING FLAT FILES .................................................................................................................... 1
EXERCISE 1.1: GETTING STARTED IN THE ANALYST TOOL ............................................................................... 1

EXERCISE 1.2: IMPORTING THE EU_CUSTORD FILE USING BROWSE AND UPLOAD ........................................ 3
EXERCISE 1.3: IMPORT US_CUSTORD.CSV FLAT FILE DEFINITION .................................................................... 6
UNIT 2 LAB A: PROFILING, RULES AND COLLABORATION ................................................................... 9
EXERCISE 2.1: COLUMN PROFILING .................................................................................................................. 10

EXERCISE 2.2: CREATE A FILTER AND APPLY TO A PROFILE ........................................................................... 12
EXERCISE 2.3: REVIEW PROFILING RESULTS, ADD COMMENTS AND EXPORT DRILLDOWNS ........................ 15
TAGS: ....................................................................................................................................................................... 21
EXERCISE 2.4: CREATE AND ASSIGN A NEW TAG.............................................................................................. 21
EXERCISE 2.5: ASSIGNING TAGS ........................................................................................................................ 22
EXERCISE 2.6: CREATE A CUSTOM PROFILE ON THE DATA OBJECT US_CUSTOMERORDERS .................... 23
UNIT 2 LAB B: RULES ............................................................................................................................................. 28
PROFILE RULES ........................................................................................................................................................ 28
APPLYING PREBUILT RULES US_CUSTOMERORDERS .......................................................................................... 28
EXERCISE 2.7: PRE-BUILT RULE - EMAIL_VALIDATION .................................................................................. 28
EXERCISE 2.8: PRE-BUILT RULES ISNUMERIC ............................................................................................... 32
EXERCISE 2.9: APPLY PRE-BUILT RULES: RULE_COMPLETENESS_MULTI_PORT......................................... 34
CREATE AND APPLY CUSTOM CLEANSING RULES EU_CUSTOMERORDERS ........................................................ 36
EXERCISE 2.10: CREATE A RULE FOR COLUMN ADDR4 (ZIP CODE) .............................................................. 36
EXERCISE 2.11: CREATE A RULE TO REFORMAT THE DATE COLUMNS ........................................................... 39
Editing the output..................................................................................................................................................... 41
Editing the Rule: ...................................................................................................................................................... 42
EXERCISE 2.12: APPLY THE RULE TO THE SHIP_DATE FIELD ....................................................................... 43
EXERCISE 2.13: TAG THE CUSTOM RULES ........................................................................................................ 43
EXERCISE 2.14: SEARCHING FOR ITEMS THAT HAVE BEEN TAGGED .............................................................. 44
UNIT 3: REFERENCE TABLE MANAGEMENT ............................................................................................ 47
EXERCISE 3.1: CREATE A REFERENCE TABLE .................................................................................................. 47

EXERCISE 3.2: IMPORTING REFERENCE TABLES FROM A FLAT FILE .............................................................. 54
EXERCISE 3.3: FIND, REPLACE AND ADD VALUES IN A REFERENCE TABLE.................................................... 57
EXERCISE 3.4: CREATE A MANAGED REFERENCE TABLE FROM AN ORACLE TABLE ..................................... 58
EXERCISE 3.5: CREATE AN UNMANAGED REFERENCE TABLE FROM AN ORACLE TABLE.............................. 61
Table of Contents
Informatica Data Quality Analyst 9.1 HF1 v
UNIT 4: SCORECARDING ................................................................................................................................. 63
SCORECARDING US_CUSTOMERORDERS ............................................................................................................ 63

EXERCISE 4.1: DEFINING THE US_CUSTOMERORDERS SCORECARD IN THE ANALYST TOOL ...................... 64
UNIT 5: WORKING WITH THE DATA QUALITY ASSISTANT ................................................................. 77
EXERCISE 5.1: DEFINING THE BAD RECORD TABLE IN THE ANALYST TOOL ................................................. 78
WORKING WITH THE DATA QUALITY ASSISTANT ................................................................................................... 80
EXERCISE 5.2: MANAGE BAD RECORDS ............................................................................................................ 80
EXERCISE 5.3: AUDIT TRAIL............................................................................................................................... 84
EXERCISE 5.4: PROFILE THE BAD CUSTOMER RECORDS TABLE..................................................................... 86
EXERCISE 5.5: DEFINING THE DUPLICATE RECORD TABLE IN THE ANALYST TOOL ..................................... 87
EXERCISE 5.6: RECORD CONSOLIDATION ......................................................................................................... 89
EXERCISE 5.7: SPLITTING A CLUSTER ................................................................................................................ 90
EXERCISE 5.8: THE AUDIT TRAIL ...................................................................................................................... 92
OPTIONAL ADDITIONAL LABS ...................................................................................................................... 93
WORKING WITH PRODUCT DATA .......................................................................................................................... 93

1 - CREATE THE PROJECT AND CONNECT TO THE DATA SOURCE ............................................................................ 93
2 - DATA PROFILING: BUSINESS REQUIREMENTS .................................................................................................... 95
3 - DATA RULES: REQUIREMENTS ............................................................................................................................ 96
4 - SCORECARD: REQUIREMENTS ............................................................................................................................. 97
vi Table of Contents
Informatica Data Quality Analyst 9.1 HF1
Introduction and Overview
The business case
GoDog.Ltd is an American-based dog food manufacturing company. Their main customers are
supermarkets across the United States, however they also sell to supermarkets in the UK and Spain.
Problems have arisen within the company relating to the quality of its customer orders and product
data.
There is little confidence in the quality of the data. Reports generated provide misleading and
inaccurate information.
Before the data can be cleansed and standardized, the specific problems must be identified. This
will be done using Analyst Profiling. The results of this analysis will define what standardization
needs to be done.
The Data Analysts know the data and can identify what needs to be done to cleanse and standardize
the data. They can collaborate with Developers to ensure the appropriate standardization and
cleansing can be done.
Known problems with the data include:
There are many variations in supermarket name - A&P, for example, appears as A&P, A and
P, A+P, A&Ps, AandP.
Invoices have been sent to the wrong people in certain supermarkets due to inadequate
contact details and in some cases, inaccurate addresses. There has been no validation of
addresses done and many addresses are incomplete.
The data is completely unstandardized; cases vary per record as well fields containing
superfluous symbols and spaces.
Auto dial, which was enabled on all phones, is failing as many phone fields contain text and
spaces in what should be, purely numeric fields.
Possible duplicate records need to be identified.
Similar problems exist with the product data, description formats differ per record, naming
conventions are inconsistent, and many duplicate records exist.
The level of data quality needs to be assessed. Once the Analyst has assessed the data, the
developer can begin the process of cleansing and standardization, and duplicates can be identified.
Data Quality Projects - Where to start?

Step 1. Profiling the Customer Order files (Analyst Task)
Data profiling is used to identify problems with completeness (what data is missing), conformity
(what data is held in non-standard format), and consistency (data in one field being inconsistent with
data in another field in the same record). We need to identify the problems in the data before we
can begin to cleanse and standardize it.
Profiling outputs include:
Reference Data Tables that can be used by developers to standardize and cleanse data.
Business Rules (custom and Pre-built) can be created and applied to profiles to identify data
anomalies. The analyst can build expression based custom rules that the developer can use
in the standardization processes. For more complex rules, the analyst can define the
requirements and the developer can build the rule/mapplet.
The data can be scorecarded to get an overall view of the level of data quality.
Lab Introduction and Overview

Informatica Data Quality Analyst 9.1 HF1 1
After profiling, a Scorecard can be created to obtain a high level view of completeness, conformity,
consistency and accuracy in the dataset. This Scorecard will allow you to track the level of data
quality over time.
Step 2. Standardizing the merged Customer Orders file (Analyst and Developer Task)
Once the data has been profiled, the analyst can identify anomalies and decide on the
standardization that needs to be performed. These requirements can be documented within the
profile which will enable analysts and developers to collaborate on projects. The analyst can create
the reference tables that the developers need to standardize, cleanse and enrich the data. Once
mapplets have been developed by the developers, they can be reviewed by the analyst and modified
if required.
The data analyst and developer can easily collaborate and transfer project related knowledge using
the Analyst and Developer Tools. This symbiotic transfer of information and ease of communication
on projects helps reduce misunderstandings and ensure project success.
Step 3. Address Validation (Developer Task using input from the Analyst)
Once the data has been cleansed and standardized the next step would be to ensure the addresses
are accurate by validating and enhancing them using definitive reference data from international
postal agencies. This is a developer task performed in the Developer Tool.
Step 4. Matching (Developer Task using input from the Analyst)

Once the data has been cleansed and standardized, duplicate records can be identified using
various different matching techniques. This is also a developer task performed in the Developer
Tool.
Step 5. Consolidation (Developer and Analyst Task)

Using business rules defined by the analyst, the developer can build mappings to automatically or
manually consolidate the matched records.
Automatic Consolidation: The developer can build mappings to consolidate the matched records
using rules defined by the analyst. They can take the best value from each field across a record
and use these to create a new master record made up of these specially selected values.
Manual Consolidation: The developer can build mappings, using rules defined by the analyst, to
populate a bad records and duplicate records table. This table will be available to the analyst to
enable them to manually correct bad records and consolidate duplicate records.
2 Lab Introduction and Overview

Unit 1: Importing Flat Files
Technical Description
The function of this lab is to familiarize users with working with Informatica Analyst and importing flat
files using the methods defined.
Objectives
US and EMEA Customer order data are in separate flat files that contain order information.
Import both Customer Order files using browse and upload and referencing the file on the
server. Once imported we will be able to perform profiling on the files in subsequent labs
Duration
30 minutes approximately
Exercise 1.1: Getting started in the Analyst Tool

This lab will demonstrate how to log into the Analyst Tool and create a new Project
Step 1. Start Informatica 9.1.0 Services

1) Select Start > Programs > Informatica 9.1.0 > Server > Start Informatica Services to start the
Informatica Services
Note: Typically you will not need to start the services as this is an Administration task.
a) It will take 2-3 minutes (approximately) to start all of the required services.
Step 2. Open the Analyst Tool and Log In

1) Start Internet Explorer
There is an icon in the quick launch tool bar to use if you wish
2) Select the Informatica Analyst entry in the Favorites menu. This URL entry in the Favorites
menu has been created for your convenience.
Unit 1: Data Objects

Note: If you receive an error, wait 30 seconds and try the URL again. It takes the Informatica9 service a few
minutes to fully start.
3) The Informatica Analyst Login screen appears
4) Log into the Informatica Analyst using Analyst/Analyst
Step 3. Create a new Project

The first thing we want to do is to create a New Project
1) At the introductory window, click Create a New Project
a) Name the Project CUSTOMER_DATA and make it a shared project.

2 Unit 1: Data Objects
Note: If you were to make it unshared, only you and users you assign permissions to would be able to
see/access it.
b) Click OK and verify the Project has been created. If needed, click the plus sign next to
the word Projects in the navigation panel.
Exercise 1.2: Importing the EU_Custord File using Browse and Upload
Next we want to go through importing our data using the various options available.
1) Locate and select the New Flat File icon in the Project Contents workspace tool bar
2) Select the Browse and Upload method to import a flat file definition.
Note: Remember this places a copy of the file on the server.
a) Browse to C:\infa_shared\PDO\EU_custord.csv and click Open. Once selected click Next.

b) If the data is not displaying properly in the Preview pane, change the code page. Select
UTF-8 encoding of Unicode code page.
(i) Ensure the Comma is set as the delimiter and the Text Qualifier as Double Quotes.

Note: The wizard allows you to import flat files with many types of Delimiters and several Text Qualifiers. Note
that double quote is not the default so you will need to set it or the data will not import correctly.
(ii) Set Import from first line for Column Names. This will tell the Analyst Tool that the
first line is the column header.
(iii) Click next.
Note: EMEA languages require a wider character set than ASCII can provide. Unicode more accurately renders
non-English characters.
Note: . By choosing to import from the first line <1>, values are automatically picked up from the first line
c) Click Next and review the Column Attributes for the data. This view has been designed to
allow the user define the data types for each column so the data can be imported
appropriately.
(i) Set the following properties for the columns specified:
Note: Note that we are not setting all the data types. If a value does not match the type defined they will
not be displayed in some views. We need to profile the data before we can be certain of the data type. If
you are unsure we suggest leaving string as default.
RECORD_ID DATA TYPE: Choose int from the Data type dropdown.
UNIT_COST DATA TYPE : number
SCALE : 2

(ii) Click Next
d) Choose to save the file in the CUSTOMER_DATA folder and give the definition the name
EU_CustomerOrders
(i) In the Description field type: EMEA Customer Orders and click Finish. Note the Data
Object appears on the Project Contents panel with the properties displayed at the
bottom.
(ii) The File Path is defined as (Uploaded) which means that a copy of the file has been
uploaded to the Server. It is stored in the cache directory specified on the Analyst
Service Flat File Cache location in the Administration Console. It will be found in the
cache location on the server with a name similar to uploaded-EU_CustomerOrders-
180770125096166.

Exercise 1.3: Import US_custord.csv flat file definition
1) Often, there is more than one way to do things in the client. To start the New Flat File wizard,
use the Actions drop-down in the upper right corner of the Project Contents screen.
2) This time select Enter a Network Path:
a) Enter:
C:\INFA_Shared\PDO\US_custord.csv into the Network Path field
Note: In Windows, the path must be shared. On the computer image this class was developed on, the path
C:\Infa_Shared has been shared as Infa_Shared. Typically you will not have a Server and Client installed on
the same machine. This location refers to a folder either on the server or accessible by the server. You will need
to substitute the name of the server in the network path. For example, if your server is named infa-server, and
C:\infa-shared has been shared as infa-shared, the path becomes:
\\infa-server\infa_shared\PDO\US_custord.csv
b) Click Next. Leave the default codepage, ensure the comma delimiter is selected and the
Text Qualifier set to Double Quote and select Import column names from first line.
c) As before click the Show button to update the preview.
(i) Scroll to the end of the preview to ensure the data appears as expected.
d) Click Next. Again the Data Types and Precision are set but these can be modified.
(i) Set:
ID DATA TYPE: Choose int from the Data type dropdown.
UNIT_COST DATA TYPE: number
SCALE : 2
e) Click Next and ensure the file is being saved in the CUSTOMER_DATA folder. Rename the
Data Object: US_CustomerOrders
(i) Description: US Customer Orders
f) Click Finish and review the results noting the file path is defined in the properties.

Unit 2 Lab A: Profiling, Rules and Collaboration
Lab A - The function of this lab is to familiarize users with Profiling in the Analyst Tool. It will also
demonstrate how Analysts can collaborate with Developers through the use of Comments and Tags.
Data profiling is used to identify problems with completeness (what data is missing), conformity
(what data is held in non-standard format), and consistency (data in one field being inconsistent with
data in another field in the same record).
The analyst will need to identify the problems in the data before the developer can begin to cleanse
and standardize it.
Profiling outputs include:
Reference Data Tables.
Business Rules (custom and Pre-built).
Scorecards.
This lab will demonstrate how to use the tool to identify possible anomalies in the data. It will
demonstrate what to look for, how to review the value frequency lists and patterns as well as using
drill downs to review the underlying data.
Lab B This will focus on Applying Pre-built Rules and building Expression Rules to identify
anomalies in the data.
Objectives
Engage in a discovery effort to determine various business problems with the data and
possible fixes.
Apply Filters to profile subsets of data.
Create Comments to track any issues discovered.
Tag project objects with appropriate tags
Export Drilldowns to files.
Apply Out of the Box rules to your data.
Build custom expression rules.
Duration
Unit 2 Profiling
Exercise 2.1: Column Profiling
Profile the EU_CustomerOrders Data

1) Select the EU_CustomerOrders Data Object and click the white area (which will turn green
when selected) in between the Name of the data object and the Type of data object.
Note: If you click on the URL, the file will be opened in Data Preview mode. If that happens, close the tab and
try again.
a) Right click the highlighted EU_CustomerOrders data object and choose New > Profile
b) It is possible to create multiple profiles at the same time however in this example we will
only profile the EU_CustomerOrders file.
(i) Leave the EU_CustomerOrders file selected and press Save and Run to create a
profile using the default settings.
10 Unit 2 - Profiling
A profile named Profile_EU_CustomerOrders is automatically created and opened.
c) Make sure all the columns are selected to be displayed in drilldowns. If not, select the
Drilldown checkbox at the top. You may need to scroll to the right to see this.
Unit 2 Profiling
Exercise 2.2: Create a Filter and apply to a profile
1) Review the results of the profile. Scroll to the end of the profile and note the field called
Record_Status.
This field tells us if the record is live or inactive. In this example we want to profile Live records only
so we will create a filter and rerun the profile applying the filter to the profile.
2) In the profile click on the Manage Filters icon.
3) We are going to create a Filter for LIVE records so in the Manage Filters dialog click on the
New button.
4) Name the filter LIVE_RECORDS and select the field RECORD_STATUS from the dropdown.
a) Set the filter to filter records where the RECORD_STATUS = LIVE and click OK. (You may
need to click out of the LIVE dialog for the OK button to become enabled.)
b) Click Save when complete.
5) We have now saved the filter with the profile but we need to apply it.
6) Choose to rerun the profile by selecting the Run Profile icon.
a) In the Run Profile dialog click Choose a Filter, then select the Filter just created and
choose Run.
Unit 2 Profiling
Note: Note the option to remove any applied Filters from the profile is option 1.
7) Check the RECORD_STATUS field and ensure that only LIVE records have been profiled.
8) An Additional example of a filter that can be created and applied is to:

Profile only the records that contain a zip code.
a) Continue using either the Live records filter or no filter.
Exercise 2.3: Review Profiling Results, add comments and export drilldowns
Note: You may need to turn off the Pop up blocker ahead of this exercise. This can be done by selecting
Tools > Pop-up Blocker > Turn Off Pop-up Blocker from the browser menu.
1) There is much you can determine about your data through Column profiling:
a) Unique Values
The number of unique values in a column is listed for each column.
b) Unique %
The percentage of unique values in each column. Therefore, a column with 100% unique
values is a key. In this case, Record_ID.
c) Null values
The number of null values in each column
d) Null %
The percentage of null values in each column
e) Datatype
Data type derived from the values for the column
f) Inferred %
Percentage of values that match the data type inferred by the Analyst tool
g) Documented Data Type
Data type declared for the column in the profiled object
h) Max Value
Maximum value in the column
i) Min Value
Minimum value in the column
j) Last Profiled
Last Profiled Date and time you last ran the profile
2) Review the Datatype columns this is the data type that was inferred by the Analyst tool
through Profiling. The Inferred % relates to the percentage of values that meet that data type.
a) For the RECORD_ID we defined the type as integer and upon review all of the data has
been identified as integer.
b) What data type has been identified for the ORDER/SHIPPING DATE fields?
c) Has the Analyst tool identified the expected data types for the rest of the data?
3) Select the ORDER_NO column and review the Values to the right.
a) The column should contain order numbers which would be expected to be numeric. From
looking at the values it is clear this is not the case.
(i) Review the patterns, there are letters, spaces and symbols in the field. It would
benefit from some data cleansing to remove any noise.
b) Open the comment view to by clicking on the Show Comments icon to the right.
Unit 2 Profiling
Note: Because the comments are saved with the profile, not the column, its a good idea to name the column
you are writing the comment about.
(i) Add a comment to ask the developer to remove the noise from this field. Once
complete press Add.
Note: To remove a comment click on the icon to the left of the comment.
(ii) Close the comment view by pressing the Hide Comment icon in the viewer.
4) Select the ORDER_DATE column and review the Values to the right.
a) Examine the values:
(i) Note the 22 occurrences of NULL
This is not valid and will need to be addressed. We will need to export these records
and send them to the data owner to be corrected.
(ii) Note the formats of the other dates in the field.
This is a European format for dates (DD/MM/YYYY). Review the other dates in the
column by clicking on the Value column to sort the data. It is evident that the date is
written as DD/MM/YYYY throughout the file.
Note: We would need to change the format of the date from European to US format before the files could be
merged.
(iii) Patterns - Check the format of the data in Patterns. Are the dates in the same format
throughout the file?
We want to add comments to the profile to track inconsistencies and flag what needs to be reviewed
through the DQ process.
(i) Create a comment to note the dates in this file are in DD/MM/YYYY format and may
need to be changed.
(ii) Close the Comment viewer when complete.
b) Make sure the NULLS are to the top of the viewer (Sort the data if required). We want to
export these records to a file to send to the data owner as they cant be processed until
the appropriate order date is defined.
(i) Select the NULL values right click and choose Drilldown.
Note: It is possible to change the columns that are displayed in the lower preview pane by deselecting the
checkbox to the right of the Last Profiled column. Columns that are not selected are not displayed in the data
preview.
Note: Check to ensure the Pop-up blocker is switched off ahead of exporting the data.
c) Select the Export Data icon in the drilldown part of the dialog to export the data to a file.
Unit 2 Profiling
d) Rename the File as Profile_EU_CustomerOrders_Invalid_ORDERDATE.
(i) Either leave the default code page or choose UTF-8 encoding of Unicode (if that was
what you imported the file as) and click the OK button.
(ii) Choose to save the file on the desktop for easy access. (If this does not open it may
be because the pop up blocker is active.)
(iii) When complete choose Open and review the contents of the file.
(iv) Once done, close Excel and return to your profiling.
5) Continue reviewing the Profiling results: COMPANY column

a) Select the Company column and click the Value column to sort the company names in
ascending order.
b) Place your cursor on the border between the Value and Frequency columns and drag the
Values column to make it wide enough to read the company names. As you scroll through
the company names you can quickly see that the names need to be standardized. Case
is mixed, name formats are mixed. There is noise that needs to be removed.
c) Add a comment noting these issues:
COMPANY column needs to be standardized. Company names should all be in upper
case, name formatting needs to be standardized, and noise needs to be removed.
d) Once complete hide the comments view.
6) Continue reviewing the Profiling results: ADDR1 Column

a) Select the ADDR1 column and note the data values.
The data values are in mixed case. They should be in upper case only. Also note the
Address Suffixes there are instances where road appears as RD, Road, Rd. etc. All of
the Address suffixes should be standardized.
Unit 2 Profiling
b) Add a comment to tell the developer that this will have to be changed to upper case and
to standardize the Address Suffixes.
7) Continue reviewing the Profiling results: COUNTRY_CODE column

a) Click the COUNTRY_CODE column
b) Country codes should be formatted to the three character ISO standard. Later we will use
a Reference table to standardize this column
c) Add a comment for the COUNTRY_CODE column:
Please use a Reference Table to standardize the COUNTRY_CODE column to ISO format
8) Adding Filters to drilldown results.

a) Drill down to display all records where the country code is GBR.
b) Click on the Edit Drilldown Filter icon to the right of the drilldown results.
c) Choose to add an additional condition to the drilldown by pressing the + button.
d) In the new dropdown choose the CURRENCY field. We want to see if there are any
records where the Country code is GBR but the country code isnt GBP. Once complete
press the Run button and review the results. Are there any records where the country is
GBR and the currency isnt correct?
e) Drill down on records with the Countrycode = ESP and check that all records with a
Spanish country code have an EUR Currency Code. You will need to edit the filter to
remove the previous query and run the new one.
If this is not the case what are the incorrect values in the Currency field?
f) Add a comment to ask the developer to create a Currency for these missing values based
on the CountryCode.
Tags:
As well as leaving a comment for the developer we can Tag fields as well. This means when the
Developer searches for Tags in the Developer tool, objects will be highlighted in association with a
Tag, for example Standardization. This will flag the object for standardization for the developer.
We want to add Tags to some of the fields we have just profiled.
Exercise 2.4: Create and assign a New Tag

Some of the fields profiled require that noise be removed. We can create a tag called Remove Noise
and apply this to the fields.
1) Open the Tag viewer by clicking on the show Tag viewer to the right.
A list of all the available Tags is displayed. These tags were imported along with the Core Accelerator
(pre-built rules and reference tables)
2) Create a new tag by clicking on the new tag icon.
Unit 2 Profiling
3) In the area below type the name: Remove Noise and also enter the description. Note that
the Tag appears immediately in the list of available tags above.
Now that the Tag has been created, it needs to be assigned to objects. The Developers will be able
to use a combination of both comments and tags to see what needs to be done in the project.
Exercise 2.5: Assigning Tags

1) Click on the Company field and click on the new Tag Remove Noise.
2) Once the field and tag are selected the Assign Tag button will become enabled.
a) Click on this button to assign the Tag to the field.
3) Assign the Standardize Tag to the same field as once the noise is removed the company
names will need to be standardized.
4) Assign appropriate Tags to the following fields:
Addr1
Country Code
Currency
Contact
5) Once complete close the Tag Viewer and close the profile.
Exercise 2.6: Create a Custom Profile on the Data Object US_CustomerOrders

When reviewing larger files/tables you may only want to profile a subset of the data, selecting only
certain columns and a certain amount of rows. Custom Profiles enable you to select the columns you
want as well as specifying the amount of rows to profile.
Step 1. Create the custom profile
1) On the Browse Project Tab, select the Data Object US_CustomerOrders
a) Right click the Data Object US_CustomerOrders and choose New Profile
2) Ensure the US_CustomerOrders file is selected and choose Next.
a) The suggested name of Profile_US_CustomerOrders is good. Leave it as is. Change the

description to read: USA Customer Orders. Verify that the project folder CUSTOMER_DATA
is selected and click Next.
Unit 2 Profiling
b) Make sure the following columns only are selected:
ID
COMPANY
CONTACT
SHIPPING_STREET
SHIPPING_CITY
SHIPPING_STATE
SHIPPING_ZIP
COUNTRY
c) Choose to run the profile on the first 100 rows only. Click Save & Run.
Note: Clicking Next will bring you into the Filters view we will apply a filter later.
d) Only the selected columns have been profiled and only the first 100 records. Review the
results of the profile.
e) Drill down on the first Value in the Company. Note how only the profiled columns are
currently being displayed in the drilldown.
Step 2. Create a New Filter
3) In our case the file is quite small so we will want to profile all of the columns and all of the
LIVE rows. To profile only the LIVE records we need to create the filter.
a) Create a new filter in the Manage Filters dialog to profile only the records where the
RECORD_STATUS = LIVE.
Note: Note the filter we created previously is not available this is because filters are saved with the profile on
which they are created.
b) Save the filter with the profile.
Step 3. Apply the Filter to the profile.
4) Now the filter has been created it needs to be applied when the profile is run. To rerun the
profile choose the Run Profile icon.
Unit 2 Profiling
a) This time make sure all the columns are selected by checking the checkbox above
Columns.
b) Ensure All Rows (complete analysis) is selected under Sampling Options.
c) The Select Columns button under Drill down options allows you to choose the columns
that are displayed when drilling down on values.
d) To apply the filter, click Choose a Filter and in the Filter dialog choose the profile you just
created.
e) Then choose Run. Make sure all the columns have been profiled this time but only
records with a Record_Status of Live should be profiled.
5) Column Profiling: ORDER_DATE
a) Review the date fields. Are they in the same format as the EU file?
6) Column Profiling: COUNTRY

a) Review the values in the COUNTRY field. This should only contain the value USA. Are
there other values in this field? If so they will need to be standardized later in the course.
7) Column Profiling: PHONE

a) Review the values in the PHONE field. This should only contain numeric values. Are there
other values in this field? If so they will need to be removed later in the course.
8) Column Profiling: CURRENCY

a) Review the Currency field in the US order file. If it does not exist the Developer will need
to create one when merging the EU and US files together. The default value will need to
be USD.
(i) Add a comment to ask the Developer to do this.
9) Review the other Columns and add Comments and Tags where appropriate.
Unit 2 Profiling
Unit 2 Lab B: Rules
Duration: 60 mins
Profile Rules
Rules provide the Analyst with a way to identify anomalies in the data and also to manipulate the
data. Rules can be built in both the developer and analyst tools. The Developer will create rules that
the Analyst will be able to test and review.
While it is possible to apply and test the rules in the Analyst tool, the data will not be changed until
the rule is applied in a mapping in Developer.
Rules in the Analyst tool have restrictions:
Only one rule can be made to data at a time. For example, you cannot apply a rule to a rule.
However, you can use Custom rules with complex expressions to make more than one
manipulation type.
Part of the Core Accelerator contains a set of prebuilt rules that are available to all customers. These
rules can be used to execute mapplets that perform fairly complex functions.
It is also possible for the Analyst to develop their own rules using expressions in the Analyst Tool.
These Rules can be used to cleanse and standardize data.
These labs will look at applying prebuilt rules as well as developing custom rules.
Note: Rules can be created and reviewed in the Analyst Tool but the data will not be modified unless
applied as mapplets in the Developer.
Applying Prebuilt Rules US_CustomerOrders
Exercise 2.7: Pre-built Rule - Email_Validation

Rules can be applied to the data through Profiles.
We want to apply a rule to check the validity of the data in the email column in the
Profile_US_CustomerOrders .
1) Click the Add Rule icon ( ) on the right panel

a) Choose Apply a Rule and click Next
The prebuilt rules from the Core Accelerator were installed to the CONTENT folder.
b) Select Projects -> CONTENT -> Rules -> Contact_Data_Cleansing and select
rule_Email_Validation
Note: Each rule contains a description explaining its function.

c) Click Next and click the drop down on the Inputs panel and select the EMAIL column
d) Click the Output Column Label field and enter: Conformant_Email
Unit 2 Profiling
e) Click Next.
The data has not changed since we ran the profile so we only need to run it for the new rule that has
been added.
(i) Click twice on the top Columns checkbox to select and deselect all of the columns.
Scroll to the bottom and select the rule column Conformant_Email.
(ii) We dont want to show results only for column in this run so deselect the checkbox
and press OK if a warning is displayed.
(a) Also verify that All Rows (complete analysis) is selected.
(iii) In Drilldown Options click the Select Columns button. We want to view all columns in
a drilldown so make sure all of the columns are selected and click OK.
f) Click Save & Run
2) At the profiling screen, select the rule column Conformant_Email

Note that there are Valid and Invalid emails in the data.
Note: If you do not select all of the columns in the Drilldown Option only the output for the rule will be visible in a
drilldown.
Unit 2 Profiling
3) Review the output of the Conformant_Email Rule over to the right.
Exercise 2.8: Pre-built Rules IsNumeric

1) Select the ORDER_NO column
a) Select Statistics
Note there are characters in some of the values. We know however, that ORDER_NO
should only contain numeric data so we will apply a rule to validate this.
2) Apply the rule_isNumeric to the ORDER_NO column.

a) This rule can be found in Projects -> CONTENT -> Rules -> General_Data_Cleansing.
b) The input column is ORDER_NO. Rename the output column Conformant_OrderNo.
c) Click Next and select the rule column Conformant_OrderNo as the column to profile.
(i) Select to view all of the columns in Drilldown Options, then Save & Run.
3) In the Profile click on the Conformant_OrderNo rule
a) Select Values and note there are approximately 32 values that need to be corrected.
Unit 2 Profiling
Exercise 2.9: Apply Pre-built Rules: rule_Completeness_Multi_Port
1) Apply another rule, this time choose the rule_Completeness_Multi_Port from the Projects ->
CONTENT -> Rules -> General_Data_Cleansing and apply it to the following fields renaming
the output appropriately:
COMPANY Completeness_Company
CONTACT Completeness_Contact
TITLE Completeness_Title
SHIPPING_STREET Completeness_Street
SHIPPING_CITY Completeness_City
SHIPPING_STATE Completeness_State
SHIPPING_ZIP Completeness_Zip
COUNTRY Completeness_Country
PHONE Completeness_Phone
EMAIL Completeness_Email
2) Press Next and ensure the fields are selected and this time choose to Save the rule. We will
apply another rule before running the profile in the next step.
3) Apply the following rule and when finished choose Save and Run making sure all of the
required columns are selected for profiling.
Rule Column Output Name
IsNumeric PHONE Conformant_Phone
4) Once complete you will have the following rules applied to the US_CustomerOrders profile
5) Close the profile.
Unit 2 Profiling
Create and Apply Custom Cleansing Rules EU_CustomerOrders
We want to apply rules that will correct some of the problems in the data. We can review the outputs
and verify they are correct. Later the developers can covert the profile to a mapping which will allow
them to utilize the cleansed data going forward.
Exercise 2.10: Create a Rule for Column ADDR4 (zip code)

1) Open the Profile_EU_CustomerOrders. Select column ADDR4
a) Click Patterns and review some of the values. Sort the patterns and note the presence of
the format 9(4).
There is a business rule that specifies there should be no 4 character zip codes. If one exists it is
because the leading 0 was removed by excel. Because of this, a zero (0) should be appended to the
front of any 4 digit codes.
Note: This is just a sample rule that could be applied.
2) Make a note of the amount of values in the 9(5) and 9(4) pattern format.
Create a custom rule to add a zero character to the front of any zip code values that are 4 characters
long.
3) Choose to Add a rule.
4) This time choose to Create a rule and name the rule: rule_LeadZeroFiveDigitZip
(i) Description: If a zip has four characters, append a 0 to the start.
a) Save as a reusable rule in the Rules > User_Data_Cleansing folder.
Note: This is a folder that was created centrally for user defined reusable rules.
Note: Because this is a Shared Project you will have read access but you will need to have been assigned
Write Permissions to save the rule here.
b) In the expression editor type the following expression:
IIF(LENGTH(ADDR4)=4, '0'||ADDR4, ADDR4)
This expression says; if the value in the ADDR4 column is four characters long, then concatenate a
zero character to the front of the rest of the value in ADDR4. If the value in the ADDR4 column is not
four characters long, then return the current value in the column.
c) Click the green tick button to validate and press Next.
Unit 2 Profiling
The Analyst will automatically check the validity of the expression syntax. If you receive a message
that the expression is invalid, carefully examine your typing to make sure it is correct. Call your
instructor if necessary.
(i) Under Columns: Choose to run only the new rule
(ii) Under Profile Results Option: Deselect the checkbox: Show results only for column, rules
selected in the current run.
(iii) Under Drilldown options: Choose Select Columns and choose to view all columns.
(iv) Then click Save & Run.

d) Select the profile column rule_LeadZeroFiveDigitZip
5) Once again review the value frequency for the patterns 9(5) and compare the frequency to
the previous values. The frequency should have increased for 9(5) and the pattern 9(4) will
be gone.
Exercise 2.11: Create a Rule to reformat the date columns
We need to create a rule that will convert the European dates to US format so the data can be
merged together into a single Customer Order file.
1) Click on the ORDER_DATE column and create a new Rule called: rule_EUtoUS_DateFormat
(i) Description: This rule converts EU Date Format to US date format
(ii) Save as a reusable rule in /Content/Rules/User_Data_Cleansing
a) In the Expression Editor, enter the following expression:
TO_CHAR(
TO_DATE(
ORDER_DATE, 'DD/MM/YYYY'), 'MM/DD/YYYY')
This will take the ORDER_DATE and change the format from European format 'DD/MM/YYYY' to US
format 'MM/DD/YYYY'.
Unit 2 Profiling
b) Click Next
(i) Under Columns: Choose to run only the new rule
(ii) Under Profile Results Option: make sure the checkbox is deselected
(iii) Under Drilldown options: Choose Select Columns and ensure all columns is selected.
(iv) Then click Save & Run.
c) In the profile, select rule column rule_EUtoUS_DateFormat
Check the dates have been converted to North American format.
Editing the output
2) We want to edit the output of the rule to set the output as ORDER_DATE_US.
a) Right click on the rule in the profile and select Edit Rule.
b) Double click the output and rename as ORDER_DATE_US.
Unit 2 Profiling
a) Save the changes.
Editing the Rule:
Note: Note that if you wanted to edit the syntax or permanently change the name of the output you
would need to select the rule from the User_Data_Cleansing folder.
Note: It may not be possible to edit the syntax for all rules. Some rules created in the developer use DQ
transformations and rules that have these transformations should be edited in the developer.
Once opened here it is possible to edit the input/output and syntax. It will also tell you where the rule is
used.
Exercise 2.12: Apply the rule to the SHIP_DATE field

1) Apply the rule to the Ship_Date field renaming the output SHIP_DATE_US.
Exercise 2.13: Tag the Custom Rules

Its a good idea to Tag the custom rules that we built so they will be identified as Standardization
Rules in searches in the Developer
1) On the Browse Projects Tab, select Actions and Show Tags.
2) Select the rule and the appropriate Tag and assign the tag to the rule.
Unit 2 Profiling
Exercise 2.14: Searching for Items that have been Tagged
1) To the right of the Analyst and choose to Search by Tag
2) Enter the name of one of the Tags that you applied and press Go.
The Results will be displayed in the Search View.

It is possible to restrict what is displayed by switching off the object type checkbox to the right.
Note: There is an error in 9.1HF1 that may prevent the results from being displayed. This will be
corrected in a future release.
a) Any of the objects can be opened by clicking on the hyper link.

3) Once you have found the items that you tagged close the viewer and return to the main
project window.
Unit 2 Profiling
Unit 3: Reference Table Management
The function of this lab is to familiarize users with working with Reference Tables.
It will cover the steps involved in creating, updating and modifying reference tables.
These reference tables will be used later in the course by the developer to standardize, cleanse and
enrich the data.
Objectives
Build reference tables using the reference table editor and also importing flat files.
Update, modify and edit values in the reference table.
Duration
Exercise 3.1: Create a Reference Table

When creating the reference table you may want to use all values, even if they belong to inactive
records. For this reason we are going to remove the filter we applied to the profile.
1) Open the Profile_EU_CustomerOrders and click Run Profile. Select All of the columns.
a) Click on the Choose a Filter button.
b) Select 1 Profile source data will not be filtered and press Run.
Unit 3 RTM
There will be a slight increase in the number of records in the profile.
Now you are ready to create the reference table.
Step 1. Create the Table
2) Select the COUNTRY_CODE column

Observe there are both two and three letter abbreviations as country codes. The codes should have
three characters, not two. In addition, there appear to be codes similar to each other and that may
refer to the same country.
One way to standardize codes like these is to use a Reference Table
a) Select the values in the frequency viewer.
Note: In HF1 if you do not select the values it will store the patterns (XX,XXX) in the ref table instead of the
values (UK, GBR). This will be corrected in the next release.
b) Locate and click the Add to Reference Table icon ( ) on the

Values/Patterns/Statistics panel.
48 Unit 3 - RTM
c) Select Create a New Reference Table and click Next.
(i) Table Name: COUNTRY_CODE_STD

(ii) Description: Standardize to three letter country codes
d) Click Next and verify the Column Attributes and Preview of the column.
e) Change the Precision to 3 and click Next
We will not include column descriptors at this time
Unit 3 RTM
f) On the Browse Project Tab, select the CONTENT>Dictionaries>Training project. It is
possible to keep notes for the IT auditor. This is important in standards compliance.
(i) Enter Created from European Customer Order file in the Audit Note and click Finish
to save the Reference Table.
To provide the cross reference standardization values, we must edit the new Reference Table.
Step 2. Modify the table structure
1) Refresh the Browse: Projects tab and open the reference table just created.
2) To Edit the Reference Table click on the Edit Table icon ( ) in the toolbar.
a) To add columns in which to enter the values that are to be mapped to the correct Country
code, click the Add Column Attributes button ( ) and add 5 more columns to cater for
variants.
(i) Ensure the COUNTRY_CODE is set as the Valid column in the table.
50 Unit 3 - RTM
(ii) Change the precision of the new columns to 20.
(iii) Click OK
The names of the additional columns are not important for the purpose of this lab. These columns
will serve to hold aliases that are to be associated with the desired value in column COUNTRY_CODE.
Therefore there is no need to change the column names at this time.
Step 3. Add the Standardization Values

1) Edit the values in the table to look like the figure below.
The attributes UK and GB should map to GBR.
The attribute ES should map to ESP.
The column containing ESP and GBR is what we set as the valid column. For consistency
we typically set the first column as the valid one.
a) To Delete unneeded rows, select the rows containing COUNTRY_CODE values ES, UK,
and GB and click the Delete icon ( ).
b) An Audit Note box pops up. Enter the text:
Country codes ES, UK and GB, are invalid and have been deleted.
Unit 3 RTM
c) Press OK and review.
Step 4. Adding Values to the Reference Table
1) Click on the Browse: Projects tab and open Profile_ US_CustomerOrders

a) Select the COUNTRY column
b) Observe the Value frequencies and the variations in how the United States is coded.
c) Select the list of values and click Add to Reference Table icon
d) Select Add to existing reference table and click Next
52 Unit 3 - RTM
e) Select Projects >CONTENT>Dictionaries>Training > COUNTRY_CODE_STD and click Next
f) Choose to add the values to the second or third column in the table (we choose the
second or third column because the precision set in the first column is only 3 and the
values would be truncated.) and click Finish.
g) Return to the Browse: Projects Tab and open the reference table.
h) The row that contains the value USA in the column is the valid row.
(i) Modify this row to hold the correct value as well as the various aliases from the
US_CustomerOrders Data Object. The data values to enter are:
USA
UNITED STATES
U.S.A.
AMERICA
US
i) Delete the extra rows with the redundant US country code values
In the Audit Note enter: Invalid COUNTRY_CODES
Unit 3 RTM
j) Click OK.
The table should appear:
2) To view the Audit Trail press the Audit Trail button as illustrated.
a) In the Audit Trail window, press the Show button to review all the entries.
b) To refine the display, add filters to the Type or Status dropdowns.

c) To refresh press the Show button again.
d) Close all open tabs and return to your CUSTOMER_DATA project.
Exercise 3.2: Importing Reference Tables from a flat file

1) We want to keep the reference tables all together in our central project so on the Browse
Project Tab open CONTENT>Dictionaries>Training folder and from the icons to the right,
choose New Reference Table.
a) Choose Import a flat file and click Next.
54 Unit 3 - RTM
b) Browse to: C:\infa_shared\Dictionaries\Training and select
standardize_company_names.dic and choose open.
c) Rename Standardize_Company_Names and press the Upload button.
d) Set the Code Page set to: UTF-8 encoding of Unicode and click Next
e) Leave the default comma delimiter and select Double quotes as the Text Qualifier. Press
Next.
Unit 3 RTM
f) To use the reference table for standardization we must have a valid column selected.
This is the master column in the table. Make sure Column1 is selected.
g) Click Finish. The table is now ready for the developer to use.
56 Unit 3 - RTM
Exercise 3.3: Find, Replace and add values in a Reference Table
1) Open the Reference Table Standardize_Company_Names if it isnt already open.
(i) Scroll to the top of the table if you arent already there.
a) Click the Find and Replace icon ( )
b) In the Find box, type: THE % and press the Next button
We want to change any invalid instances to the correct version. We selected Column1 as the Valid
Column so this means that all the other values in the row will be standardized to the value in this
cell.
c) In the first free cell for the row, enter the value CORNERS SHOPS and press Enter (or the
green tick arrow to the right) to save the changes.
d) Find PUBLIX SUPERMARKETS (or PUBLIX %) and change the value in column 7 from
PUBLIX SUPER MARKETS INC to PUBLIX SUPER MARKETS, INC as one of the variants.
2) We want to add in some new values. Press the Add Row icon
a) In the popup dialog enter the following values.

(i) When they are entered scroll down to press the OK button.
Unit 3 RTM
b) Close the reference table. It is now ready to be used by the developers.
Exercise 3.4: Create a managed Reference Table from an oracle table

1) Again - we want to keep the reference tables all together in our central content project so on
the Browse Project Tab open CONTENT>Dictionaries>Training folder and from the icons to
the right, choose New Reference Table.
a) This time choose Connect to a relational table and leave the Unmanaged Table
Unchecked. This means that the data will be imported into the Informatica 9 staging
area and the reference table can be edited and updated. It also means that any changes
to the source table are NOT reflected in Informatica. .
58 Unit 3 - RTM
b) Click Next
c) Select the Training Connection and press Next.
d) From the Tables available choose ISO_COUNTRYCODES and press Next.
Unit 3 RTM
e) This table will be used with product data by the developer where the country code should
be 2 digits so this will be set as the Valid column. Press Next.
f) Save it to the Projects\Dictionaries\Training folder and press Finish.

Note: The data from the table is imported and stored in Informatica 9. This means that it can be edited if
required.
60 Unit 3 - RTM
Exercise 3.5: Create an Unmanaged Reference Table from an oracle table
1) Again - we want to keep the reference tables all together in our central project so on the
Browse Project Tab open CONTENT>Dictionaries>Training folder and from the icons to the
right, choose New Reference Table.
a) This time choose Connect to a relational table and check the Unmanaged Table
checkbox. This means that a link is created to the table and the data will not be
imported therefore it will be unmanaged by Informatica 9. It means that any changes
to the source table are reflected in Informatica but the table cannot be edited in
Informatica.
b) Click Next
c) Select the Training Connection and press Next.
Unit 3 RTM
d) From the Tables available choose DOMAINS and press Next.
e) Set the first column as the Valid column and press Next.
f) Save it to the Projects\Dictionaries\Training folder and press Finish.
g) Note that the values in the table can NOT be editied, nor can rows be added or deleted.
62 Unit 3 - RTM
Unit 4: Scorecarding
Scorecarding US_CustomerOrders
The function of the following lab is to demonstrate how to configure a scorecard in the Analyst Tool
using a number of different techniques. These will include:
Applying and including prebuilt Rules
Using valid values from profiling.
Because we have already applied Pre-built rules to our US_CustomerOrders profile we can use this to
create the scorecard.
Objectives
Understand how to build and configure a data quality scorecard by applying Pre-built rules
and specifying the valid values per column.
Duration
Overview
The data has been profiled and we have applied both Pre-built Rules (to the US_CustomerOrders
Profile and Custom rules) to the EU_CustomerOrders Profile.
We want to track the level of data quality using a scorecard. This will provide a high level view of the
quality of the data. This is facilitated through the Data Quality Scorecard.
Scorecards can be created in the Analyst Tool and updated to include more complex Rules
developed in the Developer Tool.
Targets can be defined and a snapshot taken of the data quality measures at any stage in the
process. Once the data has been cleansed, the scorecard can be re-run to measure the level of
improvement and the targets adjusted if required.
Scorecards can be created using a blend of the following:

(i) Applying Out Of The Box Rules (Pre-built rules)
(ii) Simple selection of valid values per column in the Analyst Tool when defining the
Scorecard
(iii) Building your own rules/mapplets in the Developer Tool and applying them in the
Analyst (not covered here)
Unit 4 Scorecarding
The first step involves identifying which key columns we want to scorecard and which measures (e.g.
consistency, conformity, accuracy) will be applied.
Business Rules we have already applied to the US_CustomerOrders profile include:
Conformity: (using Pre-built rules)
The ORDER and PHONE columns should contain only numeric characters.
The EMAIL address should be a valid email in the appropriate format.
Both DATE columns should contain valid dates.
We are also going to add the following:

Accuracy: (by specifying valid values in the profile)
The COUNTRY column should contain USA, GBR or ESP.
The PAY_MTHD should be one of either DD, SO, CA, or CQ.
The PAY_TERM should be 30, 45, 90 or 120.
Exercise 4.1: Defining the US_CustomerOrders Scorecard in the Analyst Tool

1) In the CUSTOMER_DATA Project, open the Profile_US_CustomerOrders.
2) We want to scorecard all of the data so choose to run the profile. Remove the filter and run
the profile on ALL of the data, including all columns and rules.
a) Review the results to ensure that all records have been profiled.
We have already applied prebuilt rules to the profile that we can use in the Scorecard.
64 Unit 4 - Scorecarding
In the scorecard, as well as applying Pre-built rules to determine good and bad data, we can define
what the valid/invalid values per column are. This is possible in columns that do not have very many
unique values. In this section we will be able to check the following:
Valid country in the COUNTRY column is USA.
Valid values in the PAY_MTHD column are DD, SO, CA, or CQ.
Valid values in the PAY_TERM column are 30, 45, 90 or 120.
3) In the profile select the Add to Scorecard icon
a) Select the following columns to add to the Scorecard, as well as all of the rules:
(i) COUNTRY rename Accurate_Country
(ii) PAY_MTHD rename Accurate_PayMethod
(iii) PAY_TERM rename Accurate_PayTerm
Note: We are selecting only these columns as the others contain too many values for us to define as
valid/invalid.
Unit 4 Scorecarding
b) Also select the Pre-built Rules that were applied earlier. There is no need to rename the
outputs as they were named appropriately at the time.
Note: If any of these are disabled, run the profile and try again.
c) Once selected press next.
4) Choose to create a NEW scorecard.
a) Call the Scorecard US_Customer_Scorecard and create it in the CUSTOMER_DATA
Project. Select OK.
We are measuring Completeness, Conformity and Accuracy and we want to group the measures into
these categories or groups.
b) To the right of the dialog choose to create a New Group within the
US_Customer_Scorecard.
We have a default group where all of the current measures reside and we will rename this later.
Unit 4 Scorecarding
c) Enter the name Accuracy and press OK
d) Press New again and enter the name Conformity and press OK. You will now have 3
groups: Default, Accuracy and Conformity.
e) Press Next
5) Click on the first measure in the list - Accurate_Country. The list of values contained are
displayed to the right of the dialog. This is where we can define the valid values.
a) Select the USA as valid.
b) Click on the next measure Accurate_PayMethod and select CQ/CA/DD/SO as valid
values.
Unit 4 Scorecarding
c) For Accurate_PayTerm select the following as valid: 30/45/90/120
d) For the Completeness Columns select Complete
e) For Conformant_Email select Valid
f) For Conformant_OrderNo select True
g) For Conformant_Phone select True
h) Once you have defined the values for all of the columns select Finish. Open the
scorecard if it does not open automatically.
We have the scorecard with 3 groups with all of the measures in the Default group. The next step is
to rename and re organize the measures into their appropriate group. Before we can edit the
scorecard we will need to run it.
6) Run the scorecard by pressing the Run Scorecard button to the right.
a) Accept the defaults in the Run scorecard dialog and press Run.
Once the scorecard has been run we will be able to edit it.
7) Click on the Edit Scorecard icon
a) Choose the Score Groups tab.
b) Click on Default and choose to Edit.
Unit 4 Scorecarding
(i) Most of the measures in the group are for Completeness so rename Default to
Completeness and press OK.
c) Click on the Accuracy measures to the right holding down the CTRL key to select them
together, and then press Move.
d) Move them to the Accuracy Group and press OK
e) Click on the Conformity measures to the right holding down the CTRL key to select them
together, and then press Move.
f) Move them to the Conformity Group and press OK.

g) Press Save and review the new Scorecard.
Unit 4 Scorecarding
8) We now have the right measures in the appropriate Group. Drill down to view the matching
rows. From here it is possible to toggle between the Valid and Invalid rows.
9) To run the Scorecard at a later date, open the Scorecard in the project and press the Run
Scorecard icon.
a) It is possible to choose which columns in the source you want available during drilldowns
by selecting/deselecting the columns in the viewer.
b) You can also choose to use live or staged data in the drilldowns.
10) To view the history or trend, right click on the column and choose Show Trend Chart.
Unit 4 Scorecarding
You now have a scorecard:
Which was created directly from value frequencies in a profile BUT IS NOT connected to the
profile(s) from which column/virtual column originated from. This means you could delete
the profile which was used to create the scorecard, without impacting the scorecard.
That you can update to include additional scores from ANY profile which means the
scorecard can reference multiple data sources.
Note: If you delete the source then both profile and scorecard would become invalid.
Unit 5: Working with the Data Quality Assistant
The objective of the following lab is to enable users become familiar with launching the DQA, editing
bad records and consolidating records through the DQA.
Objectives
Working with the Data Quality Assistant.
Step 1. Configuring the DQA This part is done by the Developers

Create the tables
Define the connections in the Developer tool
Create the mappings to populate the bad record and duplicate record tables
Step 2. Working with the DQA This part is done by the Analysts
Define the tables in the Analyst tool
Correct Bad records
Consolidate duplicate records
In this section we will deal with Step 2. The tables have already been created and populated by
mappings run by developers in the Developer Tool. Once the data has been written to the table the
Analyst can connect to the table to review and edit the data.
Unit 5: Data Quality Assistant

Exercise 5.1: Defining the Bad Record table in the Analyst Tool
The tables have already been created and populated by mappings run by developers in the
Developer Tool. Once the data has been written to the table the Analyst can connect to the table to
review and edit the data.
1) Log into the Analyst tool if you are not in it and open the CUSTOMER_DATA project.
a) Choose to create a New Data Quality Table using the icons to the right of the screen.
b) Choose to create a Bad Records table and click Next.
c) From the established connections choose Training as this is where the customer table is.
Click Next.
d) Select the table Bad_Customer_Records from the list of tables. The DQA knows that they
associated issue table will be the original table name with _issue. Click Next
78 Unit 5: Data Quality Assistant

e) In the final window click Finish. The new Bad Records table will be displayed in the
CUSTOMER_DATA project.
f) Open the table by clicking on the hyperlink.

Working with the Data Quality Assistant
Exercise 5.2: Manage Bad Records

Once open note the bad records with the errors are displayed in bold.
1) Click on the Priority Dropdown and choose Priority 1.
a) From the Quality Issue dropdown, choose to view by invalid_company and press SHOW.
2) Select the checkbox to the left of the two KWIKSAVE records and press the Edit icon.
a) Check the box for the company field and enter the value KWIKSAVE. In the Audit Note
enter an explanation of why you are changing the name. Once complete press OK and
OK again on the pop up dialog.

b) Remove (Londis) from the first record and remove the Credit on hold from the 3 records.
Note the pencil icon to the left to note that the record has been edited but not saved.
Note the name has been changed.
3) Click on the SUPERPETZ (CREDIT ON HOLD) record and note that you are in edit mode. (If
you cant edit the field click the Show button again)
a) Correct the name of this and the Clares Pantry record by removing the words in brackets.

b) Once you have edited the records press the Save icon to save your changes. Note that
the edit icon disappears once the changes have been saved.
You can flag a record with the following status:

Accept. Flag a record as correct or acceptable for use.
Reprocess. Flag a record for reprocessing by a data quality mapping. Select this option when
you are unsure if the record is valid. Rerun data quality mapping with an updated business
rule to recheck the record.
Reject. Flag a record as not acceptable for use.
When you take action on a record you must refresh the view by pressing the Show button to remove
it from the view.
4) Select the checkbox to the left of the four records we edited. They are complete now so we
want to mark them as Accepted.
a) Once the records are selected click the Accept button to the right. The records will
remain in the view. To refresh the view press the Show button. Note the records will now
have been removed from the view.

5) Return to view all records with an invalid_zip.
It is also possible to filter records by clicking on the Filter icon or right clicking on a record and
choosing Filter.
6) Choose to create a new Filter by clicking on the Filter icon and then New.
a) Enter Spar_GBR as the name and from the dropdown select the ex_company field and
set to SPAR.
b) Press the button to add an condition line and set the ex_country = GBR.
c) Once complete press OK and Save.
Note: Note that the filter is case sensitive.

d) All of the records with an invalid zip for the company Spar in Great Britain are displayed.

e) Remove the . from the end of the 2 ADDR4 fields.
f) Save the changes and mark the two records as Reprocess and press the Show button to
refresh the view.
Exercise 5.3: Audit Trail

The Data Quality Assistant tracks changes to the staging database in an audit trail. It can be used to
review the status of the records that have passed through the record management and consolidation
processes.
1) Click on the Audit Trail button.
a) Select to view all records with a status of Updated. These are the records that we
changed.
b) An information bar at the bottom of the Audit Trail tab provides the number of records in
each status group.
Audit Trail Record Statuses
Updated Edited during Bad Record processing.
Rejected Rejected during Bad Record processing.
Accepted Accepted during Bad Record processing.
Reprocess Marked for reprocessing during Bad Record processing.

Exercise 5.4: Profile the Bad Customer Records Table
1) Right click on the Bad Customer Records table in the project view and create a new profile.
2) Create filters to profile records that:
Have not yet been updated.
Have been Accepted.
That have not been Accepted.

Manually Consolidate duplicate records
Exercise 5.5: Defining the Duplicate Record table in the Analyst Tool
The table was populated by a mapping run by the Developer in the Developer tool.
1) If you are not already logged in to the Analyst tool, log into the Analyst tool and open the
CUSTOMER_DATA project.
a) Again, choose to create a New Data Quality Table using the icons to the right of the
screen.
b) This time, choose Duplicate Records as the type of DQA source and click next.
c) Again choose Training as the established connection.
d) From the list of available tables, choose CON_CUSTOMER and click next.

e) In the final dialog choose Finish and then check that the Duplicate Record Table has
been added to the project.

Exercise 5.6: Record Consolidation
1) Double click on the table to open and view the groups of potential duplicate records, called
clusters.
Note: For the clusters to appear correctly the mapping to populate the table will need to have been run
by the developer. This was done on the training image so there is no need to run the mapping.
a) You can perform the following operations:
Identify duplicate records to consolidate into a final record.
Select a master record, and create a final record with values from the master record or from
other records in the cluster.
Identify records that do not belong in the cluster and extract these records to a new cluster.
Consolidate duplicate data to a single updated record.
The records are displayed in clusters with the first in the group being automatically selected as the
Master record. This is the record that will be updated to be the survivor record.
b) Scroll through the clusters and review the matches.
2) Scroll to Cluster 8.
a) Using the green arrow to the left of the fields, create a master record by selecting the
most appropriate field from each record.
b) Once complete select Consolidate Cluster. Note the records are removed from the view.

3) Scroll through the clusters and consolidate a few more clusters.
Exercise 5.7: Splitting a cluster

Often records that are similar, but not duplicates, can end up in the same cluster. In this instance
the clusters can be split. To split a single cluster into two clusters, deselect the checkbox for the
records you want to remove from the cluster and choose Extract Cluster. This will create two new
clusters.
1) Filter for the records SPARTAN*. Note you will need the wildcard to find these records.

2) Only one set of records is found for SPARTAN STORES.
a) These records although very similar should not be in the same cluster. Deselect the
second and last records and choose to Extract them from the cluster.
b) Once extracted consolidate the 2 clusters. (The extracted cluster will become cluster
number 1)

Exercise 5.8: The Audit Trail
We have already reviewed the Audit trail for bad record management but an audit trail is also
maintained for records that have been consolidated.
1) Click on the Audit Trail button.
a) Select to view all records with a status of Consolidated and click the Show button.
These are the duplicate records we just consolidated.
An information bar at the bottom of the Audit Trail tab provides the number of records in each status group.
Audit Trail Record Statuses

Updated
Consolidated Consolidated to a Master record during Consolidation processing.

Rematch Removed from a cluster during Consolidation processing.
Extracted Extracted from a cluster into a new cluster during Consolidation processing.
Note: All the audit details are stored in the RTM_DQA_AUDIT_TABLE table and RTM_DQA_AUDIT_DETAILS
table

Optional Additional Labs
Working with Product Data
The function of the workshop is to give users time to review what was covered on the course and
apply their knowledge to a different type of data. It should be viewed as a revision module that is
aimed at helping users become more confident with working with the Analyst Tool.
It will primarily cover Profiling, Rules and Scorecarding. The data is product data that has been
loaded into a table for you to work with.
1 - Create the Project and connect to the Data Source
a) Create a new shared Project called PRODUCT_DATA.
b) Within the new project we want to connect to the table that contains the data. From the
Actions Menu on the right choose New >Table.
(i) Choose the Training connection and press Next.
(ii) From the list of tables presented select the Product Data table and press Next, Next
and Finish.
Optional Labs
Informatica Data Quality Analyst 9.1 93
(iii) The table will be listed in your project.
c) Open the table and review the contents.
d) The table contains the following columns:
Material code Volume unit

Description GTIN
Material type Product_Length
Base unit of Product_Width
measure Product_Height
Gross weight Product unit
Net weight Created on
Weight unit Vendor ID
Product Volume
94 Optional Labs
Informatica Data Quality Analyst 9.1
Note: There is a known issue displaying large numeric values (GTIN for example) which will be corrected in the
next HF. This issue relates to how the data is displayed only and will not affect the labs.
2 - Data Profiling: Business Requirements

Build a Profile of the data and answer the questions below by adding comments or Tags to the
profile.
a) The Material code should consist of 5 numeric digits how many records do not conform
to this?
____________________________________________________________________________________
b) Review the material_desc field. How consistent do the values appear? Write a list of
some of the issues that are in this field and how they could be corrected.
_________________________________________________________________________________
__________________________________________________________________________________
c) Based on first impressions are there invalid values in the material type field? What are
they?
____________________________________________________________________________________
d) The base unit of measure should be either KG or EA. how many do not conform?
____________________________________________________________________________________
Optional Labs
e) How many of the gross and net weight values are null or 0?
____________________________________________________________________________________
f) The GTIN code should be made up of 14 numeric values. Do all of the records conform to
this?
____________________________________________________________________________________
g) Identify 5 other possible anomalies in the product data file.
____________________________________________________________________________________
____________________________________________________________________________________
____________________________________________________________________________________
____________________________________________________________________________________
____________________________________________________________________________________
3 - Data Rules: Requirements

Use a pre-built rule where possible and when necessary build and apply custom rules (making them
Shared rules available to others in the Rules > User_Data_Cleansing folder) to cover the following:
a) Check the following fields for completeness:

Description
Material Type
Created On
Vendor ID
b) Create rules in the Analyst tool that will:

(i) Verify the base unit of measure is either KG or EA.
(ii) Verify that the material code is 5 numeric characters.
(iii) Verify that the GTIN code is made up of 14 numeric values.
(iv) Verify that the Vendor ID is 4 numeric characters.
(v) Check to ensure that the appropriate material_type (from the material type field in
the record) is contained in the material_desc field, outputting 0 or Null if it does not
exist.
Note: If you need help working out which function to use the Functions in the help to assist you. You may need
to open the Developer Tool to use the help on the VM.
96 Optional Labs
4 - Scorecard: Requirements
Once complete build a scorecard on your profile to measure the following:
a) Completeness:
Material Description
Optional Labs
Material Type
Vendor ID
Created_On
b) Conformity:
Base_UOM
Material code
Gtin
Vendor ID
c) Consistency
Consistency between the Material Description and Material Type (Rule)
d) Accuracy
Material Type Valid Values:
o SP, CD, SC, PC, DT, DFM, PSC, DB, PSD, OT, CDS, PP, PS
Product Unit Valid Values:
o MM
Volume Unit Valid Values:
o PP
Weight Unit Valid Values:
o KG
2) Open the Developer Tool and review that the Projects, Profiles, Rules, Tags/Comments and
Scorecard created in the Analyst tool are available for the Developer in the Developer Tool.
Note: The solution to the workshop can be found in the

CONTENT>Training_Materials>Analyst_Training>Product Data folder.
98 Optional Labs
Notes Page
100 Notes Page
Notes Page
102 Notes Page
Notes Page
104 Notes Page
Notes Page
106 Notes Page
Notes Page
108 Notes Page

IDQ91 HF1 Analyst LG 201107 WM

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

IDQ91 HF1 Analyst LG 201107 WM

Uploaded by

Copyright:

Available Formats

Informatica Data Quality

Informatica Data Quality Analyst 9.1 HF1 iii

INTRODUCTION AND OVERVIEW .................................................................................................................. 1

THE BUSINESS CASE .................................................................................................................................................. 1

UNIT 1: IMPORTING FLAT FILES .................................................................................................................... 1

EXERCISE 1.1: GETTING STARTED IN THE ANALYST TOOL ............................................................................... 1

UNIT 2 LAB A: PROFILING, RULES AND COLLABORATION ................................................................... 9

EXERCISE 2.1: COLUMN PROFILING .................................................................................................................. 10

UNIT 3: REFERENCE TABLE MANAGEMENT ............................................................................................ 47

EXERCISE 3.1: CREATE A REFERENCE TABLE .................................................................................................. 47

SCORECARDING US_CUSTOMERORDERS ............................................................................................................ 63

UNIT 5: WORKING WITH THE DATA QUALITY ASSISTANT ................................................................. 77

OPTIONAL ADDITIONAL LABS ...................................................................................................................... 93

WORKING WITH PRODUCT DATA .......................................................................................................................... 93

Data Quality Projects - Where to start?

Lab Introduction and Overview

Step 4. Matching (Developer Task using input from the Analyst)

Step 5. Consolidation (Developer and Analyst Task)

2 Lab Introduction and Overview

Exercise 1.1: Getting started in the Analyst Tool

Step 1. Start Informatica 9.1.0 Services

Step 2. Open the Analyst Tool and Log In

Unit 1: Data Objects

4) Log into the Informatica Analyst using Analyst/Analyst

Step 3. Create a new Project

a) Name the Project CUSTOMER_DATA and make it a shared project.

a) Browse to C:\infa_shared\PDO\EU_custord.csv and click Open. Once selected click Next.

Unit 1: Data Objects

4 Unit 1: Data Objects

Unit 1: Data Objects

Unit 1: Data Objects

Profile the EU_CustomerOrders Data

6) Choose to rerun the profile by selecting the Run Profile icon.

8) An Additional example of a filter that can be created and applied is to:

5) Continue reviewing the Profiling results: COMPANY column

d) Once complete hide the comments view.

6) Continue reviewing the Profiling results: ADDR1 Column

7) Continue reviewing the Profiling results: COUNTRY_CODE column

8) Adding Filters to drilldown results.

c) Choose to add an additional condition to the drilldown by pressing the + button.

We want to add Tags to some of the fields we have just profiled.

Exercise 2.4: Create and assign a New Tag

Exercise 2.5: Assigning Tags

Exercise 2.6: Create a Custom Profile on the Data Object US_CustomerOrders

2) Ensure the US_CustomerOrders file is selected and choose Next.

a) The suggested name of Profile_US_CustomerOrders is good. Leave it as is. Change the

Step 2. Create a New Filter

Step 3. Apply the Filter to the profile.

6) Column Profiling: COUNTRY

7) Column Profiling: PHONE

8) Column Profiling: CURRENCY

Applying Prebuilt Rules US_CustomerOrders

Exercise 2.7: Pre-built Rule - Email_Validation

1) Click the Add Rule icon ( ) on the right panel

Note: Each rule contains a description explaining its function.

2) At the profiling screen, select the rule column Conformant_Email

Exercise 2.8: Pre-built Rules IsNumeric

2) Apply the rule_isNumeric to the ORDER_NO column.

5) Close the profile.

Exercise 2.10: Create a Rule for Column ADDR4 (zip code)

c) Click the green tick button to validate and press Next.

(iv) Then click Save & Run.

Exercise 2.11: Create a Rule to reformat the date columns