ETL Exception Handling

Exception Levels: 1) Information: Field level exception: a field has an improper format, if the field can be populated by a default value, the process will log the error, populate the field with the default value and continue. An error row is written with sufficient information. 2) Warning: if the field/s cannot be populated by a default value then the process should handle that error by rejecting the row and logging the error 3) Critical: the process is completely shut down, to reasons such as database space constraints or unavailability of a source system. For each above instant, all exceptions must be logged and the log retained. Rejection: 1) Field level: When the data in the field is erroneous, but the field can be populated with a default value. This will usually generate an Informational Exception 2) Row level: This corresponds to a Warning level of Exception. This data in the row is not clean enough to be loaded into the warehouse, and is rejected. An error log should be written out and the row must be written out to a rejection store 3) Process level: This corresponds to the Critical level of Exception where the entire process needs to be aborted. Once the cause of the errors is determined, the entire data set may need to be sourced again, or if possible, the same data set used, once the ETL process is fixed. Architecture Exception handling: 1) Exception handling will vary by project based on the functional requirements. Staging area is used. A central error handling logging (files or table) can be used . 2) Start by designing the ETL data flow to plan for calling the reusable error logging module whenever an exception occurs. Need: The requirement to validate virtually every data element received from the source data systems mandates the development, implementation, capture and maintenance of quality indicators.

Status: Effective management of an ETL process is monitoring the status of the data integration process executions. Possible status tags such as: Not Started, In Progress, Complete, and Failed. Control: Can Prepare a central logging module ( Control table), Insert details into the Control Table to update the status of the run :
1 2 ) End Date and Time ) Status of the Job. Strategies for Exception Handling in Power Center : 1) Central Error Table for Exception Handling: For an application, row, column and business level exceptions are written to a central error table 2) Enabling Power Center Row Error Logging: When you configure the session, you can choose to log row errors to a central location. When a row error occurs, the Integration Service logs error information that lets you determine the cause and source of the error. Exceptional row errors can be logged into relational tables or flat files