Practices and Methods adopted For Monarch R4B (Phase-I) Release Project

Best Practices Document for the CTB McGraw Hill Project

Document Details
Project Name Current Version List of Contributors Customer Contact Information Prepared by/Date CTB offshore team / 29-Sept-2006 Reviewed by/Date Approved by/Date Monarch R4B (Phase-I) Release ETL Process for Customer Hierarchy Replication 0.1 CTB offshore team

Table of Contents
1 INTRODUCTION.................................................................................................4
1.1 PURPOSE OF THE DOCUMENT........................................................................4 1.2 BACKGROUND AND OBJECTIVE.......................................................................4 1.3 INTENDED AUDIENCE ....................................................................................5

2 BEST PRACTICES FOLLOWED IN THE MONARCHR4B PROJECT.............6 3 GENERAL PRACTICES FOLLOWED................................................................7 4 PRACTICES FOLLOWED IN DEVELOPMENT.................................................8
INFORMATICA.....................................................................................................8 ORACLE & PL/SQL............................................................................................9

5 PRACTICES FOLLOWED IN TESTING...........................................................10 6 PRACTICES FOLLOWED IN REVIEW.............................................................11

Introduction

1.1 Purpose of the Document

The objective of this document is to showcase the best practices and methods adopted during the analysis, development, testing and review phases of the deliverables transferred to CTB as entire fulfillment for approved requirements for the MonarchR4B project.

1.2 Background and Objective

Monarch is the system of records for CTB. The R4B release has the following to be accomplished. R4B Scoring shall be categorized into following three releases: Customer Hierarchy: In the current scenario, the customer hierarchy resides in several of CTBs systems COMPASS, EISS, Navigator, CRM, OAS, and Monarch. CTB faces challenges in managing and synchronizing the hierarchies across these systems. CTB needs to establish a centralized view of the customer hierarchy in order to maintain accurate hierarchy in the central repository. Barcode, SGL & GIS: Scoring users have been executing several Job Control Language (JCL) to generate pre-coded student bar codes, Group Information Sheets and School Group Lists. The goal of this release is to accomplish the same functionality currently achieved through JCL via user interface. All bar code files will be received through Online Precoding so that the customer, if provided with the barcode information, can clean the precode files prior to sending to CTB. The School Group List information is currently captured through a paper form and the information is entered manually at CTB into the School Group List Application. The need is to capture this information electronically through the web so that the SGL information can be fed to downstream systems. Extended Completeness Check & Score checking: The 4A Completeness Checks provide scoring users a mechanism to validate the WinScore information (Op Unit reports) and ensure that student raw scores have been loaded to EISS (Completeness Check). However, there is no tight coupling or validation with EISS and users could still submit reports with incomplete data. This release will provide additional functionality to make the completeness checks more reliable, possibly tie into the automated research data checks and extend the functionality to have a tighter coupling with EISS and prevent users from submitting reports with incomplete data. Monarch currently provides a consolidated view of all the raw data for a student or group of students to ensure the completeness of data prior to report generation. This functionality will now be extended to also include verification of student raw

scores, individual derived scores and summary scores for data reasonableness. Analysis of theses score checks will be added to the Monarch dash board view such that users can take actions against any exceptions flagged Hierarchies will be generated only till the school level and propagated by the Monarch system. In EISS the remaining lower level hierarchy will be generated by the system. The objective of the project is to propagate the Customer Hierarchy changes that occur at Monarch to other systems like COMPASS, AMS/OE, and EISS. Also reverse feed the changes to customer hierarchy that occurs at CTB.com (AMS/OE) into Monarch.

1.3 Intended Audience

This document is intended for Wipro Professionals who are involved in development / maintenance in one of or related to: Informatica Client, Oracle 9.2, and PL/SQL. However some of the practices mentioned in Best Practices followed in the MonarchR4B Project, General Practices followed, Practices followed in Testing, Practices followed in Review can be suitable in general.

Best Practices followed in the MonarchR4B Project

1. Regular onsite meetings were held to synchronize the onsite people with the current development status of the project 2. An Issue Tracker was sent out on each day of the project development to the onsite team to maintain a single document that consolidated all the queries raised 3. A Single point of contact was maintained with the onsite team 4. All project deliverable dates were fixed at least 2 days in advance. 5. All project meetings with onsite were followed up with a MoM 6. All project members were synchronized to the latest Project information and data 7. All project deliverables were reviewed by the Wipro Onsite team and was followed up with Review Comments. 8. All Review comments were documented and used for review to prevent reoccurrences of errors. 9. Division of man hours in Analysis and requirements was kept large enough when compared to Development, to optimize development. 10. All review comments were consolidated and documented into a shared folder for verification during the next deliverable. 11. Started mapping development and UTC preparation almost parallel. So dint miss any test cases. 12. Loaded sample data and done the unit testing before the start of testing with actual data.

General Practices followed

1. Separate folders were used to maintain versioning of the mappings, workflows across different milestones of development, testing etc. 2. Versioning of technical documents were followed. 3. Versioning of codes, scripts were followed. 4. A central shared folder was used as a reference for accessing all latest documents and scripts 5. An Issue Tracker was sent out on each day of the project development to the onsite team to maintain a single document that consolidated all the queries raised 6. Standard templates were maintained for all documents from which all documents were prepared 7. A well documented system of Informatica Development standards was followed; the standards document was also regularly updated with time with the reviews suggested from the Onsite team. 8. All deliverables went through a minimum round of at least one review and in many cases through two, by the teams senior members before they were designated suitable for delivery (to onsite).

Practices followed in development

Informatica
1. Each mapping was reviewed by the same developer who had developed the mapping for all the naming conventions and descriptions. 2. The source and target tables were checked to verify if all the columns had appropriate data types before they were used in the mapping. 3. Backup for Informatica mappings and sessions were taken and stored in the shared folder at the end of each day. 4. The queries that were used in the mappings override were stored in the teams shared folder with versioning. 5. When the mapping was completed, the mapping specification document was updated with the new changes done in the mapping. This was be done by the same developer who developed the mapping. 6. The Review check list document was updated by the same developer who reviewed the mapping. 7. Only those source and target tables that were required were imported into the Repository. 8. Use of transformations was kept to a minimum to improve performance. 9. Data types and precision of columns were propagated. 10. All the sources related to that mapping were checked for data before development of the mapping 11. Reusable Lookups and Stored Procedures were used where applicable 12. Multiple groups in the router transformation were avoided and instead the Default Group was utilized. 13. Variable ports in Transformations were used sparingly and only when the need arises. 14. Update Strategy Transformation were used instead of multiple instances of the same target table for insert/Update activities. 15. In the Session level, the parameter file name and path were specified from the Informatica Root Directory and were not hard coded to improve flexibility. 16. In the Session level, the $Source and $Target Properties (Placeholders) were utilized to improve flexibility. 17. In the Session level, Descriptions were added for all pre and post sessions commands

Oracle & PL/SQL

1. The SQL queries developed for Informatica mappings were tested for various combinations of business keys before they were actually used in the mappings. 2. All the SQL queries, grants, table creation scripts, alter table scripts, synonyms, and stored procedure scripts were maintained in the shared folder. 3. All changes made to the scripts and queries were reflected in the shared folder. 4. Descriptions and due comments were given for each statement in the stored procedure for e.g. variable declarations, select queries used, cursors declarations, etc,. 5. Existing indexes were used in the Oracle objects (tables) to speedup query execution. 6. WITH Clauses were used in complicated SQLs to improve the performance and make it easily readable. 7. Capital letters were used for table names and keywords. 8. Oracle in-built functions were used in SQLs to minimize use of additional functions in Informatica Mappings. 9. Multiple explicit cursors were avoided in stored procedures to improve the performance. 10. Intermittent commit statements were used in stored procedures to improve the performance and space dislocation in rollback segment. 11. Minimized the Oracle tables in Stage schema to avoid space incumbent in Database. 12. Underscores were used to separate individual words in an identifier, and used "init cap" for each word (as in Maximum_Allowed_Error) to improve readability in coding. 13. Trailing comments and embedded comments (/* */) were used in coding. 14. Each local variable used was examined to determine if it can be declared as a constant. 15. All database related activities were handled by a single person.

Practices followed in Testing

1. Verification was done to ensure that the mapping and the Mapping Specification documents were in synchronization. 2. Verification was done to check if the needed transformation logic were applied to the ports. 3. Actual results were compared with the expected results to confirm if the result matched with those of the requirements. 4. Logs used in UTCs, were captured from Informatica session log files for Database connectivity, Path for target file & parameter files, Pre session &post session command executions and they were shown in the Actual Behavior column of the appropriate test cases. 5. Informatica database connection parameters were tested for both Valid and Invalid values. 6. Invalid connection values were used in testing to see if the connectivity fails. 7. Testing was done to ensure if the process fails when the data precision of the incoming field is higher than the precision specified in the mapping transformations. 8. Each individual field, of all the columns of the table, where business rules were involved, was extracted separately and tested if they meet the business requirements. They were tested using appropriate SQL queries and the SQL was attached in UTC as a part of steps. 9. The output of these queries was pasted in the Actual Behavior column of the test case. 10. The format of the records generated in the file, file names of the generated file, location of the file, no. of records in the generated file in UNIX were all checked with the appropriate UNIX commands. These commands were as a part of steps in UTC and the output log was captured in Actual Behavior column of the test case. 11. Documentation of the points discussed in daily Tele-Conferences and sending them in MOM to onsite helped to keep both Onsite and offshore teams in the same phase.

10

Practices followed in Review

1. The naming conventions were reviewed as per the standard naming convention document. 2. The SQL override query mentioned in source qualifier were verified and optimized. 3. Unused ports were deleted from all the transformations in the mapping. 4. The orders of ports (i.e., I/O, V, and O) were checked to see if they were ordered correctly order, as specified in Expression transformation. 5. Verification of Data type and size of column was done to check if all columns in all transformations in the mapping were the same. 6. Lookup transformations and stored procedures transformations were analyzed as to whether they could be made reusable. 7. The connection strings $Source, $Target in lookups and stored procedure transformations were verified. 8. Verification was done to check if the correct path and name were mentioned for parameter file, session log file, and bad file in session properties. 9. Verification was done to check if the correct name and path were mentioned for workflow log file in workflow properties. 10. Verification was done to check if the correct file name and path were specified if source and targets were of file type. 11. Verification was done to check if the correct name and parameters were mentioned for shell scripts in Pre and Post session command of a session. 12. Verification was done to check if all sessions were properly integrated in the final workflow. 13. A check list document of review comments was maintained for each mapping, session and workflow.

11