semantic integration explained

So what does all this talk about semantic data integration really mean? Metadata, image files, meta types? How does
semantic rationalization work? This primer explains it all.

Central to the expressor semantic data integration systemtm is the concept of semantic rationalization, the process of
mapping fields from multiple and diverse external data resources to data types and definitions that are then used
exclusively within expressor projects.

The input to the semantic rationalization process is the metadata describing the data that will be processed and
emitted by the expressor application. The output is an image file and entries within the listings of subject areas, terms,
term abbreviations, and names comprising the dictionary definitions that will be used in subsequent semantic
rationalization efforts.

Image files are specific to the information resource for which they were written but different image files will map the
same common name — the definition — to differently-named fields in their corresponding data resources. For
example, the definition part_number may be mapped in two image files to the rdbms table column
name pnum and to the COBOL data item part_no.

The expressor system represents all data as one of five expressor types — byte, date time, decimal, number, and
string. Each definition is associated with an expressor meta type, which is a descriptive alias (e.g., binary or id) for the
actual expressor data type (i.e., respectively, byte and string). The meta type assigned to a definition, and its
underlying type, may be different from the type assigned to the data in its resource, but this conversion provides for
greater consistency and ease of use across projects.

The important concept is that individuals developing expressor data integration applications work only with the
definition names and their assigned meta types and never need to refer to, or even know, the names or types used by
the actual data resources. Consequently, all of the application code and business rule development — referred to as
semantic integration — use the definition names and corresponding meta types, thereby providing a decoupling of the
integration design from the physical environment.

Let us see how definitions and meta types provide for semantic integration. Suppose the previously mentioned rdbms
column contains an alphanumeric part number [e.g., pnum varchar2(6)] while the COBOL file resource data
item identifies a numeric part number [e.g., part_no PIC 9(5)]. Note that not only are these fields different
types but they are also different sizes.
During the semantic rationalization process, each field is mapped to the definition part_number and assigned the
meta type id, an alias for the expressor string type. What is the rationale for changing the types of
the pnum and part_no fields? It is a reasonably safe assumption that mathematical operations will not be made
on the part_number field, so storing and manipulating this value as a string type will provide greater functionality
(the string manipulation and pattern matching functions will be usable), less confusing scripting (part numbers are
always represented by the same type), and memory management efficiency.

It is the responsibility of the expressor parallel data processing engine, not the integration developer, to manage the
conversions between the types in the resources and the types within the integration, so these conversions are
completely hidden from the developer, who only needs to be aware of the meta type assigned to the definition. Now
all integration projects, where a data resource contains a field mapped to the part_number definition will treat this
data as a string type.

What about the size differences? Although all part numbers are now represented by a common type, how can code
be simplified if part numbers differ in size? The answer is to write a business rule for the
definition part_number that adjusts the size of part number fields. For example, the following rule initially left
pads the part number and then extracts the right-most characters as the formatted part number value.

string.substring(string.concatenate("00000000", part_number), -8)

Note that the formatted part number actually contains eight characters, which means that the rule is usable with data
resources where part number values contain up to eight characters. Of course, the rule can be modified to format the
part number to any width using any padding character. Writing a comparable rule for a numeric field would be much
more complicated, especially when using zero as the padding digit. The first step of integration would apply this rule to
the incoming part number and all subsequent steps would work with an eight character wide string field.

As part of the expressor semantic integration paradigm, it is important to note that all transformation code and
business rules refer to the data fields using their definition names and not the names used by their respective
resources. The expressor illustrator -- a Visio based graphical development tool -- scripting window, coding aids, and
schema viewers show only the definition names and access the definition specific business rules. Consequently, all
data manipulation within the expressor processing engine uses these semantically rationalized definition names and
their corresponding expressor data types. Only when data is read into, or emitted from, the expressor integration does
the engine use the resource specific field names and types.