DATA IMPORT DOCUMENTATION

This document describes one methodology on how to set up data imports into opentaps/OFBiz.
Change Log
August 16,
2006
Initial version

2006 Open Source Strategies, Inc. All Rights Reserved.
Page 1 of 4
DATA IMPORT DOCUMENTATION
How to Import Data
O!i" co#es $ith t$o tools for i#porting data% the &ata ile tool and the 'ntit( 'ngine )*+
I#port. !oth can ,e run fro# the -e, .ools application. &ata ile is used to transfor# flat files,
including fi/ed $idth or co##a0separated files, into the 'ntit( 'ngine )*+, and these 'ntit(
'ngine )*+ files can ,e i#ported through the )*+ I#port screen in -e, .ools.
An alternative is the &ata I#port tools created for opentaps and availa,le fro# S12 via
$ svn co svn://svn.opentaps.org/dataimport/versions/0.9/trunk/ dataimport
3ou can chec4 this out into (our hot0deplo(5 director(.
.he &ata I#port tool $or4s differentl( than the standard O!i" tools ,( creating a 6,ridge7 entit(
$hich contains #ost of the fields co##on used for various ,usiness data, such as custo#ers,
and then providing services $hich $ould #ap the data into the opentaps5O!i" data #odel. .his
has a fe$ advantages over the O!i" &ata ile5'ntit( 'ngine )*+ co#,ination%
1. .he user 8ust needs to i#port data into the ,ridge entit(, $hich usuall( #aps fairl( closel(
to data in legac( s(ste#s, rather than #apping the# to the highl(0nor#ali"ed
opentaps5O!i" data #odel.
2. .he i#port into the 6,ridge7 entit( can ,e do$n $ith the native i#porting tools of
data,ases, avoiding the co#ple/it( of going through )*+ and potentiall( having to clean
up special characters.
9. .he data can then ,e #oved into opentaps5O!i" $ith transactions around each unit,
rather than the 'ntit( 'ngine, $hich $raps all records in a single transaction. .hus, if (ou
are i#porting 100,000 custo#ers, the &ata I#port tool $ill $rap each custo#er in a
transaction. If a particular custo#er failed, it $ould si#pl( s4ip past it. In contrast, the
'ntit( 'ngine tool $ould tr( to read all 100,00 custo#ers plus all their address and other
infor#ation, potentiall( over a #illion records, as one transaction. If the :;,<260th
custo#er=s address had a pro,le#, (our data i#port $ould fail as a $hole.
>urrentl( the &ata I#port tool onl( supports i#porting of custo#er records.
Importing Customers
.o use the custo#er data i#port, read (our e/isting custo#er data into the &ataI#port>usto#er
entit(, using either the O!i" datafile and entit( engine /#l i#port tools, or i#porting tools of (our
data,ase.
.hen run the 6i#port>usto#er7 service fro# ?-e, .ools@ A ?Service Reference@ AA ?Schedule@.
3ou $ill have to enter the follo$ing fields%
Initial responsi,le part(Id B part(Id of a >R*SA tea# #e#,er CAccount, Account
*anager, etc.D $ho $ill ,e initiall( responsi,le for this custo#er.
AR and offsetting E+ account Ids B E+ accounts for the accounts receiva,le outstanding
,alance of the custo#er, and an offsetting E+ account.
Organi"ation part(Id B part(Id of the internal accounting organi"ation to post the accounts
receiva,les ,alance.
How Customer Data Imports Work
>usto#er data i#port in the datai#port5 director( $or4s ,( creating a te#porar( 6,ridging entit(7
for the legac( data in the &ataI#port>usto#er entit(, defined in entit(#odelFcusto#ers./#l. It
2006 Open Source Strategies, Inc. All Rights Reserved.
Page 2 of 4
DATA IMPORT DOCUMENTATION
should #irror all the e/isting fields of (our custo#er records and has a fe$ additional fields for
#anaging the data i#port%
1. .he processed.i#esta#p is used to record $hen an i#port record is successfull(
inserted in the s(ste#. 3ou can figure out $hich records have ,een successfull(
processed or not ,( chec4ing if processed.i#esta#p is null.
2. pri#ar(Part(Id, co#pan(Part(Id, personPart(Id are filled in after i#port>usto#ers has
created records for the co#pan( and person in the custo#er record. .he( can ,e used
to help i#port other data $hich are related to (our custo#ers, such as orders and
invoices.
.he i#port>usto#er service is set up li4e an O!i" service and $ill ta4e each record of the
&ataI#port>usto#er entit( and create corresponding entities against it. CIn this case, it $ill
create Part(, Part(Eroup, Person, Part(Role, Part(Relationship, >ontact*ech, PostalAddress,
.eleco#2u#,er, Part(>ontact*ech, Part(>ontact*echPurpose, Pa(#ent*ethod, >redit>ard,
!illingAccount, !illingAccountRole, Acctg.rans, and Acctg.rans'ntr(.D If all the records are
successfull( created, then the processed.i#esta#p of the &ataI#port entit( $ill ,e updated to ,e
the ti#e of $hen the custo#er is i#ported, and the pri#ar(Part(Id, co#pan(Part(Id, and
personPart(Id $ill ,e set to the part(Ids created. If there are pro,le#s i#porting an( custo#er
record, the service $ill si#pl( s4ip past the record and tr( again $ith the ne/t one.
A fe$ i#portant differences a,out the i#port>usto#er service to note are%
1. All the entities are created using the delegator rather than ,( calling O!i" services. .his
is done to increase the efficienc( of the data i#port, as the service engine $ill add
additional overhead and slo$ it do$n.
2. .he i#port>usto#er service is not $rapped in a transaction. In the services )*+
definition, there is a flag use0transactionG7false7
9. or each i#port record, a +ist called toStore is used to put all the ne$ entities for the
custo#er record, and the delegator.storeAll #ethod is used to create all the ne$ records.
H. .he delegator.storeAll is $rapped in its o$n transaction $ith .ransactionItil.,eginCD and
.ransactionItil.co##itCD. If there is a pro,le# $ith creating the records, the
Eeneric'ntit('/ception is caught, and .ransactionItil.roll,ac4CD $ill roll,ac4 this
particular transaction, and the service continues. In contrast, #ost O!i" services $ill
a,ort and return an error on such an e/ception.
J. .here is also a catch Eeneric'/ception $hich $ill catch all other t(pes of e/ceptions,
such as nu#,er for#atting, and s4ip the record. Again, $hen a pro,le# occurs, the
service 8ust s4ips to the ne/t custo#er record instead of returning an error.
How to et Up !our Own Data Import
1. Set up a 6,ridging entit(7 to #atch the fields of (our e/isting legac( data. Add fields for
processed ti#esta#p and for I&s.
2. I#port (our legac( data into the ,ridging entit(
9. >reate a service si#ilar to i#port>usto#ers $hich $ill #ap the ,ridging entities in the
s(ste#, $rapping each record fro# the ,ridging entit( in its o$n transaction and
sta#ping it $ith a processed ti#esta#p. If there are e/ceptions $ith an( ,ridging entit(
records, si#pl( s4ip that record B do not return error or null values. &o not $rap (our
i#port service in a transaction of its o$n.
Regu"ar Imports
It is also possi,le to set up regular i#ports of data, such as inventor( infor#ation, in a si#ilar
fashion. 3ou $ould set up a ,ridge entit(, i#port (our data into it, and then run a service to
2006 Open Source Strategies, Inc. All Rights Reserved.
Page 3 of 4
DATA IMPORT DOCUMENTATION
process the#. 3ou can set the createdSta#p field of the ,ridge entit( to $hen the data is read
and then use that createdSta#p to control $hich data to process. or e/a#ple, (ou can set (our
service to process all records $hose createdSta#p is $ithin the last da( and $hose
processed.i#esta#p is null.
2006 Open Source Strategies, Inc. All Rights Reserved.
Page 4 of 4