DOW CHEMICAL ARTICLE for the DISASTER RECOVERY JOURNAL

TITLE: ______________________________
As companies adopt enterprise-wide applications to help facilitate operations throughout their
organizations, ensuring availability of these systems is absolutely essential. Applications that
were once discrete to a location or a particular business function can now be critical to the overall
business and to thousands of employees that rely on immediate use and access to these
applications to serve customers.
This is the situation that Midland, Michigan-based Dow Chemical faced when it migrated its
mainframes to an SAP environment running on mainframe DB2 databases. Dow, which has more
than 40,000 employees and presence in more than 150 countries, depends on the SAP
environment to handle all its critical enterprise financial and supply chain information worldwide.
"Our mainframe SAP environment houses our most critical data, it's relied on for the entire
supply chain all over the world, including tracking inventory, shipping, sales, order entry and
invoicing," said Bill Worsley, business continuity manager for Dow Chemical. "Not having
access to this application would have a devastating impact on us, bringing operations to a
standstill."
Additionally, because the system is highly integrated, the loss of any data could have a ripple
effect on the rest of the environment.
"If we try to reconstruct data from the previous day's backup, we wouldn't know where we left off
because a day's worth of data would be lost," explained Worsley.
With this in mind, Dow recognized it needed to establish a recovery point objective (RPO) of
zero, meaning no data loss, and a stringent recovery time of just four hours. The next step was
finding a solution that would meet these requirements.
For this, Dow turned to Comdisco to design and implement an availability solution for its SAP
environment. Dow had long used Comdisco for hot-site recovery solutions and had recently
outsourced the management of its continuity program to Comdisco.
With a requirement for no or very little data loss and a very short recovery window, Comdisco
recommended implementing a solution using E-Net's Remote Recovery Data Facility (RRDF) to
ensure the availability of Dows SAP applications. Based in San Francisco, E-Net produces
software for business continuity, remote recovery and data replication of mainframe databases.
The RRDF software, provided by E-Net through a long-standing partnership with Comdisco,
enables recovery to point-of-failure for Dow's mission critical SAP environment using real-time
remote journaling and database shadowing to a Comdisco Technology Service Center.
Implementing the Solution
Once the decision was made to proceed, the next step was to implement the RRDF solution.

Dow's enterprise SAP applications are deployed on two mainframes housed in its Midland,
Michigan data center. One mainframe runs two DB2 databases to support its operations in North
America and the Pacific region; the second mainframe runs two additional DB2 databases to
support Dow's operations in Europe and Latin America.
To help ensure against a regional disaster, the recovery environment for Dow's SAP applications
is located at Comdisco's Technology Service Center in North Bergen, N.J. An added benefit of
the RRDF solution is that it's insensitive to distance, allowing the continuity site to be thousands
of miles away with little or no additional impact on the production applications.
The RRDF software was installed on both Dow's production processors as well as on dedicated
processors at the Comdisco center. As a transaction is processed against any of Dow's DB2
databases at the production location, a duplicate copy of the database log and journal data is
captured in real-time and transmitted to Comdisco's North Bergen center instantly over a
relatively inexpensive network consisting of three dedicated T1 lines. At the Comdisco center, the
log and journal information is immediately saved to disk. Several times daily, the disks are
archived to tape, allowing the disks to be reused while ensuring that the archived data is available
should the entire database need to be recreated.
RRDF send regions running on the Dow mainframes are monitored remotely from the
Comdisco service location. Comdisco is also responsible for running and monitoring RRDF
receive regions for Dow and other remote journaling customers. RRDF buffers, filters and
compresses DB2 logstreams, thus fully utilizing the available bandwidth. Furthermore, RRDF has
fully automated spilling and gap recovery features enabling speedy recovery from day-to-day link
outages, spikes in the logging rate, or whatever software and hardware failures might occur.
In addition to live, real-time remote journaling, Dow produces daily backups, also known as
image copies, of all four of the DB2 databases at its location and ships them off-site to their
tape storage provider.
In the event of a disaster, the off-site tapes are shipped to the Comdisco North Bergen facility.
There, technicians would handle the initial restoration from those tapes and then use the RRDF
and standard DB2 recovery software to do a "roll forward" of the databases, in effect capturing all
the transactions that took place from the last tape backup to the point of failure.
The Solution is Put to the Test Under Unexpected and Potentially Difficult Circumstances
Over the past year Dow has successfully tested the solution on several occasions. But it was
during one test, when all didn't go as planned, that Dow realized additional benefits of the RRDF
solution.
The operational procedure controlling the use of DB2 log data needed for remote site recovery
had a minor problem. The Pacific region database had already been backed up on the same day
(Day Two) as the simulated disaster point, so Dow asked Comdisco to present log data from
that point forward. However, this backup tape was not available at the recovery site because it
had not yet been ejected from the tape silo to be sent off-site. It is not unusual for backup tapes to
stay at the production facility for a period of time, sometimes hours, before they are physically
sent off-site. In this case, the simulated disaster point happened to fall at an inconvenient time.

The test situation illustrated a disaster recovery planners worst nightmare sending the wrong
backup tapes or receiving unusable backup tapes. "In a traditional recovery scenario, if you don't
have the right backup tapes to restore from, you can't fully recover and you can't synchronize
your data," said Worsley. Having the Pacific region out of sync with the rest of the world as far
as database currency is concerned, could be catastrophic. We needed to have all the databases
reflect their state as of the simulated disaster point.
Using the RRDF solution, Comdisco was able to help Dow avert a test failure. As part of the
RRDF implementation, Comdisco had established a process to archive Dows data on tape and
hold these archives for several days just in case. As a result, Dow was able to use the older (and
available) database backup tape to do the initial restoration, then roll forward using two days
worth of DB2 log data from the remote journals to ultimately reach the DR test's simulated point
of failure. All databases, including the one for the Pacific region, were recovered to a consistent
point in time representing the simulated disaster point.
Dow realized during the recovery test that something was wrong, they had backups from Day
One for all databases, but the log data needed to bring the Pacific region current to the disaster
point was not provided. Tom Rechsteiner, Dows database administrator, managed the recovery
process and recognized that there was a hole in the log stream, caused by the fact that the Pacific
regions had been backed up early on Day Two. To avert the potential loss in data caused by the
gap in the Pacific region database and the disaster point, Dow worked with Comdisco to execute
a special, unplanned, RRDF reformat process to obtain the log data needed for complete recovery
of all the databases.
Comdisco has had ample experience supporting customers in disaster recoveries and recovery
tests. Each customer has unique requirements and every recovery test is different, said John
Dounis, database engineer at Comdisco. Fortunately, RRDF provides options for extracting
specific ranges of log data, enabling Comdisco to provide everything Dow needed to recover
ALL their databases.
In a disaster situation, unforeseen issues can have a significant impact on your ability to
recover, said Worsley. Knowing that we can still successfully recover - even if we have the
wrong tape or a tape that has gotten corrupted shipped to the facility - is very reassuring to us in
making certain that we have a true high-availability solution.
We are delighted Dow Chemical was able to complete their test successfully in spite of
unexpected complications, said Tom Flesher at E-Net. Dows experience shows how versatile
and forgiving recoveries can be if you have your log data. Many of our customers, including
Dow, back up their databases on a staggered basis, and some use share-level change or fuzzy
backups. Using the log, we can recover all the databases to a consistent point in time namely
the disaster point.
Flesher pointed out that the use of fuzzy backups means that the databases dont have to be
quiesced or taken offline to make the full daily backup tapes. Too often, backups for contingency
or disaster recovery require outages, compromising high availability. With the RRDF remote
journaling solution in place, Dow achieves improved availability in its day-to-day operations as
well as complete recovery at the disaster recovery site.