Using FTP Overview

By penchalaraju.yanamala

You can configure a session to use File Transfer Protocol (FTP) to read from flat
file or XML sources or write to flat file or XML targets. The Integration Service
can use FTP to access any machine it can connect to, including mainframes.
With both source and target files, use FTP to transfer the files directly or stage
them in a local directory. Access source files directly or use a file list to access
indirect source files in a session.

To use FTP file sources and targets in a session, complete the following tasks:

Create an FTP connection object in the Workflow Manager and configure the
connection attributes. For more information about creating FTP connections,
1.see FTP Connections.
Configure the session to use the FTP connection object in the session
2.properties. For more information, see Configuring FTP in a Session.

Configure an FTP connection to use SSH File Transfer Protocol (SFTP) if you
are connecting to an SFTP server. SFTP enables file transfer over a secure data
stream. The Integration Service creates an SSH2 transport layer that enables a
secure connection and access to the files on an SFTP server.

Rules and Guidelines

Use the following guidelines when using FTP with flat file or XML sources and
targets:

Specify the source or target output directory in the session properties. If you do
not specify a directory, the Integration Service stages the file in the directory
where the Integration Service runs on UNIX or in the Windows system directory.
You cannot run sessions concurrently if the sessions use the same FTP source
file or target file located on a mainframe.
If you abort a workflow containing a session that stages an FTP source or target
from a mainframe, you may need to wait for the connection to timeout before
you can run the workflow again.
To run a session using an FTP connection for an SFTP server that requires
public key authentication, the public key and private key files must be accessible
on nodes where the session will run.

Integration Service Behavior

The behavior of the Integration Service using FTP depends on the way you
configure the FTP connection and the session. The Integration Service can use
FTP to access source and target files in the following ways:

Source files. Stage source files on the machine hosting the Integration Service
or access the source files directly from the FTP host. Use a single source file or
a file list that contains indirect source files for a single source instance.
Target files. Stage target files on the machine hosting the Integration Service or
write to the target files on the FTP host.
Select staging options for the session when you select the FTP connection object
in the session properties. You can also stage files by creating a pre- or post-
session shell command to copy the files to or from the FTP host. You generally
get better performance when you access source files directly with FTP. However,
you may want to stage FTP files to keep a local archive.

Using FTP with Source Files

Use FTP in a session that reads flat file or XML file sources. You can stage the
source files for a session on the machine hosting the Integration Service. Use a
single source file or a file list for each source instance.

When you stage source files, the Integration Service copies the source file from
the FTP host to the machine hosting the Integration Service. The Integration
Service uses the local file as the source file for the session. If the local source file
exists, the Integration Service replaces the file.

The following table describes the behavior of the Integration Service using FTP
with source files:

Source Is Integration Service Behavior

Type Staged
Direct Yes Integration Service copies the file from the FTP host to the
machine hosting the Integration Service after the session
begins.
Direct No Integration Service uses FTP to access the source file directly.
Indirect Yes Integration Service reads the file list and copies the file list and
the source files to the machine hosting the Integration Service
after the session begins.
Indirect No Integration Service copies the file list to the machine hosting
the Integration Service after the session begins. The
Integration Service uses FTP to access the source files
directly.

Using FTP with Target Files

Use FTP in a session that writes to flat file or XML file targets. You can stage the
target files on the machine hosting the Integration Service before copying them to
the FTP host.

When you stage target files, the Integration Service creates a target file locally
and transfers it to the FTP host after the session completes. If you do not stage
the target file, the Integration Service writes directly to the target file on the FTP
host. If the target file exists, the Integration Service truncates the file.

If you have the Partitioning option, use FTP for multiple target partition instances.
You can write to multiple target files or a merge file on the Integration Service or
the FTP host.

Configuring FTP in a Session

Before you can configure a session to use FTP, you must create an FTP
connection object in the Workflow Manager. The Integration Service uses the
FTP connection attributes to connect to the FTP server.

After you create an FTP connection in the Workflow Manager, you can configure
a session to use FTP. To use a secure connection, select an FTP connection
object configured for SFTP. Use any session with flat file or XML sources or
targets. To configure the session, complete the following tasks for each source
and target that requires an FTP connection:

Select an FTP connection.

Configure source file properties.
Configure target file properties.

To stage the source or target file on the Integration Service machine, edit the
FTP connection in the session properties to configure the directory and file name
for the staged file.

Configuring SFTP in a Session

To run a session using an FTP connection for an SFTP server that requires
public key authentication, the public key and private key files must be accessible
on nodes where the session will run.

If the Integration Service is configured to run on primary and backup nodes,

make the key files accessible on each node configured to run the Integration
Service process.

If the Integration Service is configured to run on a grid, make the key files
accessible on each node configured to run on the grid. If you cannot put the files
on each node in the grid, create a resource in the domain and assign it to each
node where you put the files. When you create a session, configure it to use the
resource.

For example, create a custom resource called SFTP. When you create a
session, you can require the session to use the SFTP resource. The Load
Balancer will only dispatch the session to nodes where the key files are
accessible.

Selecting an FTP Connection

To configure a session to use FTP, select the connection type and the
connection object. Select an FTP connection object for each source and target
that will use the FTP connection. To use SFTP, select an FTP connection object
that is configured for SFTP. You configure connection options in the Connections
settings on the Mapping tab.

To select an FTP connection for a source or target instance:

On the Mapping tab, select the source or target instance in the Transformation
1.view.
2.Select the FTP connection type.
3.Click the Open button in the Value field.
4.Select a connection object or variable:
Use object. Select an FTP connection object. Click the Override button to
override connection attributes.
Use connection variable. Use the $FTPConnectionName session parameter,
and define the parameter in the parameter file. Override connection attributes in
the parameter file.
You can override the following attributes:
Attribute Description
Remote The remote file name for the source or target. If you use an indirect
Filename source file, enter the indirect source file name.
You must use 7-bit ASCII characters for the file name. The session
fails if you use a remote file name with Unicode characters.
If you enter a fully qualified name for the source file name, the
Integration Service ignores the path entered in the Default Remote
Directory field. The session will fail if you enclose the fully qualified file
name in single or double quotation marks.
You can use a parameter or variable for the remote file name. Use
any parameter or variable type that you can define in the parameter
file. For example, you can use a session parameter,
$ParamMyRemoteFile, as the source or target remote file name, and
set $ParamMyRemoteFile to the file name in the parameter file.
Is Staged Stages the source or target file on the Integration Service. Default is
not staged.
Is Transfer Changes the transfer mode. When enabled, the Integration Service
Mode ASCII uses ASCII transfer mode. You can use ASCII mode when
transferring files on Windows machines to ensure that the end of line
character is translated properly in text files. When disabled, the
Integration Service uses binary transfer mode. Use binary transfer
mode when transferring files on UNIX machines. Default is disabled.
5.Click OK.
Related Topics:
Overriding Connection Attributes in the Session
Overriding Connection Attributes in the Parameter File

Configuring Source File Properties

If you access source files with FTP, configure the source file properties after you
choose the FTP connection for the source instance. The source file properties
determine the source file type and the staging location. You can configure source
file properties in the Properties settings on the Mapping tab.

If you stage the source file, select the source file name, directory, and file type.

If you do not stage the source file, specify the source file type. The Integration
Service uses the remote file name and directory from the FTP connection object
and ignores the source file name and directory.

The following table describes the source file properties on the Mapping tab:

Attribute Description
Source File Indicates whether the source file contains the source data or a list of
Type files with the same file properties. Choose Direct if the source file
 contains the source data. Choose Indirect if the source file contains a
list of files.
Source File Name and path of the local source file directory used to stage the
Directory source data. By default, the Integration Service uses the service
process variable directory, $PMSourceFileDir, for file sources. The
Integration Service concatenates this field with the Source file name
field when it runs the session.
If you do not stage the source file, the Integration Service uses the file
name and directory from the FTP connection object.
The Integration Service ignores this field if you enter a fully qualified
file name in the Source file name field.
Source File Name of the local source file used to stage the source data. You can
Name enter the file name or the file name and path. If you enter a fully
qualified file name, the Integration Service ignores the Source file
directory field.
If you do not stage the source file, the Integration Service uses the
remote file name and default directory from the FTP connection
object.

Configuring Target File Properties

If you write to target files with FTP, specify the target file properties after you
specify the FTP connection for the target instance. The target file properties
determine the reject file and directory and staging location. Specify target file
properties in the Properties settings on the Mapping tab.

If you stage the target file, configure the target file name and directory and the
reject file name and directory. If you do not stage the target file, configure the
reject file and directory. The Integration Service uses the remote file name and
directory from the FTP connection object.

If you have the Partitioning option, you can also select merge file properties.

The following table describes the target file attributes on the Mapping tab:

Attribute Description
Output File Name and path of the local target file directory used to stage the
Directory target data. By default, the Integration Service uses the service
process variable directory, $PMTargetFileDir. The Integration Service
concatenates this field with the Output file name field when it runs the
session.
If you do not stage the target file, the Integration Service uses the file
name and directory from the FTP connection object.
The Integration Service ignores this field if you enter a fully qualified
file name in the Output file name field.
Output File Name of the local target file used to stage the target data. You can
Name enter the file name, or the file name and path. If you enter a fully
qualified file name, the Integration Service ignores the Output file
directory field.
If you do not stage the source file, the Integration Service uses the
remote file name and default directory from the FTP connection
object.
Partitioning FTP File Targets

When you choose an FTP connection type for the partitioned targets in a
session, you configure FTP settings for the target partitions. You can merge the
target files or individual target files for each partition.

Use the following rules and guidelines when you configure FTP settings for target
partitions:

You must use an FTP connection for each target partition.

You can choose to stage the files when selecting the connection object for the
target partition. You must stage the files to use sequential merge.
If the FTP connections for the target partitions have any settings other than a
remote file name, the Integration Service does not create a merge file.

The following table describes the actions of the Integration Service with
partitioned FTP file targets:

Merge Integration Service Behavior

Type
No Merge If you stage the files, The Integration Service creates one target file
for each partition. At the end of the session, the Integration Service
transfers the target files to the remote location.
If you do not stage the files, the Integration Service generates a target
file for each partition at the remote location.
Sequential Enable the Is Staged option in the connection object. Integration
Merge Service creates one output file for each partition. At the end of the
session, the Integration Service merges the individual output files into
a single merge file, deletes the individual output files, and transfers
the merge file to the remote location.
File List If you stage the files, the Integration Service creates the following
files:
-
Output file for each partition
-
File list that contains the names and paths of the local files
-
File list that contains the names and paths of the remote files
At the end of the session, the Integration Service transfers the files to
the remote location. If the individual target files are in the Merge File
Directory, file list contains relative paths. Otherwise, the file list
contains absolute paths.
If you do not stage the files, the Integration Service writes the data for
each partition at the remote location and creates a remote file list that
contains a list of the individual target files.
Use the file list as a source file in another mapping.
Concurrent If you stage the files, the Integration Service concurrently writes the
Merge data for all target partitions to a local merge file. At the end of the
session, the Integration Service transfers the merge file to the remote
location. The Integration Service does not write to any intermediate
output files.
If you do not stage the files, the Integration Service concurrently
writes the target data for all partitions to a merge file at the remote
location.