FTP Adapter in OIC

FTP Adapter allows OIC to integrate or communicate with FTP or SFTP servers to process files where FTP is the File Transfer Protocol and SFTP is the Secure File Transfer Protocol

OIC can communicate publicly accessible FTP and private FTP running behind the firewall with the help OIC agent

Features

• It allows reading files and writing files from SFTP servers and it also generates the Schema files format based on the sample CSV file or JSON or XML files
• It also allows downloading the files from the FTP server to OIC's internal server location to process large  files
• It also allows to move files and delete files.
• It provides support for encryption and decryption using a PGP (Pretty Good Privacy) key
• It provides various methods of authentication to communicate with FTP servers 
• It supports FTP over SSL
• It supports zipping and unzipping of the files
• It supports signing the content before sending it to the FTP server and it also verifies the signature after downloading the file from FTP   

Restrictions

• FTP Adapter does not support the IBM SFTP server
• Unzipping is not supported with file download operation when the Connectivity agent is used
• FTP adapter does not support processing of the files from the same directory by two integrations
• FTP adapter does not support heterogeneous array in JSON

How to create connectivity between OIC and FTP server

Before creating the connectivity first get the below details

1. How to authenticate the SFTP server

There are multiple ways to authenticate the user while login to the FTP server

• FTP Server Access Policy: This will enable to use of User Name and Password
• FTP Public-key Authentication: This will enable to use user name with a private key and optionally passphrase  
• FTP Multi-Level Authentication: This will enable two levels of authentication with the combination of password and public key (i.e. combination of the above two policies) in the order that you specify

2. Does File contents request encrypted or decrypted using OIC

• Check if the file content needs to be encrypted before sending it to the FTP server
• Check if the file content needs to be decrypted after downloading the file
• Encrypted means converting the plain content into a non-human readable format 
• Decrypted means converting the encrypted content into a human-readable format
• If the plain content is Vivek Kumar then its encrypted content will be Vml2ZWsgS3VtYXI
• If encrypted content is Vml2ZWsgS3VtYXI then its decrypted content will be Vivek Kumar
• For Encryption/decryption OIC enables PGP public & Private keys where the PGP public key will be used to encrypt the message payload and PGP private key will be used to decrypt the message payload

 3. Collect FTP host IP address and port

4. Ensures that FTP user has read permissions on FTP folders from which file will be read

5. Ensure that FTP user has to write permission on FTP folders to which file will be written

To create the connection navigate to Integrations > Connections then click on Create then search FTP and click on FTP then click on Select like below

Then Create Connection page will be populated

Enter the Connection name, Identifier, and keys and select the role and role may be any one of the below as per requirement

• Trigger: will act as a source
• Target: will act as the target
• Trigger and Invoke: will act as both source and target

After entering these details click on Create Connection, Connection page will be opened having three sections

• Connection Properties: To configure the FTP host address, FTP server port, SFTP Connection, SSL certificate, and many more options 
• Security: To provide the security policy to login and PGP public key and private key for encryption and decryption
• Agent Group: If the FTP server is behind the firewall

After entering all these details click on test and if the test is successful save the connection

Now connection with the FTP server from OIC has been established. Now question what are the operations that you may perform on the FTP server

You can perform the below operations

1. List File

2. Read File

3. Write File

4. Delete File

5. Move File

6. Download the File to OIC

Let's understand each operation and when to use these 

1. List File: This operation will be used when you want to get the no of files available under the directory. This operation will be useful when multiple files are under a folder and you want to know the file details like file name, creation date, etc.

As per the excerpt, there are fours files, and using the list file operation we will get these four files

Now create the integration and configure the list file operation and provide the parameter like file name pattern, the input directory, and the minimum age then click next

The response from the list file will contain a number of fetched files, and a repeated File element which will contain directory, filename, last modified time, creation date, and size, and all these files will be in the ascending order based on the last modified time.

2. Read File: This operation allows you to read the file and it enables two transfer modes ASCII and Binary and you may select anyone. ASCII will allow transferring special control characters to format the data and Binary to transfer the raw bytes of the file data. You also need to provide the input directory and filename.

Whenever you are reading files, sometimes you might be required to define the schema format for the files and sometimes you may not require the schema format, just you need opaque files like PGP, GIP, etc. So whenever you need to define the structure of the file select, Yes, and whenever you need an Opaque file then select No.

If you select the structure of the file then you must have to define the structure by any one of the below methods

• Sample delimited document (e.g. CSV)
• XML Schema XSD document
• Sample XML document (Single or No namespace)
• Sample JSON document 

Then You need to select the file and provide the Record Name, Record Set Name, Field Delimiter, Character Set, and Terminated By. Where Field delimiter maybe 

• Single Space
• Comma
• Semicolon
• Tab
• Pipe

If your file contains the first row as a header for the file then you may ignore it while reading the file at runtime by selecting Use First Row as Column Headers. You also mark some fields optional and some fields mandatory by selecting mandatory and optional from the drop-down as per your requirement.

You may also define the datatype for each column.

In the end, it will generate the Schema file.

Note: Read file will read the entire file and file size should be less than 10 MB otherwise integration will fail

Response from the read file will be like below

2. Write File: This operation enables you to write the file. Whenever you write the file then you have to define the structure as well.

While writing file you may encrypt and decrypt the files based on the requirement. Let's say you have to write encrypted file at external FTP then you may use PGP public key to encrypt the file. In the similar way you may decrypt the file before sending to FTP server using PGP private key.

If no encryption/decryption required then you may select No PGP encryption/decryption on the file to be sent to external FTP server

Optionally you may sign or verify the signature before sending the file to FTP server

Sometimes while writing the files you might requires first row as a header of the files.

There are two ways to achieve this. 

• You may create the duplicate element of the File write element containing columns of the file  and then map hard coded header values for each column
• Second option is you can provide the header element in the schema file.

We will elaborate both the options

While Configuring the write operation provide the file name pattern and  Output directory to place the file. It also allow to append the content to the existing file.

You may write the opaque type files without defining the structure of the file as well as you define the file structure in the similar way that you defined in the read file operation

First we will see without defining the structure of the file so select No as below and you can not modify the file content with this option

In this case write file request will contain ICSFile element having File Refence and File properties

Where File Reference is reference of the file in the string format and File properties will have filetype, directory, filename, last modified time, creation time, size and check sum

just provide the File reference and file name and directory to write the file

How can you get the file reference

If you are reading the files from FTP server then you can get the file reference as a response from read file

If you are reading the file from Stage activity then also you may get the file reference

Note: Whenever you are writing the files and you want to specify the directory and file name at runtime in write file operation then specify these values under FTPWrite Request > outboundFTPHeaderType

then map with Directory and FileName like below. If you will specify the filename under ICSFile property section then it will not be rendered at runtime and file won't be placed at specified loactoin

Now let's write the file with content

Provide the parameter list for write operation along with output directory, filename, PGP if required and then select Yes as the structure of the content of the file and then select any one of the below to specify the structure

• Simple delimited document
• XML Schema
• Sample XML document
• Sample JSON document

We will select the delimited document then next and define the structure of the content as below and select at least of the field as mandatory fields other wise it will throw error in next step while generating the schema

Then map the employees details and save the integrations and run it. After successful run file will be placed but there won't be header. Our requirement is first line should be header

Note: Even though output folder or directory is not available under FTP, Integration will not throw the error so be careful while mapping the Folder and filename and provide these details in the similar way that you did in the above write file

As you saw there are two ways to provide the header in the above excerpt of the file.

Option 1: Once Schema file will be created, copy it and save as .xsd file and add the below highlighted line after adding your header columns separated by some delimiters like comma, tab, semicolon etc.

            nxsd:hasHeader="true"

            nxsd:headerLines="1"

    nxsd:outboundHeader="Name,Age,address,Company,Employment-type,joining-date ${eol}"

            nxsd:headerLinesTerminatedBy="${eol}"

And then define the structure based on this XSD file and complete the configuration by clicking next --> next and then finish

Then Test the integration... Now file will be generated with header file.

Option 2: Second Option is use the mapper and create the duplicate element of the Record name that you have entered while configuring the schema of the file. To do this right click on the record name and then click on Repeat Node

At the end you have two employee element and add the hardcoded values for header against each column of one the repeated employee node

This will also generate the file with headers as above.

4. Delete File: This operation will delate the file from FTP server. To delete the file select the file and directory and click on done and it will delete the file from SFTP

5. Move File: This operation will move the file from one location to other. To move file you have to specified Source directory and source file name and Target Directory and Target File name. optionally you may overwrite the file.

Response from Move file operation will be like below 

6. Download File: This operation allows to download the file from SFTP to local OIC server temp folder

This temp folder will be generated at run time with the name that you specify while configuring this operation. This temp folder will be available until instance of the integration is running state.

You have to provide Input directory, File name and Download directory (local to OIC ie. temp folder of OIC). Below features are also available with Download file operation

• You may also unzip the file
• You may decrypt the encrypted file
• you also verify the signature 

Response from Download file operation will be like below

Conclusion: We have seen various operations of FTP adapter and its uses