Get List and Data

Operation Name

Get List and Data

Function overview

Get the list of files and file contents.

Data model

Data model of this component is XML type.

Properties

For information about using variables, refer to "variables".

Basic settings

Item name	Required / Optional	Use of Variables	Description	Remarks
Name	Required	Not Available	Enter the name on the script canvas.

Required settings

Item name	Required / Optional	Use of Variables	Description	Remarks
Destination	Required	Not Available	Select Global Resources. Refer to "the global resource properties" to set up a global resource. [Add]: Adds new global resource. [Edit list]: Global resource settings can be edited with "Edit Resource list".
Path	Required	-	The path. Each column can be operated with the following buttons. [Add]: Adds the column. [Up]: Moves the order of the selected column upwards by one. [Down]: Moves the order of the selected column downwards by one. [Delete]: Deletes the column.
Path / Remote path	Required	Available	Enter file/directory on the FTP server with absolute path beginning from "/".	The file name can use wildcard ("","?"). Example: /upload/.csv Files with csv extension is under /upload directory correspond. Example: /upload/sales data??.xls Xls files which begins with "sales data" followed by two characters under /upload directory correspond. Case-sensitive.
Path / Local path	Required	Available	Enter the absolute path name beginning with "/" (slash)on the FTP client (DataSpiderServer).	[Remote path] if use a wildcard ("*","?"), please enter the directory name. Not case sensitive.
Path / Data type	Required	Not available	Select the data type. [ASCII]: [BINARY]: (default)

Read settings

Item name	Required / Optional	Use of Variables	Description	Remarks
Directory option	Required	Not available	Select the directory when updating data. [Ignore]: (Default) Ignores the directory and do not transfer. Directory information is not included in the list. [Recursive]: Downloads file/directory of the specified hierarchy recursively.
Date option	Required	Not available	Select whether to include date information. [Exclude]: (default) Results do not include date information. [Include]: Results include date information.
Check for updates when transfer files	Optional	Not available	Select whether to check to update check when transferring files to a remote file. [Checked]: (default) [Criteria for updating] transfer only if meet the condition specified. [Not Checked]: Forced to transfer.
Criteria for updating	Required	Not available	Select the condition when updating the files. [Date modified]: (default) Compares the update date between files on FTP server side and files on the DataSpiderServer, if files on DataSpiderServer side are newer, then transfer the files. The FTP server must support MDTM command. [Exist]: Transfers if file/directory dose not exist in the path of DataSpider side.	[Check for updates when transfer files] check to enable.

Comment

Item name	Required / Optional	Use of Variables	Description	Remarks
Comment	Optional	Not Available	You can write a short description of this adapter. The description will be reflected in the specifications.

Schema

Input Schema

None.

Output Schema

<?xml version="1.0"?>
<ftpserver name="" port="" user="">
  <pathnames>
    <pathname date="" filetype="" local="" remote="" size="" status="" type=""/>
  </pathnames>
</ftpserver>

Element name	Attribute name	Description	Remarks
ftpserver	-
	name	Host name of connected FTP server
	port	Port number of connected FTP server
	user	User name to connect to the server
pathnames	-
pathname	-	Emerge accordingly to the received file/directory.
	date	Get the last modified date of files. The output format is yyyyMMddHHmmss. Example: "20071016131522" means 1:15 pm and 22 seconds on October 16, 2007.	[Date option] is gotten only when [Include]. The data of directory is empty.
	filetype	Outputs the file type of received file/directory. File:File Directory:Directory
	local	Outputs the absolute path on DataSpider file system of the received file/directory.
	remote	Outputs the absolute path on FTP server of the received file/directory.
	size	Size of retrieved filesThe unit is byte.	The data of directory is empty.
	status	Download status Updated: File is downloaded, and file on DataSpider is updated. Uptodated: File is not downloaded. This status is output when you select [Date modified] at [Criteria for updating] and file on FTP server is not newer than file on DataSpider. Exists: File is not downloaded. This status is output when you select [Exist] at [Criteria for updating], and file on FTP server is existed on DataSpider. [NG]: Not downloaded because of some reason.
	type	Data Types ASCII: ASCII output when format is ASCII. BINARY: output when format is BINARY.	The data of directory is empty.

Loading schema in Mapper

Schema is loaded automatically.

See "Edit Schema" for details with regards to defining a schema.

Mass Data Processing

Mass data processing is not supported.

PSP Usage

PSP is not supported.

Available component variables

Component variable name	Description	Remarks
count	Stores numbers of the loaded file/directory.	The value defaults to null. This value is the sum of "directory_count" and "file_count".
directory_count	The number of directory retrieved returns.	The value defaults to null.
file_count	The number of files retrieved returns.	The value defaults to null.
message_category	Stroes the category to which corresponding message code belongs to, when an error occurs.	The value defaults to null.
message_code	Stores its corresponding message code of occured error.	The value defaults to null.
message_level	Stores the severity of the corresponding message code of the occured error.	The value defaults to null.
error_type	Returns the error type when error occurred.	The value defaults to null. Error is represented in the format depicted below. Example:java.io.FileNotFoundException The message may vary depending on DataSpider Servista version.
error_message	Return the error message when error occurred.	The value defaults to null. The message may vary depending on DataSpider Servista version.
error_trace	Return trace information when error occurred.	The value defaults to null. The message may vary depending on DataSpider Servista version or the client application used.

Specification Limits

ACCOUNT (ACCT) is not supported.

If you get a text file containing ASCII character codes other than code, [BINARY] must be obtained.

Issue an MDTM command and receive the modification date of files from the FTP server to see whether to update.
Some FTP servers may not use this command.If so, do not check [Check for updates when transfer files], put a check [Check for updates when transfer files], select [Exist] in [Criteria for updating].
In addition, if MDTM does not return GMT (Greenwich Mean Time) you may have trouble in determining the update.

MDTM command can be used in FTP server, we can use [Date modified] in [Check for updates when transfer files] only when MDTM command returns GMT (Greenwich Mean Time).
In addition, this feature is not available if there is a difference between FTP server and DataSpiderServer host.

In DataSpider, the modification time of files on DataSpider can not be freely set.Therefore,time of files received from FTP server is set from DataSpider host.
Therefore, there may be an error if there is time difference between FTP server and DataSpiderServer.

To download the file, you must have read permission for the file to be downloaded.Also, you need to download the path write permission.

The list of FTP files depend on the result that NLST command output to the FTP server.Sorting through files and directories is not done.

File property that can be gotten is the "Directory", "file", "symlink" .For symbolic links, obtain the data from link destination.Block special file, character special files, named pipes, AF_UNIX sockets, are not supported.Files included files that cannot be handled in the reading process will be ignored.In other words, not for downloading, not included in the results.

The error "com.enterprisedt.net.ftp.FTPException: Permission denied" may occur if the access permission to the parent directory specified in [Path / Remote path] is not authorized to users.

Main exception

Exception name	Causes	Solution
ResourceNotFoundException Resource definition could not be found.Name: []	[Destination] is not specified.	Please specify [Destination] .
ResourceNotFoundException Resource definition could not be found.Name: [<name of Global Resources>]	Resource definition selected in [Destination] is not found.	Please check the global resource specified in [Destination]
RemoteFileListParseException Failed analyze the list of remote files. The encoding specified may be incorrect.	Failed analyze the list of remote files.	Ensure the [Encoding] specified in [Destination] and the FTP server settings are correct.
FTPException	Exception if any error returned by the FTP server.	Check the contents of the message.
com.enterprisedt.net.ftp.FTPException	Exception if any error returned by the FTP server.	Check the contents of the message.
java.net.UnknownHostException	Exception if FTP server is not found.	Please check your FTP server configuration.
java.net.ConnectException	Exception if can not Connect to FTP Server.	Check if port number or FTP is running.

Notes

File path used by adapters is absolute path.Input and Output parts of the path and file path are all absolute path.

The separator used for all adapters is "/" .

Characters cannotbe used as file name.
- Not supported by dataSpiderServer and extended character encoding or external characters can not be used.
- Characters unable to connect to the FTP server cannot be used.String can not be used is as follows.
  Shift-JIS character code in parentheses .
  - ―(0x815c)
  - ～(0x8160)
  - ‖(0x8161)
  - －(0x817c)
  - ¢(0x8191)
  - £(0x8192)
  - ¬(0x81CA)
If use wildcards in file paths, can not specify wildcards in file paths.
- File path that contains wildcard as follows can be set.
  - /home/user/getdata/*
  - /home/user/getdata/*.txt
- File path that contains wildcard as follows cannot be set.
  - /home/user/*/sample.txt
  - /home/user/*data/*.txt
Start the download process, except when an error occurs when downloading files network failure, the log level "WARN" log output to the error, and then continue later.
The output schema, sets the status attribute of the file that occurs an error "NG".