Read File/Folder

Operation name

Read File/Folder

Function overview

Retrieves files/folders that are on Amazon S3 and writes them to the directory on HULFT Square Storage.

Data model

The data model of this component is XML type.

Tutorials

For details about how to create a connection to Amazon S3, refer to the following tutorial:

Create a connection to Amazon S3

Properties

= Remarks =

For details on use of variables, refer to Variables.

Basic settings

Item name	Required/Optional	Use of variables	Description	Remarks
Name	Required	Not available	Enter a name that is used on the script canvas.

Required settings

Item name	Required/Optional	Use of variables	Description	Remarks
Destination	Required	Not available	Select a connection resource. Add A new connection resource can be added. Edit list Connection resource settings can be edited in > HULFT INTEGRATE > Connections.
Bucket name	Required	Available	Specify a bucket.	For Bucket name, if you specify a bucket in which a large number of files exist, it may take time to update the list in Folder path.
Folder path	Required	Available	Enter an Amazon S3 folder path.	Note Specify the absolute path of Amazon S3 as the folder path. With the exception of the path separator "/", the Restricted characters of HULFT Square Storage can't be used.
File/folder name	Required	Available	Specify a file/folder.	Wildcards ("*" and "?") can be used. "/" can't be included.
Local directory	Required	Available	Enter a directory path of HULFT Square Storage. The Browse button launches a directory chooser that allows you to select a directory.	Note Specify an absolute path of HULFT Square Storage for the directory path. With the exception of the path separator "/", the Restricted characters of HULFT Square Storage can't be used. Case-insensitive.
Overwrite	Optional	Not available	Select whether to overwrite when a file with the same name exists in the specified local directory. Selected (Default) Overwrite when a file with the same name exists in the specified local directory. Not selected Output an error without overwriting if a file with the same name exists in the specified local directory.
Replace invalid file name	Optional	Not available	Select whether to replace any Restricted characters of HULFT Square Storage or tabs in the name of a file/folder on Amazon S3 before the name is retrieved. Selected When the name of a file/folder on Amazon S3 contains restricted characters of HULFT Square Storage or tabs, those characters are replaced with "_" and then the name is retrieved. Not selected (Default) A name of a file/folder on Amazon S3 isn't replaced.	For details, refer to About a name of a file/folder with invalid characters.
Recursive processing	Optional	Not available	For a specified folder, select whether to get the file/folder recursively or not. Selected If the specified folder contains folders, get the file/folder recursively. Not selected (Default) Get only the files in the folder you specify.
Include permissions in result	Optional	Not available	Select whether to include file access right information in the results. Selected Include file access right information in the results. Not selected (Default) Don't include file access right information in the results.

Decryption settings

Item name	Required/Optional	Use of variables	Description	Remarks
Secret Key	Optional	Available	Enter the file path of the secret key to decrypt the client encrypted file. The Browse button launches a file chooser that allows you to select a file.	If the secret key isn't specified, the client encrypted file will be read without decryption. Note Specify an absolute path of HULFT Square Storage for the file path. With the exception of the path separator "/", the Restricted characters of HULFT Square Storage can't be used.

Item name

Required/Optional

Use of variables

Description

Remarks

Secret Key

Optional

Available

Enter the file path of the secret key to decrypt the client encrypted file.

The Browse button launches a file chooser that allows you to select a file.

If the secret key isn't specified, the client encrypted file will be read without decryption.

Note

Specify an absolute path of HULFT Square Storage for the file path.
With the exception of the path separator "/", the Restricted characters of HULFT Square Storage can't be used.

Data processing method

Item name	Required/Optional	Use of variables	Description	Remarks
Data processing method	Required	Not available	Select a data processing method. Apply script settings (Default) The setting for mass data processing in the script properties is applied to the connector. Do not perform mass data processing Mass data processing isn't performed. Perform mass data processing Mass data processing is performed.

Item name

Required/Optional

Use of variables

Description

Remarks

Data processing method

Required

Not available

Select a data processing method.

Apply script settings

(Default)

The setting for mass data processing in the script properties is applied to the connector.

Do not perform mass data processing

Mass data processing isn't performed.

Perform mass data processing

Mass data processing is performed.

Comment

Item name	Required/Optional	Use of variables	Description	Remarks
Comment	Optional	Not available	You can write a short description of this connector.

Schemas

Input schema

None.

Output schema

<?xml version="1.0"?>
<server>
  <bucket name="" status="" >
    <file etag="" localpath="" name="" public="" remotepath="" size="" status="" storageclass="" type="" updated="" />
  </bucket>
</server>

Element name	Attribute name	Description	Remarks
server	-
bucket	-	Appears for the bucket that is specified in Bucket name.
	name	Outputs the name that is specified in Bucket name.
	status	Outputs the status of the bucket that is specified in Bucket name. Exist The bucket exists. ErrorBucketNotFound The specified bucket doesn't exist in Amazon S3. The specified Bucket name may be incorrect. Error The bucket couldn't be retrieved because an error has occurred.
file	-	Appears once for each retrieved file/folder.
	etag	Outputs the ETag of the retrieved file/folder.	The value is blank when the file/folder isn't retrieved correctly.
	localpath	Outputs the local path of the retrieved file/folder.	If status is "ErrorRemoteFolderNotFound", then the value is blank.
	name	Outputs the name on Amazon S3 of the retrieved file/folder.	If status is "ErrorLocalFolderNotFound", then the value is blank. If status is "ErrorRemoteFolderNotFound", then the value is blank.
	public	Outputs the access right of the retrieved file/folder. true Public false Private	The value is blank when the file/folder isn't retrieved correctly. When "READ" permissions is given to "AllUsers Group", it's "Public". If Include permissions in result isn't selected, then the value is blank.
	remotepath	Outputs file path on Amazon S3 of the retrieved file/folder.	If status is "ErrorLocalFolderNotFound", then the value is blank.
	size	Outputs the size of the retrieved file/folder. The unit is byte.	The value is blank when the file/folder isn't retrieved correctly. In the case of a folder, "0" is always output.
	status	Outputs the status of the retrieved file/folder. Exist The file/folder exists. Virtual The folder specified in Folder path doesn't exist on Amazon S3 but files and/or folders exist in the folder. ErrorRemoteFolderNotFound The specified Folder path doesn't exist on Amazon S3, or a file is specified. ErrorRemoteFileNotFound The specified file/folder doesn't exist in the specified Folder path on Amazon S3. ErrorLocalFileNotFound The specified Local directory doesn't exist, or a file is specified. ErrorLocalFileExist Overwrite isn't selected and a file with the same name exists in the specified Local directory. ErrorClientEncryption Reading has failed because the specified Secret Key is incorrect. Error The file/folder couldn't be read due to an error.
	storageclass	Outputs Storage Class of the retrieved file/folder. STANDARD Standard REDUCED_REDUNDANCY RRS(Reduced Redundancy Storage) STANDARD_IA Standard_IA	The value is blank when the file/folder isn't retrieved correctly.
	type	Outputs the classification of the retrieved file/folder. File File Folder Folder
	updated	Outputs the last updated date of the retrieved file/folder.	The value is blank when the file/folder isn't retrieved correctly.

Loading schema in Mapper

The schema is loaded automatically.

= Remarks =

For details, refer to Edit Schema.

Mass data processing

Mass data processing is supported.

Parallel Stream Processing

PSP isn't supported.

Available component variables

Component variable name	Description	Remarks
count	The number of retrieved folders and files is stored.	The default value is null. The sum of "folder_count" and "file_count".
folder_count	The number of retrieved folders is stored.	The default value is null.
file_count	The number of retrieved files is stored.	The default value is null.
message_category	When an error occurs, the category of the message code corresponding to the error is stored.	The default value is null.
message_code	When an error occurs, the code of the message code corresponding to the error is stored.	The default value is null.
message_level	When an error occurs, the severity of the message code corresponding to the error is stored.	The default value is null.
error_type	When an error occurs, the error type is stored.	The default value is null. The format of the error type is as follows. Example: java.io.FileNotFoundException
error_message	When an error occurs, the error message is stored.	The default value is null.
error_trace	When an error occurs, the trace information for the error is stored.	The default value is null.

About a name of a file/folder with invalid characters

Behavior about Replace invalid file name is as follows:

When this item is selected

A file/folder name on Amazon S3 isn't modified at all.
In a case where the below conditions are all satisfied, obtain the file name with a serial number at the end (before an extension if any), like "<File Name>_1.csv".
- A name of a file/folder retrieved from Amazon S3 contains restricted characters of HULFT Square Storage or tabs.
- The name after replacing restricted characters of HULFT Square Storage or tabs is the same as another file/folder name which was obtained by the same process.
When the file/folder name is the same as the file/folder name in the specified local directory, behavior depends on the Overwrite setting.

When this item isn't selected

An error occurs when a file/folder whose name has restricted characters in HULFT Square Storage is created on local.

Message codes, exception messages, and limitations

Connector	Message code	Exception message	Limitations
Messages and limitations of the Amazon S3 connector