Read File/Folder(Data)

Operation Name

Read File/Folder(Data)

Function overview

Read the files/folder on Amazon S3.

Data Model

Data model of this type of component is XML type for input and table model type for output.
Refer to "Schema" for input/output schema.

Properties

For information about using variables, refer to "variables".
Basic settings
Item name Required / Optional Use of Variables Description Remarks
Name Required Not available Enter the name on the script canvas.  
Input data Required Not available Select the component on the script canvas.  
Required settings
Item name Required / Optional Use of Variables Description Remarks
Destination Required Not available Select Global Resources.
  • [Add]:
    Adds new global resource.
  • [Edit list]:
    Global resource settings can be edited with "Edit Resource list".
 
Retrieve ACL information Optional Not available Select whether to include file access right information in the results.
  • [Checked]:
    Include file access right information in the results.
  • [Not Checked]: (default)
    Do not include file access right information in the results.
 
Decryption settings
Item name Required / Optional Use of Variables Description Remarks
Secret Key Optional Available Enter file path of secret key to decrypt client encrypted file.

Click [Browse] button to activate the file Selectr and Select the file.
Data processing method
Item name Required / Optional Use of Variables Description Remarks
Mass data processing Required Not available Select a data processing method.
  • [Use script settings]: (default)
    Apply mass data processing settings of script property to adapter.
  • [Disable]:
    Mass data processing is not performed.
  • [Enable]:
    Mass data processing is performed.
 
Comment
Item name Required / Optional Use of Variables Description Remarks
Comment Optional Not available You can write a short description of this adapter.
The description will be reflected in the specifications.
 

Schema

Input Schema

<?xml version="1.0"?>
<server>
  <bucket name="" >
    <file remotepath="" type="" />
  </bucket>
</server>
Element Name Attribute Name Description Remarks
server -    
bucket - Specify each Bucket to read.  
name Specify a name of a Bucket to read.  
file - Specify each file/folder to read.  
remotepath Specify a path on Amazon S3 to read.
  • The delimiter of a path is "/".
  • A path needs to start with "/".
  • A path of a folder do not need to end with "/".
  • The root folder ("/") cannot be read.
type Specify a type of a file/folder to read.
  • File: File
  • Folder: Folder
 
Output data of Get File/Folder List operation can directly be input to this operation.

Output Schema

<?xml version="1.0"?>
<table>
  <row>
    <column>bucketname</column>
    <column>etag</column>
    <column>filename</column>
    <column>public</column>
    <column>remotepath</column>
    <column>size</column>
    <column>status</column>
    <column>storageclass</column>
    <column>type</column>
    <column>updated</column>
    <column>data</column>
    <column>contenttype</column>
  </row>
  <row>
    :
  </row>
</table>
Column Name Description Remarks
bucketname Outputs the Bucket name of the retrieved file/folder.
  • The value of the name attribute of the bucket element specified in the input data.
etag Outputs the ETag of the retrieved file/folder.
  • If file/folder information could not be retrieved successfully, the value is blank.
filename Outputs the name of the retrieved file/folder.
  • The value generated from the remotepath attribute of the file element specified in the input data.
public Output access right of retrieved files/folder.
  • true: Public
  • false: Private
  • If [Retrieve ACL information] is [Not Checked] then the value is blank.
  • If file/folder information could not be retrieved successfully, the value is blank.
remotepath Output file path on Amazon S3 of the retrieved file/folder.
  • The value of the remotepath attribute of the file element specified in the input data.
size Outputs the size of the file/folder retrieved. The unit is byte.
  • If file/folder information could not be retrieved successfully, the value is blank.
  • In case of folder, "0" is output.
status Output the status of the gotten file/folder.
  • Exist: Status that file/folder exists.
  • Virtual:Status that the folder specified in remotepath does not exist on Amazon S3 and files and/or folders exist in the folder.
  • ErrorBucketNotFound: The status that on Amazon S3, the specified Bucket does not exist in the specified bucketname.
  • ErrorRemoteFolderNotFound: The status that on Amazon S3, the specified folder does not exist in the specified remotepath.
  • ErrorRemoteFileNotFound: The status that on Amazon S3, the specified file does not exist in the specified remotepath.
  • ErrorClientEncryption:The status that failed to read, due to existence of error in the specified [Secret Key].
  • Error: The status that because an error occurred file/folder information could not be got.
 
storageclass Output Storage Class of the gotten file/folder.
  • STANDARD: Standard
  • REDUCED_REDUNDANCY: RRS(Reduced Redundancy Storage)
  • If file/folder information could not be retrieved successfully, the value is blank.
type Output the classification of gotten file/folder.
  • File: File
  • Folder: Folder
  • The value of the type attribute of the file element in the input data.
updated Output the last updated date of the retrieved file/folder.
  • If file/folder information could not be retrieved successfully, the value is blank.
data Output the received file contents in binary format.
  • In case of file, output value.
  • In case of folder, does not output value.
contenttype Output the Content-Type of the file/folder to read on Amazon S3.
  • If file/folder information could not be retrieved successfully, the value is blank.
One row is output per a file or a directory.

Loading schema in Mapper

Schema is loaded automatically.
See "Edit Schema" for details with regards to defining a schema.

Mass data processing

Mass data processing is supported.

Usage in a PSP script

Cannot be used in a PSP script.

Available component variables

Component variable name Description Remarks
count Return the number which is the total number of the gotten folder number and file number.
  • The value defaults to null.
  • The sum of "folder_count" and "file_count".
folder_count Returns the number of retrieved folder.
  • The value defaults to null.
file_count Returns the number of retrieved file.
  • The value defaults to null.
message_category Stroes the category to which corresponding message code belongs to, when an error occurs.
  • The value defaults to null.
message_code Stores its corresponding message code of occured error.
  • The value defaults to null.
message_level Stores the severity of the corresponding message code of the occured error.
  • The value defaults to null.
error_type Stores the type of the occured error.
  • The value defaults to null.
  • Error is represented in the format depicted below.
    Example:java.io.FileNotFoundException
  • The message may vary depending on the DataSpider Servista version.
error_message Stores the error message for the occured error.
  • The value defaults to null.
  • The message may vary depending on the DataSpider Servista version.
error_trace Stores stack trace of the occurred error.
  • The value defaults to null.
  • The message may vary depending on the DataSpider Servista version or the client application used.

Client decryption key length limitation

When using the first client decryption, error message of decryption may appear.
Client decryption requires maximum 256 Bit. However, Java Cryptography Extension(JCE) jurisdiction policy file restricts maximum key length used in decryption to 128 Bit.
You can dodge this restriction by installing Java Cryptography Extension(JCE) unlimited intense jurisdiction policy file.

Specification limitations

None.

Main exceptions

Exception name Causes Solution
ResourceNotFoundException
Resource definition could not be found.Name: []
[Destination] is not specified. Please specify [Destination].
ResourceNotFoundException
Resource definition could not be found.Name: [<name of Global Resources>]
Resource definition selected in [Destination] is not found. Please check the global resource specified in [Destination].
IllegalArgumentException
Invalid input data: <message>
The <message> is one of the following messages.
  • Bucket name is not specified.
  • Remote path is not specified.
  • File/folder type is not specified.
  • File/folder type<value> is invalid.
  • Element<element name> is incorrect.
Invalid data is input. Check the input data. Please check Schema.
InvalidPropertyConfigurationException
Specified secret key does not exist.
File path specified in [Secret Key] does not exist.; Specify the existing file path to [Secret Key].
InvalidPropertyConfigurationException
Specified secret key is directory.
[Secret Key] is set as directory path. Specify file path for [Secret Key].
Status Code: 403, AWS Request ID: XXXXXXXXXXXXXXXX, AWS Error Code: InvalidAccessKeyId, AWS Error Message: The AWS Access Key Id you provided does not exist in our records., S3 Extended Request ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX [Access Key ID] of global resources specified in [Destination] is not correct. Check the settings of the global resource specified in [Destination].
Status Code: 403, AWS Request ID: XXXXXXXXXXXXXXXX, AWS Error Code: SignatureDoesNotMatch, AWS Error Message: The request signature we calculated does not match the signature you provided. Check your key and signing method., S3 Extended Request ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX [Secret Access Key] of the global resources specified in [Destination] is incorrect. Check the settings of the global resource specified in [Destination].
com.amazonaws.AmazonClientException
Unable to execute HTTP request: <endpoint>
[Endpoint] of the global resource specified in [Destination] is incorrect. Check the settings of the global resource specified in [Destination].

Notes