Write File/Folder

Operation Name

Write File/Folder

Function overview

Write file/folder on DataSpider files system to Amazon S3.

Data Model

Data model of this component is XML type.

Properties

For information about using variables, refer to "variables".

Basic settings

Item name	Required / Optional	Use of Variables	Description	Remarks
Name	Required	Not available	Enter the name on the script canvas.

Required settings

Item name	Required / Optional	Use of Variables	Description	Remarks
Destination	Required	Not available	Select Global Resources. [Add]: Adds new global resource. [Edit list]: Global resource settings can be edited with "Edit Resource list".
Local directory	Required	Available	Specify a directory path of DataSpider filesystem. Select a directory in the directory Selectr that is launched by pressing [Browse] button.	Specify the absolute path of the DataSpider file system as the file path. Constrained characters of DataSpider File System cannot be used except for path separator "/". Does not distinguish on upper/lower cases.
Local file/directory name	Required	Available	Specify file/directory on DataSpider files system.	Wildcards("*" and "?") are available. "/" Can not be included.
Recursive processing	Optional	Not available	For specified local directory, select whether to process file/directory recursively. [Checked]: If the directory is contained in specified local directory, then process file/directory recursively. [Not Checked]:(default) Only process files in the specified local directory.
Bucket name	Required	Available	Specify Bucket.	If specifying Bucket which large number of files exist in [Bucket name], it may takes time to update list of [Folder path]. For avoidance method, refer to "Notes" section.
Create a bucket when it doesn't exist	Optional	Not available	Select whether to create a bucket when specified bucket doesn't exist. [Checked]: If the specified bucket doesn't exist, then creates a bucket. [Not Checked]:(default) If there is no specified bucket, then doesn't create bucket and sets an error.
Folder path	Required	Available	Enter Amazon S3 folder path.	The Amazon S3 folder path specified needs to be absolute path. Constrained characters of DataSpider File System cannot be used except for path separator "/".
Create folder when it doesn't exist	Optional	Not available	Select whether to create when the specified folder does not exist. [Checked]:(default) If the specified folder doesn't exist, then creates folder. [Not Checked]: If there is no specified folder, then doesn't create folder and sets an error.
Overwrite	Optional	Not available	Select whether to overwrite when file with the same name exists on the specified folder. [Checked]:(default) Overwrites when file with the same name exists in the specified folder. [Not Checked]: Sets an error without overwriting when file with the same name exists in the specified folder.
Storage class	Required	Not available	Select Storage class of specified file. [STANDARD]: (default) Set Storage class of the writing file to Standard. [REDUCED_REDUNDANCY]: Set Storage class of the writing file to RRS (Reduced Redundancy Storage). [STANDARD_IA]: Set Storage class of the writing file to Standard_IA
Permissions	Required	Not available	Select the access permission of the specified file. [Public]: Set access permissions of the writing file to "public". [Private]: (default) Set access permissions of the writing file to "private".	If setting "public", "READ" right is granted to "AllUsers Group".
Include permissions in result	Optional	Not available	Select whether to include file access right information in the results. [Checked]: Include file access right information in the results. [Not Checked]: (default) Do not include file access right information in the results.

Encryption settings

Item name	Required / Optional	Use of Variables	Description	Remarks
Encryption	Required	Not available	Select whether to encrypt file or not. [No]:(default) Does not encrypt. [Server side] Encrypts in Amazon S3 after uploading. [Client side] Writes in Amazon S3 after encryption.	For notes about client encryption, refer to "Client encryption key length limitation".
Public Key	Required	Available	Enter file path of public key to encrypt. Click [Browse] button to activate the file Selectr and Select the file.	Valid, if you select [Client side] in [Encryption]. [Public Key] needs to be the key created with RSA method of cryptographic algorithm. Specify the absolute path of the DataSpider file system as the file path. Constrained Characters of DataSpider File System except for path separator "/" cannot be used.

Data processing method

Item name	Required / Optional	Use of Variables	Description	Remarks
Mass data processing	Required	Not available	Select a data processing method. [Use script settings]: (default) Apply mass data processing settings of script property to adapter. [Disable]: Mass data processing is not performed. [Enable]: Mass data processing is performed.	This property item is to configure the processing method of result data and it does not affect files/folders to be written.

Comment

Item name	Required / Optional	Use of Variables	Description	Remarks
Comment	Optional	Not available	You can write a short description of this adapter. The description will be reflected in the specifications.

Schema

Input Schema

None.

Output Schema

<?xml version="1.0"?>
<server>
  <bucket name="" status="" >
    <file etag="" localpath="" name="" public="" remotepath="" size="" status="" storageclass="" type="" updated="" />
  </bucket>
</server>

Element Name	Attribute Name	Description	Remarks
server	-
bucket	-	Appear in each Bucket which is gotten.
	name	Output the name of the retrieved bucket.
	status	Output the status of the retrieved bucket. Exist: Status that the Bucket exists. ErrorBucketNotFound: The status that specified Bucket does not exist in Amazon S3. The specified [Bucket name] may be incorrect. Error: The status that because error has occurred, Bucket could not be obtained.
file	-	Appear in each written file/folder.
	etag	Output ETag of each written file/folder.	If file/folder is not written correctly, the value is blank.
	localpath	Output local path of written file/folder.	If status is "ErrorRemoteFolderNotFound" then the value is blank.
	name	Output name of written file/folder.	If status is "ErrorLocalFolderNotFound" then the value is blank. If status is "ErrorRemoteFolderNotFound" then the value is blank.
	public	Output access right of written files/folder. true: Public false: Private	If file/folder is not written correctly, the value is blank. Should the "READ" authority be granted to "AllUsers Group", the access authority becomes "Public". If [Include permissions in result] is [Not Checked] then the value is blank.
	remotepath	Output file path of Amazon S3 written file/folder.	If status is "ErrorLocalFolderNotFound" then the value is blank.
	size	Output size of written file / folder. The unit is byte.	Output when file/folder is written correctly. In case of folder, "0" is output.
	status	Output status of written file/folder. Exist: Status that file/folder exists. Virtual:Status that the folder specified in [Folder path] does not exist on Amazon S3 and files and/or folders exist in the folder. ErrorLocalFileNotFound: Status that the specified [Local directory] does not exist, or the specified file/directory does not exist. ErrorUnsupportedFileSize: The status that could not write due to size of the specified [Local file/directory name] exceeds 5G-byte, the state could not write. ErrorRemoteFolderNotFound: The status that the specified [Folder path] is not on the Amazon S3, or file is specified. ErrorRemoteFileExist: The status that [Overwrite] is [Not checked], and on Amazon S3, files with the same name exist in specified [Folder path]. ErrorClientEncryption:The status that failed to read, due to existence of error in the specified [Public Key]. Error: The status that because an error occurred file/folder could not be written.
	storageclass	Output Storage Class of the gotten file/folder. STANDARD: Standard REDUCED_REDUNDANCY: RRS(Reduced Redundancy Storage) STANDARD_IA:Standard_IA	If file/folder is not written correctly, the value is blank.
	type	Output classification of written file/folder. File: File Folder: Folder
	updated	Output the latest update date of written file/folder.	If file/folder is not written correctly, the value is blank. Output by the format of [xmlfw.daterenderingformat] inside system property.If it has not been set, the default format will be "yyyy-MM-dd'T'HH: mm: ss.SSSZZ "is printed on. Example :2007-10-16T13: 15:22.738 +0900 Time zone is the time zone of DataSpiderServer.

Loading schema in Mapper

Schema is loaded automatically.

See "Edit Schema" for details with regards to defining a schema.

Mass data processing

Mass data processing is supported.

Transaction

Transaction is not supported.

PSP Usage

PSP is not supported.

Available component variables

Component variable name	Description	Remarks
count	Return the number which is the total number of the written folder number and file number.	The value defaults to null. The sum of "folder_count" and "file_count".
folder_count	Return the number of written folder.	The value defaults to null.
file_count	Return the number of written file.	The value defaults to null.
message_category	Stroes the category to which corresponding message code belongs to, when an error occurs.	The value defaults to null.
message_code	Stores its corresponding message code of occured error.	The value defaults to null.
message_level	Stores the severity of the corresponding message code of the occured error.	The value defaults to null.
error_type	Stores the type of the occured error.	The value defaults to null. Error is represented in the format depicted below. Example:java.io.FileNotFoundException The message may vary depending on the DataSpider Servista version.
error_message	Stores the error message for the occured error.	The value defaults to null. The message may vary depending on the DataSpider Servista version.
error_trace	Stores stack trace of the occurred error.	The value defaults to null. The message may vary depending on the DataSpider Servista version or the client application used.

Client encryption key length limitation

When using the first client encryption, error message of encryption may appear.

Example of error message:

Unable to build cipher: Illegal key size
Make sure you have the JCE unlimited strength policy files installed and configured for your JVM

Client encryption requires maximum 256 Bit. However, Java Cryptography Extension(JCE) jurisdiction policy file restricts maximum key length used in encryption to 128 Bit.
You can dodge this restriction by installing Java Cryptography Extension(JCE) unlimited intense jurisdiction policy file.

How to get policy file
Downloads "Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files for JDK/JRE 8" from Java SE downloading page (http://www.oracle.com/technetwork/java/javase/downloads/index.html).
Uses policy file generated after deployment (local_policy.jar/US_export_policy.jar).

How to install
1. Stop DataSpiderServer and DataSpider Studio.
2. Copy the received policy file (local_policy.jar/US_export_policy.jar) under the directory $DATASPIDER_HOME/jre/lib/security.
  We recommend taking backups of policy file just in case.
3. Start DataSpiderServer and DataSpider Studio.

Specification limitations

None.

Main exceptions

Exception name	Causes	Solution
ResourceNotFoundException Resource definition could not be found.Name: []	[Destination] is not specified.	Please specify [Destination].
ResourceNotFoundException Resource definition could not be found.Name: [<name of Global Resources>]	Resource definition selected in [Destination] is not found.	Please check the global resource specified in [Destination].
InvalidPropertyConfigurationException <Property Name> is not specified.	[<Property Name>] is not specified.	Please specify [<Property Name>].
InvalidPropertyConfigurationException Local file/directory name includes "/" .	[Local file/directory name] includes "/".	Please specify the file/folder name which does not include "/" in [Local file/directory name].
InvalidPropertyConfigurationException Public key is not set.	[Public Key] is not specified.	Specify [Public Key], or Specify [Client side] value to [Encryption].
InvalidPropertyConfigurationException Specified public key does not exist.	File path specified in [Public Key] does not exist.	Specify the existing file path to [Public Key].
InvalidPropertyConfigurationException Specified public key is directory.	[Public Key] is set as directory path.	Specify file path for [Public Key].
Status Code: 403, AWS Request ID: XXXXXXXXXXXXXXXX, AWS Error Code: InvalidAccessKeyId, AWS Error Message: The AWS Access Key Id you provided does not exist in our records., S3 Extended Request ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX	[Access Key ID] of global resources specified in [Destination] is not correct.	Check the settings of the global resource specified in [Destination].
Status Code: 403, AWS Request ID: XXXXXXXXXXXXXXXX, AWS Error Code: SignatureDoesNotMatch, AWS Error Message: The request signature we calculated does not match the signature you provided. Check your key and signing method., S3 Extended Request ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX	[Secret Access Key] of the global resources specified in [Destination] is incorrect.	Check the settings of the global resource specified in [Destination].
com.amazonaws.SdkClientException Unable to execute HTTP request: <endpoint>	[Endpoint] of the global resource specified in [Destination] is incorrect.	Check the settings of the global resource specified in [Destination].

Notes

Use HTTPS for communicating between Amazon S3 and this adapter.

If the setting of Region in which Bucket exist and endpoint is different, because it takes time to propagate the status of Bucket and file/folder, you may not get the status of latest Bucket and file/folder and fail to execute the operation.

If specifying Bucket which large number of files exist in [Bucket name], it may takes time to update list of [Folder path].
in these cases, please set the following property items and do not show items in the list.

Key	Location	Description	Remarks
itemmaxsize	$DATASPIDER_HOME/server/plugin/data_processing/modules/amazon_s3_adapter/META-INF/ adapter.properties	The setting procedure is as follows. Stop DataSpiderServer. Open adapter.properties file, add "itemmaxsize=0" and save. Start DataSpiderServer.	If you set this property, please input the path of the folder to [Folder path] directly.

Region of Bucket created newly in this operation may change as follow depending on the setting of endpoint.

[Endpoint] Setting	The Region Bucket	Remarks
Not set	US Standard
s3.amazonaws.com	US Standard
s3-us-west-1.amazonaws.com	US-West (Northern California)
s3-ap-southeast-1.amazonaws.com	Asia Pacific (Singapore)

When running, if an error unrelated to the connection (error which is not described in "main exception") occurs, set [Error] to status attribute of file element of the file / folder in which error occurs, then output as result data and continue the further process. When running, if an error unrelated to the connection (error which is not described in "main exception") occurs, set [Error] to status attribute of file element of the file / folder in which error occurs, then output as result data and continue the further process.

Bucket names must follow the rules below.
1. Begin with alphanumeric letters or, numbers, and include number, periods (.), Hyphens (-) only.
2. Must not be IP address format (example: 192.168.5.4).
3. Must not contain Underscore (_).
4. Alphabetic characters must be lower case letters.
5. Length must be from 3 to 63 characters.
6. Trailing must not be hyphen (-).
7. Must not contain consecutive periods (..).
8. Must not contain adjacent hyphens and a period (-.) or (.-).

Depending on Amazon S3 specification, if the length (remotepath attribute of (file element) of file path of written file exceeds 1025, (byte number after converting to UTF-8 in case of it include Japanese), then you can not write file.

Content-Type of the file written will be set automatically by the extension.