Copy File/Folder

Operation Name

Function Overview

Data Model

Properties

Basic Settings

Required settings

Item name	Required/Optional	Use of Variables	Description	Remarks
Destination	Required	Not Available	Select a global resource. For details about how to configure global resource, refer to "Global Resource Properties". [Add]: Adds a new global resource. [Edit list]: You can edit the global resource settings by "Edit resource list".
Copy source bucket name	Required	Available	Select or enter a copy source bucket.	When you specify a Bucket with large number of files, it may take time to update the list of [Copy source folder path]. For more details on how to avoid it, refer to "Notes".
Copy source folder path	Required	Available	Select or enter Amazon S3 folder path of copy source.	Specify the absolute path of Amazon S3 as the folder path. Constrained characters of DataSpider File System except path separator "/" cannot be used.
Copy source file/folder name	Required	Available	Select or enter a copy source file/folder.	Wild card ("*" and "?") is available. "/" Can not be included. The folder is copied recursively when it is specified.
Copy destination bucket name	Required	Available	Select or enter a copy destination bucket.	When you specify a bucket with large number of files, it may take time to update the list of [Copy destination folder path]. For more details on how to avoid it, refer to "Notes".
Create copy destination bucket when it doesn't exist	Optional	Not Available	Select whether to create a bucket when the specified copy destination bucket doesn't exist. [Checked]: If the specified copy destination bucket doesn't exist, then creates a bucket. [Not Checked]:(default) If there is no specified copy destination bucket, then doesn't create bucket and sets an error.
Copy destination folder path	Required	Available	Select or enter Amazon S3 folder path of copy destination.	Specify the absolute path of Amazon S3 as the folder path. Constrained characters of DataSpider File System except path separator "/" cannot be used.
Create copy destination folder when it doesn't exist	Optional	Not Available	Select whether to create folder when the specified copy destination folder doesn't exist. [Checked]:(default) If the specified copy destination folder doesn't exist, then creates folder. [Not checked]: If there is no specified copy destination folder, then doesn't create folder and sets an error.
Copy destination file/folder name	Optional	Available	Select or enter a copy destination file/folder.	"/" Can not be included. When omitted, it is copied with the name of copy source file/folder. When wild card is used in [Copy source file/folder name], it is copied with the name of copy source file/folder.
Overwrite a copy destination file when it exists	Optional	Not Available	Select whether to overwrite the file when a file with the same name exists on the specified copy destination folder. [Checked]:(default) Overwrites when file with the same name exists in the specified copy destination folder. [Not checked]: Sets an error without overwriting when file with the same name exists in the specified copy destination folder.
Include permissions in result	Optional	Not Available	Select whether to include information of copy destination file/folder access rights in the result. [Checked]: Include information of access rights in the result. [Not checked]:(default) Does not include information of access rights in the result.

Data processing method

Comment

Schema

Input Schema

Output Schema

<?xml version="1.0"?>
<server>
  <bucket source_name="" destination_name="" status="" >
    <file etag="" name="" public="" source_remotepath="" destination_remotepath="" size="" status="" storageclass="" type="" source_updated="" destination_updated="" />
  </bucket>
</server>

Loading Schema on Mapper

Mass Data Processing

PSP usage

Available component variables

Specification limits

• You cannot copy when the bucket specified in [Copy source bucket name] and [Copy destination bucket name] exists in different Region.


• You cannot copy the files and folders using Server-Side Encryption with Encryption Keys (SSE-C).
  For more details, refer to "Protecting Data Using Server-Side Encryption with Customer-Provided Encryption Keys (SSE-C)"(https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html).

Main Exceptions

Notes

• HTTPS is used in the communication between Amazon S3 and this adapter.


• When the settings of endpoint and the Region where bucket exists is different, it takes time to propagate the bucket and file/folder state. Hence, the operation may fail to execute without getting the state of latest bucket and file/folder.


• When you specify a bucket with large number of files in [Copy source bucket name] and [Copy destination bucket name], it may take time in updating the list in [Copy source folder path] and [Copy destination folder path].
  In such case, you can hide the items of the list by setting the following property items.
  
  

  Key	Location	Description	Remarks
  itemmaxsize	$DATASPIDER_HOME/server/plugin/data_processing/modules/amazon_s3_adapter/META-INF/ adapter.properties	Setting procedure is as follows. Stop the DataSpiderServer. Open adapter.properties file and save after adding "itemmaxsize=0". Launch the DataSpiderServer.	After setting this property, enter the folder path directly in the following items. [Copy source folder path] [Copy destination folder path]



• Region of bucket created newly in this operation changes according to the endpoint settings as follows.
  
  

  [Endpoint] settings	Region of bucket	Remarks
  unset	US Standard
  s3.amazonaws.com	US Standard
  s3-us-west-1.amazonaws.com	US-West (Northern California)
  s3-ap-southeast-1.amazonaws.com	Asia Pacific (Singapore)



• At the time of execution, when errors other than connection-related (errors not described in "Main Exceptions") occur, "Error" is set to the status attribute of file element of error occurred file/folder, output it as the result data and continue with the subsequent operation.


• You must follow the below rules for bucket name.
  1. It should begin with an alphabet or number and should contain only alphabets, numbers, period(.) and hyphen(-).
  2. It should not be in the format of an IP address (Ex: 192.168.5.4).
  3. It should not contain an under score(_).
  4. Alphabets should be in lower case.
  5. The length of the text should be more than or equal to 3 characters and less than 63 characters.
  6. It should not end with a hyphen(-).
  7. It should not contain consecutive periods(..).
  8. Periods and hyphens should not be adjacent ("-." or ".-").


• Based on the specification of Amazon S3, when the length (number of bytes after converting to UTF-8 when it contains Japanese) of copy destination file path (destination_remotepath attribute of file element) exceeds 1025, you cannot copy the file.


• The number of folders whose status attribute of file element is "Virtual" is output as the number of virtual folders in the execution log.