Read File/Folder

Operation Name

Read File/Folder

Function Overview

Read data from a file/folder on Google Cloud Storage onto the DataSpider file system.

Data Model

The data model of this component is XML type.

Properties

For details on using variables, refer to "Variables".

Basic Settings

Item name	Required/Optional	Use of variables	Description	Remarks
Name	Required	Not Available	Enter the name on the script canvas.

Required Settings

Item name	Required/Optional	Use of variables	Description	Remarks
Destination	Required	Not Available	Select Global Resources. [Add]: Adds a new global resource. [Edit list]: Allows you to edit the global resource settings on the "Edit resource list" screen.
Bucket name of the read source	Required	Available	Select or enter the bucket to be read.
Folder path of the read source	Required	Available	Select or enter the folder path on Google Cloud Storage.	Specify a folder path as an absolute path on Google Cloud Storage. Specify slash (/) at the beginning and the end. Consecutive slashes (//) cannot be included.
File/Folder name of the read source	Required	Available	Select or enter the file/folder on Google Cloud Storage.	Wild cards (asterisk (*) and question mark (?)) can be used.
Save destination directory	Required	Available	Enter a directory path of DataSpider File System. Click the [Browse] button to launch the directory chooser, and select the directory.	Specify the directory path as an absolute path of DataSpider file system. Constrained characters of DataSpider File System except path separator "/" cannot be used. Case-insensitive.
Overwrite when one exists	Optional	Not Available	Select whether or not to overwrite a file when a file with the same name exists in the specified save destination directory. [Checked]: Overwrite a file when a file with the same name exists in the specified save destination directory. [Not checked]: (default) When a file with the same name exists in the specified save destination directory, raise an error without overwriting.

Property Action

Item name	Description	Remarks
Update the list of bucket names of the read source	Get bucket names in the specified destination and, set them in [Bucket name of the read source].	If [Destination] is edited, or a bucket has been edited on Google Cloud Storage, you can apply the changes by using this property action. Disabled when [Destination] is not specified.
Update the list of folder paths of the read source	Get folder paths in the specified bucket, and set them in [Folder path of the read source].	If [Bucket name of the read source] is edited, or folder path has been edited on Google Cloud Storage, you can apply the changes by using this property action. Disabled when [Destination] or [Bucket name of the read source] is not specified.
Update the list of file/folder names of the read source	Get file/folder names in the specified folder path and, set them in [File/Folder name of the read source].	If [Folder path of the read source] is edited, or file/folder name has been edited on Google Cloud Storage, you can apply the changes by using this property action. Disabled when [Destination], [Bucket name of the read source], or [Folder path of the read source] is not specified.

Data Processing Method

Item name	Required/Optional	Use of variables	Description	Remarks
Data processing method	Required	Not Available	Select the data processing method. [Apply script settings]: (default) Apply mass data processing settings of script property to adapter. [Do not perform mass data processing]: Mass data processing is not performed. [Perform mass data processing]: Mass data processing is performed.

Comment

Item name	Required/Optional	Use of variables	Description	Remarks
Comment	Optional	Not Available	You can write a short description of this adapter. The description will be applied in the specifications.

Schema

Input Schema

None

Output Schema

<?xml version="1.0"?>
<bucket name="" status="" >
  <file errorcode="" localpath="" name="" remotepath="" size="" status="" statuscode="" type="" updated=""/>
</bucket>

Element name	Attribute name	Description	Remarks
bucket	-	Appears in each bucket which is gotten.
	name	Outputs name of the bucket acquired.
	status	Outputs the status of the gotten bucket. Exist: Status that bucket exists. ErrorBucketNotFound: The specified [Bucket name of the read source] does not exist on Google Cloud Storage. The specified bucket name may be incorrect.
file	-	Appears in each read file/folder.
	errorcode	When an error occurs on Google Cloud Storage, output the API error code specified on Google Cloud Storage.	If no error has occurred on Google Cloud Storage, the value is a blank. For more details about API error code defined on Google Cloud Storage, refer to the following Web site: https://cloud.google.com/storage/docs/json_api/v1/status-codes
	localpath	Output the local path of the read file/folder.	When status is "RemoteFileNotFound", the value is a blank. When status is "Error", the value may be a blank.
	name	Output the name on Google Cloud Storage of the read file/folder.	When status is "RemoteFileNotFound", the value is a blank. When status is "LocalFileNotFound", the value is a blank. When status is "Error", the value is a blank.
	remotepath	Output the file path on Google Cloud Storage of the read file/folder.	When status is "LocalFileNotFound", the value is a blank. When status is "LocalFileExist" and type is "Folder", the value is a blank. When status is "Downloaded" and type is "Folder", the value is a blank. When status is "Error", the value may be a blank.
	size	When the read file/folder is a file, the file size is output. The unit is bytes.	When the file/folder was not read correctly, the value is a blank. If it is a folder, the value is a blank.
	status	Output the status of the read file/folder. Downloaded: The file/folder was read. RemoteFileNotFound: The specified [File/Folder name of the read source] does not exist in the folder path specified in [Folder path of the read source] on Google Cloud Storage. LocalFileNotFound: The specified [Save destination directory] does not exist. LocalFileExist: A file/folder with the same name as the [Folder path of the read source] or [File/Folder name of the read source] exists in the [Save destination directory]. However, when [Overwrite file when one exists] is [Checked], the file is overwritten and then the status changes to "Downloaded". Error: The file/folder could not be read, because an error occurred.
	statuscode	When an error occurs on Google Cloud Storage, output the API status code specified on Google Cloud Storage.	If no error has occurred on Google Cloud Storage, the value is a blank. For more details about the API status codes defined on Google Cloud Storage, refer to the following Web site: https://cloud.google.com/storage/docs/json_api/v1/status-codes
	type	Output the type of read files/folders. File: File Folder: Folder
	updated	Output the latest update date when the read file/folder is a file.	When the file/folder was not read correctly, the value is a blank. In the case of a folder, the value is a blank. It is output in the format set in “xmlfw.daterenderingformat” in system properties. When it is not set, it is output in the default format of "yyyy-MM-dd'T'HH:mm:ss.SSSZZ". Example: 2007-10-16T13:15:22.738+0900 Time zone is the time zone of DataSpiderServer.

Loading Schema in Mapper

Schema is loaded automatically.

For details, refer to "Edit Schema".

Mass Data Processing

Mass data processing is supported.

PSP Usage

PSP cannot be used.

Available Component Variables

Component variable name	Description	Remarks
count	Stores the number of target files/folders.	The default value is null.
message_category	When an error occurs, the category of the message code corresponding to the error that occurred is stored.	The default value is null.
message_code	When an error occurs, the code of the message code corresponding to the error that occurred is stored.	The default value is null.
message_level	When an error occurs, the severity of the message code corresponding to the error that occurred is stored.	The default value is null.
error_type	When an error occurs, the type of error that occurred is stored.	The default value is null. Error type is in the following format: Example: "java.io.FileNotFoundException" The contents to be stored may vary depending on the version of DataSpider Servista.
error_message	When an error occurs, the message of the error that occurred is stored.	The default value is null. The contents to be stored may vary depending on the version of DataSpider Servista.
error_trace	When an error occurs, the trace information of the error that occurred is stored.	The default value is null. The contents to be stored may vary depending on the version and client applications of DataSpider Servista.

Specification Limits

For more details on specification limits common to each operation, refer to "Google Cloud Storage Adapter Limitations".

Major Exceptions

Exception Name	Cause	Solution
ResourceNotFoundException Resource definition is not found. Name:[]	[Destination] is not specified.	Specify [Destination].
ResourceNotFoundException Resource definition is not found. Name:[<Name of global resources>]	Resource definition selected in [Destination] is not found.	Check the global resource specified in [Destination].
InvalidPropertyConfigurationException An incorrect folder path is specified.	The format of the [Folder path of the read source] is invalid.	Specify slash (/) at the beginning and the end, and specify a folder path that does not contain two slashes (//).
InvalidPropertyConfigurationException An incorrect file/folder name is specified.	The format of [File/Folder name of the read source] is invalid.	Specify "/" at the end for a folder, do not include "/" for a file.
InvalidCredentialException A service account authentication key is incorrect.	[Service account authentication key] in [Destination] is incorrect.	Confirm [Service account authentication key] in [Destination].
CloudStorageServiceException Failed in the operation for Google Cloud Storage.	Failed in the operation for Google Cloud Storage.	Identify the cause of the error from the status code and error code by referring to the error message list page of Google Cloud Storage (https://cloud.google.com/storage/docs/xml-api/reference-status#standardcodes), and then resolve the problem.

Notes

HTTPS is used in the communication between Google Cloud Storage and this adapter.

At the time of execution, when errors other than connection-related (errors not described in "Main Exceptions") occur, "Error" is set to the status attribute of file element of error occurred file/folder, output it as the result data and continue with the subsequent operation.