Read File/Directory

Operation Name

Read File/Directory

Function Overview

Write the specified file or directory which exists in Microsoft Azure BLOB Storage to another specified directory in DataSpider File System.

Data Model

XML type.

Properties

For details on use of variables, refer to "Variables".

Basic settings

Item name	Required/Optional	Use of variables	Description	Remarks
Name	Required	Not available	Enter the name to show on the script canvas.

Required settings

Item name	Required/Optional	Use of variables	Description	Remarks
Destination	Required	Not available	Select a global resource. [Add]: Add a new global resource. [Edit list]: Global resource settings can be edited in “Edit resource list” screen.
Container name	Required	Available	Specify the container.	For details on naming conventions of container name in Microsoft Azure, refer to ”Notes”. When specifying root container, create a new container named $root and select it.
Virtual directory	Optional	Available	Select or enter a virtual directory.	When omitted, it will be root directory “/”.
File/Directory name	Required	Available	Select or enter a file/directory name.	Wild cards ("*" and "?") can be used. Case-sensitive.
Set directory as target as well	Optional	Not available	Select whether or not to set the directory of [File/Directory name] as target as well. [Checked]: Files and directories of [File/Directory name] will be targets. [Not checked]: (default) Files of [File/Directory name] will be targets.
Get only file information	Optional	Not available	Select whether or not to get only file information without downloading the file in local. [Checked]: Get only file information, without downloading the file in local. [Not checked]: (default) Download the file in local.
Local directory	Required	Available	Enter a directory path of DataSpider File System. You can click [Browse] button to launch the directory chooser and find the directory.	Specify the directory path as an absolute path of DataSpider File System. Except for the path separator "/”, restricted characters of DataSpider File System cannot be used. Disabled when [Get only file information] is checked. Case-insensitive.
Overwrite when file exists	Optional	Not available	Select whether or not to overwrite when a file with the same name exists in the specified local directory. [Checked]: (default) Overwrite when a file with the same name exists in the specified local directory. [Not checked]: When a the file with the same name exists in the specified local directory, it will be an error without overwriting.	Disabled when [Get only file information] is checked.

Property actions

Item name	Description	Remarks
Update container name list	Get container names from the specified destination and set them in [Container name].	When [Destination] is changed or the container in Microsoft Azure BLOB Storage is modified, those changes can be reflected with this property action. Disabled when [Destination] is not specified.
Update virtual directory list	Get virtual directories in the specified container and set them in [Virtual directory].	When [Container name] is changed or the virtual directory in Microsoft Azure BLOB Storage is modified, those changes can be reflected by this property action. Disabled when [Destination] or [Container name] is not specified.
Update file/directory name list	Get file/directory names in the specified virtual directory and set them in [File/Directory name].	When [Virtual directory] is changed or the file in Microsoft Azure BLOB Storage is modified, those changes can be reflected with this property action. When [Set directory as target as well] is checked, directory names will be got as well. Disabled when [Destination], [Container name], or [Virtual directory] is not specified.

Data processing method

Item name	Required/Optional	Use of variables	Description	Remarks
Data processing method	Required	Not available	Select a data processing method. [Apply script settings]: (default) Apply the mass data processing settings of the script property to the adapter. [Do not perform mass data processing]: Mass data processing will not be performed. [Perform mass data processing]: Mass data processing will be performed.

Comment

Item name	Required/Optional	Use of variables	Description	Remarks
Comment	Optional	Not available	You can write a short description of this adapter. The description will be reflected in the specifications.

Schema

Input schema

None.

Output schema

<?xml version="1.0"?>
<server>
  <container name="" status="" >
    <file etag="" localpath="" name="" size="" status="" type="" updated="" uri="" />
  </container>
</server>

Element name	Attribute name	Description	Remarks
server	-
container	-
	name	Outputs the container name of the got file.
	status	Outputs the container status of the file. [Exist]: Status that container exists. [ErrorContainerNotFound]: Status that the container does not exist. The specified container name might be incorrect. Error: Status that the existence of container could not be confirmed due to an error.
file	-	Appears for each got file.
	etag	Outputs the ETag of the file.	Outputs when the specified container and the file are got normally.
	localpath	Outputs the local path of the file.	Does not output when [Get only file information] is checked.
	name	Outputs the file name in Azure BLOB Storage.
	size	Outputs the file size. Unit is byte.	Outputs when the specified container and the file are got normally.
	status	Outputs the status of the file. [Exist]: Status that file exists. [ErrorRemoteFileNotFound]: Status that file does not exist in Microsoft Azure BLOB Storage. [ErrorLocalFileNotFound]: Status that the specified [Local directory] does not exist or a file is specified. [ErrorLocalFileExist]: Status that [Overwrite when file exists] is [Not checked] and a file with the same name exists in the specified [Local directory]. [Error]: Status that the file could not be read due to an error.
	type	Outputs the type of the file.	Outputs when the specified container and the file are got normally.
	updated	Outputs the last updated date and time of the file.	Output in yyyyMMddHHmmss format. Example: “20071016131522” means October 16, 2007 13:15:22. Time zone will be in DataSpiderServer's time zone. Output when the specified file is updated normally.
	uri	Outputs the URI of the file.	Outputs when the specified container and the file are got normally.

Loading schema in Mapper

Schema is loaded automatically.

For more details, refer to “Edit Schema”.

Mass Data Processing

Mass Data Processing is supported.

PSP Usage

PSP is not supported.

Available Component Variables

Component variable name	Description	Remarks
count	The number of got files is stored.	Default value is null.
message_category	When an error occurs, category of the corresponding message code is stored.	Default value is null.
message_code	When an error occurs, code of the corresponding message code is stored.	Default value is null.
message_level	When an error occurs, severity of the corresponding message code is stored.	Default value is null.
error_type	When an error occurs, the error type is stored.	Default value is null. The error type would be in the following format. Example: “java.io.FileNotFoundException” The contents that will be stored may vary depending on DataSpider Servista versions.
error_message	When an error occurs, the error message is stored.	Default value is null. The contents that will be stored may vary depending on DataSpider Servista versions.
error_trace	When an error occurs, trace information of the error is stored.	Default value is null. The contents that will be stored may vary depending on DataSpider Servista versions and client applications.

Specification Limits

Files with the following names cannot be downloaded. status attribute of file element will be “Error”.
- When any of restricted characters of DataSpider File System other than the path separator "/” is included.
- When the file name ends with the path separator “/”.

Major Exceptions

Exception name	Cause	Solution
ResourceNotFoundException Resource definition is not found. Name: []	[Destination] is not specified.	Specify [Destination].
ResourceNotFoundException Resource definition is not found. Name: [<global resource name>]	The resource definition selected in [Destination] is not found.	Check the global resource specified in [Destination].
InvalidPropertyConfigurationException <property name> is not specified.	[<property name>] is not specified.	Specify [<property name>].
InvalidPropertyConfigurationException The local directory does not exist.	The directory path specified in [Local directory] does not exist.	Specify a directory path that exists in [Local directory].
InvalidPropertyConfigurationException The path specified in local directory is not a directory.	The path specified in [Local directory] is not a directory.	Specify a directory path in [Local directory].
InvalidPropertyConfigurationException Could not write to the local directory.	Could not write to the directory specified in [Local directory].	Specify a writable directory path in [Local directory].
com.microsoft.azure.storage.StorageException	Status unavailable to connect due to the following causes. There is something wrong with [Account] or [Access key] of the global resource specified in [Destination]. 2 byte characters are included in [Account] of the global resource specified in [Destination].	Check the settings of the global resource specified in [Destination] and the connection.
java.security.InvalidKeyException Storage Key is not a valid base64 encoded string.	There is something wrong with [Access key] of the global resource specified in [Destination].	Check the settings of the global resource specified in [Destination].

Notes

HTTPS is used in communication between Microsoft Azure BLOB Storage and this adapter.

Naming conventions of container name in Microsoft Azure
- Containers should be named in compliance with the following 4 rules.
  1. A container name starts with alphabets or numerics. It should contain only alphabets, numerics, or hyphens (-).
  2. A hyphen (-) should be both just before and after alphabets or numerics.
  3. Alphabets should be in lower case letters.
  4. The length of a container name should be between 3 and 63 characters.
When an error other than connection related errors (an error that are not described in "Major Exceptions") occurs, sets "Error" in status attribute of file element of the file/container in which the error has occurred of the result data, outputs the result data, and continues to the subsequent process.

When reading files of huge volume in multi-threads, an error may occur (status attribute of file element of the relevant file ends with "Error").
In that case, reduce the multiplicity of threads and execute it.