Bulk Get Data(Query)

Operation Name

Function Overview

This operation performs a Query operation with Bulk API and creates CSV file.
Job Creation, batch creation, batch result obtainment, and job closing will be done subsequently as one operation.

For details on the processes to be executed, please refer to the "Process flow". When API 50.0 or later and [API Type] of the global resource is selected as [Bulk API 2.0], Note that the process flow is different.

For the API specifications used in this operation, please refer to the version of the API document, selected in the global resource, in the below mentioned link.

"Salesforce Developer Documentation"-(https://developer.salesforce.com/docs)
- Bulk API 2.0 and Bulk API Developer Guide

Properties

For information about using variables, refer to "variables".

Basic Settings

Item name	Required/Optional	Use of Variables	Description	Supplement
Name	Required	Not Available	Enter the name that appears on the script canvas.

Required Settings

Item name	Required/Optional	Use of Variables	Description	Supplement
Destination	Required	Not Available	Select Global Resources. [Add...]: Add new global resource. [Edit...]: Global resource settings can be edited by [Edit Resource list].	A global resource on API 29.0 or later versions can be specified.
SOQL	Required	Available	Enter SOQL.	Bulk API query does not support the following SOQL. COUNT ROLLUP SUM GROUP BY CUBE OFFSET Nested SOQL queries Relationship fields Base64 fields
File	Required	Available	Input the file path of the CSV format. By pressing the [Browse] button, an file chooser will be activated and a file can be chosen.	Please specify absolute path on the DataSpider File System. The prohibited characters on the DataSpider File System except for the path delimiter "/" cannot be used.
Encoding	Required	Available	Select or input write file encode. When input, you can specify encode supported in Java SE Runtime Environment. Refer to "Supported Encodings"(http://docs.oracle.com/javase/8/docs/technotes/guides/intl/encoding.doc.html) for details.	Default value is "UTF-8".
New line code	Required	Not available	Select a new line code. [AUTO]:(default) As the new line code of DataSpiderServer running OS. Windows: CR+LF UNIX: LF [CR]: [LF]: [CR+LF]:

Property Action

Item name	Description	Supplement
Preview	Display the contents of the file specified by [File] by activating a previewer.	This becomes invalid if the specified file does not exist or a variable is set for [File]. The contents is displayed in the character encoding specified by [Encoding]. The contents of the file cannot be edited. The maximum file size to be displayed by the previewer is 1MB. If the file size exceeds 1MB, the contents up to 1MB from the beginning of the file will be displayed.

Bulk Operation Settings

Item name	Required/Optional	Use of Variables	Description	Supplement
Concurrency mode	Required	Not Available	Select the job's simultaneous execution mode. [Parallel]:(default) The job is created with the parallel processing mode. [Serial]: The job is created with the sequential processing mode.	If parallel processing is performed, database rivalry can occur. If the competition is large, readings can fail. If the consecutive mode is used, the batches will be reliably processed one after another. However, with this option, the reading processing time can increase greatly. Not displayed when API 50.0 or later and [API Type] of the global resource is selected as [Bulk API 2.0]. Due to the API specification, Runs in parallel mode
Job status after execution	Required	Not Available	Select whether or not a created job will be closed, after execution. [Close]:(default) The job will be closed after execution. [Do not close]: The job will not be closed after execution. It will remain open.	Not displayed when API 50.0 or later and [API Type] of the global resource is selected as [Bulk API 2.0]. Due to the API specification, Close the Job.
Obtain the batch results after waiting until the task is completed	Required	Not Available	Select whether or not to monitor the situation until the created process are finished, and to obtain results. [Checked]:(default) Obtain processing results. The obtained results can be passed on to the subsequent proccessing from the output schema. [Not Checked]: Do not obtain processing results.	If [Checked] is selected, the item specified as the schema definition, as well as the result information item, can be passed onto the subsequenct proccessing as Table Model type data. Please refer to "Schema" for information to be obtained. Please refer to "Usage Examples of Get Results" for examples. Display of item names depends on the settings of the selected resource. API 43.0 or earlier [Obtain the batch results after waiting until the task is completed] is displayed. API 50.0 or later and [API Type] is set to [Bulk API] [Obtain the batch results after waiting until the task is completed] is displayed. API 50.0 or later and [API Type] is set to [Bulk API 2.0] [Obtain the results after waiting until the task is completed] is displayed.
Obtain the results after waiting until the task is completed	Required	Not Available
Wait Time before the start of monitoring (sec)	Optional	Available	Input the waiting time, until the beginning of the processing state monitoring, in seconds.	The default value is "3". Will be valid only when [Obtain the batch results after waiting until the task is completed] or [Obtain the results after waiting until the task is completed] is checked. A value of 3 or greater and 10800 or smaller may be specified. If a value is specified outside of this range, an error will occur.
Batch monitoring intervals (sec)	Optional	Available	Input the intervals, which determine whether or not the results have been obtained and the process has completed, in seconds.	The default value is "10". Will be valid only when [Obtain the batch results after waiting until the task is completed] or [Obtain the results after waiting until the task is completed] is checked. A value of 10 or greater and 600 or smaller can be specified. If a value is specified outside of this range, and error will occur. Display of item names depends on the settings of the selected resource. API 43.0 or earlier [Batch monitoring intervals (sec)] is displayed. API 50.0 or later and [API Type] is set to [Bulk API] [Batch monitoring intervals (sec)] is displayed. API 50.0 or later and [API Type] is set to [Bulk API 2.0] [Monitoring intervals (sec)] is displayed.
Monitoring intervals (sec)	Optional	Available

Transaction

Item name	Required/Optional	Use of Variables	Description	Supplement
Enable transaction	Required	Not Available	Choose whether to perform transaction processing. [Checked]: Perform transaction processing. [Not Checked]:(default) Transaction processing is not performed. Refer "File type adapter transaction" for more information.

Comment

Item name	Required/Optional	Use of Variables	Description	Supplement
Comment	Optional	Not Available	You can write a short description of this adapter. The description will be reflected in the specifications.

Schema

Input Schema

None.

Output Schema

None.

Transaction

Transactions is supported.

Transaction is enabled only when [Enable transaction] in [Transaction] is checked..

Usage on PSP Script

Cannot be used on PSP Script.

Available Component variables

Component Variable Name	Description	Supplement
job_id	The IDs of the jobs created are stored.	The default value is null.
batch_id	The IDs of the batches created are stored.	The default value is null. When API 50.0 or later and [API Type] of the global resource is selected as [Bulk API 2.0], the value is not stored.
number_records_processed	Stores the number of records processed by Salesforce.	The default value is null.
number_records_failed	Stores the number of records processed by Salesforce.	The default value is null. When API 50.0 or later and [API Type] of the global resource is selected as [Bulk API 2.0], the value is not stored.
created_date	Processing start time of the Salesforce is stored.	The default value is null.
state	The processing status of Salesforce is stored.	The default value is null.
state_message	The processing status message of Salesforce is stored.	The default value is null.
apex_processing_time	Stores the processing time (ms) of triggers and other processes related to the batch.	The default value is null. When API 50.0 or later and [API Type] of the global resource is selected as [Bulk API 2.0], the value is not stored.
api_active_processing_time	Stores the processing time (ms) of the batch. Does not include time waiting for a queue or for serialization / parallelization.	The default value is null. When API 50.0 or later and [API Type] of the global resource is selected as [Bulk API 2.0], the value is not stored.
total_processing_time	Stores the total processing time (ms) of Salesforce. Does not include time waiting for a queue.	The default value is null.
system_modstamp	The processing update date and time of Salesforce is stored.	The default value is null.
queryResultCount	Number of files created on the server is stored.	The default value is null.
server_url	The end point URL after Login is stored.	The default value is null.
session_id	The session Id is stored.	The default value is null.
message_category	In the case that an error occurs, the category of the message code corresponding to the error is stored.	The default value is null.
message_code	In the case that an error occurs, the code of the message code corresponding to the error is stored.	The default value is null.
message_level	In the case that an error occurs, the importance of the message code corresponding to the error is stored.	The default value is null.
operation_api_exception_code	The ExceptionCode of the occured error, in a case of API error, is stored.	The default value is null. For any error other than an API Error, the value is not stored. The content to be stored may change according to the version of DataSpider Servista.
operation_error_message	If an error occurs, the error message of the occured error is stored.	The default value is null. The content to be stored may change according to the version of DataSpider Servista.
operation_error_trace	When an error occurs, the trace information of the occured error is stored.	The default value is null. The content to be stored may change according to the version of DataSpider Servista.

Process Flow

Process Flow will differ depending on the settings of the Global Resource.

API 43.0 or earlier, or API 50.0 or later and [API Type] is set to [Bulk API]

Jobs and batches, required for data reading, are created and managed.
Job creation and batch creation will be performed definitely.
Whether or not to execute job/batch information obtainment, batch result obtainment, and job update (Close) can be selected by the property setting value.

In the Query job, split files of up to 1 GB is created on Salesforce server.
This component downloads each split file contents and outputs as a single file.

Number in the Image	Name	Description	Supplement
(1)	Required Processes	These are mandatory processes that will take place regardless of the property settings.
(2)	Selection of Whether or Not to Execute the Batch Result Obtainment and Job/Batch Information Obtainment	Whether or not to execute can be selected by the value of the [Obtain the batch results after waiting until the task is completed] in the [Bulk Operation Settings] tab.	If the [Obtain the batch results after waiting until the task is completed] is checked, the process will take place.
(3)	Selection of whether or not to execute the Job Update (Close)	Whether or not to execute can be selected by the value of the [Job status after execution] in the [Bulk Operation Settings] tab.	If "Close" is selected for the [Job status after execution], the process will take place.
(4)	Repetition of the Job/Batch Result Obtainment	Until the process is completed ,the job/batch information obtainment will repeat at intervals of the [Batch monitoring intervals (sec)], of the [Bulk Operation Settings] tab.
(5)	Repetition of the Batch Result Obtainment	Batch results will be obtained repeatedly, the same amount of times as the number of query result files.

API 50.0 or later and [API Type] is set to [Bulk API 2.0]

Jobs required for data writing, are created and managed.
Job creation is always performed when input data exists.
Job update (Close) is done reliably regardless of property settings.
Whether or not to execute job information obtainment, and processing result obtainment can be selected by the property setting value.

Due to the API specification, batch processing is performed automatically by Salesforce, so it is not performed within this operation.

Number in the Image	Name	Description	Supplement
(1)	Required Processes	These are mandatory processes that will take place regardless of the property settings.
(2)	Selection of Whether or Not to Execute the processing Result Obtainment and Job Information Obtainment	Whether or not to execute can be selected by the value of the [Obtain the results after waiting until the task is completed] in the [Bulk Operation Settings] tab.	If the [Obtain the results after waiting until the task is completed] is checked, the process will take place.
(3)	Repetition of the Job Result Obtainment	Until the process is completed ,the job information obtainment will repeat at intervals of the [Monitoring intervals (sec)], of the [Bulk Operation Settings] tab.
(4)	Repetition of the Processing Result Obtainment	Repetition until all query results are obtained.

Specification Limits

In order to perform data reading and writing with this adapter, API's of Salesforce organization must be published.

All permissions for reading/writing data and other operations, as well as data access control, depend upon the profile, shared rule setting and API specifications in Salesforce.

The objects and fields that can be used may differ by API version. For details, please refer to the Salesforce documentation.

An item whose data type is "base 64" cannot be handled in this component.

In this adapter, the job contet type "CSV" is used.
It is not possible to create a job with another contet type, or to create batches for jobs with other content types.

Main exceptions

Exception Name	Reason	Resolution
ResourceNotFoundException Resource Definition is Not Found. Name:[]	[Destination] is not specified.	Specify [Destination].
ResourceNotFoundException Resource Definition is Not Found. Name:[<Global Resource Name>]	The resource definition selected in [Destination] cannot be found.	Verify the global resource specified in [Destination]
java.net.UnknownHostException	This exception occurs when the PROXY server specified in the global resource cannot be found.	Verify the condition of the PROXY server. Or verify [Proxy Host] of the global resource specified in the [Destination].
java.net.SocketTimeoutException connect timed out	A time-out has occurred while connecting to Salesforce.	Verify the network condition and Salesforce server condition. Or check [Connection timeout(sec)] of the global resource specified in the [Destination].
java.net.SocketTimeoutException Read timed out	A time-out has occurred while waiting for a responce from the server after connecting to Salesforce.	Verify the network condition and Salesforce server condition. Or check [Timeout(sec)] of the global resource specified in the [Destination].
jp.co.headsol.salesforce.adapter.exception.SalesforceAdapterIllegalArgumentException	Invalid value is set for the property of Salesforce Bulk adapter.	Check the error message, and verify the settings.
com.sforce.soap.partner.fault.LoginFault	Login to Salesforce has failed.	Check the ExceptionCode or error message, and refer to the information about this type of error in Salesforce-related documents etc.
com.sforce.async.AsyncApiException	An error has occurred in the batch or job executed in the Salesforce Bulk adapter.	Check the ExceptionCode or error message, and refer to the information about this type of error in Salesforce-related documents etc.
jp.co.terrasky.salesforce.bulk.api.rest.exception.RestApiException	An error has occurred in the process executed in the Salesforce Bulk adapter.	Check the ExceptionCode or error message, and refer to the information about this type of error in Salesforce-related documents etc.

Notes

If the number of query result was zero (0), this operation does not output CSV file.

If batch process is otherwise "Completed", this operation does not output CSV file.
When API 50.0 or later and [API Type] of the global resource is selected as [Bulk API 2.0], the CSV file will not be output unless the job processing is otherwise "JobComplete".

If an unexpected error occurs while writing to a CSV file, the content written up to the point of failure will not be Rolled Back. To allow for Rolling Back, please use the [Transaction] functionality.

The format of the data written to the CSV file depends on the API specification.