Data Check

Operation name

Data Check

Function overview

Checks the values and formats of input data and outputs the result in selected format.

Properties

= Remarks =

For details on use of variables, refer to Variables.

Basic settings

Item name	Required/Optional	Use of variables	Description	Remarks
Name	Required	Not available	Enter a name that is used on the script canvas.
Input data	Required	Not available	Select a component on the script canvas.

Required settings

Item name	Required/Optional	Use of variables	Description	Remarks
Use a connection setting for data check condition	Required	Not available	Select whether or not to use a connection resource for the data check condition. Selected Use a connection resource. Not selected (Default) Do not use a connection resource.
Data check condition definition	Required	Not available	Select a data check condition definition registered in a connection resource. = Remarks = Items that can be set in a connection resource are the same as those of Data check condition list. Add A new connection resource can be added. Edit list Connection resource settings can be edited in > HULFT INTEGRATE > Data Formats.
Data check condition list	Optional	-	Specify conditions with which input data columns will be checked. Each column can be operated with the following buttons: Up Moves the selected column upward by one row. Down Moves the selected column downward by one row. Add Adds a column. Delete Deletes the column.	This field is displayed when Use a connection setting for data check condition isn't selected.
Data check condition list/Column number	Required	Available	Enter a column number of input data to be checked.
Data check condition list/Required	Optional	Not available	Select whether or not entry of data for Column number is required. Selected Entry is required. Not selected (Default) Entry isn't required.	When the input data for a column where Required is selected is null or an empty string, the subsequent checks for Type, Minimum, Maximum, Option, and Regular expression won't be performed.
Data check condition list/Type	Required	Not available	Select the type to check the data for Column number with. String (character count) (Default) String (byte count) Integer Decimal Date/Time	When specifying Integer or Decimal in Type and the input data to be checked is a non-numeric value, such as "abc", the subsequent checks for Minimum, Maximum, Option, Regular expression won't be performed.
Data check condition list/Minimum	Optional	Available	Enter the minimum number of digits, minimum byte count, or minimum value to allow for data for Column number.	When String (character count) is specified in Type, Minimum will be handled as number of digits. When String (byte count) is specified in Type, Minimum will be handled as byte count. When Integer or Decimal is specified in Type, Minimum will be handled as value. When Date/Time is specified in Type, the value set in Minimum will be ignored.
Data check condition list/Maximum	Optional	Available	Enter the maximum number of digits, maximum byte count, or maximum value to allow for data for Column number.	When String (character count) is specified in Type, Maximum will be handled as number of digits. When String (byte count) is specified in Type, Maximum will be handled as byte count. When Integer or Decimal is specified in Type, Maximum will be handled as value. When Data/Time is specified in Type, the value set in Maximum will be ignored.
Data check condition list/Option	Optional	Available	Specify the option to check the data for Column number with.	For more details on how to specify option, refer to Option.
Data check condition list/Regular expression	Optional	Available	Specify the regular expression to check the data for Column number with.	The available regular expression patterns conform to java.util.regex.Pattern. For more details on Pattern, refer to "Java(TM) Platform, Standard Edition 8 API Specification" (http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html).
Output record	Required	Not available	Select a record type to be output. Output only records matching conditions (Default) Output only records that match the conditions. Output only records not matching conditions Output only the records that don't match the conditions. Output all records and check results Assign check results to all the read records and output them.	When Output all records and check results is selected, the columns of the input data plus the check result column are output.

Condition evaluation settings

Item name	Required/Optional	Use of variables	Description	Remarks
Evaluation method of multiple conditions	Required	Not available	Set how to evaluate conditions when multiple data check conditions exist. AND evaluation (Default) The check is passed when all the conditions are satisfied. OR evaluation The check is passed when any one of the conditions is satisfied. = Remarks = When there are multiple data check conditions for one column number in Data check condition list, you can set the evaluation method for that using Specify evaluation method of multiple conditions for same column number and Evaluation method for same column number.
Specify evaluation method of multiple conditions for same column number	Optional	Not available	Select whether or not to specify how to evaluate conditions when multiple data check conditions exist for one column number. Selected Use the evaluation method specified in Evaluation method for same column number to check data. Not selected (Default) Use the evaluation method specified in Evaluation method of multiple conditions to check data even when there are multiple data check conditions for one column number.
Evaluation method for same column number	Optional	Not available	Set how to evaluate conditions when multiple data check conditions exist for one column number. AND evaluation (Default) The check is passed when all the conditions specified for one column number in Data check condition list are satisfied. OR evaluation The check is passed when any one of the conditions specified for one column number in Data check condition list is satisfied.	Enabled when Specify evaluation method of multiple conditions for same column number is selected.

Item name

Required/Optional

Use of variables

Description

Remarks

Evaluation method of multiple conditions

Required

Not available

Set how to evaluate conditions when multiple data check conditions exist.

AND evaluation

(Default)

The check is passed when all the conditions are satisfied.

OR evaluation

The check is passed when any one of the conditions is satisfied.

= Remarks =

When there are multiple data check conditions for one column number in Data check condition list, you can set the evaluation method for that using Specify evaluation method of multiple conditions for same column number and Evaluation method for same column number.

Specify evaluation method of multiple conditions for same column number

Optional

Not available

Select whether or not to specify how to evaluate conditions when multiple data check conditions exist for one column number.

Selected

Use the evaluation method specified in Evaluation method for same column number to check data.

Not selected

(Default)

Use the evaluation method specified in Evaluation method of multiple conditions to check data even when there are multiple data check conditions for one column number.

Evaluation method for same column number

Optional

Not available

Set how to evaluate conditions when multiple data check conditions exist for one column number.

AND evaluation

(Default)

The check is passed when all the conditions specified for one column number in Data check condition list are satisfied.

OR evaluation

The check is passed when any one of the conditions specified for one column number in Data check condition list is satisfied.

Enabled when Specify evaluation method of multiple conditions for same column number is selected.

Data processing method

Item name	Required/Optional	Use of variables	Description	Remarks
Data processing method	Required	Not available	Select a data processing method. Apply script settings (Default) The setting for mass data processing in the script properties is applied to the connector. Do not perform mass data processing Mass data processing isn't performed. Perform mass data processing Mass data processing is performed.

Item name

Required/Optional

Use of variables

Description

Remarks

Data processing method

Required

Not available

Select a data processing method.

Apply script settings

(Default)

The setting for mass data processing in the script properties is applied to the connector.

Do not perform mass data processing

Mass data processing isn't performed.

Perform mass data processing

Mass data processing is performed.

Comment

Item name	Required/Optional	Use of variables	Description	Remarks
Comment	Optional	Not available	You can write a short description of this connector.

Option

In the Option item, type-specific settings can be assigned in the "key=value" format. When setting multiple keys, delimit them using "," (comma).

Options that can be set for each type are as follows:

Corresponding type	Key	Meaning	Value	Remarks
String (byte count)	CS	Set the encoding to be used in calculation of byte count.	Specify the encoding supported by Java SE Runtime Environment 8. = Remarks = For details, refer to "Supported Encodings" (http://docs.oracle.com/javase/8/docs/technotes/guides/intl/encoding.doc.html).	When the option isn't specified, the default encoding selected for Encode of file type connectors is UTF-8.
Integer	AF	Set whether to allow multibyte numeric values.	true Multibyte numeric values are allowed. false (Default) Multibyte numeric values aren't allowed.
Decimal	I	Set the maximum number of digits for the integer part.	Allowed values are from 1 to 2147483647.
Decimal	D	Set the maximum number of digits for the fractional part.	Allowed values are from 1 to 2147483647.
Date/Time	DF	Set the date format.	Date formats supported in java.text.SimpleDateFormat can be set. = Remarks = For further details about SimpleDateFormat, refer to "Java(TM) Platform, Standard Edition 8 API Specification" (http://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html).

Schemas

Input schema

Same as the schema of input source component.

Output schema

Same as the schema of input source component.

When Output all records and check results is selected in Output record, a string type column is added to the schema of the result data, and the check result for the record will be output as "OK" or "NG".

= Remarks =

For schema structure, refer to Table model type .

Loading schema in Mapper

The schema needs to be loaded manually.

Specify the schema of the data to be loaded.

= Remarks =

For details, refer to Edit Schema.

Mass data processing

Mass data processing is supported.

Parallel Stream Processing

PSP isn't supported.

Available component variables

Component variable name	Description	Remarks
count	The number of all read records is stored.	The default value is 0.
matched_count	The number of records that match the data check conditions is stored.	The default value is 0.
unmatched_count	The number of records that don't match the data check conditions is stored.	The default value is 0. Detailed information of the data that doesn't match the conditions can be gotten using Get Data Check Error Details operation.
message_category	When an error occurs, the category of the message code corresponding to the error is stored.	The default value is null.
message_code	When an error occurs, the code of the message code corresponding to the error is stored.	The default value is null.
message_level	When an error occurs, the severity of the message code corresponding to the error is stored.	The default value is null.
error_type	When an error occurs, the error type is stored.	The default value is null. The format of the error type is as follows. Example: java.io.FileNotFoundException
error_message	When an error occurs, the error message is stored.	The default value is null.
error_trace	When an error occurs, the trace information for the error is stored.	The default value is null.

Specification limits

Only table model type component can be specified for the input data.

Exception messages

Exception name	Cause	Solution
ResourceNotFoundException Resource definition could not be found. Name: []	Data check condition definition isn't specified.	Specify Data check condition definition.
ResourceNotFoundException Resource definition could not be found. Name: [<connection resource name>]	The resource definition selected in Data check condition definition isn't found.	Check the connection resource specified in Data check condition definition.
InvalidPropertyConfigurationException The value of [<item name>] in row [<row number>] of the check condition list is incorrect.	The check conditions aren't specified properly.	Confirm the check conditions.
InvalidPropertyConfigurationException Value is not entered in [Column number] specified in row [<row number>] of the check condition list.	Column number of the check condition list isn't entered.	Enter Column number.
InvalidPropertyConfigurationException The value of [Column number] specified in row [<row number>] of the check condition list exceeds the number of columns of input data [<the number>].	Column number of the check condition list isn't specified properly.	For Column number, specify a number that is less than or equal to the number of columns in the input data.