Handling of null characters in data

DataMagic allows you to select how to process a null character (0x00) when it is found in the input data.

The following are the four methods of processing a null character.

Stop processing

: When a null character is found, an error is generated and processing stops.

Replacement with spaces

: Any null characters are replaced with a space.

Replacement with 0s (for database tables only)

: A null character is replaced with a 0.

Data cutoff

: The portion up to the null character is identified as data, and the portion thereafter is cut off.

Treat a null character as part of a field

Without converting the null character, treats as null character.

Such as data processing of text format, if the conditions do not meet the following description, the null character is treated as is.

If the CSV file contains a null character, the null character will be truncated in the preview of the file.

(1) Stop processing

When Null character present in format character data is selected in the Stop conditions in the Options tab of the Data Processing Settings screen, if a null character (0x00) is found in the input data, the processing stops.

This is effective when the field type of the input data is M, N, X, or D.

When stop conditions are set, all subsequent processing stops, and therefore settings such as Replacement of null characters with spaces and Handling of null values in numeric fields become invalid.

(2) Replacement with spaces

If Replace is set for Replacement of null characters with spaces in the Options tab of the Data Processing Settings screen, null characters (0x00) in the input data are replaced with halfwidth spaces.

This is effective for Format, Multi format, and Hierarchy format.

The field types of the input data are M, N, X, and D.

Depending on the input kanji code type, halfwidth spaces are replaced with the following:

Shift_JIS, EUC, UTF-8, and JIS

: 0x20

UTF-16 (Big Endian)

: 0x0020

UTF-16 (Little Endian)

: 0x2000

EBCDIC (JEF, IBM, KEIS, and NEC)

: 0x40

Note

A NULL character inside a fullwidth value that contains control characters and shift encoding is replaced with a halfwidth space without exception.

= Remarks =

Even if 1 (Yes) is set for "Treat a null character as part of a field (ed_fmt_null_flag)" in the system environment settings file, the setting takes priority.

(3) Replacement with 0s (for database tables only)

For migration of database tables, if Zero (0) is set for Handling of null values in numeric fields in the Input Settings screen, if the input data contains a null character (0x00), it is replaced with a 0.

If Null is set for Handling of null values in numeric fields, a null character is handled as null in the conversion. Note, however, that a null character is handled as a null of type M.

(4) Data cutoff

When Null character present in format character data is not set for Stop conditions and Do not replace is set for Replacement of null characters with spaces in the Options tab of the Data Processing Settings screen, if a null character (0x00) is contained in input data other than a numeric field, the portion up to the null character is determined to be data, and the portion thereafter is cut off. (If the input kanji code type is UTF-16, the null character is 0x0000).

Figure 3.19 A null character in the data

The above handling applies to the following fields:

  • Fields of format types X, M, N, and D

  • CSV fields

Note

If a null character exists in the text or an attribute value of XML, the input XML file will generate a loading error.

(5) Treat a null character as part of a field

In Format information, without converting the null character, you can treat as null character. If you want to treat as null character, specify to 1 (Yes) for "Treat a null character as part of a field (ed_fmt_null_flag)" in the system environment settings file.

= Remarks =

When one of the following is specified in the Options tab of the Data Processing Settings screen, even if 1 (Yes) is set for "Treat a null character as part of a field (ed_fmt_null_flag)" in the system environment settings file, the setting in the Data Processing Settings screen takes priority:

  • Null character present in fixed-format character data is selected as the stop condition.

  • For the input behavior, in Replacement of null characters with spaces, Replace is selected.

(6) Data cutoff for text format

If you select <b>Enable mapping</b> in the input settings, the portion from a null character up to a line break code is cut off regardless of other settings.