Exclusive control of Networked Files

DataMagic lets you use files on your network for purposes such as processing input data or output data. The network file functionality allows you to access files on your network and applies locks on the files. In DataMagic terminology, network files are files that are not managed by DataMagic hosts on which DataMagic is installed, but are managed by other hosts in the network.

When DataMagic functionality accesses a network file, a lock can be applied to the file. The lock functionality is referred to as a network file lock. If the access is made to the locked file, the access results in an error immediately. After a network file is accessed and processed, the network file is unlocked.

Figure 6.2 Overview of the network file functionality

Enabling network file lock guarantees integrity of data when network files are processed. The network file lock also reduces your time for copying input data from the network to your local host before processing the data. It also reduces your time for saving the output data back to the file server after processing the data. This facilitates management of files to be processed. For details on how to lock network files, see (5) About setting network file locks.

Before you use the network file functionality, see (7) Cautions to review cautionary notes on using network files.

(1) Example of using the network file functionality

The network file functionality can be used to process data in a file server, as illustrated below:

Figure 6.3 Example of using the network file functionality

(2) Specifying the path

To specify the path to a network file, enter the path using the following format in the file name field in the screen or dialog box.

Note

If you are using Windows, follow the Universal Naming Convention (UNC). In addition, note that you cannot specify network drives.

(3) Requirements and the pre-install check tool for the network file functionality

This section explains requirements for using the DataMagic network file functionality.

The pre-install check tool for the network file functionality lets you verify whether the file server on your network satisfies the requirements discussed in this section. Use this tool and perform verification if you want to use the network file functionality of DataMagic. This tool is available at our web page describing the operation environment for DataMagic. Our website is located as follows.

 

URL: https://www.hulft.com

 

a) Requirements common to all the operating systems

Access permissions must be set to target paths.

Read and write permissions must be set for the following paths:

  • The target network file and network path

  • Directory path specified for the Path to Shared Environment by HULFT Products (hulsharepath) of the system environment settings file (huledenv.conf)

Note

Read permission must be set for the following files:

  • Input files

  • Matching files

  • Files specified as virtual tables

  • Files specified on the Import Fields screen in CSV - Detailed Information

b) Requirements specific to UNIX and Linux

The file server must support NFS version 3 (NFS v3).

The file server managing network files must support the distributed file protocol NFS version 3 (NFS v3) or higher.

The attribute cache for the target files and paths must be disabled

The attribute cache of the NFS client must be disabled when the directory containing the target files is mounted.

The following is an example of mounting from Red Hat Enterprise Linux 5:

Example of the mount command:

 # # mount -o noac server:/data /mnt/server/data/ #

c) Requirements specific to Windows

The file server must support SMB/CIFS.

The file server managing network files must support SMB/CIFS, which is a file-sharing protocol for Windows.

oplock (Opportunistic Lock) must be disabled

oplock must be disabled if files on the file server are shared by applications using a file-sharing protocol other than SMB/CIFS. For details on how to make settings on oplock, see the documentation for your operating system or file server.

You do not need to disable oplock if files on the file server are shared by applications using the SMB/CIFS file-sharing protocol.

(4) Scope of application

You can specify network files or paths when you perform the following operations in DataMagic:

  • Perform operations on input/output files.

  • Perform operations on matching files.

  • Perform operations on virtual tables.

  • Specify the Path to Shared Environment by HULFT Products (hulsharepath) of the system environment settings file (huledenv.conf).

  • Specify the file name in the Retrieve records screen from the CSV - Detailed Information screen

  • Specify a database name in the Database Connection - Detailed Information screen, when the product type is SQLite3.

= Remarks =

The Path to Shared Environment by HULFT Products (hulsharepath) of the system environment settings file (huledenv.conf) is the path set to implement exclusive control of network files. For details, see (5) About setting network file locks.

(5) About setting network file locks

To enable a network file lock, configure your system environment settings. You need to configure system environment settings that match your system configuration. For example, you might need to configure environment settings for a system consisting of multiple DataMagic hosts or for a system consisting of HULFT and other HULFT family products.

a) If files are to be shared by multiple HULFT family products

If network files need to be shared by multiple DataMagic or HULFT hosts, you need to configure system environment settings on each of the hosts. When you configure system environment settings on the hosts, you need to specify the exclusive control management file to be shared on the network. For DataMagic, specify the Path to Shared Environment by HULFT Products (hulsharepath). Make sure that the same path is specified on each of the hosts.

Note

If the specification of the Path to Shared Environment by HULFT Products differs among the DataMagic hosts sharing the same file, a lock might not be performed properly and the files might be damaged.

Figure 6.4 Sharing of a network file

In Figure 6.4 , File 1 on the file server is shared by HULFT on Host A and DataMagic on Host B. The path of the directory \\server\data on the network is set as the Path to Shared Environment by HULFT Products (hulsharepath), based on the conventions used by the respective operating systems. Because of this setting, a lock is applied when Host A or Host B accesses the file, guaranteeing data integrity.

For details on system environment settings, see the online Help.

b) If DataMagic is installed on the file server

In which DataMagic is installed on the file server that contains files shared as network files with other hosts, for the DataMagic instance installed on the file server, you need to set below:

  • Set the pathname to hulsharepath (path to shared environment by HULFT products) of the system environment settings file (huledenv.conf).

  • Set Mode 1 (network file lock is used on local files) to localfile_lockmode (local file lock mode) of the system environment settings file (huledenv.conf).

Note

  • If files are shared when Local file lock mode is Mode 0 on the file server where DataMagic is installed, locks might not be applied properly and the files might be damaged. Set Mode 1 to the local file lock mode in the DataMagic on the file server.

  • In which DataMagic that do not shared files with other hosts, we recommend to set Mode 0 to localfile_lockmode (local file lock mode).

Figure 6.5 Settings for locking local files

In figure Figure 6.5 , File 1 on the file server is shared by DataMagic on Host A and DataMagic on the file server.

In this case, if you specify the path to the network file to access File 1 from DataMagic on the file server, the file might be handled as a local file during the system processing. Consequently, the network file lock functionality might not apply locks properly. If Local file lock mode is set Mode 1, the network file lock functionality can apply locks properly when local files are accessed. Therefore, a lock is applied when File 1 is accessed from DataMagic on Host A and the file server.

The settings for locking local files are unneeded on Host A, which is not a file server.

(6) Accounts for connecting to a file server

Use the following accounts to access a file server on the network when DataMagic is in use.

a) For UNIX and Linux

Operation

Account for connection

Starting the DataMagic Request Acknowledge daemon (huledd) as the root user.

root

Starting the DataMagic Request Acknowledge daemon (huledd) as a non-root user.

The account used for starting the huledd daemon

Running the command for executing data processing (utled).

The account used for starting the utility mentioned in the left column

b) For Windows

Settings or operations

Account for connection

The local system account is set to the DataMagic service in the Log on tab of the Properties dialog box for the service.

SYSTEM

An account is set for the DataMagic service in the Log on tab of the Properties dialog box for the service.

The account specified as mentioned in the left column

Running the command for executing data processing (utled).

The account used for starting the utility mentioned in the left column

If you want to set an account to the DataMagic service, you need to make the following settings before setting the account.

File server settings

Make sure that you allow connections from the account that is set to DataMagic service.

No settings on user management and user permissions are needed unless you impose restrictions on connections.

Settings on user management and user permissions

Specify an account in the Log on tab of the Properties dialog box for DataMagic service.

(7) Cautions

Keep the following in mind when using network files.

a) Unsupported behavior

The network file lock functionality of DataMagic might not work correctly with non-HULFT-family applications. Depending on the file server, locks might not be applied if the lock mechanism differs among applications. In this case, DataMagic and other applications cannot recognize the status of a lock even if the conditions explained in (3) Requirements and the pre-install check tool for the network file functionality are met. Consequently, this might damages the files. It is recommended that network files should only be shared by HULFT-family applications.

b) Notes on locks on files

The network file lock functionality of DataMagic automatically applies a lock on files in the directory that contains the file to be locked. The name of the lock file starts with .#HULLOCK, which is followed by the name of the file to be locked. The network file lock functionality uses this file to apply locks. Do not delete the lock file when the processing for a network file lock is in progress. DataMagic cannot apply locks properly if the lock file is deleted while the processing is in progress.

c) Notes on the names of network files

In one of the following tabs or screens, in File name or Database name, if you specify a network file, make sure that the same network file is not specified in any other tab or screen:

  • Layout tab of the Input Settings screen

  • Merging tab of the Input Settings screen

  • Layout tab of the Output Settings screen

  • Layout tab of the Matching Settings screen

  • Layout tab of the Virtual Table Settings screen

  • Database Connection - Detailed Information screen (if Product type is SQLite3)

For example, when merging files, you must specify different file names (not the same file name) in the following locations: File name in the Layout tab of the Input Settings screen, File name in the Merging tab of the Input Settings screen, and File name in the Layout tab of the Output Settings screen.

If the same file name is specified, correct data-processing results are not guaranteed.