Parallel uploads

The transfer speed over the network from the receiving-side host to object storage may be slower than the transfer speed over the network from the sending-side host to the receiving-side host. To take measures against this problem, the receive data is split into multiple parts and uploaded in parallel.

An overview of the procedure for parallel uploads is shown below.

Figure 1.2 Procedure for parallel uploads

1) File transfer

A file is sent from the sending-side host to the receiving-side host.

2) Parallel uploads

Each time a certain amount of receive data is accumulated, that data is uploaded to object storage as one part in a sequence of uploads.

When the file transfer in 1) is higher speed than the parallel uploads, by the time one part of the data is updated, the next part of the data will have been accumulated.

Therefore, each part of the data is uploaded in parallel.

The uploaded data is stored in the object storage as temporary data.

3) Commit request

When the upload of all receive data is complete, the receiving-side host issues a commit request to the object storage.

4) Commit

When the object storage receives the commit request, the temporary data is combined to create an object.

The temporary data is deleted after the commit is complete.

5) Completion of commit request

When the commit request ends successfully, the receiving-side host determines that the receiving of the file has ended successfully.

When "Receive Completion" is specified for Notification in the Receive Management Information, a notification of receive completion is sent to the sending-side host at this time.

Note

  • If the time for uploading to object storage is too long, a timeout occurs during communication with object storage.

    For HULFT Cloud Storage Option Ver.8.5.1 or higher, you can specify the wait time until a timeout occurs for HTTP requests (storage timeout) during communication with object storage. For details, refer to Receive Storage Management Information settings.

  • Due to the load on the memory of the receiving-side host and the load on the network between the receiving-side host and the object storage, a maximum number of parallel uploads that can be performed simultaneously is set and restricted.

    If the number of parallel uploads reaches the maximum number, any additional uploads will be placed in the queue until one or more running uploads are finished and a slot opens in the available number of uploads. In this case, if the wait time is too long, a timeout occurs during communication with HULFT.

    The wait time until a timeout occurs during communication with HULFT can be specified in the System Environment Settings of the sending-side host. For details, refer to System Environment Settings.

  • If a network error or other error occurs before the commit request is issued, temporary data may remain on the object storage as follows:

    For Azure Blob Storage

    If the transfer is canceled, temporary data remains.

    For Amazon S3 and Google Cloud Storage

    • For HULFT Cloud Storage Option Ver.8.4.1

      If the transfer is canceled, temporary data is usually deleted.

      However, temporary data may remain depending on the timing of the cancelation or other factors.

      For details on measures to resolve this problem, refer to Deletion of Cloud Storage temporary data.

    • For HULFT Cloud Storage Option Ver.8.5.0 or higher

      If the transfer is canceled, the temporary data is deleted after the file transfer that is in-progress completes. Therefore, temporary data is not remained.

      However, if a timeout occurs when canceling and the transfer is forcefully terminated, the temporary data is not deleted and remains.

      For details on measures to resolve this problem, refer to Deletion of Cloud Storage temporary data.

  • When "Successful Job Completion" is specified for Notification in the Receive Management Information, the receiving is considered to have ended unsuccessfully if the data reception ends successfully and then the post-receive job ends unsuccessfully (the same is true for receive operations for disk files).

    However, even if "Restore" is specified for Error Handling, the object is not restored to the status before receiving. This is due to the fact that the commit processing cannot be canceled on the object storage.