Parallel downloads

The transfer speed over the network from object storage to the sending-side host may be slower than the transfer speed over the network from the sending-side host to the receiving-side host. To take measures against this problem, the send data is split into multiple parts and downloaded in parallel.

An overview of the procedure for parallel downloads is shown below.

Figure 1.3 Procedure for parallel downloads

1) Parallel downloads

The send data is split into parts of a determined size and that data is downloaded from object storage one part at a time in a sequence of downloads.

Starting from the first part and continuing sequentially, each part of the data is downloaded in parallel.

2) File transfer

A file is sent from the sending-side host to the receiving-side host.

Note

  • If the time for downloading to object storage is too long, a timeout occurs during communication with object storage.

    For HULFT Cloud Storage Option Ver.8.5.1 or higher, you can specify the wait time until a timeout occurs for HTTP requests (storage timeout) during communication with object storage. For details, refer to Send Storage Management Information settings.

  • Due to the load on the memory of the sending-side host and the load on the network between the object storage and the sending-side host, a maximum number of parallel downloads that can be performed simultaneously is set and restricted.

    If the number of parallel downloads reaches the maximum number, any additional downloads will be placed in the queue until one or more running downloads are finished and a slot opens in the available number of downloads. In this case, if the wait time is too long, a timeout occurs during communication with HULFT.

    The wait time until a timeout occurs can be specified in the System Environment Settings of the receiving-side host. For details, refer to System Environment Settings.

  • If multiple downloads are performed simultaneously for one receive file, the transfer remains in-progress until a timeout occurs.

    The transfer cannot be canceled, either.

    Therefore, do not perform multiple downloads simultaneously for one receive file.

  • On the object storage, exclusive control of send files is not available. For details on exclusive control of send files (exclusive control of objects) when downloading, refer to HULFT functional restrictions for downloads from object storage.