# What are fast ways to make bulk transfers? **URL:** \ **Category:** Discussion Zone\ **Tags:** data, file-transfer, globus, data-transfer\ **Created:** [March 16, 2018, 1:22am UTC](https://ask.cyberinfrastructure.org/t/what-are-fast-ways-to-make-bulk-transfers/114 "2018-03-16T01:22:21Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![christophernhill](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/c/f08c70/32.png) [@christophernhill](https://ask.cyberinfrastructure.org/u/christophernhill)\ **Post date:** [March 16, 2018, 1:22am UTC](https://ask.cyberinfrastructure.org/t/what-are-fast-ways-to-make-bulk-transfers/114/1 "2018-03-16T01:22:21Z")
**COMMENT** This question may not have a single answer, but answers that highlight different possible ways to achieve good data transfer rates in different situations could all be useful. In all cases good transfer speeds will require that the end-point systems and the intermediate networks are configured appropriately. There is an extensive discussion of end-point configuration for fast data transfers here ( [https://fasterdata.es.net/host-tuning/](https://fasterdata.es.net/host-tuning/) ). **CURATOR:** Chris Hill ---
**Author:** ![christophernhill](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/c/f08c70/32.png) [@christophernhill](https://ask.cyberinfrastructure.org/u/christophernhill)\ **Post date:** [March 29, 2018, 3:04pm UTC](https://ask.cyberinfrastructure.org/t/what-are-fast-ways-to-make-bulk-transfers/114/2 "2018-03-29T15:04:12Z")
One option is bbftp ([http://software.in2p3.fr/bbftp/](http://software.in2p3.fr/bbftp/)). It is used by a number of NASA locations. A typical bbftp transfer command requires software to be installed at both the sending and receiving location. The software can be installed by an end user. A typical bbftp command is ```auto bbftp -s -E PATH-TO-BBFTPD-ON-REMOTE -R bbftprc -V -p 8 -r 5 -u USER -i LIST-OF-COMMANDS REMOTE-MACHINE ``` where - `PATH-TO-BBFTPD-ON-REMOTE`: is the location on the remote system of the bbftpd server command. - `USER`: is the user name on the remote system - `REMOTE-MACHINE`: is the remote system to transfer files to - `LIST-OF-COMMANDS`: is a file with a list of commands to perform a transfer e.g. A typical `LIST-OF-COMMANDS` file contains entries like those shown here ```auto setoption keepaccess setoption keepmode setoption nocreatedir put /nobackupp8/cnhill1/hawaii_npac/0001171008_V_10800.8150.1_1080.3720.90 /nfs/cnhlab003/cnh/llc4320/incoming/hawaii_npac/ setoption keepaccess OK setoption keepmode OK setoption nocreatedir OK ``` ---
**Author:** ![christophernhill](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/c/f08c70/32.png) [@christophernhill](https://ask.cyberinfrastructure.org/u/christophernhill)\ **Post date:** [March 29, 2018, 3:16pm UTC](https://ask.cyberinfrastructure.org/t/what-are-fast-ways-to-make-bulk-transfers/114/3 "2018-03-29T15:16:47Z")
grid-ftp is another option ---
**Author:** ![christophernhill](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/c/f08c70/32.png) [@christophernhill](https://ask.cyberinfrastructure.org/u/christophernhill)\ **Post date:** [March 29, 2018, 3:16pm UTC](https://ask.cyberinfrastructure.org/t/what-are-fast-ways-to-make-bulk-transfers/114/4 "2018-03-29T15:16:59Z")
globus is another option… ---
**Author:** ![christophernhill](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/c/f08c70/32.png) [@christophernhill](https://ask.cyberinfrastructure.org/u/christophernhill)\ **Post date:** [March 29, 2018, 3:26pm UTC](https://ask.cyberinfrastructure.org/t/what-are-fast-ways-to-make-bulk-transfers/114/5 "2018-03-29T15:26:20Z")
It is possible to script bulk uploads to facilities like Dropbox too. A set of scripts such as the ones here ( [https://github.com/cpausmit/PyCox](https://github.com/cpausmit/PyCox) ) can be used to make large scale transfers from multiple end-points to achieve good throughput. ---
**Author:** ![christophernhill](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/c/f08c70/32.png) [@christophernhill](https://ask.cyberinfrastructure.org/u/christophernhill)\ **Post date:** [March 29, 2018, 4:05pm UTC](https://ask.cyberinfrastructure.org/t/what-are-fast-ways-to-make-bulk-transfers/114/6 "2018-03-29T16:05:43Z")
Aspera ([http://asperasoft.com/software/](http://asperasoft.com/software/)) is another option. It is a commercial too, but is widely used in the bioinformatics community. ---
**Author:** ![christophernhill](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/c/f08c70/32.png) [@christophernhill](https://ask.cyberinfrastructure.org/u/christophernhill)\ **Post date:** [March 29, 2018, 4:14pm UTC](https://ask.cyberinfrastructure.org/t/what-are-fast-ways-to-make-bulk-transfers/114/7 "2018-03-29T16:14:15Z")
Amazon S3 provides another useful solution for intermediate staging of large files. The article here ( [https://arxiv.org/abs/1708.00544](https://arxiv.org/abs/1708.00544) ) provides a description of the mechanisms and experience doing this. ---
**Author:** ![rberger](https://ask.cyberinfrastructure.org/user_avatar/ask.cyberinfrastructure.org/rberger/32/56_2.png) [@rberger](https://ask.cyberinfrastructure.org/u/rberger)\ **Post date:** [June 28, 2019, 11:15pm UTC](https://ask.cyberinfrastructure.org/t/what-are-fast-ways-to-make-bulk-transfers/114/8 "2019-06-28T23:15:29Z")
Besides the software solutions to this problem, there are other means to achieve good data transfer rates for bulk transfers. It can be cheaper and faster to use the “FedEx protocol”. Take a bunch of disks and mail them. The latency is not so great, but you can’t beat the bandwidth. ---
**Author:** ![tru](https://ask.cyberinfrastructure.org/letter_avatar_proxy/v4/letter/t/3e96dc/32.png) [@tru](https://ask.cyberinfrastructure.org/u/tru)\ **Post date:** [July 9, 2019, 3:29pm UTC](https://ask.cyberinfrastructure.org/t/what-are-fast-ways-to-make-bulk-transfers/114/9 "2019-07-09T15:29:44Z")
[http://moo.nac.uci.edu/~hjm/HOWTO\_move\_data.html](http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html) maybe?