A 1.18 TB copy between two seedboxes in the same data centre took 14.5 hours. The network between them did 8 Gbit/s on a single iperf3 stream. The ping was 0.29 ms. My tool averaged 21.6 MiB/s.
The tool is Hopsync, an open-source Rust service I'm building for moving files between SSH servers from a web UI or a CLI. The other side only needs a normal SSH account with SFTP. Its main trick is parallel "lanes": several SSH connections, with big files split across them. Over a 200 ms link that roughly doubles scp's speed, and on 300 small files it finishes in 39 s where scp takes 249 s.
On two hard drives sitting next to each other, the same trick made it slower than rclone. Here's why, and what fixed it.
Measure the ceiling first
Before touching code I wanted to know what "fast" meant on these boxes. The simplest possible pipe, one sequential reader, one SSH stream, one sequential writer:
dd if=big.bin iflag=direct bs=16M \ | ssh user@dst 'dd of=big.bin bs=16M conv=fdatasync' 32 GiB in 4 m 26 s: 123 MiB/s, with the destination disk busy over 90 % most of the time. That's the real limit: the slowest disk, not the network or the cipher (one OpenSSH stream carried ~400 MiB/s).
Then the same file with the tools:
| rclone 1.53 (1 stream) | Hopsync (old, 6 lanes) | |
|---|---|---|
| Average speed | 61.4 MiB/s | 36.3 MiB/s |
| Source disk busy | 37 % | 72 % |
| Source iowait | 10 % | 48 % |
| Destination disk busy | 37 % | 20 % |
Hopsync kept the source disk twice as busy as rclone and moved about half as much data. The destination was mostly idle. The source disk was working hard at something other than reading.
Root cause: the disk was seeking
The old design cut each file into chunks of 8 to 64 MiB and gave each lane its own chunk. Every lane read from its own offset, with 16 reads of ~256 KiB in flight each, all through one shared file descriptor.
Two things went wrong:
- Readahead gave up. Linux keeps readahead state per open file. Sixteen readers jumping between offsets look like random access, so the kernel stops reading ahead.
- The head moved constantly. A spinning disk does ~150 MiB/s sequentially and a tiny fraction of that when it seeks between chunks. Parallel reads that help on SSDs and fast networks turn into a seek storm on an HDD.
There was more on top: up to 64 requests in flight per job, two jobs on the same disk, and at one point 405 threads stuck in D state. Every 8 to 64 MiB chunk also ended with an fsync@openssh.com.
What the fast tools do
I read how rsync, rclone and libtorrent handle this:
- rsync: one file at a time, one sequential reader, no fsync unless you ask.
- rclone: a sequential reader with an async read-ahead buffer, and up to 64 SFTP writes in flight at increasing offsets.
- libtorrent: pieces arrive in random order, but the writes land in RAM first and the kernel sorts dirty pages before write-back.
The lesson that changed the design:
Writes can be absorbed and reordered by RAM. Reads can't.
A read waits for the disk the moment you issue it, so the reading side must be sequential and large. The writing side only needs offsets that move forward, so the kernel can merge them.
The fix: lanes carry buffers, they don't choose offsets
The new push path:
- One sequential reader per file. It reads 16 MiB blocks in order, with
posix_fadvise(SEQUENTIAL), into buffers from a fixed pool.O_DIRECTwhen the filesystem allows it, so the seedbox's page cache stays out of it. - A bounded read-ahead queue (64 MiB, 4 blocks). When the queue is full the reader waits, so memory has a hard cap.
- Lanes only carry data. Each block is cut into SFTP writes of the server's
max_writesize and handed to whichever lane has room. All writes in flight sit in one sliding window, so the destination sees offsets moving forward a few MiB at a time. - One big file per source disk at a time, daemon-wide. Two queued jobs on the same disk share it instead of doubling the seeks. Different disks still run in parallel.
- Fewer fsyncs. A durability checkpoint every 1 GiB instead of every chunk, with only one in flight at a time.
- Lane auto-tuning stays, because a 200 ms link still needs it.
Results
Same 32 GiB file, same boxes, all four runs back to back in the same hour:
| Tool | Time | Speed | Source busy / iowait | Destination busy |
|---|---|---|---|---|
| Hopsync 1.3 | 3 m 31 s | 155 MiB/s | 69 % / 8 % | 90 % |
| `dd \ | ssh \ | dd` | 4 m 25 s | 124 MiB/s |
| rclone 1.53 (SFTP) | 9 m 11 s | 60 MiB/s | 38 % / 10 % | 32 % |
| Hopsync 1.2 | 15 m 30 s | 35 MiB/s | 72 % / 47 % | 22 % |
The source disk is as busy as before, but iowait fell from 47 % to 8 %, because it's streaming instead of seeking. It moves 4.4x as much data. The destination disk is now the bottleneck, which is where it should be. Peak memory was 162 MB.
It also beats the "ideal" single dd pipe. My best guess is that several SSH streams keep the destination's write queue fuller than one stream can, but I haven't isolated that yet.
Caveats, because benchmarks lie easily:
- These are shared seedboxes. Other users' load varies between runs, which is why all four ran back to back.
- One file size, one pair of boxes. I haven't compared tuned rsync or lftp yet.
- An earlier near-final build hit 172.8 MiB/s. I can't fully explain the 10 % gap yet.
The full method, raw CSVs and scripts are in the repo: results, design doc, scripts.
What I took away
- Measure the ceiling before optimizing. The
dd | ssh | ddnumber told me the network was never the problem. - Parallelism has a physical cost. What helps on a high-latency link hurts on a spinning disk. The same tool needs both behaviours, picked from what it measures.
- Look at disk busy and iowait, not only throughput. 72 % busy for 36 MiB/s was the clue.
Try it
Hopsync is a single binary and needs no root, so it runs on shared seedboxes:
curl -fsSL https://github.com/YourNeelsOG/hopsync/releases/latest/download/install.sh | sh hopsync setup It also has resume after a dropped connection (even kill -9), a job queue that survives restarts, a preview before copying, and hopsync push / pull on the CLI. Host keys are always pinned, and there's no default password.
It's early software and hasn't had an independent security review yet. Issues, benchmarks from your own boxes and PRs are welcome: github.com/YourNeelsOG/hopsync
Have you hit the same "parallel is slower" wall on HDDs? I'd like to hear how you solved it.


