(Oops, strange enough I left this as a "Draft" in the forum but I thought that I created a post)
Also, note, the input/output from the hard disk was very slow due to 600K files in a directory and using NTFS on Linux (and I think that this was part of the problem, more details at the bottom)
What is the problem you are having with rclone?
rclone reports file sent more than 100%
What is your rclone version (output from rclone version)
This one sadly is an oldish one from Debian:
rclone v1.49.5
- os/arch: linux/amd64
- go version: go1.12.10
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Debian 64 bit
Which cloud storage system are you using? (eg Google Drive)
Object Storage S3 alike (Scality)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone --verbose --config config-file --include 201701182* copy . bucket-name://id/dir1/dir2/dir3/
A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)
2020/01/03 12:58:46 INFO :
Transferred: 2.289M / 1.103 GBytes, 0%, 65.190 kBytes/s, ETA 4h55m11s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 574, 0%
Elapsed time: 35.9s
Transferring:
* 20170118214625atr.pol: 27% /1.968M, 15.468k/s, 1m34s
* 20170118214628atr.pol: 36% /1.969M, 13.651k/s, 1m34s
* 20170118214630atr.pol: 30% /1.968M, 18.532k/s, 1m15s
* 20170118214633atr.pol: 21% /1.968M, 11.609k/s, 2m16s
2020/01/03 12:59:46 INFO :
Transferred: 4.625M / 1.103 GBytes, 0%, 49.355 kBytes/s, ETA 6h29m5s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 574, 0%
Elapsed time: 1m35.9s
Transferring:
* 20170118214625atr.pol: 46% /1.968M, 8.291k/s, 2m10s
* 20170118214628atr.pol: 92% /1.969M, 39.188k/s, 3s
* 20170118214630atr.pol: 55% /1.968M, 9.177k/s, 1m37s
* 20170118214633atr.pol: 40% /1.968M, 6.030k/s, 3m19s
2020/01/03 13:00:46 INFO :
Transferred: 8.771M / 1.103 GBytes, 1%, 57.586 kBytes/s, ETA 5h32m15s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 574, 0%
Elapsed time: 2m35.9s
Transferring:
* 20170118214625atr.pol: 74% /1.968M, 1.910k/s, 4m27s
* 20170118214628atr.pol:199% /1.969M, 7.797k/s, -
* 20170118214630atr.pol: 93% /1.968M, 2.531k/s, 53s
* 20170118214633atr.pol: 77% /1.968M, 3.229k/s, 2m18s
2020/01/03 13:01:46 INFO :
Transferred: 9.700M / 1.103 GBytes, 1%, 45.992 kBytes/s, ETA 6h55m39s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 574, 0%
Elapsed time: 3m35.9s
Transferring:
* 20170118214625atr.pol: 93% /1.968M, 15.277k/s, 8s
* 20170118214628atr.pol:199% /1.969M, 166/s, -
* 20170118214630atr.pol:100% /1.968M, 1.634k/s, 0s
* 20170118214633atr.pol: 99% /1.968M, 11.415k/s, 0s
2020/01/03 13:02:46 INFO :
Transferred: 10.617M / 1.103 GBytes, 1%, 39.396 kBytes/s, ETA 8h4m52s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 574, 0%
Elapsed time: 4m35.9s
Transferring:
* 20170118214625atr.pol:100% /1.968M, 866/s, 0s
* 20170118214628atr.pol:212% /1.969M, 2.503k/s, -
* 20170118214630atr.pol:126% /1.968M, 18.774k/s, -
* 20170118214633atr.pol:100% /1.968M, 262/s, 0s
2020/01/03 13:03:46 INFO :
Transferred: 12.960M / 1.103 GBytes, 1%, 39.503 kBytes/s, ETA 8h2m32s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 574, 0%
Elapsed time: 5m35.9s
Transferring:
* 20170118214625atr.pol:146% /1.968M, 22.715k/s, -
* 20170118214628atr.pol:218% /1.969M, 2.615k/s, -
* 20170118214630atr.pol:161% /1.968M, 24.786k/s, -
* 20170118214633atr.pol:132% /1.968M, 12.566k/s, -
2020/01/03 13:04:46 INFO :
Transferred: 13.535M / 1.103 GBytes, 1%, 35.002 kBytes/s, ETA 9h4m18s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 574, 0%
Elapsed time: 6m35.9s
Transferring:
* 20170118214625atr.pol:146% /1.968M, 484/s, -
* 20170118214628atr.pol:231% /1.969M, 8.915k/s, -
* 20170118214630atr.pol:176% /1.968M, 9.504k/s, -
* 20170118214633atr.pol:132% /1.968M, 267/s, -
2020/01/03 13:05:46 INFO :
Transferred: 13.843M / 1.103 GBytes, 1%, 31.089 kBytes/s, ETA 10h12m38s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 574, 0%
Elapsed time: 7m35.9s
Transferring:
* 20170118214625atr.pol:146% /1.968M, 10/s, -
* 20170118214628atr.pol:231% /1.969M, 189/s, -
* 20170118214630atr.pol:176% /1.968M, 202/s, -
* 20170118214633atr.pol:148% /1.968M, 5/s, -
Please note:
This rclone is trying to read files from a directory with 600k files from a USB 3 hard disk using NTFS (on a Debian machine). When doing the initial scan it takes ages (often around 7 to 10 minutes). Doing a simple "ls" it also takes too long and it can't be "Control+C"ed (kernel blocking it?)
I have been working uploading directories from here for a week (on and off) and I haven't seen this problem before. I'll try to reproduce but I don't think that will be possible.
Thanks,