What is the problem you are having with rclone?
Running on Debian server.
rclone copy will eventually stop after no specific time period but essentially never completes a full specified upload of media (anywhere from 10-1000GB). I've been able to complete it with single files.
The only thing I see are "input/output" errors in the log with -vv. The progress for the actual command in CLI just stops. It is not possible to kill the rclone copy command and eventually it becomes a zombie or "d" process. Only way (I've found) to rid a failed copy process is to reboot. Everything runs ok still though including mounts and copy command can be run again.
This worked fine over 3 years. I've had a few theories but none was solid. One I haven't investigated much is related to characters. I recall about the time I started getting error some newly added files had chinese characters and they were nested in multiple deep folders. I also noted some of the file names are very long because of the encryption. Perhaps this could be something but I'm not sure? Really not seeing much verbosity. Maybe it broke something. It's my best guess at this point.
What is your rclone version (output from rclone version
)
v1.57.0
First discovered this problem spring 2021. I believe it was 1.55.x
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone copy '/media/main/media/movies' media:'movies' --log-file /opt/rclone/logs/upload-movies-202112142222.log -vv --exclude-from /opt/rclone/excludes --bwlimit 8M --config /opt/rclone/rclone.conf --progress --transfers 1 --checkers=100 --min-age 15m --crypt-show-mapping --stats 5s --buffer-size 2G --cache-dir /media/main2/temp/rclone/cache
This is the latest version of the command I've tried. I've played around with different variants of the command including removing many options and adding them. --cache-dir was an attempt to avoid the same disk in case there are hw issues (input/output error). Made no difference and the "transfers", "checkers", "min-age", "crypt-show-mapping", "stats", "buffer-size" vars has made no impact either.
The rclone config contents with secrets removed.
[gd]
type = drive
client_id = REDACTED
client_secret = REDACTED
token = REDACTED
[gd-crypt]
type = crypt
remote = gd:/media
filename_encryption = standard
password = REDACTED
password2 = REDACTED
[media]
type = crypt
remote = /media/main/cloud/google-drive-encrypted-mount-media
filename_encryption = standard
password = REDACTED
password2 = REDACTED
A log from the command with the -vv
flag
This is a small excerpt of a few files failing. Really not getting more info than this.
2021/12/14 20:00:48 ERROR : REDACTEDMEDIANAME (2021)/REDACTEDMEDIANAME/REDACTEDMEDIANAME.mkv: Failed to copy: write /media/main/cloud/google-drive-encrypted-mount-media/oq6ahiprcma266vifvt2gqg8fk/mr3cgpq2fvmv4qio6icibn85lk/e2i3f9bqoho1kdft7i2mba35l73iqu17uq3n2hc5ai7qhsjgmp89mmm37gjjcui6vjoge6pk4vhjpl5efos1s76ispln93vvuh22iv8/qj3dj8cs0vjhmi8du70o70bmiu0fdcd7km60c5be0n7lhdt495squg2k6su5vlsir7v9961pm1f3ljsjpip4p42mam6aq017mkejehqpjic1gpoba62rh0hq9u63rn83: input/output error
2021/12/14 20:00:48 DEBUG : REDACTEDMEDIANAME (2021)/REDACTEDMEDIANAME/REDACTEDMEDIANAME.nfo: md5 = a79088efaf2f9f3f246e6050bee44f71 OK
2021/12/14 20:00:48 INFO : REDACTEDMEDIANAME (2021)/REDACTEDMEDIANAME/REDACTEDMEDIANAME.nfo: Copied (replaced existing)
2021/12/14 20:00:49 DEBUG : REDACTEDMEDIANAME (2021)/REDACTEDMEDIANAME/REDACTEDMEDIANAME.jpg: md5 = b5e1e756bc02d26f0ea2d9f1bb1b05e8 OK
2021/12/14 20:00:49 INFO : REDACTEDMEDIANAME (2021)/REDACTEDMEDIANAME/REDACTEDMEDIANAME.jpg: Copied (replaced existing)
2021/12/14 20:57:53 DEBUG : REDACTEDMEDIANAME2 (2021)/REDACTEDMEDIANAME2/REDACTEDMEDIANAME2.mkv: md5 = a76f7508da2ba3acec031bb487a0368d OK
2021/12/14 20:57:53 INFO : REDACTEDMEDIANAME2 (2021)/REDACTEDMEDIANAME2/REDACTEDMEDIANAME2.mkv: Copied (replaced existing)
2021/12/14 21:05:33 NOTICE: cijc2cqvn09876rmjm24e6j6gk/a8a2921m666kqfhb3p30hfc69sshpph0fer50c5pstgmq7ppb0841g9a15fk6mq0g494duul86l3bbtcophcpl6c8ld59317871270g/98c13tpo6i9lv4de04vih9922fe3m0nbrq7q968ra2c5f7q980159rm3ahogufv5poldk2j80tdh5oc053ceegtt6h676jut83dij2o: Removing partially written file on error: write /media/main/cloud/google-drive-encrypted-mount-media/oq6ahiprcma266vifvt2gqg8fk/cijc2cqvn09876rmjm24e6j6gk/a8a2921m666kqfhb3p30hfc69sshpph0fer50c5pstgmq7ppb0841g9a15fk6mq0g494duul86l3bbtcophcpl6c8ld59317871270g/98c13tpo6i9lv4de04vih9922fe3m0nbrq7q968ra2c5f7q980159rm3ahogufv5poldk2j80tdh5oc053ceegtt6h676jut83dij2o: input/output error
2021/12/14 21:05:33 ERROR : REDACTEDMEDIANAME3 (2002)/REDACTEDMEDIANAME3/REDACTEDMEDIANAME3.mkv: Failed to copy: write /media/main/cloud/google-drive-encrypted-mount-media/oq6ahiprcma266vifvt2gqg8fk/cijc2cqvn09876rmjm24e6j6gk/a8a2921m666kqfhb3p30hfc69sshpph0fer50c5pstgmq7ppb0841g9a15fk6mq0g494duul86l3bbtcophcpl6c8ld59317871270g/98c13tpo6i9lv4de04vih9922fe3m0nbrq7q968ra2c5f7q980159rm3ahogufv5poldk2j80tdh5oc053ceegtt6h676jut83dij2o: input/output error
2021/12/14 21:13:12 NOTICE: 4ja2cf8h3r974u4uet8vaca07uiskpnmsabof7vlkdnmjmgg1hig/v50a8u53s9nqt8m7qb6lr89nvj2fd8mrt10qp4l79msoemhc29d0r1fr0nkcelc5jgveetn3reuisf11tqojaooenmvlidif5avrhfo/ior8h5bgkdib3ti8dm2l3db1o8u7nmnk0o8g2iqfh014lb1a6pjob4tcg3q71k5iulcnemap96p5hsjpf3gcgfs70126l20plr9usiqm91quk5htjf9j0f5cn8bkfrmj: Removing partially written file on error: write /media/main/cloud/google-drive-encrypted-mount-media/oq6ahiprcma266vifvt2gqg8fk/4ja2cf8h3r974u4uet8vaca07uiskpnmsabof7vlkdnmjmgg1hig/v50a8u53s9nqt8m7qb6lr89nvj2fd8mrt10qp4l79msoemhc29d0r1fr0nkcelc5jgveetn3reuisf11tqojaooenmvlidif5avrhfo/ior8h5bgkdib3ti8dm2l3db1o8u7nmnk0o8g2iqfh014lb1a6pjob4tcg3q71k5iulcnemap96p5hsjpf3gcgfs70126l20plr9usiqm91quk5htjf9j0f5cn8bkfrmj: input/output error
2021/12/14 21:13:19 ERROR : REDACTEDMEDIANAME4 (2021)/REDACTEDMEDIANAME4/REDACTEDMEDIANAME4.mkv: Failed to copy: write /media/main/cloud/google-drive-encrypted-mount-media/oq6ahiprcma266vifvt2gqg8fk/4ja2cf8h3r974u4uet8vaca07uiskpnmsabof7vlkdnmjmgg1hig/v50a8u53s9nqt8m7qb6lr89nvj2fd8mrt10qp4l79msoemhc29d0r1fr0nkcelc5jgveetn3reuisf11tqojaooenmvlidif5avrhfo/ior8h5bgkdib3ti8dm2l3db1o8u7nmnk0o8g2iqfh014lb1a6pjob4tcg3q71k5iulcnemap96p5hsjpf3gcgfs70126l20plr9usiqm91quk5htjf9j0f5cn8bkfrmj: input/output error
➜ ~
Thanks for reading and looking forward to replies. Was hoping to solve this myself but after the better part of a year spending the odd hour here and there I turn to the wiser and knowledgeable.