So I have my rclone setup with cache to Gdrive unencrypted with temp-upload-path enabled. All is well. However today I want to upload 30GB of data. In the middle of rclone uploading, I had to stop/restart rclone service few times and reboot the server. it is now looks like rclone is in idle state and doesn’t try to upload the rest of the data.
How can I manually tell rclone to upload the rest of data ?
root@jupiter:~/media/upload# du -csh
16G .
16G total
rclone v1.41-061-g96977547β
os/arch: linux/amd64
go version: go1.10.1
service rclone status
● rclone.service - Mount and cache Google drive to /mnt/Plexdrive
Loaded: loaded (/etc/systemd/system/rclone.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2018-06-02 10:12:56 JST; 39min ago
Process: 4790 ExecStop=/bin/fusermount -uz /mnt/Plexdrive (code=exited, status=0/SUCCESS)
Main PID: 4845 (rclone)
Tasks: 18
Memory: 97.6M
CPU: 1min 16.757s
CGroup: /system.slice/rclone.service
└─4845 /usr/sbin/rclone mount GDCache: /mnt/Plexdrive --cache-tmp-upload-path=/home/aaa/media/upload --cache-workers=5 --dir-cache-time=1h --cache-chunk-size=20M --cache-info-age=2h --buffer-size=0M --attr-timeout=1s --config=
Just found out that the upload for the past 2 days weren’t retried until completion. this was the 2 days that I restarted rclone service during the upload. so I guess it passed the x number of tries and give up or a service restart event trigger this?
Luckily the incident is only for the 2 days part, and I can always just delete the whole data and redownload from internet… currently I moved the 16GB data out of upload folder, purged cache, and re upload that 16GB again. If this feature is implemented the same way as ‘rclone move’ then I still have hope that by the end of the upload all files will be good/readable again.
So after manually retrying the upload, it seems that it works. Rclone normally upload the rest of the data and even when I stop/restart rclone service, it retries to upload the rest of data. I didn’t try to reboot the server this time. But seems like it works.
It sounds like one and it shouldn’t do that. Can you provide some steps to be able to reproduce this? How many files were there pending?
I think if the restart happens during the actual upload the file might remain idle as rclone will not attempt to upload one that’s marked as started but all the others that never started should resume after rclone starts up.
In a screen session, ‘mv /tv-shows/ /mnt/Gdrive-Cache’. This was one tv shows folder with less than 20 episodes, each one is 2GB.
confirm using nethogs that rclone starts uploading.
‘service rclone restart’ or “service rclone stop/start” few times.
in nethogs it seems that rclone will stil upload.
reboot server
rclone will not finish the upload.
After a reboot there were 3 files pending. in another case, there were 6 files pending. all of them sits in upload folder and rclone doesn’t seem to retries.
I have tried 4 times. and in the 4 times after reboot rclone will not resume file transfer.
shouldn’t the logic be:
check internal rclone list && physical files in upload folder.
if list == empty && there is file in upload folder, reupload the files anyway, because last time the operation might not fully finished.
Checking actual files in the folder doesn’t help. There’s metadata required for an upload to be completed. Just looking at the files isn’t enough to do it, that’s why a persistent state is needed.
Some -v logs would likely have revealed where the queue gets stuck but I’ll also try to replicate it in a VM.
Just to let you know that I have seen this behaviour as well on OSX High Sierra:
Large upload queue
Upload interrupted by a reboot
Remaining queue does not get uploaded.
Note: a umount then rclone mount works fine, queue recommences uploading. EDIT: after issuing umount command, must wait for the rclone process to complete uploading the current file, then the process will quit. Then rclone mount to continue uploading.