Rclone getting stuck on mishandled file in crypt remote: unexpected EOF

Help and Support
1

What is the problem you are having with rclone?

While attempting to rclone sync, the copy part of the command seems to get stuck on a file that was mishandled by me. I deleted it and the folder on the local and google drive remote. I emptied the trash, ran rclone cleanup remote:, and I thought the file was gone.

The mishanding occurred since I might have deleted the file, while it was in the process of being transferred, and maybe that's why the partial~ file existst. The file no longer exists nor do I want it. I just want my rclone sync command to finish all the way through.

rclone v1.58.0-beta.6018.ec72432ce

  • os/version: debian 11.2 (64 bit)
  • os/kernel: 5.10.0-11-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.7
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive/TeamDrive

rclone sync Remote1_Crypt:path/to/file/X /mnt/local/path/to/file/X --backup-dir /mnt/path/to/file/ToBeDeleted/$(date +%Y-%m-%d)/X -vv --log-file=/home/username/logs/rclone_sync.log

2022/04/02 01:37:20 DEBUG : T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~: Reopening on read failure after 5533925376 bytes: retry 1/10: unexpected EOF
2022/04/02 01:37:20 DEBUG : Remote1: Loaded invalid token from config file - ignoring
2022/04/02 01:37:20 DEBUG : Saving config "token" in section "Remote1" of the config file
2022/04/02 01:37:20 DEBUG : Keeping previous permissions for config file: -rw-r--r--
2022/04/02 01:37:20 DEBUG : Remote1: Saved new token in config file
2022/04/02 01:37:21 DEBUG : T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~: Reopen failed after 5533925376 bytes read: unexpected EOF
2022/04/02 01:37:21 DEBUG : T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~: multi-thread copy: stream 4/4 failed: multipart copy: read failed: unexpected EOF
2022/04/02 01:37:21 DEBUG : T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~: Reopening on read failure after 5031067648 bytes: retry 1/10: context canceled
2022/04/02 01:37:21 DEBUG : T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~: Reopen failed after 5031067648 bytes read: open file failed: Get "https://www.googleapis.com/drive/v3/files/1n_d0X9tH5OQZA-1-PacWALKDSXasa>
2022/04/02 01:37:21 DEBUG : T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~: multi-thread copy: stream 1/4 failed: multipart copy: read failed: context canceled
2022/04/02 01:37:21 DEBUG : T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~: multi-thread copy: stream 2/4 failed: context canceled
2022/04/02 01:37:21 DEBUG : T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~: multi-thread copy: stream 3/4 failed: context canceled
2022/04/02 01:37:21 DEBUG : T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~: Received error: multipart copy: read failed: unexpected EOF - low level retry 3/10
2022/04/02 01:37:23 DEBUG : T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~: Starting multi-thread copy with 4 parts of size 5.154Gi
2022/04/02 01:37:23 DEBUG : T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~: multi-thread copy: stream 4/4 (16601972736-22135926784) size 5.154Gi starting
2022/04/02 01:37:23 DEBUG : T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~: multi-thread copy: stream 2/4 (5533990912-11067981824) size 5.154Gi starting
2022/04/02 01:37:23 DEBUG : T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~: multi-thread copy: stream 1/4 (0-5533990912) size 5.154Gi starting
2022/04/02 01:37:23 DEBUG : T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~: multi-thread copy: stream 3/4 (11067981824-16601972736) size 5.154Gi starting

Above is a snippet of the log file and it's pretty much this error repeating. The issue is that the rclone sync command is never able to finish, and can never delete files on the local array, causing huge overflow issues.

2

The file looks broken on the source somehow.

Can you delete it on the source? Something like this should do it (check the path to the remote - you didn't write the exact command you are using for rclone)

rclone deletefile -i "Remote1_Crypt:/T/Tunes of Glory (1960)/Tunes.of.Glory.1960.Criterion-Randommmm.mkv.partial~"

3

I deleted it on my rclone mount which uses the following command:

rclone mount UnionDrive: mounttpoint --config=/home/user/.config/rclone/rclone.conf --log-level INFO --log-file /home/user/logs/gdrive.log --allow-non-empty --use-mmap --fast-list --drive-chunk-size 128M --vfs-read-chunk-size 64M --vfs-read-chunk-size-limit 512M --vfs-cache-mode full --cache-dir /home/user/cache --vfs-cache-max-age 120h --vfs-cache-max-size 100G --dir-cache-time 72h --buffer-size 16M --vfs-read-ahead 16G

Question: Is deletefile significantly different? Should I not be deleting files from the mount directly?

4

Deleting it from the mount is fine if you found that easier. They both do the same thing.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.