What is the problem you are having with rclone?
Random, unreproduceable issues with an rclone union remote. See below for more detail.
The issues have not shown up when I try the exact same operations directly on the individual remotes, so I suspect the issue is with the union and not with them (and so not with the storage provider either).
That, combined with the fact that my testing of their service is using (for now) free accounts could be construed as abuse, is the reason I'm omitting the provider name.
This is the first time I'm really using the Union remote, so my objective with this post (and the reason I'm posting it to the "Help and Support" section, and not to the "Suspected Bug" section) is to hear from all of you rclone Union experts/veterans: are you having any problems? Do you see anything obviously wrong with my setup?
Thanks in advance for any responses.
What is your rclone version (output from rclone version)
rclone v1.55.1
- os/type: linux
- os/arch: amd64
- go/version: go1.16.3
- go/linking: static
- go/tags: none
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Linux 64 bits (Devuan 3.0 Beowulf and and Ubuntu 18.04).
Which cloud storage system are you using? (eg Google Drive)
Union remote, on top of a series of other remotes which themselves use free accounts.
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Issues are really diverse and unreproduceable, but usually show up after I copy a series of files to the union via:
rclone copy LOCALDIR UNION:REMOTEDIR
and then mount the remote via:
rclone mount UNION: ~/LOCALMOUNTPOINT --low-level-retries=1000 --dir-cache-time 10m --max-read-ahead 256k --vfs-cache-mode=full
and finally use MD5SUM to verify the copy with a command like:
(cd ~/LOCALMOUNTPOINT/REMOTEDIR && md5sum -c REMOTEDIR.md5 | grep -v ': OK$')
The problems are only detected when I run the last command above, and they range from corrupted files (which, when compared to the originals, shows many and varied-sized blocks full of zeroes instead of the supposed data), missing files and even whole directories, etc.
In other words, these errors are totally silent and don't show up until the whole copied data is verified file-by-file and byte-by-byte.
Also, at least so far the issues seem to be totally random and irreproducible: if I purge the destination directory for the copy, and then repeat the exact same commands, then the data is copied and verified OK (or other, different errors are detected instead).
The rclone config contents with secrets removed.
[UNION]
type = union
upstreams = rem0001: rem0002: rem0003: rem0004: rem0005: rem0006: rem0007: rem0008: rem0009: rem0010: rem0011: rem0012: rem0013: rem0014: rem0015: rem0016: rem0017: rem0018: rem0019: rem0020: rem0021: rem0022: rem0023: rem0024: rem0025: rem0026: rem0027: rem0028: rem0029: rem0030: rem0031: rem0032: rem0033: rem0034: rem0035: rem0036: rem0037: rem0038: rem0039:
create_policy = mfs
[rem0001]
type = REDACTED
user = REDACTED
pass = REDACTED
hard_delete = true
[rem0002]
type = REDACTED
user = REDACTED
pass = REDACTED
hard_delete = true
[rem0003]
type = REDACTED
user = REDACTED
pass = REDACTED
hard_delete = true
(and so on, ad nauseam for all 39 remotes, each one referring to a free account created on a storage provider who shall remain redacted for the reasons pointed above).
A log from the command with the -vv flag
Logs show nothing, I've activated -vv everywhere and even used backend-specific debug flags, and then gone through it all with a fine comb and didn't find anything unusual/abnormal.
So I will refrain from posting them at least until I can at least get a reproducible issue.