What is the problem you are having with rclone?
I am copying from S3-compatible storage. S3 does not actually have a hierarchy of directories — slashes in the blob keys are not inherently meaningful (though many tools present a directory-like facade).
If I have a blob with key remoteStorageBlob/documents/notes/ffc86831-153e-447f-b217-f65c0de612ee in a bucket, I also have blobs with keys remoteStorageBlob/documents/notes and remoteStorageBlob/documents.
When I use rclone to copy my blobs to a remote file system (using SSH, but that should make no difference), rclone wants to create a directory hierarchy (which in many scenarios is what is needed, but not in mine). rclone copies the blob remoteStorageBlob/documents, but then refuses to copy the blobs remoteStorageBlob/documents/notes and remoteStorageBlob/documents/notes/ffc86831-153e-447f-b217-f65c0de612ee, because it want to create the directory remoteStorageBlob/documents, but that path in the destination file system is already used for a file:
2024/07/16 21:57:36 ERROR : doug-store.hominidsoftware.com/remoteStorageBlob/documents/notes/ffc86831-153e-447f-b217-f65c0de612ee: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: is a file not a directory
Is there a flag or option or workaround to copy all the blobs from an S3 bucket to a single directory or archive, without needing to created a hierarchy of directories? Presumably, slashes in the S3 keys would need to be mapped to another character, for the file system.
The proposed --flatten argument, as described (flat copy/sync without preserving dirs: `--flatten` and `--flatten-mode` · Issue #2755 · rclone/rclone · GitHub) would not help, as I need to preserve the whole S3 key.
Run the command 'rclone version' and share the full output of the command.
rclone v1.60.1-DEV
- os/version: debian 12.6 (64 bit)
- os/kernel: 6.1.0-really95-rh-20240624003055.xenU.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.8
- go/linking: dynamic
- go/tags: none
The latest version would appear to be v1.67.0
Which cloud storage system are you using? (eg Google Drive)
Garage (https://garagehq.deuxfleurs.fr/) using the S3-compatible API. (Any S3-compatible system would have this problem.)
I'm writing to a file system using SSH, but that shouldn't make a difference.
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone copy garage: --include '*-store.hominidsoftware.com/**' bakop:armadietto-modular-test
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
$ cat .config/rclone/rclone.conf
[garage]
type = s3
provider = Other
access_key_id = G...4
secret_access_key = 4...5
region = garage
endpoint = http://127.0.0.1:3900
location_constraint = garage
acl = private
[bakop]
type = sftp
host = host2.bakop.com
user = ftpz
key_use_agent = true
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum
A log from the command that you were trying to run with the -vv flag
$ rclone -vv copy garage: --include '*-store.hominidsoftware.com/**' bakop:armadietto-modular-test
2024/07/17 00:09:35 DEBUG : rclone: Version "v1.60.1-DEV" starting with parameters ["rclone" "-vv" "copy" "garage:" "--include" "*-store.hominidsoftware.com/**" "bakop:armadietto-modular-test"]
2024/07/17 00:09:35 DEBUG : Creating backend with remote "garage:"
2024/07/17 00:09:35 DEBUG : Using config file from "/home/armadietto/.config/rclone/rclone.conf"
2024/07/17 00:09:35 DEBUG : Creating backend with remote "bakop:armadietto-modular-test"
2024/07/17 00:09:36 DEBUG : sftp://ftpz@host2.bakop.com:22/armadietto-modular-test: New connection 74.50.55.206:55992->72.249.182.215:22 to "SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u3"
2024/07/17 00:09:36 DEBUG : sftp://ftpz@host2.bakop.com:22/armadietto-modular-test: Shell type "unix" from config
2024/07/17 00:09:36 DEBUG : sftp://ftpz@host2.bakop.com:22/armadietto-modular-test: Relative path resolved to "/home/ftpz/armadietto-modular-test"
2024/07/17 00:09:36 DEBUG : sftp://ftpz@host2.bakop.com:22/armadietto-modular-test: Using root directory "/home/ftpz/armadietto-modular-test"
2024/07/17 00:09:36 DEBUG : doc-beta.hominidsoftware.com: Excluded
2024/07/17 00:09:36 DEBUG : michigander-beta.hominidsoftware.com: Excluded
2024/07/17 00:09:36 DEBUG : first: Excluded
2024/07/17 00:09:36 DEBUG : shared-store.hominidsoftware.com/remoteStorageBlob: Size and modification time the same (differ by -608ms, within tolerance 1s)
2024/07/17 00:09:36 DEBUG : shared-store.hominidsoftware.com/remoteStorageBlob: Unchanged skipping
2024/07/17 00:09:36 ERROR : shared-store.hominidsoftware.com/remoteStorageBlob/documents: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: is a file not a directory
2024/07/17 00:09:36 DEBUG : shared-store.hominidsoftware.com/remoteStorageAuth/userMetadata: Size and modification time the same (differ by -738ms, within tolerance 1s)
2024/07/17 00:09:36 DEBUG : shared-store.hominidsoftware.com/remoteStorageAuth/userMetadata: Unchanged skipping
2024/07/17 00:09:36 ERROR : shared-store.hominidsoftware.com/remoteStorageBlob/documents/notes: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: is a file not a directory
2024/07/17 00:09:36 ERROR : shared-store.hominidsoftware.com/remoteStorageBlob/documents/notes/03d41531-5168-4688-a74f-0a5f2b7171eb: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: is a file not a directory
...