Transferred: 0 B / 0 B, -, 0 B/s, ETA - when copying from AZ to S3 300k small files

What is the problem you are having with rclone?

I am trying to copy files from azure blob storage to crypted S3 bucket. The source contains over 300k files in the same dir. The average size is >1Mb. It looks like rclone is copying nothing. The longest execution took over 12H till I ended it up. It was without any verbosity. But there were no files on the destination. It definitely works for smaller container ~5k files. It worked for this container with a smaller amount of files passed to the include-from flag. I stopped that because I missed pagination in az output. So this file list contained 5000 files at max. Rclone runs in a container with Mem limits 2GiB and CPU 1core.

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-1041-azure (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.1
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Azure Blob storage as a source
AWS S3 Bucket as a backend for crypt

The command you were trying to run (eg rclone copy /tmp remote:tmp)

    rclone --s3-disable-checksum --s3-no-head --max-backlog 1000 --transfers 64 --checksum  --s3-upload-cutoff 100MiB --dump requests \
    --dump responses --azureblob-disable-checksum --azureblob-no-head-object\
    --azureblob-sas-url "https://${AZ_ACCOUNT}.blob.core.windows.net/${AZ_CONTAINER}?$(cat /root/az/sas_token)" --log-level=DEBUG -P --no-traverse copy az:$AZ_CONTAINER \
    crypt:$DATE_MARK/ --include-from blob-list.txt && \

This is the last iteration of the command. I played a lot with flags during the research of docs and topics with the same problem. blob-list.txt contains file names I need to copy. Every file name with new lane.

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[az]
type = azureblob
account = XXX
tenant = XXX

[crypt]
type = crypt
filename_encryption = off
password = XXX
password2 = XXX
remote = s3:mt-api-bu-alphanumerichash
directory_name_encryption = false

[s3]
type = s3
provider = AWS
env_auth = true
region = eu-central-1
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

deleted part marked with dots - list of 5k files.

You probably meant --include-files here assuming that your file has filenames one per line. With --include-files you've written 300k lines into the filters rules which will be extremely slow.

Use one of these instead.

  --files-from stringArray       Read list of source-file names from file (use - to read from stdin)
  --files-from-raw stringArray   Read list of source-file names from file without any processing of lines (use - to read from stdin)

Because azure and s3 have a compatible checksum then I would use --checksum to do the transfer - this will save HEAD requests on the objects.

No, I actually meant --include from. I picked it up from Rclone Filtering and rclone copy I see no flag called --include-files. But now I understand the problem. I need to pass a list of files I want to copy. If possible, I want to avoid getting such a list from the source.
I tried both proposed flags — the same thing both times. This time I deleted --dump flags to make the log readable.

2023/09/27 16:48:04 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "--s3-disable-checksum" "--s3-no-head" "--max-backlog" "1000" "--transfers" "64" "--checksum" "--azureblob-disable-checksum" "--azureblob-no-head-object" "--azureblob-sas-url" "https://alphanumerichash.blob.core.windows.net/main?sas_sig_here" "--log-level=DEBUG" "-P" "--no-traverse" "copy" "az:main" "crypt:2023-09-27-16-45/" "--files-from" "blob-list.txt"]
2023/09/27 16:48:04 DEBUG : Creating backend with remote "az:main"
2023/09/27 16:48:04 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/09/27 16:48:04 DEBUG : az: detected overridden config - adding "{Op8Dn}" suffix to name
2023/09/27 16:48:04 DEBUG : fs cache: renaming cache item "az:main" to be canonical "az{Op8Dn}:main"
2023/09/27 16:48:04 DEBUG : Creating backend with remote "crypt:2023-09-27-16-45/"
2023/09/27 16:48:04 DEBUG : Creating backend with remote "s3:mt-api-bu-alphanumerichash/2023-09-27-16-45/.bin"
2023/09/27 16:48:04 DEBUG : s3: detected overridden config - adding "{n3mmc}" suffix to name
2023/09/27 16:48:04 DEBUG : fs cache: renaming cache item "s3:mt-api-bu-alphanumerichash/2023-09-27-16-45/.bin" to be canonical "s3{n3mmc}:mt-api-bu-alphanumerichash/2023-09-27-16-45/.bin"
2023/09/27 16:48:04 DEBUG : Creating backend with remote "s3:mt-api-bu-alphanumerichash/2023-09-27-16-45"
2023/09/27 16:48:04 DEBUG : s3: detected overridden config - adding "{n3mmc}" suffix to name
2023/09/27 16:48:04 DEBUG : fs cache: renaming cache item "s3:mt-api-bu-d011fafd09e8/2023-09-27-16-45" to be canonical "s3{n3mmc}:mt-api-bu-alphanumerichash/2023-09-27-16-45"
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.9sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         1.4sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         1.9sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         2.4sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         2.9sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         3.4sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         3.9sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         4.4sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         4.9sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         5.4sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         5.9sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -

In 3 hours of execution, it shows something new

*                  bl9obxhw5uk19z5pfw96i1t6btz4:  0% /0, 0/s, -
 *                  bla1h9l0iwb0es2s7o36hhwv510l: transferring
 *                  blal4f9rdyc8urg8gwh07kwqeg62:  0% /0, 0/s, -
 *                  blalgfbs8q0lxrs3noa5wplzywco: transferring
 *                  blam2f70ohd2inotpxyy6ruhv5rv: transferring
 *                  blbp03xds7tf592rjlzvtmoc796c: transferring
2023-09-28 06:41:36 DEBUG : bpnbsepl20qsoa9zbea0oqusp4at: Need to transfer - File not found at Destination
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:             97195 (retrying may help)
Transferred:            0 / 1064, 0%
Elapsed time:   2h38m19.4s
Transferring:
 *                  bl28b8megoxoc46hrbeypb3h24fl:  0% /0, 0/s, -
 *                  bl2ujva0x5kcih7dg9hu3vfcb9l0:  0% /0, 0/s, -
 *                  bl34au3yn3yahv9elxapql9ybbe3:  0% /0, 0/s, -
 *                  bl35g2kiz3zkv32nrsf8ncrs0dns:  0% /0, 0/s, -
 *                  bl36re7ydsip4rn3qiixwahf8i34:  0% /0, 0/s, -
 *                  bl3e3x50tsbgy79gjz3bvl9r7291:  0% /0, 0/s, -
 *                  bl3ieftl3j0y70qr8qaeapda3ki6:  0% /0, 0/s, -
 *                  bl3k8ura7qy93cd9w23chxcf36qw:  0% /0, 0/s, -
 *                  bl3lez73lnl9sffvqew5uecs4c88:  0% /0, 0/s, -
 *                  bl3oycovr6azvpikhn2ey1tngayt:  0% /0, 0/s, -
 *                  bl3wspfvlgaf6csl2tfrrmtwxil7:  0% /0, 0/s, -
 *                  bl42oubvl6arauw9wc7jrihptwkx:  0% /0, 0/s, -
 *                  bl439t7ykjmuw3vyi2zgj1gnmagl:  0% /0, 0/s, -
 *                  bl44t2tvu2ewo8ry6yrwrdczj7w3:  0% /0, 0/s, -
 *                  bl46peeq5z6e2plhjbwpzp6htbqs:  0% /0, 0/s, -
 *                  bl47qd9nqr5sq2gm36aobg8ecktj:  0% /0, 0/s, -
 *                  bl49tk68nazdxlkr9na9cqchdvxa:  0% /0, 0/s, -
 *                  bl4dgabm15et97wpgtofe9gkdz5r:  0% /0, 0/s, -
 *                  bl4h6caign492t9lrvc1ea2titxx:  0% /0, 0/s, -
 *                  bl4mqbt5tirm6e49fa5hx8sglras:  0% /0, 0/s, -
 *                  bl4x6jv49zgt55kcquf8gqsjfh5t:  0% /0, 0/s, -
 *                  bl54ain00ahy52unrv91jq1k301v:  0% /0, 0/s, -
 *                  bl57884lqt1231cakf847lese00v:  0% /0, 0/s, -
 *                  bl5fe21ie0n2iak93yefwq3epr57:  0% /0, 0/s, -
 *                  bl5gj3dp0ot0jiudfmmoth2fnji8:  0% /0, 0/s, -
 *                  bl5hwfi496eref0ncl1jlcfawv3w:  0% /0, 0/s, -
 *                  bl5i62jek1atbgcpbxeeu6rqaefh:  0% /0, 0/s, -
 *                  bl5tw1g7xnuq6bdomk1hkk7tnroq:  0% /0, 0/s, -
 *                  bl5xan2rbvs1p4fkj2eokscu3x1q:  0% /0, 0/s, -
 *                  bl5zkt60bs9tgtlc2d3upqrebrxh:  0% /0, 0/s, -
 *                  bl64nhju15y4yegafsm321ctxfse:  0% /0, 0/s, -
 *                  bl6etbdurb8hx6ysrm757y8wa5t0:  0% /0, 0/s, -
 *                  bl6gbvkn5jomoyjm6ys8zmp0ajxx:  0% /0, 0/s, -
 *                  bl6tht7hmchnau3ccwuu9erib0pd:  0% /0, 0/s, -
 *                  bl6u14igfg791x0hgv2fgs31rurs:  0% /0, 0/s, -
 *                  bl6ugd1p0lfedgxrhfn8mc71jmq3:  0% /0, 0/s, -
 *                  bl6zudqpoi6wkjfc8t78hb5wzzwg:  0% /0, 0/s, -
 *                  bl72pkburu53hk7w99xvnafle73y:  0% /0, 0/s, -
 *                  bl7c9jcpwasbzpolm5r5t52fq3q2:  0% /0, 0/s, -
 *                  bl7kmvdlzabz2el4hc17qxjc7wb3:  0% /0, 0/s, -
 *                  bl7r7u21ax3c368kuc7vx4hcyptj:  0% /0, 0/s, -
 *                  bl7u6hllhu0k3on8bc7kye807yz8:  0% /0, 0/s, -
 *                  bl7yct3sxfzmsejamvy3pvml54ih:  0% /0, 0/s, -
 *                  bl8609j5hajnm466v87b7l6qhlm9:  0% /0, 0/s, -
 *                  bl86edmuj7bgber35c08id2x0uj0:  0% /0, 0/s, -
 *                  bl89r4qklj44r3t6kgno22z17xzv:  0% /0, 0/s, -
 *                  bl8bxqtpml0svjdz6zt2xgc8bz75:  0% /0, 0/s, -
 *                  bl8kyyit4mnzd1uy76tkdc1q9bsa:  0% /0, 0/s, -
 *                  bl8tzcgkn64pz7sl7ddmtv6aolii:  0% /0, 0/s, -
 *                  bl933p3v5qlhlpbohkw8dohbdfw1:  0% /0, 0/s, -
 *                  bl963o8mjugv3zbrptfjaed6tfmd:  0% /0, 0/s, -
 *                  bl97xfcj5bnf5kklkndine02eluj:  0% /0, 0/s, -
 *                  bl98em7bl210i2rd56o9n5didx6b:  0% /0, 0/s, -
 *                  bl9c4hb2ijqxml6smyxaaupg11ki:  0% /0, 0/s, -
 *                  bl9c65b2pd5ajl3bj4terce4hd73:  0% /0, 0/s, -
 *                  bl9fh2x645el62lwwk9oe1hneiqu:  0% /0, 0/s, -
 *                  bl9gbcjckxepip9hggr20q56v72e:  0% /0, 0/s, -
 *                  bl9obxhw5uk19z5pfw96i1t6btz4:  0% /0, 0/s, -
 *                  bla1h9l0iwb0es2s7o36hhwv510l: transferring
 *                  blal4f9rdyc8urg8gwh07kwqeg62:  0% /0, 0/s, -
 *                  blalgfbs8q0lxrs3noa5wplzywco: transferring
 *                  blam2f70ohd2inotpxyy6ruhv5rv: transferring
 *                  blbp03xds7tf592rjlzvtmoc796c: transferring
 *                  blbpo6h73lwueh1eudpswrce3tw1: transferring
2023-09-28 06:41:36 ERROR : bl34au3yn3yahv9elxapql9ybbe3: Failed to copy: failed to prepare upload: AccessDenied: Access Denied
	status code: 403, request id: NQ0KPCPPAYNNC0H6, host id: rKMA0NtZY7yPgAz3fpQJvyV9RWZ43RiRJfkoQAilUIlfhDnNPC37QQs9Z5mDSk0bGxFp+bjy/fM=
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:             97196 (retrying may help)
Transferred:            0 / 1065, 0%
Elapsed time:   2h38m19.5s
Transferring:
 *                  bl28b8megoxoc46hrbeypb3h24fl:  0% /0, 0/s, -
 *                  bl2ujva0x5kcih7dg9hu3vfcb9l0:  0% /0, 0/s, -
 *                  bl34au3yn3yahv9elxapql9ybbe3:  0% /0, 0/s, -
 *                  bl35g2kiz3zkv32nrsf8ncrs0dns:  0% /0, 0/s, -
 *                  bl36re7ydsip4rn3qiixwahf8i34:  0% /0, 0/s, -
 *                  bl3e3x50tsbgy79gjz3bvl9r7291:  0% /0, 0/s, -

I tried a simple az blob download on the 403 file and not found at Destination one with the same SAS token. Works well.

["rclone" "--s3-disable-checksum" "--s3-no-head" "--max-backlog" "1000" "--transfers" "64" "--checksum" "--azureblob-disable-checksum" "--azureblob-no-head-object" "--azureblob-sas-url" "https://alphanumerichash.blob.core.windows.net/main?sas_sig_here" "--log-level=DEBUG" "-P" "--no-traverse" "copy" "az:main" "crypt:2023-09-27-16-45/" "--files-from" "blob-list.txt"]

Add --s3-no-check-bucket to fix this.

Also add --checkers 64 (or larger) to speed up gap from starting to when things transfer.

Thanks, I actually investigated whole day the reason. Found that this 403 was about s3 and not Azure. But pretty strange thing:

  • It worked before
  • It works for 2 other containers/buckets

Other buckets use the same policy. Not same in the meaning of similarity. One policy for all 3 buckets. Buckets were created as clones of the first one. However, a command for those 2 buckets is:

rclone --fast-list --update --use-server-modtime --s3-disable-checksum --s3-no-head --azureblob-sas-url "https://${AZ_ACCOUNT}.blob.core.windows.net/${AZ_CONTAINER}?$(cat /root/az/sas_token)" copy az:$AZ_CONTAINER crypt:$DATE_MARK/ --include-from blob-list.txt

This was the original one. I made changes to find the workaround for big batches of files. As you can see there are no --s3-no-check-bucket flags. But actually, it helped. I'm just sharing info about strange actions. Thanks a lot.

So is it running a bit better now?

It is running. It achieves goals. That is enough for me :grinning:. It took 2.5h but it is just the initial batch. The next batches will be far lower in numbers. And it was without checkers/transfers improvement because I want to keep resource usage low. I'll play with them if needed. --s3-no-check-bucket by itself solved the problem. Just highlighted a strange thing that it wasn't needed earlier. And it wasn't needed for other buckets.

Thanks for the advice and thanks for your software.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.