Rclone Copy command taking forever to start the copy between remotes

Help and Support
1

What is the problem you are having with rclone?

I am trying to copy a folder from Box Storage that has around 872k files in it and the total size of 650GB to an AWS S3 bucket.
But it is taking a long time to start the actual copy and am trying to understand how much time it may take before it starts the actual copy.

Until now i tried to do a rclone copy of a smaller sized folder that had around 300 files from a Box folder and the total size of around 217MB to the same AWS S3 bucket and it worked and completed quickly.

Run the command 'rclone version' and share the full output of the command.

PS C:\rclone-v1.68.2-windows-amd64\rclone-v1.68.2-windows-amd64> .\rclone version
rclone v1.68.2

  • os/version: Microsoft Windows Server 2022 Datacenter 21H2 (64 bit)
  • os/kernel: 10.0.20348.2849 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.23.3
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Box and AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

.\rclone copy BoxtoS3:/GCX/ CWProd-S3:/elektra-file/pega-gcx --progress

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[AWSS3]
type = s3
region = us-east-1
secret_access_key = XXX
access_key_id = XXX

[BoxtoS3]
type = box
token = XXX

[CWProd-S3]
type = s3
region = us-east-1
secret_access_key = XXX
access_key_id = XXX
provider = AWS
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

PS C:\rclone-v1.68.2-windows-amd64\rclone-v1.68.2-windows-amd64> .\rclone copy BoxtoS3:/GCX/ CWProd-S3:/elektra-file/pega-gcx --progress -vv
2024/12/05 20:54:56 DEBUG : rclone: Version "v1.68.2" starting with parameters ["C:\\rclone-v1.68.2-windows-amd64\\rclone-v1.68.2-windows-amd64\\rclone.exe" "copy" "BoxtoS3:/GCX/" "CWProd-S3:/elektra-file/pega-gcx" "--progress" "-vv"]
2024/12/05 20:54:56 DEBUG : Creating backend with remote "BoxtoS3:/GCX/"
2024/12/05 20:54:56 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2024/12/05 20:54:57 DEBUG : fs cache: renaming cache item "BoxtoS3:/GCX/" to be canonical "BoxtoS3:GCX"
2024/12/05 20:54:57 DEBUG : Creating backend with remote "CWProd-S3:/elektra-file/pega-gcx"
2024/12/05 20:54:57 DEBUG : fs cache: renaming cache item "CWProd-S3:/elektra-file/pega-gcx" to be canonical "CWProd-S3:elektra-file/pega-gcx"
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         21m16.1s
2

welcome to the the forum,

box lacks support for ListR, therefore, does not support --fast-list
for a deeper look, try --dump=headers

also, not sure your use-case but might consider --no-traverse and --no-check-dest

fwiw, what i do, when the dest is an aws bucket.
run rclone on cheap cloud ec2 vm in the same region as the bucket.
no need for windowz to be the middleman.

for AWSS3:, it is is missing provider = AWS

for most providers, get better performace when creating client id+secret.
not sure that applies to box

3

Just a quick update - the rclone copy command eventually started after a delay of an hour and a little over.
I guess it does some kind of read of the source folder and if it is a very big folder then it takes time for it to start copying the data.