Rclone chunker newbie

Help and Support
1

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

rclone-chunker is not working

Run the command 'rclone version' and share the full output of the command.

rclone v1.42

  • os/arch: linux/amd64
  • go version: go1.9.4

Which cloud storage system are you using? (eg Google Drive)

chunker

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --config /u04/share/dom_dba.conf ls chunker-test:
2024/08/25 12:35:46 Failed to create file system for "chunker-test:": didn't find filing system for "chunker"

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

Paste config here

A log from the command that you were trying to run with the -vv flag

Paste  log here
2

Your version is about 8-9 years old. Update.

3

woah, thank you! we'll update for sure.

4

how to update the version from 1.42 to 1.67, i already download the .rpm file, I am not sure if rclone selfupdate [flags] will work.

Options

      --beta             Install beta release
      --check            Check for latest release, do not download
  -h, --help             help for selfupdate
      --output string    Save the downloaded binary at a given path (default: replace running binary)
      --package string   Package format: zip|deb|rpm (default: zip)
      --stable           Install stable release (this is the default)
      --version string   Install the given rclone version (default: latest)
5

Self update was introduced many years after your version:) so won’t work.

Remove/uninstall your old version. Probably part of your OS packages.

Then install the latest one directly form rclone website:

When done provide all details of your issue. As so far it is not clear at all what does not work.

6

Thanks, am I doing it right?

1: delete existing rclone
rm -r /usr/sbin/rclone

2: install
yum install rclone-v1.67.0-linux-amd64.rpm

3: Copy binary file
sudo cp rclone /usr/sbin/
sudo chown root:root /usr/sbin/rclone
sudo chmod 755 /usr/sbin/rclone

7

hi,

run rclone version.

otherwise,

  1. uninstall using the same technique are you installed the old version, os package manager or however.
  2. install rclone the easy way using the official script - https://rclone.org/install/#script-installation
  3. run rclone version

otherwise,

  1. download+install pre-compiled binary - https://rclone.org/install/#linux-precompiled
  2. run rclone version
8

Thanks everyone @asdffdsa @kapitainsky @Animosity022 , I am able to install rclone latest version 1.67 , chunker is now working. I am working on a project where I will use chunker to transfer big dumps sample 4TB. And I am getting this from the logs, is this okay? note that I set the chunk-size to 100G .

2024/09/04 09:21:08 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405: Need to transfer - File not found at Destination
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405: size: 100Gi, parts: 10000, default: 5Mi, new: 11Mi; default chunk size insufficient, returned new chunk size
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405.rclone_chunk.001_acuyqn: open chunk writer: started multipart upload: 32e685d8-5c4b-ba9b-1fce-014bed83eadb
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405: multipart upload: starting chunk 0 size 11Mi offset 0/100Gi
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405: multipart upload: starting chunk 1 size 11Mi offset 11Mi/100Gi
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405: multipart upload: starting chunk 2 size 11Mi offset 22Mi/100Gi
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405: multipart upload: starting chunk 3 size 11Mi offset 33Mi/100Gi
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405.rclone_chunk.001_acuyqn: multipart upload wrote chunk 2 with 11534336 bytes and etag "3c315415c41040836f156d23743a499f"
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405.rclone_chunk.001_acuyqn: multipart upload wrote chunk 1 with 11534336 bytes and etag "983f3eb7bc68809901d41554b31346bf"
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405: multipart upload: starting chunk 4 size 11Mi offset 44Mi/100Gi
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405: multipart upload: starting chunk 5 size 11Mi offset 55Mi/100Gi
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405.rclone_chunk.001_acuyqn: multipart upload wrote chunk 4 with 11534336 bytes and etag "43d6e1d9efdbb259e19e6646ba2c83aa"
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405: multipart upload: starting chunk 6 size 11Mi offset 66Mi/100Gi
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405.rclone_chunk.001_acuyqn: multipart upload wrote chunk 3 with 11534336 bytes and etag "c82dfcac88140c2c897cecfb7c5cc7b9"
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405: multipart upload: starting chunk 7 size 11Mi offset 77Mi/100Gi
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405.rclone_chunk.001_acuyqn: multipart upload wrote chunk 6 with 11534336 bytes and etag "f7cee036d6b26aef364e2f55d45e5fee"
2024/09/04 09:21:09 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405: multipart upload: starting chunk 8 size 11Mi offset 88Mi/100Gi
2024/09/04 09:21:10 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405.rclone_chunk.001_acuyqn: multipart upload wrote chunk 9 with 11534336 bytes and etag "8e9aa1e20d8d80d28891c4e2233a2ce2"
2024/09/04 09:21:10 DEBUG : df_D01FYTQ_864987818_4a2vqfgl_24714_1_1_20240713_1174224405: multipart upload: starting chunk 9 size 11Mi offset 99Mi/100Gi

9

It is some log fragment without any details. Not sure what do you want really.... nobody can see what you are doing.

10

yes, it is okay.

i am going to assume that remote provider is S3, based on:

  • parts: 10000 from --s3-max-upload-parts
  • default: 5Mi from --s3-chunk-size

now, might be a bit confusing, as there are two types of chunks.

  1. size of chunk as used by chunker.
  2. size of chunk as used by S3 provider when uploading the file.

with chunker, you specifced the chunk size as 100GiB. remember, each chunk is uploaded as a single file.
rclone wants to upload that 100GiB file, but using default values, the max file size is limited to 48GiB.

rclone noticed that problem, and instead of erroring out,
rclone increased the --s3-chunk-size from 5Mi, to 11Mi
check the docs - https://rclone.org/s3/#s3-chunk-size

now, need to post the output of:

  • rclone config redacted
  • top 20 lines of the debug output
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.