Can rclone tar files off a directory for a copy in flight

Help and Support
1

What is the problem you are having with rclone?

I need to upload around 5TB worth data from disk onto Object Storage. To do this, I'm compressing the data first into a tar file and then uploading with rclone copy command.

Can you pls suggest if the same compression can be done by rclone copy in flight to avoid the double processing times and steps. One to tar and one for rclone copy for the tarred file. The data mostly consists of gz files already, so I'd like to stick to only creating a tar file. tar.gz is not giving any further compression benefit.

What is your rclone version (output from rclone version)

# rclone version
rclone v1.53.1
- os/arch: linux/amd64
- go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Oracle Linux 7.8 64 bit

Which cloud storage system are you using? (eg Google Drive)

Oracle Cloud Infrastructure Object Storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy /tmp/custom-test.tar <rclone-profile>:<object-storage-bucket>

The rclone config contents with secrets removed.

[rclone-profile]
type = s3
provider = IBMCOS
access_key_id = <redacted>
secret_access_key = <redacted>
region = us-ashburn-1
endpoint = <redacted_namespace>.compat.objectstorage.us-ashburn-1.oraclecloud.com
location_constraint = us-ashburn-1
server_side_encryption = AES256
force_path_style = true

A log from the command with the -vv flag

None

2

https://rclone.org/commands/rclone_rcat/

i got this from @calisro

tar -zcvf - / --exclude-from=/tmp/exc | rclone rcat robgs-cryptp:/vpsbackups/data_date "+%Y-%m-%d_%H:%M:%S".tar.gz -v
3

Thanks for the suggestion on rcat. Tried out the command but ran into issues. Does this look related to rclone limits or the type of cloud storage?

tar cf - /tmp/sync-dir/ | rclone rcat <rclone-profile>:<os-bucket-name>/custom-test-rcat.tar

tar: Removing leading `/' from member names
2020/10/02 20:02:48 NOTICE: S3 bucket <os-bucket-name>: Streaming uploads using chunk size 5M will have maximum file size of 48.828G
2020/10/02 20:14:09 Failed to rcat with 2 errors: last error was: multipart upload failed to upload part: InvalidPart: The part number must be between 1 and 10000
	status code: 400, request id: <req-id>, host id:
4

uploading a large file to s3 requires multipart upload; the file is uploaded in chunks.
using copy/move, rclone knows the size of the file before upload and sets the correct value for https://rclone.org/s3/#s3-chunk-size

when using rcat, rclone does not know the file size and you must set the correct value.

5

...so what you want (assuming the file is 5TB) is a chunk size of 5TB / 10000 which is 500 MB

Add a bit to that for safety say 600M

--s3-chunk-size 600M

Note that these chunks are buffered in memory and there will be a few of them so expect rclone to use multiple GB of RAM

You'll see in the NOTICE that rclone prints if you got it right.

6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.