Re-assembling chunked files on a remote

What is the problem you are having with rclone?

I have made the mistake of using the chunker when I shouldn't have. I was transferring large files between 2 different private compute clusters I have access to and the chunker was necessary to this process (although I must be honest I'm not entirely sure why. It didn't work without it but did work with it).

I copied files from one to the other because one cluster was going offline. However, on the cluster that is still active I have a bunch of chunked files but can't do anything with them. Is there any way to rechunk the files on the cluster that is still online without using the one that is now offline?

Run the command 'rclone version' and share the full output of the command.

I do not have access to the first cluster (the one I ran the command from) so cannot run any rclone commands but I am pretty sure it was using the most recent version of rclone as of 25th April 2023.

On the second cluster the output of rclone version is:

rclone v1.62.2
- os/version: centos 7.2.1511 (64 bit)
- os/kernel: 3.10.0-327.36.3.el7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

I am using the sftp protocol to transfer between clusters.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I do not have access to it anymore unfortunately. I can attempt to reconstruct it if deemed necessary but cannot guarantee the accuracy of such a reconstruction.

The rclone config contents with secrets removed.

Again I do not have access to the old cluster that I ran everything from.

lets say it is your rclone.config, we have to add new NoChunks remote:

[sftp]
type = sftp
...

[chunker]
type = chunker
remote = sftp:Path/To/ChunksFolder

[sftp-NoChunks]
type = alias
remote = sftp:Path/To/NoChunksFolder

and now:

rclone move --server-side-across-configs -vv chunker: sftp-NoChunks:

test with --dry-run or/and with small data sample first

if you have shell access to computer where data is now:

[chunker]
type = chunker
remote = /Path/To/ChunksFolder

rclone move chunker: /Path/To/NewNoChunksFolder

I think this will probably fail with the --server-side-across-configs flag alas.

It should work without but that does mean an upload and a download.

Was not aware of this alias limitation - always learning.

Let's see what @hahahasan situation is. If no shell access we can always try to approach it differently.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.