Cop large file create segments - hown to dowload te original file

Help and Support
1

Hi, I'm new with rclone.

What is your rclone version (output from rclone version)

rclone v1.53.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Proxmox backup server (debian)

Which cloud storage system are you using? (eg Google Drive)

swift on ovh

The command you were trying to run (eg rclone copy /tmp remote:tmp)

,

I have successfully copy large file from a server to swift on OVH using te following command to backup tis large file:

rclone -v --log-file=clone.log --transfers=10 --checkers=10 --drive-chunk-size=16384k copy --max-age 1d /mnt/datastore//bck02/nfs/dump swift:/path

There is no error and te logfile return successfull copy and size copied

In the swiwft:/path file as a size of 0
In the swift:/path_segments I found te chunks files
Using rclone lsd swift:/path_segments I can "see" te right downloaded file

Now I want to transfert my backup file on the orginal server, Witch command may I use to download te fragmented files and reassemble then on oruginal server.
I ave try rclone copy source destination ... doesn't function

Thx for your experience

2

hello and welcome to the forum,

use the same command and switch the source and test.
you can test with https://rclone.org/docs/#n-dry-run

for example,
rclone copy swift:/path /mnt/datastore//bck02/nfs/dump --dry-run -vv --log-file=clone.log --transfers=10 --checkers=10 --drive-chunk-size=16384k --max-age 1d

3

Thankx I have copy back my file as you describe

In the log I see that all data are well copied without error`

2020/12/21 09:28:43 INFO  :
Transferred:       26.149G / 26.391 GBytes, 99%, 24.799 MBytes/s, ETA 9s
Transferred:            1 / 6, 17%
Elapsed time:      18m1.0s
Transferring:
 *     1608399498.089495943/28337417804/00000000: 99% /5G, 5.050M/s, 9s
 *     1608399498.089495943/28337417804/00000002: 98% /5G, 5.009M/s, 11s
 *     1608399498.089495943/28337417804/00000003: 98% /5G, 5.131M/s, 15s
 *     1608399498.089495943/28337417804/00000001: 99% /5G, 5.198M/s, 8s
 *     1608399498.089495943/28337417804/00000004: 99% /5G, 4.339M/s, 4s

2020/12/21 09:29:33 INFO  : 1608399498.089495943/28337417804/00000004: Multi-thread Copied (new)
2020/12/21 09:29:33 INFO  : 1608399498.089495943/28337417804/00000003: Multi-thread Copied (new)
2020/12/21 09:29:40 INFO  : 1608399498.089495943/28337417804/00000000: Multi-thread Copied (new)
2020/12/21 09:29:41 INFO  : 1608399498.089495943/28337417804/00000001: Multi-thread Copied (new)
2020/12/21 09:29:41 INFO  : 1608399498.089495943/28337417804/00000002: Multi-thread Copied (new)
2020/12/21 09:29:41 INFO  :
Transferred:       26.391G / 26.391 GBytes, 100%, 23.769 MBytes/s, ETA 0s
Transferred:            6 / 6, 100%
Elapsed time:     18m58.2s

Now on te file server I have directory with 5 segments files ...how to "merge" these to get m originaaal file back ?

root@proxback02:~/tmp/1608399498.089495943/28337417804# ls -l
total 27673344
-rw-r--r-- 1 root root 5368709120 Dec 19 01:47 00000000
-rw-r--r-- 1 root root 5368709120 Dec 19 01:47 00000001
-rw-r--r-- 1 root root 5368709120 Dec 19 01:47 00000002
-rw-r--r-- 1 root root 5368709120 Dec 19 01:47 00000003
-rw-r--r-- 1 root root 5368709120 Dec 19 01:47 00000004
-rw-r--r-- 1 root root 1493872204 Dec 19 01:47 00000005
root@proxback02:~/tmp/1608399498.089495943/28337417804# du -H
27673348        .
4

what is a segment?
did you use some tool to split a file in smaller parts?

rclone copies files, and as you wrote, rclone did that without error.

5

i,
When I first transfert te file to OVH (swift storage) in a directoryy (swift:path) my file is well in this directory with a size of 0 Kb, but a new dir is created in swift:path_segments were a found files with name as follow (the rclone help talk about chunk and te name of chunks files)..I understand that rclone split te file in some fragments or segments.

vzdump-qemu-101-2020_12_13-11_19_25.vma.zst/1607860637.847729008/282109612679/00000021
vzdump-qemu-101-2020_12_13-11_19_25.vma.zst/1607860637.847729008/282109612679/00000022
vzdump-qemu-101-2020_12_13-11_19_25.vma.zst/1607860637.847729008/282109612679/00000023
vzdump-qemu-101-2020_12_13-11_19_25.vma.zst/1607860637.847729008/282109612679/00000024

If I launch "rclone lsd swith:path_segments", directory listing return the original name of my file.
When I download back the file to te original server, I found theses multiples files in the destination path.

I think I'm missing something.

Te help of rclone says :slight_smile:

"The chunker overlay transparently splits large files into smaller chunks during upload to wrapped remote and transparently assembles them back when the file is downloaded. This allows to effectively overcome size limits imposed by storage providers."

It's that the "chunker" doesn't assembles te splited files.

Thxs

6

are you using the chunker remote?

you never posted the rclone config file, redact secrets like id/password, so hard to know what is going on?

7

Thx for your responses,
Now, I have understand how to reslove issue.
In my config file I don't have a config for chunker remote.( section with type = chunker and path to directoy were chunks files are stored)
With this new config, now when downloading back a large file that had splited in mutliple small files during the initial transfert to swift storage, these files are well reassembled.
All was in te documentation.
Tx for your responses

8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.