Should I archive files before uploading them to remote repository?

Help and Support
1

What is the problem you are having with rclone?

It is actually a question, sorry if I opened this in wrong category. Should I archive files before uploading them to remote repository?

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.1

  • os/version: ubuntu 24.04 (64 bit)
  • os/kernel: 6.8.0-48-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.23.1
  • go/linking: static
  • go/tags: none

Are you on the latest version of rclone? You can validate by checking the version listed here: Rclone downloads
--> Yes

Which cloud storage system are you using? (eg Google Drive)

Hetzner Storage Box

The command you were trying to run (eg rclone copy /tmp remote:tmp)

This is a question, I was not trying to run a command.

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[storagebox]
type = sftp
host = XXX
user = XXX
port = 23
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r

[storagebox_webdav]
type = webdav
url = XXX
vendor = other
user = XXX
pass = XXX

A log from the command that you were trying to run with the -vv flag

I was not trying to run a command, this is a question.

2

welcome to the forum,

do you mean creating a compressed file such as .zip, .7z and then upload that single file?
if so, it depends on how many individual files and type of remote and internet speed+latency?

for example, i also, use hetzner storagebox.
sftp, can be very slow for lots of small files, but rclone can verify file transfers using checksum.
webdav, can be faster than sftp for lots of small files, but rclone cannot verify file transfers using checksum.

3

Yes, I exactly mean that. The backup has lots of small files. I was planning on using WebDAV though. In my case, which way should I go? Should I use SFTP & archive?

4

really, hard to provide an absolute answer, other than to suggest try a few different ways and see what happens.

for me, i will always choose sftp over webdav. i want file verification for transfers using checksums.

5

And what would you recommend for archiving? Should I archive them using a compression method like gzip, pigz etc., should I just send all of these in a single folder or should I just send all of these in a single folder but this time with the gzip compression rclone offers?

6

that is experimental. i would never use it.

this is an example of real command from my backup script using 7a

7z a "\\\\ts-vserver.falconer.en.keepass\\falconer.en.keepass\\falconer\\en10\\user01\\keepass\\zip\\current\\keepass_20241112.094631.7z"  "b:\\falconer\\vss\\keepass_20241112.094631\\users\\user01\\keepass"       !ZIPPWD -bb3  >  "//ts-credserver01/agent_en10_rw/backups/keepass/logs/20241112.094631/zip.log"'
7

So you are using SFTP with archiving?

8

yes, something like this

  1. create file.7z
  2. rclone copy file.7z storagebox:
9

Got it. I'll try applying the same and let you know, thank you.

10

for testing, use --dry-run and -vv

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.