Pipeline with large&cryped files on dropbox

Help and Support
1

What is the problem you are having with rclone?

I would like a second opinion on how I made my pipeline and if this is the correct way of avoiding the --size-only fallback. I want my data to be checked for consistency on a reliable way and this was the only way I could think of.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1

  • os/version: debian 12.1 (64 bit)
  • os/kernel: 6.1.0-10-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.6
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox Business

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --checksum

The rclone config contents with secrets removed.

crypt -> hasher (md5,dropbox) -> chunker (240G) -> dropbox

A log from the command that you were trying to run with the -vv flag

n.a.

Would you do it differently? If yes, how and why?

2

IMO your way is over engineered.

I would do:

chunker->crypt->dropbox

where chunker setup:

[chunker]
type = chunker
hash_type = sha1all

enforces storing sha1 hashes for all files.

3

Would that still do the hash check on the storage system? As dropbox does only support their own hash that chunker doesn't?

EDIT:
As I am using chunker to make same size chunks I am afraid that a remote file would pass as equal when it in fact is another file with the same size (as dropbox does not support modtime too)

4

Not sure I understand your question. Dropbox does do support hashes (its own though - non standard).

Why you would like to check hashes between chunker and dropbox?

Or give some example what is your concern... what exactly you want to achieve?

5

You ideally want to check the consistency from source to destination file. From how I understand your suggested configuration would only check the source file against chunker, but would not check the dropbox checksum?

6

You have only one destination in given rclone command. so depends where is your destination (chunker or dropbox) you only check against this destination.

Can you show command you think will fail when you run it? because of not checking right checksums.?

7

I do a rclone sync --checksum localfile crypt:folder

But in a chained setup like this one, does the last step (dropbox) still check the checksums and replace the file when they differ? If yes: consider the question answered by your first response.

Thanks for your time!

8

crypt does not support hashes hence my suggestion to change remotes order to:

chunker->crypt->dropbox

and make sure that chunker enforces hashes for all files:

[chunker]
type = chunker
...
hash_type = sha1all

This way when you run:

rclone sync --checksum localfile chunker:folder

your localfile calculated SHA1 hash can be compared to chunker stored SHA1

and yes dropbox remote will use its own dropbox hashes still

9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.