Copying from GD to Jottacloud, running out of local space

Help and Support
1

Hi,

I noticed that rclone 1.43 now supports Jottacloud, so I signed up for a month to see how it fares against Google Drive. I’m trying to copy my library from GD to Jottacloud using a very simple copy command:

rclone copy gd: jotta:

Now, what I am seeing is that my root partition is filling up, presumably because rclone needs to temporarily download the files from GD before uploading to Jottacloud.

Obviously, my root partition running out of space causes all sorts of issues, including failures in the rclone transfer itself (something about md5 calculations failing because of no free space).

Is there any easy solution to this? Can I change where rclone stores temporary files?

2

From the documentation for the Jottacloud backend:

Note that Jottacloud requires the MD5 hash before upload so if the source does not have an MD5 checksum then the file will be cached temporarily on disk (wherever the TMPDIR environment variable points to) before it is uploaded. Small files will be cached in memory - see the --jottacloud-md5-memory-limit flag.

You can change the TMPDIR variable to a directory you have more space to avoid your root running full.
You can also reduce the number of parallel transfers.

3

My apologies, I must’ve missed that in the documentation! Thanks for the response.

4

Hmm, Drive should have MD5SUMs for each file though so I wouldn’t have thought it should buffer things temporarily

5

Nick is correct, if the source can provide MD5 hashes no local caching should be necessary.
I‘ll double check when I find the time

6

I did a quick test, copying a single file from GD to Jottacloud, and from what I can see you are correct - Jottacloud backend does find md5 from source and skips the function that temporarily downloads the entire file to disk to calculate md5.

1 Like
7

My apologies, I should’ve mentioned that my Google Drive (gd:) remote is actually a crypt remote since my files in GD are encrypted, while my Jottacloud remote (jotta:) is a normal unencrypted remote.

I assume rclone can’t get the md5 hash from a crypt remote? :slight_smile:

8

Yes that will be the problem.

9

I see. Any plans to be able to get the hash from a crypt remote, or is that not feasible?

10

I don’t think the md5 is stored anywhere so rclone would have to download, decrypt and compute the sum.

The sum might be computed on the fly so in theory you could spare the local space by downloading and computing the sum on the fly then download again and upload at the same time (all the operations on small chunks). Not sure if it’s worth it, be it for 1GB or for 50GB … in most situations if you can afford to transfer that much without trouble multiple times you can spare some temporary space (anyway faster and less wasteful than transferring from “cloud”) as well.

11

Yes, you nailed it.

There have been some ideas about storing md5sums as metadata on crypt but they haven’t born fruit yet.

12

Sorry to bring up a maybe old topic :slight_smile:
I’m experiencing the same issue while copying a local path to Jottacloud, i.e. something like:

rclone copy /home/my/important/data jotta:my/important/backup

Is there no shortcut or hack somehow that could calculate the MD5 from the source location directly instead of copying / “downloading” it to TMP before doing it?

Best,

Nils

13

It needs to calculate it somewhere. So not really. Closing this thread.

15

Only if the source has the MD5. This should include the local disk though. Are you using the latest rclone - see the latest release? If that doesn’t work then please start a new topic - thanks.

1 Like