Jottacloud crypt - 3 GB copy runs for a week without completing

Suspected Bug
1

What is the problem you are having with rclone?

When I'm trying to copy a 3 GB file to my remote it runs for days without completing the copy, the file size just keeps going up beyond the file's total size.

What is your rclone version (output from rclone version)

rclone v1.54.0-beta.4971.e45716cac

  • os/arch: windows/amd64
  • go version: go1.15.6

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Jottacloud (Crypt)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy "C:\Users\=REDACTED=" jottacrypt_test:"=REDACTED=" -P -vvv

The rclone config contents with secrets removed.

[OpenDrive]
type = opendrive
username = =REDACTED=
password = =REDACTED=

[GDrive]
type = drive
token = {"access_token":"=REDACTED=","token_type":"Bearer","refresh_token":"=REDACTED=","expiry":"=REDACTED="}
root_folder_id = =REDACTED=

[Jottacrypt]
type = crypt
remote = Jottacloud:private
filename_encryption = standard
directory_name_encryption = true
password = =REDACTED=
password2 = =REDACTED=

[Jottacloud]
type = jottacloud
user = =REDACTED=
mountpoint = Archive
token = {"access_token":"=REDACTED=","token_type":"bearer","refresh_token":"=REDACTED=","expiry":"=REDACTED="}
tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token
configVersion = 1

[Backblaze]
type = b2
account = =REDACTED=
key = =REDACTED=

[jotta_test]
type = jottacloud
tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token
token = {"access_token":"=REDACTED=","token_type":"bearer","refresh_token":"=REDACTED=","expiry":"=REDACTED="}
configVersion = 1
client_id = jottacli
client_secret = 

[jottacrypt_test]
type = crypt
remote = jotta_test:private
filename_encryption = standard
directory_name_encryption = true
password = =REDACTED=
password2 = =REDACTED=

A log from the command with the -vv flag

2020/12/06 21:29:24 DEBUG : rclone: Version "v1.54.0-beta.4779.c797494d8" starting with parameters ["rclone" "copy" "C:\\Users\\=REDACTED=" "jottacrypt_test:=REDACTED=" "-P" "-vvv"]
2020/12/06 21:29:24 DEBUG : Creating backend with remote "C:\\Users\\=REDACTED="
2020/12/06 21:29:24 DEBUG : Using config file from "C:\\Users\\=REDACTED=\\.config\\rclone\\rclone.conf"
2020/12/06 21:29:24 DEBUG : fs cache: adding new entry for parent of "C:\\Users\\=REDACTED=", "//?/C:/Users/=REDACTED="
2020/12/06 21:29:24 DEBUG : Creating backend with remote "jottacrypt_test:=REDACTED="
2020/12/06 21:29:24 DEBUG : Creating backend with remote "jotta_test:private/=REDACTED='private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-07 02:04:02 DEBUG : =REDACTED=: Received error: Post "https://069-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 1/10
2020-12-07 02:04:02 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-07 02:04:11 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-07 02:04:12 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-07 02:04:12 DEBUG : jotta_test: Saved new token in config file
2020-12-07 02:47:34 DEBUG : =REDACTED=: Received error: Post "https://069-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 2/10
2020-12-07 02:47:34 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-07 03:04:12 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-07 23:29:11 DEBUG : =REDACTED=: Received error: Post "https://070-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 3/10
2020-12-07 23:29:12 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-07 23:29:30 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-07 23:29:30 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-07 23:29:30 DEBUG : jotta_test: Saved new token in config file
2020-12-08 00:29:30 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-08 09:54:08 DEBUG : =REDACTED=: Received error: Post "https://070-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 4/10
2020-12-08 09:54:08 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-08 09:54:16 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-08 09:54:16 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-08 09:54:16 DEBUG : jotta_test: Saved new token in config file
2020-12-08 10:54:16 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-08 17:11:33 DEBUG : =REDACTED=: Received error: Post "https://069-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 5/10
2020-12-08 17:11:34 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-08 17:11:53 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-08 17:11:53 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-08 17:11:53 DEBUG : jotta_test: Saved new token in config file
2020-12-08 18:11:53 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-08 20:46:17 DEBUG : =REDACTED=: Received error: Post "https://070-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 6/10
2020-12-08 20:46:17 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-08 20:46:24 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-08 20:46:25 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-08 20:46:25 DEBUG : jotta_test: Saved new token in config file
2020-12-08 21:46:26 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-09 02:57:07 DEBUG : =REDACTED=: Received error: Post "https://070-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 7/10
2020-12-09 02:57:07 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-09 02:57:16 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-09 02:57:16 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-09 02:57:16 DEBUG : jotta_test: Saved new token in config file
2020-12-09 03:57:16 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-09 08:49:03 DEBUG : =REDACTED=: Received error: Post "https://069-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 8/10
2020-12-09 08:49:03 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-09 08:49:10 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-09 08:49:10 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-09 08:49:10 DEBUG : jotta_test: Saved new token in config file
2020-12-09 09:49:10 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-09 14:45:33 DEBUG : =REDACTED=: Received error: Post "https://069-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 9/10
2020-12-09 14:45:33 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-09 14:45:40 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-09 14:45:40 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-09 14:45:40 DEBUG : jotta_test: Saved new token in config file
2020-12-09 15:45:40 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-09 23:28:23 ERROR : =REDACTED=: Failed to copy: Post "https://070-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host.
2020-12-09 23:28:23 ERROR : Attempt 1/3 failed with 1 errors and: Post "https://070-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host.
2020-12-09 23:28:23 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-09 23:28:23 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-09 23:28:23 DEBUG : jotta_test: Saved new token in config file
2020-12-09 23:28:24 DEBUG : =REDACTED=: Bad size for decrypt: file is too short to be encrypted
2020-12-09 23:28:24 DEBUG : =REDACTED=: Bad size for decrypt: file is too short to be encrypted
2020-12-09 23:28:24 DEBUG : =REDACTED=: Bad size for decrypt: file is too short to be encrypted
2020-12-09 23:28:24 DEBUG : =REDACTED=: Sizes differ (src 3090499593 vs dst 0)
2020-12-09 23:28:24 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-10 00:16:13 DEBUG : =REDACTED=: Received error: Post "https://064-up.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 1/10
2020-12-10 00:16:13 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-10 00:28:23 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-10 10:53:42 DEBUG : =REDACTED=: Received error: Post "https://061-up.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 2/10
2020-12-10 10:53:42 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-10 10:53:50 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-10 10:53:50 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-10 10:53:50 DEBUG : jotta_test: Saved new token in config file
2020-12-10 11:53:50 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-11 09:57:09 DEBUG : =REDACTED=: Received error: Post "https://062-up.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 3/10
2020-12-11 09:57:09 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-11 09:57:17 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-11 09:57:17 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-11 09:57:18 DEBUG : jotta_test: Saved new token in config file
2020-12-11 10:57:17 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-11 11:35:03 DEBUG : =REDACTED=: Received error: Post "https://069-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 4/10
2020-12-11 11:35:03 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-11 11:35:11 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-11 11:35:11 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-11 11:35:11 DEBUG : jotta_test: Saved new token in config file
2020-12-11 12:35:11 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-12 01:22:06 DEBUG : =REDACTED=: Received error: Post "https://069-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 5/10
2020-12-12 01:22:06 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-12 01:22:13 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-12 01:22:13 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-12 01:22:13 DEBUG : jotta_test: Saved new token in config file
2020-12-12 02:22:13 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-12 23:47:57 DEBUG : =REDACTED=: Received error: Post "https://061-up.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 6/10
2020-12-12 23:47:57 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-12 23:48:16 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-12 23:48:16 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-12 23:48:16 DEBUG : jotta_test: Saved new token in config file
2020-12-13 00:19:17 DEBUG : =REDACTED=: Received error: Post "https://070-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 7/10
2020-12-13 00:19:17 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-13 00:48:16 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-13 03:59:55 DEBUG : =REDACTED=: Received error: Post "https://070-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 8/10
2020-12-13 03:59:55 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-13 04:00:05 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-13 04:00:05 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-13 04:00:05 DEBUG : jotta_test: Saved new token in config file
2020-12-13 04:02:03 DEBUG : =REDACTED=: Received error: Post "https://065-up.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 9/10
2020-12-13 04:02:03 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-13 05:00:05 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-13 13:47:32 ERROR : =REDACTED=: Failed to copy: Post "https://065-up.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host.
2020-12-13 13:47:32 ERROR : Attempt 2/3 failed with 1 errors and: Post "https://065-up.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host.
2020-12-13 13:47:32 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-13 13:47:32 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-13 13:47:32 DEBUG : jotta_test: Saved new token in config file
2020-12-13 13:47:32 DEBUG : =REDACTED=: Bad size for decrypt: file is too short to be encrypted
2020-12-13 13:47:32 DEBUG : =REDACTED=: Bad size for decrypt: file is too short to be encrypted
2020-12-13 13:47:32 DEBUG : =REDACTED=: Bad size for decrypt: file is too short to be encrypted
2020-12-13 13:47:32 DEBUG : =REDACTED=: Sizes differ (src 3090499593 vs dst 0)
2020-12-13 13:47:32 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-13 14:47:32 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-14 05:48:46 DEBUG : =REDACTED=: Received error: Post "https://070-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 1/10
2020-12-14 05:48:46 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-14 05:48:54 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-14 05:48:55 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-14 05:48:55 DEBUG : jotta_test: Saved new token in config file
2020-12-14 06:48:55 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
2020-12-14 15:42:11 DEBUG : =REDACTED=: Received error: Post "https://070-up-e.jotta.cloud/files/v1/upload/=REDACTED=": write tcp =REDACTED=->=REDACTED=: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 2/10
2020-12-14 15:42:11 DEBUG : =REDACTED=: Computing MD5 hash of encrypted source
2020-12-14 15:42:21 DEBUG : jotta_test: Loaded invalid token from config file - ignoring
2020-12-14 15:42:21 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-12-14 15:42:21 DEBUG : jotta_test: Saved new token in config file
2020-12-14 16:42:21 DEBUG : jottacloud root 'private/=REDACTED=': Token expired but no uploads in progress - doing nothing
Transferred:        7.804G / 10.216 GBytes, 76%, 11.478 kBytes/s, ETA 2d13h11m38s
Transferred:            0 / 1, 0%
Elapsed time:  198h2m21.3s
Transferring:
 *                =REDACTED=: 16% /2.878G, 8.476k/s, 82h52m8s
2

hi,

it is normal for tokens to expire
when that happens, rclone will save the new token to the config file.

based on this, we can see that tokens have a expiry, tho you decided to redact that.
i would compare that value against the rclone logs entries....

token = {"access_token":"=REDACTED=","token_type":"bearer","refresh_token":"=REDACTED=","expiry":"=REDACTED="}

3

Okay, then the token expiring is probably not why I'm unable to upload anything.
I have edited my post.

4

ok, i see the edit.
3GB is a very small size, tho it depends on your internet connection's upload speed.

as per the log, the file size is 10.216 GBytes?

so you have been running one rclone command for an entire week without stopping it?

5

My internet bandwith is 500/500 Mbp/s, but my upload will only use around 20-40 Kbp/s.
I tried running the same command on the stable release v1.53.3 for a few minutes earlier today, but I wasn't getting any higher upload speed.

As I said in my first post, the upload keeps going even when it's gone past the original file size.
My file is 2,87 GB.

Yes, that's correct. The timestamps in the log are accurate.

6

based on the log, it seems the server keeps closing the connection.
and then rclone tries to re-upload the file over and over again.

something really strange is going on in that log, i have no idea.
but we have many experts that should stop by soon....

if you try to upload that same file to gdrive and opendrive, what happens?

7

When uploading it to google drive it was able to finish in under 5 minutes with an upload speed of 10-15 Mbp/s.

8

How much data do you have total on jotta? On some of their subscriptions they limit the upload speed based on how much data you have – the more you have the slower the upload

9

I have around 59 TB

10

what plan to do you have?

11

I have the unlimited plan

12

how much do you pay for that?

13

$85 if I recall correctly.

14

per month, per year, per lifetime?

https://blog.jottacloud.com/unlimited-storage-for-all-your-devices-45c6963daa02

15

That's per year

16

and when did this slow upload speed to jottacloud start to happen?

and about that 59TB, how was that uploaded, with rclone or.....?

17

I used rclone to upload everything. I started having issues uploading and getting slow speeds around five months ago.

18

five months?

as a paying customer of unlimited storage, when you contacted jottacloud, how did they respond?

If you want to continue using the old subscription, you can still do that for a while longer. Within a year or so we will automatically move customers over to the new subscription.

have you tried to use the jottacloud cli tool, what happened?

19

Last time I spoke to them about issues with upload speed, they refused to help me with that.
I don't think my plan has been changed since it still says I have Jottacloud Personal.
I haven't tried uploading anything with jotta-cli because I don't trust it won't overwrite something else.

20

well,
given this from the logs, An existing connection was forcibly closed by the remote host
if jottacloud will not help a paid customer and you have not tested using their cli, not sure we can say this is a rclone bug.

should be easy to create a new 3GB file and use their cli to upload it
then we will know if this is a rclone bug or not?

i use this for many years, to create a file of random, non-compressible data.
https://www.mynikko.com/dummy/

image