Is it possible to move files that are missing locally to a backup folder on the remote server?

MrDemocracy · September 6, 2019, 1:53pm

What is the problem you are having with rclone?

I have a lot of deleted files that are still in the same folder on the remote server, as the folder I deleted them from locally.
Is it possible to move all the files that are missing locally, to a backup folder on the remote server, while still keeping the folder structure?

What is your rclone version (output from `rclone version`)

rclone v1.48.0-191-g712f7e38-beta

os/arch: windows/amd64
go version: go1.12.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Jottacloud

thestigma · September 6, 2019, 2:00pm

Maybe I am misunderstanding what you asking for here, but wouldn't a simple server-side move (or copy) do what you want? I don't quite understand why it would be relevant if you have have the files locally anymore or not. I assume there is something more to this I am not understanding, or else the problem seems pretty trivial.

You could backup manually, or depending on your needs you could look into the --backup-folder parameter if you want automatic backsups of 1 previous revision of all files. Some cloud-systems also feature file revisions natively - but I don't have the experience with Jotta to know if that is a thing there.

MrDemocracy · September 6, 2019, 2:23pm

I often delete files locally to save space, but I don't delete the same files on the remote server. Because of this my cloud backup is much larger than the capacity of my HDD. I want to have a folder on the remote server that contains the current state of my HDD, and a "deleted files backup"-folder that contains only the files that are missing on my local HDD.

ncw · September 6, 2019, 3:40pm

What you want to do is do an rclone sync but include the --backup-dir parameter. Rclone will put any deleted files in there.

MrDemocracy · September 6, 2019, 4:00pm

I have tried using --backup-dir. Here's my command:

rclone sync D:\ Jottacrypt:"/Jottacloud Backup/D" --backup-dir Jottacrypt:"/Jottacloud Backup/Backup Archive/" -P --ignore-existing --exclude-from C:\Commands\rclone\files-from.txt -L

Nothing is moved to /Jottacloud Backup/Backup Archive/. The folder is empty.

thestigma · September 6, 2019, 4:24pm

That's weird - they should be moved there whenever deleted or replaced with an newer file.

Are you sure it's not something in your filter-file that is misconfigured?
rclone ls Jottacrypt:"/Jottacloud Backup/Backup Archive/" -P --ignore-existing --exclude-from C:\Commands\rclone\files-from.txt -L
What is the result of this command? Does it accurately cover the locations you are interested in syncing to?

MrDemocracy · September 6, 2019, 4:28pm

files-from.txt

$RECYCLE.BIN/**
System Volume Information/**

Yes, it's syncing everything like it's supposed to. Except for some files with too long names.

thestigma · September 6, 2019, 4:59pm

The next step is probably to check what the debug log says with parameters:
-vv (enable debug output)
--log-file=mylogfile.txt (output log to file, as it gets rather long quickly)

That should detail every operation rclone does in exact detail and clue us in to what is happening.

As the test: delete one file on the remote, so we can see how rclone handles the backup step.

MrDemocracy · September 6, 2019, 5:16pm

Sorry, didn't see that part.

Here's the output from that (without -P as it didn't show any useful information):

rclone ls Jottacrypt:"/Jottacloud Backup/Backup Archive/" -L --ignore-existing --exclude-from C:\Commands\rclone\files-from.txt -vv
2019/09/06 19:10:29 DEBUG : rclone: Version "v1.48.0-191-g712f7e38-beta" starting with parameters ["rclone" "ls" "Jottacrypt:/Jottacloud Backup/Backup Archive/" "-L" "--ignore-existing" "--exclude-from" "C:\\Commands\\rclone\\files-from.txt" "-vv"]
2019/09/06 19:10:29 DEBUG : Using config file from "C:\\Users\\(username)\\.config\\rclone\\rclone.conf"
2019/09/06 19:10:29 DEBUG : 7 go routines active
2019/09/06 19:10:29 DEBUG : rclone: Version "v1.48.0-191-g712f7e38-beta" finishing with parameters ["rclone" "ls" "Jottacrypt:/Jottacloud Backup/Backup Archive/" "-L" "--ignore-existing" "--exclude-from" "C:\\Commands\\rclone\\files-from.txt" "-vv"]

thestigma · September 6, 2019, 5:27pm

Was your backup location completely empty? Sorry, I should have specified that we probably need at least 1 file to exist there. The whole point of the exercise would be that we double-checked that rclone can see that location properly. Assuming there was at least 1 file present and it reported nothing like this, then something would be wrong that we could investigate further.

But if it was empty I don't think we can read much from it. Feel free to check again after copying 1 file over.

MrDemocracy · September 6, 2019, 5:34pm

rclone ls Jottacrypt:"/Jottacloud Backup/Backup Archive/" -L --ignore-existing --exclude-from C:\Commands\rclone\files-from.txt -vv
2019/09/06 19:29:52 DEBUG : rclone: Version "v1.48.0-191-g712f7e38-beta" starting with parameters ["rclone" "ls" "Jottacrypt:/Jottacloud Backup/Backup Archive/" "-L" "--ignore-existing" "--exclude-from" "C:\\Commands\\rclone\\files-from.txt" "-vv"]
2019/09/06 19:29:52 DEBUG : Using config file from "C:\\Users\\(username)\\.config\\rclone\\rclone.conf"
        0 test.txt
2019/09/06 19:29:53 DEBUG : 5 go routines active
2019/09/06 19:29:53 DEBUG : rclone: Version "v1.48.0-191-g712f7e38-beta" finishing with parameters ["rclone" "ls" "Jottacrypt:/Jottacloud Backup/Backup Archive/" "-L" "--ignore-existing" "--exclude-from" "C:\\Commands\\rclone\\files-from.txt" "-vv"]

It's not that folder that is being synced to though. That's the folder I want to use as "recycle bin" or archive on the remote server.
My HDD is synced to /Jottacloud Backup/D/.

thestigma · September 6, 2019, 5:56pm

I understand you, but at least know we are sure rclone would have no problem finding the "recycle bin"" folder.

I would proceed with the -vv --log-file logname.txt --> delete one file test I described a few posts up. That should tell us a lot of what is happening under the hood when the "move to backup" step happens. If it ignores it, if it errors ect.

MrDemocracy · September 7, 2019, 3:15am

Here's the log file output. I stripped most of the lines because they were just Destination exists, skipping. There are many files that has been deleted locally that are still on the remote, but there aren't any "move to backup" messages in the logs.

2019/09/06 19:19:12 DEBUG : rclone: Version "v1.48.0-191-g712f7e38-beta" starting with parameters ["rclone" "sync" "D:\\" "Jottacrypt:/Jottacloud Backup/D" "--backup-dir" "Jottacrypt:/Jottacloud Backup/Backup Archive/" "-P" "--ignore-existing" "--exclude-from" "C:\\Commands\\rclone\\files-from.txt" "-L" "-vv" "--log-file=mylogfile.txt"]
2019/09/06 19:19:12 DEBUG : Using config file from "C:\\Users\\(username)\\.config\\rclone\\rclone.conf"
2019/09/06 19:19:13 DEBUG : $RECYCLE.BIN: Excluded
2019/09/06 19:19:13 DEBUG : System Volume Information: Excluded
2019/09/06 19:19:13 DEBUG : =HIDDEN= : Excluded
2019/09/06 19:19:13 DEBUG : =HIDDEN= : Excluded
2019/09/06 19:22:07 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:22:07 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:22:08 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:23:17 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:27:40 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:27:41 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:27:41 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:28:05 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:28:05 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:28:17 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:28:17 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:28:17 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:28:17 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:28:28 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:28:28 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:28:40 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:28:41 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:29:04 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:29:04 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:29:04 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:29:05 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:29:30 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:29:31 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:32:45 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:33:42 DEBUG : Jottacloud: Loaded invalid token from config file - ignoring
2019/09/06 19:33:42 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2019/09/06 19:33:42 DEBUG : Jottacloud: Saved new token in config file
2019/09/06 19:33:52 DEBUG : =HIDDEN= : Token expired but no uploads in progress - doing nothing
2019/09/06 19:33:52 DEBUG : =HIDDEN= : Token expired but no uploads in progress - doing nothing
2019/09/06 19:33:52 DEBUG : =HIDDEN= : Token expired but no uploads in progress - doing nothing
2019/09/06 19:34:09 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:35:29 INFO  : =HIDDEN= : Copied (new)
2019/09/06 19:37:39 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:37:39 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:39:03 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:39:26 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:39:53 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:39:55 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:44:30 DEBUG : pacer: low level retry 1/10 (error read tcp 192.168.1.100:64046->185.179.130.27:443: wsarecv: An existing connection was forcibly closed by the remote host.)
2019/09/06 19:44:30 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2019/09/06 19:44:30 DEBUG : pacer: Reducing sleep to 15ms
2019/09/06 19:44:30 DEBUG : pacer: Reducing sleep to 11.25ms
2019/09/06 19:44:30 DEBUG : pacer: Reducing sleep to 10ms
2019/09/06 19:45:42 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:45:43 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:45:44 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 19:46:17 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 20:03:48 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/06 20:30:34 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 20:30:45 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 20:30:49 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 20:30:50 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 20:30:56 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/06 20:33:32 DEBUG : Jottacloud: Loaded invalid token from config file - ignoring
2019/09/06 20:33:33 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2019/09/06 20:33:33 DEBUG : Jottacloud: Saved new token in config file
2019/09/06 21:17:32 INFO  : =HIDDEN= : Copied (new)

thestigma · September 7, 2019, 10:35am

Oh, well this looks fairly straightforward actually.

As you see from those copy errors (presumably from the files that were supposed to be copied to archive when deleted or overwritten) it can not perform the operation because the resulting path name would be too long for the underlying filesystem to handle. I would assume different cloud services have differing limits of path names - but in this case it's 255Bytes/255characters. That makes sense as your "backup archive" folder is significantly longer than your main backup directory (although both of them could make due with shortening in order to prevent the same from happening later on even longer filenames)

Suggested solution:

Shorten your folder names as much as possible without them being too cryptic for you to remember
"arc" and "bck" for example would probably be sufficient as an aggressive naming example?
Flatten your folder structure where possible rather than having them all nested inside each-other. For example in this case you might want to have the backup archive sit next to the regular backup directory rather than inside it as that adds extra path-length. How much you actually save on that obviously depends on how if you significantly shortened their names or not though.
The same goes for the contents inside the actual backup-folder too of course. I don't know how that looks, but if you have any obvious unnecessarily long names or nested folder structures there then consider shortening/flattening them too as it makes sense to you.

Doing all of the above is probably not needed in practice, but the more free characters you have at your disposal in this path-tree the less chance you ever have to encounter this problem later.

Suggestion:
I think there is room for better handling of this problem in general. Perhaps some options to stop and alert user more explicitly if detected - or an option to smartly auto-truncate paths that go too long. The idea would be to make sure that this type of error doesn't occur without the user knowing and coming back to bite you later because you didn't have files you expected to have...
So it might be an idea to

See if there is an existing issue on github to improve this and upvote it to brint it attention
Make your own issue describing the problem and linking this thread is none currently exists

If this doesn't turn out to be the problem after all then say so and we will try digging deeper, but this looks like a familiar problem.

MrDemocracy · September 7, 2019, 10:52am

I forgot to mention that I'm using rclone crypt for my remote. I think that's why I get that error message. The "255 characters" error only shows files that has not been backed up yet.

thestigma · September 7, 2019, 2:23pm

Crypt does tend to use a bit more of the character limit, but it's the same basic problem usually solved by shorter folder names and flattening the directory tree. I don't know the exact math behind it but shorter names of files and folder should encrypt into shorter hashes. At least my encrypted files have very much varying lengths.

I can't see the exact details of that error. We would also need "--dump headers" to look inside it's contents. If you want to do another run with that enabled and just make sure that you do at least one deletion or sync at least one file that is modified/newer and thus will be overwriting something already existing.

It's not unlikely that those same errors can be occurring on both a few of your "sync to Jotta" and also some of the "move to archive" operations. Headers should reveal the details of this. If this is the case then the solution should remain the same - but it would be nice to confirm it solidly rather than guess-and-try

MrDemocracy · September 12, 2019, 2:38pm

The log file ended up being almost 3GB. I had to install atom to even be able open the file. Is there a specific part of the log file you want me to paste here? I don't know how easy or necessary it would be to share the entire log.

Here's some of the log near the beginning:

2019/09/07 16:30:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/09/07 16:30:57 DEBUG : HTTP REQUEST (req 0xc0001ada00)
2019/09/07 16:30:57 DEBUG : GET /jfs/=HIDDEN=/Jotta/Archive/private/=HIDDEN= HTTP/1.1
Host: www.jottacloud.com
User-Agent: rclone/v1.48.0-191-g712f7e38-beta
Authorization: XXXX
Accept-Encoding: gzip

2019/09/07 16:30:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/09/07 16:30:57 DEBUG : =HIDDEN= : Destination exists, skipping
2019/09/07 16:30:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/09/07 16:30:57 DEBUG : HTTP REQUEST (req 0xc0000b9a00)
2019/09/07 16:30:57 DEBUG : GET /jfs/=HIDDEN=/Jotta/Archive/private/=HIDDEN= HTTP/1.1
Host: www.jottacloud.com
User-Agent: rclone/v1.48.0-191-g712f7e38-beta
Authorization: XXXX
Accept-Encoding: gzip

And here's some of it further down:

2019/09/07 16:33:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/09/07 16:33:51 ERROR : =HIDDEN= : Failed to copy: HTTP error 400 (400 Bad Request) returned body: "{\"code\":400,\"message\":\"JFS file names may not be longer than 255 characters\",\"cause\":\"\",\"error_id\":\"InvalidArgumentException\",\"x-id\":\"b2-=HIDDEN=\"}"
2019/09/07 16:33:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/09/07 16:33:51 DEBUG : HTTP REQUEST (req 0xc023d72a00)
2019/09/07 16:33:51 DEBUG : GET /jfs/=HIDDEN=/Jotta/Archive/private/=HIDDEN= HTTP/1.1
Host: www.jottacloud.com
User-Agent: rclone/v1.48.0-191-g712f7e38-beta
Authorization: XXXX
Accept-Encoding: gzip

2019/09/07 16:33:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/09/07 16:33:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/09/07 16:33:51 DEBUG : HTTP RESPONSE (req 0xc023f3e000)
2019/09/07 16:33:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/09/07 16:33:51 DEBUG : HTTP RESPONSE (req 0xc0229d7a00)
2019/09/07 16:33:51 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Accept-Ranges: bytes
Content-Type: text/xml
Date: Sat, 07 Sep 2019 14:33:53 GMT
Strict-Transport-Security: max-age= 31536000
X-Id: 121187728420
X-Resp: 55

2019/09/07 16:33:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/09/07 16:33:51 DEBUG : HTTP/1.1 400 Bad Request
Transfer-Encoding: chunked
Access-Control-Allow-Headers: Accept, Accept-Encoding, Authorization, Content-Type, Content-Encoding, User-Agent, X-ID, X-Forwarded-For, Host, Origin
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Sat, 07 Sep 2019 14:33:53 GMT
Date: Sat, 07 Sep 2019 14:33:53 GMT
Strict-Transport-Security: max-age= 31536000
X-Id: b2-=HIDDEN=

thestigma · September 12, 2019, 6:26pm

The only part we need to look at is specifically where a file is being deleted or overwritten. That's why it's usually best to just test test and log that one spesific thing and then exit - rather than doing a full sync of a gazillion files. if it's hidden somewhere inside a 3GB long log of otherwise irrelevant lines that could be tricky to sort through. Even better if you note the spesific filename you are testing with as that makes it real easy to look up in the log.

it may be plausible to zip down the logfile (as text compresses extremely well) and share the file that way, but I'd much prefer a more focused log. I'm no wizard when it comes to interpreting these. I don't think the snippets you provide capture what we wanted to. I do note that you continue to see a lot of 400 bad requests due to too long filepaths. I'd highly recommend you follow the tips outlined earlier to correct that. It may not be at the root of this particular problem but I certainly would not discount it either - and either way it's something you definitely want to fix.

I question if you can say that with confidence... how are you determining this exactly?

MrDemocracy · September 12, 2019, 8:30pm

Yes, I'm 100% sure. I know this because I checked if the files in the log with too long names, were available on the remote and they weren't.

But I tried another thing just now. I made a folder on my local D:\ drive called "test" with a file called "test.txt" inside it. Then I used the same sync command I used above, but with D:\test\ Jottacrypt:"/Jottacloud Backup/D/test" instead. After the file was backed up to the remote I deleted the file locally and ran the sync command again. This time the "test.txt" file was successfully moved to Jottacrypt:/Jottacloud Backup/Backup Archive/ when it discovered it being missing locally. And after checking the /Backup Archive folder I'm kind of happy the other command didn't work, because test.txt didn't keep it's folder structure /test/test.txt, it was just moved to the root of /Jottacloud Backup/Backup Archive/.

I still don't know why the other command doesn't work, but I want to make sure the folder structure is kept when the files are moved, before I change something that makes the command work. Is keeping the file path for the --backup-dir possible to do already, or is this a feature that would have to be added?

thestigma · September 12, 2019, 8:51pm

Well this is the thing - the feature seems to work when you test it in a more controlled circumstance, so I don't suspect it's bugged. Some other factor is at play here - and that's what we need to find. When you don't know the exact point of failure the best thing to do is usually to eliminate possibilities and test basic scenarios first - then expand complexity from there until you start hitting problems. That point of failure should give you a lot of good clues. The logs will also help a lot.

If you can either get the full logs to us or (more ideally) manage to replicate the error in a smaller and more controlled test that doesn't involve thousands of file-operations then that will surely help. If I can't pinpoint the problem, then @NCW or @Animosity022 probably can as they are more proficient at parsing logs them me.

Regarding the backup dir not keeping the hierarchy, I think that is user-error. It should keep the hierarchy.

When using sync , copy or move any files which would have been overwritten or deleted are moved in their original hierarchy into this directory.

I think the problem you had was simply that you synced D:\test\ , and that means the sync operation is only working on the files in that folder (and not the folder structure above that point, indluding the test folder). This is just how rclone's syntax works. The sync operation won't know anything about the file structure above the point at which you start it at, so in this case it only sees the files and doesn't know about the D:\test\ part. Had you synced D:\ then it would have preserved the test folder hierachy. When it moved the file to the backup dir it didn't make any folders there because there were no folders inside the sync operation you defined, only 1 file.

Does that make sense for you? If not, then repeat the test once more, but this time try using folder D:\folder1\folder2\test.txt
if you then sync D:\folder1\ then you should see that folder2\test.txt should appear in your backup directory after you delete locally and run the second sync. This hopefully illustrates how this works so that there is no confusion

TLDR: The hierarchy that is preserved in backup dir will be relative to what point in the hierarchy the sync operation works on

Is it possible to move files that are missing locally to a backup folder on the remote server?

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Which cloud storage system are you using? (eg Google Drive)

What is your rclone version (output from `rclone version`)