How to check EOF on rclone?

Help and Support
1

What is the problem you are having with rclone?

I want to check EOF file on drive.

What is your rclone version (output from rclone version)

rclone v1.54.0-beta.4942.f7d9b1570

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu

Which cloud storage system are you using? (eg Google Drive)

Google Drive 2 remotes one crypt, one type drive.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync
rclone check

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

Paste  log here

Hello, I have 2 remotes on my rclone config.
One of them is a drive and the other one is a crypt (which is based on the first one), I noticed that when uploading files my third party program was doing it wrong, i.e. it was uploading a file which was not even complete.
An example: A file that weighs 100GB that was uploaded with only 20GB (no problem with rclone it's the third party program)
So I tried to use rclone sync to try to copy a corrupted file (one another drive so another remote) and I get an error that helps me "unexpected EOF".
If I copy all the content to another drive it won't upload any corrupted file and I can tell the difference with the "rclone check" command however I would like to know if there is a way to make it without copy all?
Thanks in advance!

2

Best to use the help template to collect all the needed details to answer you.

3

Edited, i'm sorry!

4

I think you are asking if rclone can skip files that are open? Is that correct?

5

Hello, no, my program that used rclone uploaded already opened files (in the process of writing and that's why they are corrupted) I would like to know if there is a way to detect all corrupted files (EOF) without having to copy or sync to another remote.

6

hi,
perhaps you can use this?
rclone check

7

You can use rclone check?

8

@asdffdsa used rclone check but I have to copy/synchronize everything on another remote to see if there are corrupted files (unless I don't know how to use the command?)
Note that the names of the files and folders are encrypted, here is my configuration:

Name                 Type
====                 ====
SDrive               drive
SDrive_crypt         crypt

There is also the cryptcheck command but I don't see how to do it without copying/syncing everything.

9

Nobody have an solution? Thanks!

10

The solution is to use rclone cryptcheck as noted above or ensure your file system is set to copy from.

11

If use this command:
rclone cryptcheck SDrive: SDrive_crypt:
I have a lot of error with the same message File not in Encrypted drive 'SDrive_crypt:'

12

Seems like you are missing files on the destination then as that is why that errors and you'd want to recopy them.

13

Hmm, it's the same content because:

[SDrive]
type = drive
scope = drive
team_drive = id
[SDrive_crypt]
type = crypt
remote = SDrive:
filename_encryption = standard
directory_name_encryption = true
14

You need to share a log file if you want further asstiance.

15

An example of lines for rclone cryptcheck:

16

I can't figure out what you are trying to do.

You have SDrive as a Google remote and you have SDrive as the root of an encrypted drive meaning you have both unencrypted and crypted content in the same thing?

The content is either encrypted or not. If you didn't upload to the encrypted remote, it would be decrypted content.

Most folks setup a folder to be encrypted and you can move to that folder.

As what you have now, it won't work since content is all mixed up I'd imagine.

felix@gemini:~$ rclone mkdir GD:test
felix@gemini:~$ rclone copy /etc/hosts GD:test
felix@gemini:~$ rclone mkdir gcrypt:test
felix@gemini:~$ rclone copy GD:test gcrypt:test
felix@gemini:~$ rclone cryptcheck -vv GD:test gcrypt:test
2021/03/18 10:01:14 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "cryptcheck" "-vv" "GD:test" "gcrypt:test"]
2021/03/18 10:01:14 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/03/18 10:01:14 DEBUG : Creating backend with remote "GD:test"
2021/03/18 10:01:14 DEBUG : Creating backend with remote "gcrypt:test"
2021/03/18 10:01:14 DEBUG : Creating backend with remote "GD:crypt/191qpocr6dhlo35vn2gk2h8gf4"
2021/03/18 10:01:15 INFO  : Using MD5 for hash comparisons
2021/03/18 10:01:15 DEBUG : Encrypted drive 'gcrypt:test': Waiting for checks to finish
2021/03/18 10:01:16 DEBUG : hosts: OK
2021/03/18 10:01:16 NOTICE: Encrypted drive 'gcrypt:test': 0 differences found
2021/03/18 10:01:16 NOTICE: Encrypted drive 'gcrypt:test': 1 matching files
2021/03/18 10:01:16 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         2.0s

2021/03/18 10:01:16 DEBUG : 7 go routines active

GD is my root remote gcrypt is my encrypted remote which is a folder in GD. I compare GD:test to gcrypt:test and they match up validating md5sums.

17

Indeed I think that we do not have the same configuration.
I use cloudplow (which you know from your guide :wink: ) which uses the rclone move command, and I select SDrive_crypt:/ for my local folder on my server (which is not yet encrypted) and so what I am going to upload will be directly encrypted.

Seeing your example you have two copies on your gdrive, one that is encrypted another not, for my part everything is encrypted and so I have only the encrypted content, is it therefore impossible for me to check for corrupted files?

Thanks!

18

Is it correct understanding that you have moved files from local folder into SDrive_crypt:? And then you want to perform a check of the files (only) stored at SDrive_crypt:?

19

You want to use a crypt remote as a folder unless you plan to encrypt your whole remote as things are going to break if you don't.

Even if that's the case, OP is mixing crypted items with non encrypted items based on his rclone.conf

I only use my encrypted remote as I do not have two copies on my Google Drive.

You need to isolate your encrypted content from your non encrypted content.

20

I agree.

But is this known to be the case here? Only if actually using the drive storage directly, via rclone remote SDrive: or otherwise, and not just through rclone remote SDrive_crypt:?

My point was: To perform check (or cryptcheck) you need two copies to compare. So if OP is moving files into SDrive_crypt:, and only that, there is nothing to compare with. OP wants to detect if any of the moved files are corrupted at destination. Will rclone check SDrive_crypt: SDrive_crypt: do that? Or am I totally off track here..? :thinking: