Rclone, Sync, eCryptFS

Help and Support
1

Hello
I setup a new Ubuntu based NAS at home, to replace the old one… so new HW (Dell T140 versus HP Microserver) and new WD Red HDD.
My local filesystem is partially encrypted with eCryptFS: let me say… movies are not encrypted, but personal data are.
Anyway, everything is synced to GDrive with rclone and with its encryption.
So, to be clear:

  • movies are locally not ecrypted, but are encrypted on GDrive with Rclone
  • personal data are locally encrypted with eCryptFS and again encrypted on GDrive with Rclone.

I copied all the data from the old NAS to the new NAS.
In the new NAS I setup Rclone and performed a dry-run with RClone to what is going to happen and…
SURPRISE
I see thousands of messages saying “not copying as dry run”. What ???
The new source is the copy of the old one!!!

So… at the end of the story the files that RClone is going to sync on the new NAS are all inside the eCrypt folder and… checking 3 times to be sure inside the old NAS, inside the new NAS and inside GDrive… yes… the Rclone in the old NAS is not going to copy these files, but the Rclone in the new NAS is ready to copy them

Example of a file that the new NAS is going to copy and of course does not exist on Gdrive and is skipped with no warning by the old NAS
Crypt_MISC/ECRYPTFS_FNEK_ENCRYPTED.FWaX7o9q77vxzUYvJgfXcV7Py0vaeZfgOEiB1N6CqXK9EkeASxVhaXN7gU–/ECRYPTFS_FNEK_ENCRYPTED.FWaX7o9q77vxzUYvJgfXcV7Py0vaeZfgOEiB3lZ-2y9IzYsDbhAeKEBc8—/EN88TE~2

Rclone version in the old NAS is 1.41 and in the new is 1.46

2

Can you run a single copy of source to destination and use the -vv and share the output?

That will tell you why something is not being copied.

You can use dry-run along with that as well.

My example:

[felix@gemini ~]$ rclone copy /etc/hosts GD:
[felix@gemini ~]$ rclone copy /etc/hosts GD: -vv
2019/04/07 11:10:31 DEBUG : rclone: Version "v1.46" starting with parameters ["rclone" "copy" "/etc/hosts" "GD:" "-vv"]
2019/04/07 11:10:31 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2019/04/07 11:10:31 DEBUG : hosts: Size and modification time the same (differ by -56.293µs, within tolerance 1ms)
2019/04/07 11:10:31 DEBUG : hosts: Unchanged skipping
2019/04/07 11:10:31 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 1 / 1, 100%
Transferred:            0 / 0, -
Elapsed time:       400ms

2019/04/07 11:10:31 DEBUG : 4 go routines active
2019/04/07 11:10:31 DEBUG : rclone: Version "v1.46" finishing with parameters ["rclone" "copy" "/etc/hosts" "GD:" "-vv"]

1.41 is pretty old so you should upgrade that as there isn’t much point to trying to debug 1.41.

3

The log file produced is HUGE. I will paste just some relevant rows here
https://paste.kodi.tv/qijeviteke.kodi

here you can see some examples of files that Rclone 1.41 does not copy or does not delete.

I know that 1.41 is old, I do not pretend debug, but just trying to understand if we had or have problems with such kind of files, produced typically by eCryptFS

Regards.

4

Yeah, but if you want to work on it, upgrade to the latest and rerun.

5

maybe I am not clear:
the log I uploaded (can you see it?) is produced by RClone 1.46 running on the NEW nas.
It detects file\folders to be synced: files and folders to be copied to the GDrive and files and folders to be deleted from the GDrive

Running Rclone 1.41 on the OLD NAS, with the very same data, the result is OK… nothing to be added to the remote or to be deleted from.

The question is (maybe Nick has the answer): does the old version 1.41 of Rclone have some problems with that type of files, tipically prodiced by eCryptFS ?

it is just to be sure… these are my data!!! :slight_smile:

6

rclone wouldn’t know anything about eCrypFS as files are just files so it really doesn’t care.

I’m really not following what the actual question on the sync is though.

The logs show why a file may or may not be copied with the -vv output so if the size/mod time are in the same realm, it doesn’t copy.

Is there a particular line in the log that isn’t clear?

7

Have you checked a subset of the files yourself with the remote directly from the path that rclone would read from? And then cross compared to the nas size and times? The log says the files differ by size and modification time.

Real files are really stored in /home/.ecryptfs/$USER if you’ve encrypted the home. Might want to look they’re also. Perhaps it’s the presentation of the files (links to them) that’s confusing rclone.

Edit: rereading I’m confused also by your question. The log says they are the same and won’t be copied. Isn’t that what you’d expect? I’m not seeing anything wrong. I think those dryrun messages are misleading rclone already said they are the same but I personally think it’s confusing. It copies files and it’s just telling you it isn’t going to do anything since you specified dryrun.

To make this clear to yourself, create a copy of your data for one directory giving you problem messages and actually run a sync on the original. You’ve got a backup copy.

8

Again, I am sorry, but maybe I am not so clear.
Look please at the log I linked.
Row number 10 (this is an example… we have thousands of this…). The log produced by Rclone 1.46 on the NEW NAS says that the file “EN88TE~2” needs to be copied to the remote, but it is not because I am running a dry-run.
It means that the file exists in the source, but does not in the remote.
OK?

Then I run the sync with the OLD NAS (the disks are the same), and in the full verbose log there is NO evidence of this file… nothing at all… even if the file exists in the local filesystem to be synced!!!

Look at row number 16: RClone is going to delete something from the remote because it does not exist in the source.
Looking at the full verbose log of RClone 1.41 in the old nas, the entry for this file is not present, and indeed the file is not in the local filesytem, so… why RClone was not deleting it from the remote and moreover producing an entry in the log file?

I am fully aware that Rclone is looking at a local filesystem and does not care about eCryptFS… but these misalignments (thousands!) are just in the eCryptFS filesystem and (maybe) OK… RClone does not care about eCryptFS but the eCryptFS naming convention is not nice to RClone!
Expecially file or folders with “~” inside…

9

The log you provided is only a tiny portion. Grep that file for the file names of those files on lines 10 and 16. Is there other entries about those files?

Also did you look at those files and their date and times on the local file systems and see if they match?

After thought… Are the length of those filenames larger than what crypt allows?

10

I already looked deep in the log file and no…
Row 10 refers to “EN88TE~2” that is a 200 MB local file that RClone 1.46 is ready to copy to the remote, while RClone 1.41 never mentions this file… it is not tracked in the 1.41 log file… nothing at all.

If the file esists in the local filesystem, but not in the remote… I can not cross-check date and time… simply it needs to be copied.

I also checked the filename lengh limit and… no… it is not my problem, because when I copy a file to the encrypted local folder, if I exceed the limit, it warns me and, at the end of the story… I never had any problem with eCryptFS on local filesystem, so it works!
When I decided to encrypt with eCryptFS 2 years ago, I created the encrypted folder, then I used rsync to copy there everything deleting from source. I had many files not compatible with eCryptFS because of file length limit and these have been skipped, so still in the source folder, then I manually adjusted the name and rsynced again.
For new files I create in the encrypted folder… if there is something wrong… it warns and stop me.

To be honest there is a difference: the old NAS with Rclone 1.41 is working in an ext3 filesystem while Rclone 1.46 in the new NAS is with ext4

Anyway, I can confirm: I have these erros ONLY in the encrypted folders. The other folders are OK.

11

Ext4 has a milisecond precision over ext3 but that doesn’t look like your problem.

12

Just for grins. I downloaded 1.41 and installed ecryptfs. Stored a bunch of stuff in it. Then synced the stored data encrypted to google drive crypt like you did with 1.41. I then tried the sync again with 1.46 and no files were flagged for transfer.

You really need to take a small sample and figure out whats different between the files its trying to transfer. I can’t recreate it with the given info.

13

many thanks for your effort.

Since you installed ecryptfs… do you have file in your encrypted folders like “EN88TE~2” ?
Usually ecryptfs naming conventions uses something like ECRYPTFS_FNEK_ENCRYPTED…

14

I’m not sure specifically about the tilde but I deleted the folder. I’ll create a bunch of test data and retry and verify I have some with the tilde.

15

all of them are 8 characters with the tilde in the 7th position.
They are sometimes huge
I have folders with two of them with the same name inside ( !!! )

16

What encryption scheme are using added bit? Aes?

17

mount -t ecryptfs -o no_sig_cache,verbose,ecryptfs_cipher=aes,ecryptfs_key_bytes=32,ecryptfs_enable_filename=y,ecryptfs_passthrough=n,ecryptfs_enable_filename_crypto=y

18

It worked okay. It didn’t recopy any files even ones with ~ in names/directories.

The only thing I can think of is that isn’t really a tilde but a special character perhaps that got copied as such. Then the new version isn’t finding that file because its technically different. Honestly just a pure guess.

On the NAS box: echo filename | hd
compare that to the new box.

You could try with 1.41 on your new server to see if that changes something. I’d also suggest that you actually download that file from the NAS to the new server and compare

19

echo EN88TE~2 | hd
00000000 45 4e 38 38 54 45 7e 32 0a |EN88TE~2.|
00000009

does it help?

20

You’d have to compare to your other server.