Problems with Onedrive Personal Ransomware Protection

Jenssaibe · April 16, 2021, 2:28pm

What is the problem you are having with rclone?

Hello,

I'm having an error while running a sync command (every 5 minutes with cronjob).

couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS

What's strange is that it looks its properly syncing but failing with the "sync" command checks.

I have to add that I'm syncing through an ecrypted remote and that I received a few days ago an email from Onedrive warning me that I could be victim of a ransomware attack.

Onedrive might be confused with the encrypted filenames...

I told Onedrive that everything is fine but I'm still having this strange issue.

If it does not settle itself, I might try to log again.

Anyway I think it might be a bug worth looking into.

It might be linked to those issues :

What is your rclone version (output from `rclone version`)

1.55.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Raspbian 10

Which cloud storage system are you using? (eg Google Drive)

Onedrive Personnal

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

sudo rclone sync /var/lib/docker/volumes/nextcloud/_data/data/JS/files/Pro Cabinet: -v --delete-excluded --log-file=/home/pi/scripts/cabinet-cron.log --create-empty-src-dirs

The rclone config contents with secrets removed.

[Bitwarden]
type = crypt
remote = Onedrive:Backup/Bitwarden
filename_encryption = standard
directory_name_encryption = true
password = XXX
password2 = XXX

[NextcloudDB]
type = crypt
remote = Onedrive:Backup/Nextcloud/db
filename_encryption = standard
directory_name_encryption = true
password = XXX
password2 = XXX

[NextcloudDATA]
type = crypt
remote = Onedrive:Backup/Nextcloud/data
filename_encryption = obfuscate
directory_name_encryption = true
password = XXX
password2 = XXX

[Cabinet]
type = crypt
remote = Onedrive:Backup/Cabinet
filename_encryption = obfuscate
directory_name_encryption = true
password = XXX
password2 = XXX

[Onedrive]
type = onedrive
token = {"access_token":"XXX","expiry":"2021-04-16T16:59:53.193994821+02:00"}
drive_id = XXX
drive_type = personal

A log from the command with the `-vv` flag

couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS```

Additionnal comment

I see that my token is due to expire today.
I think I will wait for it to renew by itself. Is that automatic ?
I still find it weird that some commands are working and others not.

asdffdsa · April 16, 2021, 2:33pm

hi,

not sure what makes this a bug?

can you post a complete debug log?

have you searched the forum for Unable to initialize RPS
OneDrive sync fail after few minutes
Rclone errors with OneDrive sync

Jenssaibe · April 16, 2021, 2:42pm

Hi,

What would make it a bug is that Onedrive and rclone crypt do not behave correctly together because Onedrive sees rclone crypt as a ransomware attack and prevent syncing ?

I have searched the forum as my post indicates.

I think a renewal of the token will solve this problem.

But it might have broke because of this ransomware protection mecanism from Onedrive?

I will post a full debug log as soon as I can.

asdffdsa · April 16, 2021, 2:51pm

sorry, not seeing a rclone bug.

if onedrive has a false positive with a rclone crypt, that would not be a rclone bug.
the solution would be for microsoft to have an option to disable ransomware detection.

what would you want rclone to do?

i use onedrive with a crypt remote, never had that problem.
tho i am using a business, not personal.

i will do a test with onedrive personal

Jenssaibe · April 16, 2021, 2:55pm

no problem I just wanted to add my humble contribution. I think it might be worth mentionning.

I must add that I'm using rclone crypt alongside Onedrive for a few weeks with no issues.

I just had this mail from Onedrive yesterday and it this error started to appear at the same time

I think it might be related to this issue and might be a Onedrive issue.

asdffdsa · April 16, 2021, 2:59pm

i will do some testing with onedrive personal.
if true, then we need to update the documentation to warn users.

can you check what exact file triggered the warning, was it a rclone crypt file or not?

Jenssaibe · April 16, 2021, 3:11pm

I can't really tell. In the mail I spoke about, they tell me that 125 files seems compromised. I can't tell what are exactly the files mentionned.

Onedrive gives you a link, and then I saw some of my encrypted files but nothing really helpful.

It happened during my 5min backups, so I can telle you that rclone is only using obfuscation with these.

What's weird is that I worked fine for weeks beforce I got this mail.

I just tried with another remote with standard encryption and it worked well:

2021/04/16 17:12:35 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/04/16 17:12:35 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "sync" "/var/lib/docker/volumes/Bitwarden" "Bitwarden:" "-vv" "--delete-excluded" "--log-file=/home/pi/scripts/bitwarden-cron.log" "--create-empty-src-dirs"]
2021/04/16 17:12:35 DEBUG : Creating backend with remote "/var/lib/docker/volumes/Bitwarden"
2021/04/16 17:12:35 DEBUG : Creating backend with remote "Bitwarden:"
2021/04/16 17:12:35 DEBUG : Creating backend with remote "Onedrive:Backup/Bitwarden"
...
2021/04/16 17:12:44 DEBUG : Encrypted drive 'Bitwarden:': Waiting for transfers to finish
2021/04/16 17:12:44 DEBUG : Waiting for deletions to finish
2021/04/16 17:12:44 INFO  : There was nothing to transfer
2021/04/16 17:12:44 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:               635 / 635, 100%
Elapsed time:         9.7s

2021/04/16 17:12:44 DEBUG : 3 go routines active

I'm sorry I can't post a DEBUG log of the other remote at the moment because the files are privileged. But at the moment it's still getting the same error during the checks.

asdffdsa · April 16, 2021, 3:30pm

are you getting that error only on the files that onedrive has warned you about?

what command do you use to check?

Jenssaibe · April 16, 2021, 7:31pm

It’s really hard to tell if a specific file leads to an error as they are encrypted on one side. So I can’t tell you if one specific file causes an error when obfuscated in a crypt remote.

As for the checks, I’m only referring to the ones I see in the logs, for example :

ˋ

2021/04/16 21:27:16 ERROR : Encrypted drive 'Cabinet:': not deleting files as there were IO errors
2021/04/16 21:27:16 ERROR : Encrypted drive 'Cabinet:': not deleting directories as there were IO errors
2021/04/16 21:27:16 ERROR : Attempt 1/3 failed with 9 errors and: march failed with 8 error(s): first error: couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS
2021/04/16 21:27:18 ERROR : Cabinet/Acceuil téléphonique: error reading destination directory: couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS
2021/04/16 21:27:21 ERROR : Compta/2013/Note d'honoraires: error reading destination directory: couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS
2021/04/16 21:27:30 ERROR : Compta/2021/1 - 1er semestre: error reading destination directory: couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS
2021/04/16 21:27:58 ERROR : XXX : error reading destination directory: couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS
2021/04/16 21:28:03 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 4 (retrying may help)
Checks: 20198 / 20198, 100%
Elapsed time: 3m1.5s
ˋ

What I call a check : when sync lists the file to know if there is any change. Sorry if I’m misleading.

asdffdsa · April 16, 2021, 7:44pm

need to figure out if this a crypt issue or a onedrive issue?

if you copy files to Onedrive:, you do not get any errors?

you posted this, did you try the solution at the end of the topic?
https://forum.rclone.org/t/onedrive-sync-fail-after-few-minutes/11916/2?u=asdffdsa

i see you are using just one backtick before and after the text; you need to use three so it will look like this.

2021/04/16 21:27:16 ERROR : Encrypted drive 'Cabinet:': not deleting files as there were IO errors
2021/04/16 21:27:16 ERROR : Encrypted drive 'Cabinet:': not deleting directories as there were IO errors
2021/04/16 21:27:16 ERROR : Attempt 1/3 failed with 9 errors and: march failed with 8 error(s): first error: couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS
2021/04/16 21:27:18 ERROR : Cabinet/Acceuil téléphonique: error reading destination directory: couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS
2021/04/16 21:27:21 ERROR : Compta/2013/Note d'honoraires: error reading destination directory: couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS
2021/04/16 21:27:30 ERROR : Compta/2021/1 - 1er semestre: error reading destination directory: couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS
2021/04/16 21:27:58 ERROR : XXX : error reading destination directory: couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS
2021/04/16 21:28:03 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 4 (retrying may help)
Checks: 20198 / 20198, 100%
Elapsed time: 3m1.5s

sweh · April 16, 2021, 10:44pm

I've seen OneDrive previously complain about my files looking ransom-wared because of obfuscated filenamed. But it didn't block me, just made me jump through horrible hoops the next time I logged into the website.

It's not an rclone bug; it's a OneDrive "misfeature" masquerading as a security defense (particularly for Windows users where ransomware on the desktop would propagate to their OneDrive).

I don't know any way of stopping OneDrive from doing this check.

One way I can think that rclone could work around it would be to have a "index" mapping. That would be a big change 'cos it'd need a map file "file1 == <real_name>, file2 == <real_name2>". There were discussions similar to this in the past for storing metadata. I don't think anything came of it, though.

asdffdsa · April 16, 2021, 11:13pm

yeah, the main thing is this is not a rclone bug.

good one!

bad one!

Jenssaibe · April 17, 2021, 1:09pm

Of course I agree with you :

it is not a rclone bug (I would be glad to edit the FP accordingly but I can't)
it might be useful to mention this issue with onedrive in the manual

This is what I tried today :

refreshed my Onedrive token
updated my rclone.conf file
did a new sync with my usual command :

sudo rclone sync /var/lib/docker/volumes/nextcloud/_data/data/JS/files/Pro Cabinet: -v --delete-excluded --log-file=/home/pi/scripts/cabinet-cron.log --create-empty-src-dirs

Unfortunately, I'm still getting errors. What's weird is that rclone is able to access Onedrive and list files to search for changes but some of them gets an error.

Here is my partial debug log showing one error :

2021/04/17 14:59:46 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/04/17 14:59:46 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "sync" "/var/lib/docker/volumes/nextcloud/_data/data/JS/files/Pro" "Cabinet:" "-vv" "--delete-excluded" "--log-file=/home/pi/scripts/cabinet-cron.log" "--create-empty-src-dirs"]
2021/04/17 14:59:46 DEBUG : Creating backend with remote "/var/lib/docker/volumes/nextcloud/_data/data/JS/files/Pro"
2021/04/17 14:59:46 DEBUG : Creating backend with remote "Cabinet:"
2021/04/17 14:59:46 DEBUG : Creating backend with remote "Onedrive:Backup/Cabinet"
2021/04/17 14:59:48 DEBUG : BDD/IMG_6691.png: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/04/17 14:59:48 DEBUG : BDD/IMG_6691.png: Unchanged skipping
2021/04/17 14:59:48 DEBUG : BDD/tgip_annuaire_complet_2020.pdf: Size and modification time the same (differ by -389.662851ms, within tolerance 1s)
2021/04/17 14:59:48 DEBUG : BDD/tgip_annuaire_complet_2020.pdf: Unchanged skipping
2021/04/17 14:59:52 DEBUG : Signatures/Signature HTML/Signature 2017.fld/image001.png: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/04/17 14:59:52 DEBUG : Signatures/Signature HTML/Signature 2017.fld/image001.png: Unchanged skipping
2021/04/17 14:59:52 DEBUG : Signatures/Signature HTML/Signature 2017.fld/image002.png: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/04/17 14:59:52 DEBUG : Signatures/Signature HTML/Signature 2017.fld/image002.png: Unchanged skipping
2021/04/17 14:59:52 DEBUG : Signatures/Signature HTML/Signature 2017.fld/image003.png: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/04/17 14:59:52 DEBUG : Signatures/Signature HTML/Signature 2017.fld/image003.png: Unchanged skipping
2021/04/17 14:59:52 ERROR : Compta/2013/Compta: error reading destination directory: couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS
DEBUG : Encrypted drive 'Cabinet:': Waiting for checks to finish
2021/04/17 15:05:35 DEBUG : Encrypted drive 'Cabinet:': Waiting for transfers to finish
2021/04/17 15:05:35 ERROR : Encrypted drive 'Cabinet:': not deleting files as there were IO errors
2021/04/17 15:05:35 ERROR : Encrypted drive 'Cabinet:': not deleting directories as there were IO errors
2021/04/17 15:05:35 ERROR : Attempt 3/3 failed with 16 errors and: march failed with 15 error(s): first error: couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS
2021/04/17 15:05:35 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                16 (retrying may help)
Checks:             44848 / 44848, 100%
Elapsed time:      5m49.4s

2021/04/17 15:05:35 DEBUG : 3 go routines active
2021/04/17 15:05:35 Failed to sync with 16 errors: last error was: march failed with 15 error(s): first error: couldn't list files: InvalidAuthenticationToken: Unable to initialize RPS

There is nothing special about the documents getting errors.
If it might help, my new rclone.conf file reflecting the token refresh :

[Bitwarden]
type = crypt
remote = Onedrive:Backup/Bitwarden
filename_encryption = standard
directory_name_encryption = true
password = XXX
password2 = XXX

[NextcloudDB]
type = crypt
remote = Onedrive:Backup/Nextcloud/db
filename_encryption = standard
directory_name_encryption = true
password = XXX
password2 = XXX

[NextcloudDATA]
type = crypt
remote = Onedrive:Backup/Nextcloud/data
filename_encryption = obfuscate
directory_name_encryption = true
password = XXX
password2 = XXX
[Cabinet]
type = crypt
remote = Onedrive:Backup/Cabinet
filename_encryption = obfuscate
directory_name_encryption = true
password = XXX
password2 = XXX

[Onedrive]
type = onedrive
token = {"access_token":"XXX","expiry":"2021-04-17T15:47:11.0072204+02:00"}
drive_id = XXX
drive_type = personal

asdffdsa · April 17, 2021, 1:39pm

not sure what FP is? but if you want to change the topic title, click on the or i would be glad to do that for you.

as for onedrive issue, @sweh stated "misfeature" masquerading as a security defense, does not affect access to the files. in effect, the files are not quarantined.

as for the RPS errors, perhaps create a new remote and test.

Jenssaibe · April 17, 2021, 1:48pm

I don't have any button to edit the first post.

Of course you can do it if you can.

As for the issue, it is strange that it occurs now after weeks of use withtout any troubles.
As you can see I have different crypt remotes, the ones with errors are the ones with obfuscation activated. The problem is that with standard filename encryption the file path is too long for Onedrive.

Anyway I don't know what's suddenly causing these errors...

I will try without filename encryption.

asdffdsa · April 17, 2021, 1:57pm

for what it is worth,

i have a 1TB onedrive as part of my offece365 account, but i do not use it much.

as i use rclone mostly as a secondary backup, i prefer to pay a few bucks per month for reliable storage
aws s3 deep glacier is $1.00/month and wasabi, a s3 rclone for hot storage, is $6.00/month
that combination works well for me.

Jenssaibe · April 17, 2021, 2:11pm

Thanks for your advice. I agree that the simplest solution would be to choose a more suitable cloud provider.

But I'm mainly using Onedrive at the moment so I'd like to be able to sync with it without issues.

I started a new sync without filename & filepath encryption, it looks like it's working.

This issue with Onedrive might be linked with obfuscation as I have no issues with my other crypt remote with standard filename encryption.

jared · April 17, 2021, 2:13pm

now hoooooooooooold on my monkey friend. what is it that makes you consider OneDrive personal "unreliable"? and while you're at it, do you have the same feeling over SharePoint/OneDrive Business?

Jenssaibe · April 17, 2021, 2:18pm

Hum, looks like it's not linked with filename encryption as I have just tried to sync a crypt remote without filename encryption and I'm still getting random errors :

2021/04/17 16:17:02 INFO  : _data/apps/contactsinteraction/l10n/nl.json: Copied (new)
2021/04/17 16:17:03 INFO  : _data/apps/contactsinteraction/l10n/pl.js: Copied (new)
2021/04/17 16:17:03 INFO  : _data/apps/contactsinteraction/l10n/pl.json: Copied (new)
2021/04/17 16:17:03 INFO  : _data/apps/contactsinteraction/l10n/pt_BR.js: Copied (new)
2021/04/17 16:17:03 INFO  : _data/apps/contactsinteraction/l10n/pt_BR.json: Copied (new)
2021/04/17 16:17:03 ERROR : _data/apps/contactsinteraction/l10n/ro.json: Failed to copy: InvalidAuthenticationToken: Unable to initialize RPS
2021/04/17 16:17:04 INFO  : _data/apps/contactsinteraction/l10n/ro.js: Copied (new)
2021/04/17 16:17:04 INFO  : _data/apps/contactsinteraction/l10n/sc.json: Copied (new)
2021/04/17 16:17:04 INFO  : _data/apps/contactsinteraction/l10n/sc.js: Copied (new)
2021/04/17 16:17:04 INFO  : _data/apps/contactsinteraction/l10n/sk.js: Copied (new)
2021/04/17 16:17:05 INFO  : _data/apps/contactsinteraction/l10n/sk.json: Copied (new)
2021/04/17 16:17:05 INFO  : _data/apps/contactsinteraction/l10n/sl.js: Copied (new)
2021/04/17 16:17:05 INFO  : _data/apps/contactsinteraction/l10n/sl.json: Copied (new)
2021/04/17 16:17:05 INFO  : _data/apps/contactsinteraction/l10n/tr.js: Copied (new)
2021/04/17 16:17:06 INFO  : _data/apps/contactsinteraction/l10n/tr.json: Copied (new)
2021/04/17 16:17:06 INFO  : _data/apps/contactsinteraction/l10n/zh_CN.json: Copied (new)
2021/04/17 16:17:06 INFO  : _data/apps/contactsinteraction/l10n/zh_CN.js: Copied (new)
2021/04/17 16:17:07 INFO  : _data/apps/contactsinteraction/l10n/zh_HK.js: Copied (new)
2021/04/17 16:17:07 INFO  : _data/apps/contactsinteraction/l10n/zh_HK.json: Copied (new)
2021/04/17 16:17:07 INFO  : _data/apps/contactsinteraction/l10n/zh_TW.js: Copied (new)
2021/04/17 16:17:07 ERROR : _data/lib/public/Authentication/IApacheBackend.php: Failed to copy: InvalidAuthenticationToken: Unable to initialize RPS
2021/04/17 16:17:08 INFO  : _data/apps/contactsinteraction/l10n/zh_TW.json: Copied (new)
2021/04/17 16:17:08 INFO  : _data/lib/public/Authentication/IAlternativeLogin.php: Copied (new)
2021/04/17 16:17:09 INFO  : _data/custom_apps/bruteforcesettings/l10n/fr.json: Copied (new)
2021/04/17 16:17:09 INFO  : _data/custom_apps/bruteforcesettings/l10n/fr.js: Copied (new)
2021/04/17 16:17:09 INFO  : _data/custom_apps/bruteforcesettings/l10n/fo.json: Copied (new)
2021/04/17 16:17:09 INFO  : _data/custom_apps/bruteforcesettings/l10n/gl.js: Copied (new)
2021/04/17 16:17:10 INFO  : _data/custom_apps/bruteforcesettings/l10n/gl.json: Copied (new)
2021/04/17 16:17:10 INFO  : _data/custom_apps/bruteforcesettings/l10n/he.js: Copied (new)
2021/04/17 16:17:10 INFO  : _data/custom_apps/bruteforcesettings/l10n/he.json: Copied (new)

sweh · April 17, 2021, 2:30pm

It's definitely the filename encryption/obfuscation that's triggering the problem because it makes the filenames look like gibberish, and Microsoft are using that as an indicator.

I have a family O365 account, so 6*1TB of storage. Two of them I use for backups:

% rclone about OneDriveBackups: 
Total:   1.005T
Used:    753.709G
Free:    275.291G
Trashed: 731.979G

% rclone about OneDriveBackups2:
Total:   1.005T
Used:    720.914G
Free:    308.086G
Trashed: 244.630G

Drive 1 does not have filename encryption, so everything just gets the ".bin" extension and doesn't trigger the alert. Drive 2 has a combination of no-encryption and obfuscation and it does trigger the alert; I just logged into the web site and it popped up the nasty message;

I hadn't noticed 'cos rclone was still working nicely without error