pCloud to local: no hashes in common

szakeetm · September 29, 2020, 9:42am

What is the problem you are having with rclone?

When syncing from a pCloud backend to the local (NTFS) filesystem, there are no hashes in common.
According to the documentation, both MD5 and SHA1 hashes should be available.

pCloud supports MD5 and SHA1 type hashes, so you can use the --checksum flag.

What is your rclone version (output from `rclone version`)

rclone v1.53.1

os/arch: windows/amd64
go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Win10 x64

Which cloud storage system are you using? (eg Google Drive)

pCloud (EU data region)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync pcloud:test_folder C:\test_folder

After succesful sync:

rclone sync pcloud:test_folder C:\test_folder --checksum

The rclone config contents with secrets removed.

[pcloud]
type = pcloud
hostname = eapi.pcloud.com
token = {"access_token":"[masked]","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"}

A log from the command with the `-vv` flag

2020/09/29 11:40:48 DEBUG : rclone: Version "v1.53.1" starting with parameters ["C:\\ProgramData\\chocolatey\\lib\\rclone.portable\\tools\\rclone-v1.53.1-windows-amd64\\rclone.exe" "sync" "pcloud:test_folder" "c:\\test_folder" "--checksum" "-vv"]
2020/09/29 11:40:48 DEBUG : Using config file from "C:\\Users\\[masked]\\.config\\rclone\\rclone.conf"
2020/09/29 11:40:48 DEBUG : Creating backend with remote "pcloud:test_folder"
2020/09/29 11:40:48 DEBUG : Creating backend with remote "c:\\test_folder"
2020/09/29 11:40:48 DEBUG : fs cache: renaming cache item "c:\\test_folder" to be canonical "//?/c:/test_folder"
2020/09/29 11:40:48 DEBUG : Local file system at //?/c:/test_folder: Waiting for checks to finish
2020/09/29 11:40:48 NOTICE: Local file system at //?/c:/test_folder: --checksum is in use but the source and destination have no hashes in common; falling back to --size-only
2020/09/29 11:40:48 DEBUG : Unknown.pdf: Size of src and dst objects identical
2020/09/29 11:40:48 DEBUG : Unknown.pdf: Unchanged skipping
2020/09/29 11:40:48 DEBUG : Local file system at //?/c:/test_folder: Waiting for transfers to finish
2020/09/29 11:40:48 DEBUG : Waiting for deletions to finish
2020/09/29 11:40:48 INFO  : There was nothing to transfer
2020/09/29 11:40:48 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.5s

2020/09/29 11:40:48 DEBUG : 3 go routines active

szakeetm · September 29, 2020, 9:51am

A possible cause: pcloud MD5 hash seems to be an empty string (see last command):

PS C:\Users\mszakacs> rclone hashsum SHA-1 c:\testsync
9ff96ef2b4c6a8351a25c1cd53ebb2320ee171cd  Unknown.pdf
PS C:\Users\mszakacs> rclone hashsum MD5 c:\testsync
e0aee9161a43d390c43f5b7f3ae33e50  Unknown.pdf
PS C:\Users\mszakacs> rclone hashsum SHA-1 pcloud:pCloud_priv/Attestations
9ff96ef2b4c6a8351a25c1cd53ebb2320ee171cd  Unknown.pdf
PS C:\Users\mszakacs> rclone hashsum MD5 pcloud:pCloud_priv/Attestations
                                  Unknown.pdf

If this is indeed the cause, is there a way to ask rclone to use the SHA-1 hash instead of MD5?

ncw · September 29, 2020, 2:41pm

I had a look through the source and, yes, you are correct this message

"--checksum is in use but the source and destination have no hashes in common; falling back to --size-only"

Is caused by there being no MD5 hash for that particular object.

I think the message is a bit mis-leading since the source and destination do have hashes in common but the object in question just happens to have a missing MD5 for some reason.

There isn't at the moment.

I could make rclone check the SHA1 if the md5 was missing with a bit more code...

How did you upload an object with a missing MD5? All my objects have both?

szakeetm · September 29, 2020, 3:06pm

Thank you @ncw for your quick answer, I was just reading your implementation of the backend when you replied.
I have uploaded the files in a combination of WebDav, native sync client, virtual P drive and rclone methods, but curiously NONE of them have MD5 hashes (they have SHA-1).

Even more interesting is that I have uploaded a test file right now with the web interface and it has no MD5 hash either.

Are you also in the EU data region?
To be clear: when you issue rclone hashsum MD5 pcloud: you get MD5 hashes? If that's the case, I might open a ticket with pCloud.

Animosity022 · September 29, 2020, 3:21pm

Can you run:

rclone backend features pcloud:

And share the output.

szakeetm · September 29, 2020, 3:22pm

{
        "Name": "pcloud",
        "Root": "",
        "String": "pcloud root ''",
        "Precision": 1000000000,
        "Hashes": [
                "MD5",
                "SHA-1"
        ],
        "Features": {
                "About": true,
                "BucketBased": false,
                "BucketBasedRootOK": false,
                "CanHaveEmptyDirectories": true,
                "CaseInsensitive": false,
                "ChangeNotify": false,
                "CleanUp": true,
                "Command": false,
                "Copy": true,
                "DirCacheFlush": true,
                "DirMove": true,
                "Disconnect": false,
                "DuplicateFiles": false,
                "GetTier": false,
                "IsLocal": false,
                "ListR": false,
                "MergeDirs": false,
                "Move": true,
                "OpenWriterAt": false,
                "PublicLink": true,
                "Purge": true,
                "PutStream": false,
                "PutUnchecked": false,
                "ReadMimeType": false,
                "ServerSideAcrossConfigs": false,
                "SetTier": false,
                "SetWrapper": false,
                "SlowHash": false,
                "SlowModTime": false,
                "UnWrap": false,
                "UserInfo": false,
                "WrapFs": false,
                "WriteMimeType": false
        }
}

Animosity022 · September 29, 2020, 3:24pm

I do believe the rclone method would have both, but if the other methods are only doing 1 or the other, you'd have to wait for @ncw to modify to check for both if one is missing it sounds like.

ncw · September 29, 2020, 3:34pm

Hmm...

OK

No, I'm using the old non-EU region.

I don't currently have an account in the EU region to test with.

Yes that is correct. I think opening a ticket would be a good idea.

$ rclone md5sum --max-depth 1 pcloud:
0084467710d2fc9d8a306e14efbe6d0f  HELLO.txt
334d023f9f0dd51339c913f68320b61a  cover.jpg
d41d8cd98f00b204e9800998ecf8427e  empty.txt
d41d8cd98f00b204e9800998ecf8427e  empty2.txt
d41d8cd98f00b204e9800998ecf8427e  hello2.txt

It would also be easy enough to make a config param to make the pcloud backend only advertise SHA1 or MD5

szakeetm · September 30, 2020, 6:50am

Thank you. I'll open a ticket with a link to this thread and report back.

szakeetm · October 8, 2020, 12:30pm

I've created two free test accounts, one in the US and one in the EU data region.
The US account has MD5 hashes, the EU region is missing them.

It seems therefore that all EU region accounts have the MD5 hash functionality broken.

(I wrote to pCloud support, and they forwarded my email to the developers, though there wasn't any reply in the last 8 days).

@ncw One temporary patch could be to use SHA-1 hash for the EU data region (since all users are affected), until a fix comes up on pCloud's side. I hope they didn't just consider MD5 obsolete and discontinued on their new servers.

ncw · October 8, 2020, 5:16pm

I tried that here - can you give it a go?

I don't really like that fix so won't merge it until we hear back from pcloud whether this was accidental or on-purpose!

v1.54.0-beta.4815.f9750719f.fix-pcloud-eu-hash on branch fix-pcloud-eu-hash (uploaded in 15-30 mins)

szakeetm · October 8, 2020, 6:48pm

First of all, thank you very much for your prompt fix, I really appreciate it as I'm the only one who reported the issue so far.

First, I did a check with --checksum to compare the behavior. In the test below, rclone is the system-wide rclone install (1.53), while ./rclone is the beta (1.54).

PS C:\Users\Marton\Downloads\rclone-v1.54.0-beta.4815.f9750719f.fix-pcloud-eu-hash-windows-amd64> rclone sync pcloud:Joplin D:\pcloud\Joplin --dry-run -P --checksum
2020-10-08 20:38:25 NOTICE: Local file system at //?/D:/pcloud/Joplin: --checksum is in use but the source and destination have no hashes in common; falling back to --size-only
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:              3856 / 3856, 100%
Elapsed time:      1m25.4s
PS C:\Users\Marton\Downloads\rclone-v1.54.0-beta.4815.f9750719f.fix-pcloud-eu-hash-windows-amd64> ./rclone sync pcloud:Joplin D:\pcloud\Joplin --dry-run -P --checksum
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:              3856 / 3856, 100%
Elapsed time:      1m23.2s

As you can see, your beta did not report the checksum error and did the comparison correctly.

But I also tested whether hashes actually work for a modified file (same size but different content):

PS C:\Users\Marton\Downloads\rclone-v1.54.0-beta.4815.f9750719f.fix-pcloud-eu-hash-windows-amd64> rclone sync pcloud:hashtest C:\hashtest --checksum --verbose
2020/10/08 20:46:52 NOTICE: Local file system at //?/C:/hashtest: --checksum is in use but the source and destination have no hashes in common; falling back to --size-only
2020/10/08 20:46:52 INFO  : There was nothing to transfer
2020/10/08 20:46:52 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.6s

PS C:\Users\Marton\Downloads\rclone-v1.54.0-beta.4815.f9750719f.fix-pcloud-eu-hash-windows-amd64> ./rclone sync pcloud:hashtest C:\hashtest --checksum --verbose
2020/10/08 20:47:01 INFO  : modified.txt: Copied (replaced existing)
2020/10/08 20:47:01 INFO  :
Transferred:             8 / 8 Bytes, 100%, 16 Bytes/s, ETA 0s
Checks:                 1 / 1, 100%
Transferred:            1 / 1, 100%
Elapsed time:         0.9s

As you can see, 1.53 did not transfer the modified file (it had the same size), but 1.54 beta discovered the different hash and updated the local file.

In short: this fix works exactly as described, I'll update this thread once pCloud replies, and I agree with you not the pull these changes to the master branch before. Awesome work, thank you.

szakeetm · October 9, 2020, 10:01am

(Also, pasting pCloud's generic reply below)

Thanks for your e-mail.
We have escalated the case to our developers but please note that we opened the European server few months ago and it's possible that Rclone haven't designed their code to work on the European servers. Please also contact rclone regarding the issue.

ncw · October 9, 2020, 1:06pm

Great

I've found them helpful in past email conversations so hopefully they will come up with an explanation.

I think rclone has done the correct amount of work-around for the EU region, but there may be more I guess!

Sven_Heithecker · November 21, 2020, 11:54pm

Just going to inform I encountered the same issue: pCloud EU region, file change not detected with --checksum, as MD5 hash is empty, with pCloud backend (no pDrive or WebDAV).
rclone version v1.53.3-DEV
go version 1.14.10
linux (4.12.14, x86-64)
Was about creating an own topic when I detected this one. The fix above works perfectly, and I agree that it should not be necessary at all ...

ncw · November 22, 2020, 12:12pm

Did we ever hear back from pCloud that this was an intended change or not?

I can merge the fix above if we know that is what they intended. The docs here don't mention it though

szakeetm · November 22, 2020, 1:49pm

I will send them a reminder and report back when they reply.

@ncw To avoid getting the “check with rclone” answer again, can you help me which pcloud api function is rclone calling?

Then I could argue that the api returns an empty hash which is not expected, and that is independent from rclone, thus they should really investigate on their end. Thank you in advance.

ncw · November 23, 2020, 12:15pm

Great

It is the one I linked above: https://docs.pcloud.com/methods/file/checksumfile.html

The example shows both an md5 and an sha1 and there is no mention in the text that one or the other might not be there.

If you want to show an actual transaction then run rclone md5sum pcloud:path/to/file -vv --dump bodies and you'll see in there the request to the API above and the response.

szakeetm · November 25, 2020, 11:17am

I've written them, will post their reply.

There is however something very interesting after running your -vv command: it looks like they moved from MD5 to SHA256 hash for the European region, and that is causing the empty md5 hash:

The same command for a file in the US region:

Still, they should update their API documentation or get back MD5 hash, so I consider it a bug.

ncw · November 25, 2020, 12:31pm

That is very interesting!

Ideally rclone could ask pCloud what hashes are supported so it can know what to advertise.

Yes.