Gpg errors/file size discrepencies in rclone operations with protondrive

I transferred just under 40gb of media files from proton drive to desktop. The dry run showed the correct amount of data. The destination shows the correct amount of data. Rclone log shows around 15gb more than what was supposed to be transferred.

Dry run which shows the correct amount of data that was and still is at source.

--dry-run
Transferred:       36.390 GiB / 36.390 GiB, 100%, 10.700 MiB/s, ETA 0s
Transferred:          865 / 865, 100%
Elapsed time:      4m47.1s
2023/12/14 04:56:58 NOTICE: 
Transferred:   	   36.390 GiB / 36.390 GiB, 100%, 10.700 MiB/s, ETA 0s
Transferred:          865 / 865, 100%
Elapsed time:      4m47.1s

Here is what rclone says was transfered......I not sure how, these are my boys football pics and videos, i know there wasnt almost 50gb of film and pics???

Checks:               863 / 863, 100%
Transferred:          865 / 865, 100%
Elapsed time:   8h51m20.9s
2023/12/14 13:56:46 INFO  : 
Transferred:   	   51.722 GiB / 51.722 GiB, 100%, 39.220 KiB/s, ETA 0s
Checks:               863 / 863, 100%
Transferred:          865 / 865, 100%
Elapsed time:   8h51m20.9s

2023/12/14 13:56:46 DEBUG : 7 go routines active

Here is whats at the destination aftter transfer....


$ du -sh ~/Videos/footballnew
37G	/home/xxx/Videos/footballnew

$ ls -lah ~/Videos/footballnew
total 0
drwxr-xr-x 1    552 Dec 14 12:24  .
drwxr-xr-x 1    38 Dec 14 04:39  ..
drwxr-xr-x 1   1.6K Dec 14 11:02  AndersonB_09-3-23
drwxr-xr-x 1   472 Dec 14 09:26  blockingNW
drwxr-xr-x 1   1.4K Dec 14 09:25  KingsBlack_10-1-23
drwxr-xr-x 1   1.4K Dec 14 08:13  KingsRed_08-20-23
drwxr-xr-x 1   864 Dec 14 12:17 'kings vs love 09-2-23'
drwxr-xr-x 1   1.3K Dec 14 13:56  Loveland_08-26-23
drwxr-xr-x 1   1.8K Dec 14 12:00  MilfordR_KingsB_08-27-23
drwxr-xr-x 1   3.5K Dec 14 10:02  NWest_10-7-23
drwxr-xr-x 1   22 Dec 14 12:48  OHR_Rnd1_10-28-23
drwxr-xr-x 1   24 Dec 14 09:26  OpenShotProjects_FB
drwxr-xr-x 1   48 Dec 14 05:13  OpenShotReels
drwxr-xr-x 1   480 Dec 14 11:09  pancake
drwxr-xr-x 1   984 Dec 14 12:51  practice_08-31-23
drwxr-xr-x 1   600 Dec 14 05:12  Practice_09-5-23
drwxr-xr-x 1   1.2K Dec 14 06:21  Practice_09-7-23
drwxr-xr-x 1   240 Dec 14 13:03  SuperBowl_11-5-23
drwxr-xr-x 1   1.5K Dec 14 12:25  Sycamore_10-14-23

Here is size of source after transfer which is the same as it was before transfer.....


$ rclone size myproton:/football -vv
2023/12/14 18:55:49 DEBUG : rclone: Version "v1.65.0" starting with parameters ["rclone" "size" "myproton:/football" "-vv"]
2023/12/14 18:55:49 DEBUG : Creating backend with remote "myproton:/football"
2023/12/14 18:55:49 DEBUG : Using config file from "/home/xxx/.config/rclone/rclone.conf"
2023/12/14 18:55:49 DEBUG : proton drive root link ID 'football': Has cached credentials
2023/12/14 18:55:53 DEBUG : proton drive root link ID 'football': Used cached credential to initialize the ProtonDrive API
2023/12/14 18:55:55 DEBUG : fs cache: renaming cache item "myproton:/football" to be canonical "myproton:football"
Total objects: 865
Total size: 36.390 GiB (39072929695 Byte)
2023/12/14 19:00:37 DEBUG : 5 go routines active

Here is full log of transfer...

https://dpaste.com/D7GWDWA2E

chatgpt had a look at it.......


# Revisiting the rclone log file for a more comprehensive analysis, focusing on the discrepancy in data size

def detailed_analysis_of_rclone_log(log_contents):
    lines = log_contents.split('\n')

    # Variables to track various aspects
    errors = []
    data_transferred_info = []
    total_size_info = []
    file_operations = []

    for line in lines:
        if "ERROR" in line:
            errors.append(line)
        if "Transferred:" in line and "ETA" in line:  # Summary of data transferred
            data_transferred_info.append(line)
        if "Total size:" in line:  # Summary of total size from 'rclone size'
            total_size_info.append(line)
        if "/:" in line:  # Potential file operation details
            file_operations.append(line)

    return {
        "errors": errors,
        "data_transferred_info": data_transferred_info,
        "total_size_info": total_size_info,
        "file_operations": file_operations
    }



# Running the detailed analysis on the log file
detailed_rclone_analysis = detailed_analysis_of_rclone_log(rclone_log_contents)
detailed_rclone_analysis



Result
{'errors': ['2023/12/14 10:50:11 ERROR : Loveland_08-26-23/S1000055.MP4: Failed to copy: gopenpgp: unable to read symmetric packet: openpgp: invalid data: tag byte does not have MSB set',
  '2023/12/14 10:50:12 ERROR : Loveland_08-26-23/S1000054.MP4: Failed to copy: gopenpgp: unable to read symmetric packet: openpgp: invalid data: tag byte does not have MSB set',
  '2023/12/14 13:08:04 ERROR : Attempt 1/3 failed with 2 errors and: gopenpgp: unable to read symmetric packet: openpgp: invalid data: tag byte does not have MSB set',
  '2023/12/14 13:56:46 ERROR : Attempt 2/3 succeeded'],
 'data_transferred_info': ['Transferred:       36.390 GiB / 36.390 GiB, 100%, 10.700 MiB/s, ETA 0s',
  'Transferred:   \t   36.390 GiB / 36.390 GiB, 100%, 10.700 MiB/s, ETA 0s',
  'Transferred:       12.627 GiB / 36.390 GiB, 35%, 1.582 MiB/s, ETA 4h16m18s',
  'Transferred:       12.639 GiB / 36.390 GiB, 35%, 1.797 MiB/s, ETA 3h45m35s',
  'Transferring:                                                                                              Transferred:       12.639 GiB / 36.390 GiB, 35%, 1.684 MiB/s, ETA 4h38s',
  'Transferring:                                                                                            Transferred:       12.642 GiB / 36.390 GiB, 35%, 1.829 MiB/s, ETA 3h41m32s',
  'Transferred:       51.722 GiB / 51.722 GiB, 100%, 39.220 KiB/s, ETA 0s',
  'Transferred:   \t   51.722 GiB / 51.722 GiB, 100%, 39.220 KiB/s, ETA 0s'],
 'total_size_info': [],
 'file_operations': []}


This is probably way over kill, tbh i hope im doing this rite.......f it....

Discrepancy in data size and file count between Proton Drive remote (source) and desktop destination after using rclone.
Source (Proton Drive remote) and Destination (Desktop) both show approximately 36.390 GiB, but rclone reports transferring 51.722 GiB.
Rclone errors related to two specific files (S1000055.MP4 and S1000054.MP4) in the Loveland_08-26-23 directory.
Initial Analysis:

Examined the rclone log file, which indicated a transfer of 36.390 GiB (865 files).
Found errors with the two mentioned files, suggesting potential issues with encryption or data integrity.
Additional Details Provided:

A dry run of rclone and the rclone size command confirmed the source size as 36.390 GiB.
The du command on the destination directory showed a size of 37G, consistent with the source size.
The ls -lah command output showed a normal directory and file setup at the destination.
Confusion arose due to rclone reporting a transfer of 51.722 GiB, significantly more than the actual data size.
Comprehensive Analysis of Log File:

Conducted a detailed examination of the rclone log.
Identified multiple entries of data transferred, showing both the expected 36.390 GiB and the inconsistent 51.722 GiB.
Confirmed the errors with the two specific MP4 files.
No detailed file operation entries or total size information was found that could explain the discrepancy.
Possible Causes Considered:

Double Counting or Re-Transfer: Rclone might have double-counted or re-transferred some data.
Log Inconsistencies: Potential inconsistencies in the log file entries.
Recommended Steps for Further Troubleshooting:

Verify File Integrity: Check the integrity of files at both source and destination.
Rclone Version: Ensure the use of the latest version of rclone.
Seek Rclone Forum or Support Assistance: Engage with the rclone community or support for insights.
Re-run with Verbose Logging: Consider re-running the transfer with the highest level of verbose logging for more detailed information.

What is the problem you are having with rclone?

  • Discrepancy in data size and file count between Proton Drive remote (source) and desktop destination after using rclone.
  • Source (Proton Drive remote) and Destination (Desktop) both show approximately 36.390 GiB, but rclone reports transferring 51.722 GiB.
  • Rclone errors related to two specific files (S1000055.MP4 and S1000054.MP4) in the Loveland_08-26-23 directory.

Run the command 'rclone version' and share the full output of the command.


$ rclone version
rclone v1.65.0
- os/version: debian 12.4 (64 bit)
- os/kernel: 6.1.0-15-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.4
- go/linking: static
- go/tags: none


Are you on the latest version of rclone? You can validate by checking the version listed here: Rclone downloads
-->
yes

Which cloud storage system are you using? (eg Google Drive)

ProtonDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy  --progress -vv myproton:/football /home/xxx/Videos/footballnew

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

$ rclone config redacted
[myproton]
type = protondrive
username = xxx
password = XXX
2fa = 
client_uid = XXX
client_access_token = XXX
client_refresh_token = XXX
client_salted_key_pass = XXX

A log from the command that you were trying to run with the -vv flag

https://dpaste.com/D7GWDWA2E

If someone could please help me ascertain the reason for the file errors as well as data size discrepancy that would be cool, and if not that's cool too... Thanks for your time

Anybody, anybody at all….

Page not found.

Secondly proton drive remote is marked as beta - so it is appreciated that you test it and find some issues but it also means that it is not ready for production usage yet.

I bring it to the attention of proton drive remote author - @henrybear327 could you please have a look at this one?

Honest mistake, I looked up and down for the edit but cant seem to find to plug in good paste but this should be good....

https://dpaste.com/C9ZAXP6AX

1 Like

May be some of your files are re-transferred. Cloud storage is not always 100% reliable and rclone does retries and low level retries to deal with errors. If data is copied over and files are not corrupted/damaged everything should be good. Rclone can use checksums on some remotes to verify data integrity, but I think it won't work with protondrive (mega does not support checksums, because its encrypted and protondrive is basically the same). Not sure if protondrive verifies the file integrity with signatures (it may not).

When I used icedrive via webdav (not reliable) it did almost always do that. Transferred calculates the total amount of data transferred not the total size of files

I am sorry but this is simply not true. Proton drive remote supports SHA1. It is very different remote then mega - not sure where you have idea about similarities between these two.

Is it SHA1 saved with the file then? Or SHA1 of the encrypted data? Protondrive server is not able to calculate the checksum of unencrypted data right?

Yes. The same as with onedrive, google drive, any S3, B2 etc. No server calculates hashes on demand maybe with exception of local and ftp/sftp. When you ask server for hash it only serves what was stored there as an attribute during upload. This is typical mode of operations for cloud storage.

And back to this particular issue unfortunately Proton drive remote is still in beta. Number of issues on this forum attests for its status. Abysmal performance and potentially plain bugs pending resolution.

There is no official API documentation available and it has been developed by one person - not very responsive recently. Implementation relies on GitHub - henrybear327/Proton-API-Bridge: A third-party, open-source Proton API bridge (mainly focusing on Drive for now) and there are likely quite a few errors in this library, as there isn't official documentation available.

Unless somebody fluent in go and proton drive workings gives a hand here and with Proton-API-Bridge it might not have the bright future. Open source dark side:)

I appreciate y’all replying and your time , it’s nice to have “somewhat” of an understanding/answer.

All my files seem to be good (the MP4’s play and the JPG’s show…So I’m gonna let it ride

Now that I think about I just had to reinstall the pd app on my iPhone a few months back because of discrepancies in how much storage it was showing and number of files ….

Proton in general seems buggy, I have issues w the vpn, and also proton drive app for iPhone.

I moved some pictures from desktop to proton remote today and it just finished (with errors) but again all this files are good and work.

I can post errors if it helps move along the development, but I don’t see how it would do so with only one person working on the project, and not being active …

EDIT: I think with the exception of digging deeper into the specific file errors that were shown in the rclone operation from this topic, and the new ones that I got in todays rclone operation, this is solved.

I’m not in a hurry to jump ship, but if I’m still having errors and issues by the time my son’s football season starts next spring I’m gonna go a different route/cloud service so it’s easier to upload and share files

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.