Network drive to OneDrive mapped as network drive folder transfer errors: "Corrupted on transfer, file sizes differ" with cache mode on, and "Can’t read and write without --vfs-cache-mode >= minimal" with cache mode off

Help and Support
1

Hi all,

I have installed rclone to use for transferring approximately 215 GB of content from an existing network drive (let's call it J Drive) to a OneDrive location mapped as a network drive via rclone (let's call it R Drive).

Conf and version details:

rclone.conf (C:\Users\Userdeets\AppData\Roaming\rclone):

[onedrive]
type = onedrive
drive_type = business
client_id = x
client_secret = x
token = {"access_token":"x"}
drive_id = x
drive_type = documentLibrary
chunk_size = 250Mi


C:\Users\Userdeets> version
rclone v1.67.0
- os/type: windows
- os/arch: amd64
- go/version: go1.22.4
- go/linking: static
- go/tags: cmount

now for the initial command...

rclone copy "J:\folder1\folder2" onedrive:folder3

A bottleneck occurs around the 30 GB transfer point when syncing the entire 215 GB folder top directory. Even when transferring in smaller batches of, say, 10 GB, it always slows to around 10 KB/s (completely unfeasible), with all attempts resulting in the same errors that I suspect are causing the issue: 'corrupted on transfer: sizes differ'. For example:

2022/08/30 14:00:45 ERROR : master/sapb1_oibq.xml: corrupted on transfer: sizes differ 422438 vs 422421

I have tried various cache modes, including full, minimal, and write, all resulting in the same errors. I have also used the --ignore-checksum flag to bypass checksum mismatches.

rclone mount onedrive: X: --network-mode --vfs-cache-mode full --vfs-cache-max-size 100G --vfs-cache-max-age 24h

then

rclone sync "J:\folder1\folder2" onedrive:folder3

I have tried optimising chunk sizes to avoid these bottlenecks, e.g...

rclone mount onedrive: X: --network-mode --vfs-cache-mode full --vfs-cache-max-size 100G --vfs-cache-max-age 24h

then

rclone sync "J:\folder1\folder2" onedrive:folder3 --onedrive-chunk-size 250M --transfers 32 --checkers 32 --retries 3 --low-level-retries 10

After these steps, I still receive the same plethora of errors saying that x corrupted on transfer, files sizes differ.

Under the impression that no cache mode is needed to simply transfer files (correct me if I'm wrong here), I attempted the same transfer without any cache mode...

rclone mount onedrive: X: --network-mode

then

rclone sync "J:\folder1\folder2" onedrive:folder3 --onedrive-chunk-size 250M --transfers 32 --checkers 32 --retries 3 --low-level-retries 10

which resulted in a new error:

"Can’t read and write to file without --vfs-cache-mode >= minimal"

Upon which when setting cache mode to minimal, this results in 'corrupted on transfer: sizes differ' error again.

Please note, we can rule out network speed. Tested. It's fine.

Has anyone encountered either or both of these errors before?

  1. For the "corrupted on transfer: sizes differ" error:
  • Why does this occur and what can I do to address it?
  1. For the "Can’t read and write to file without --vfs-cache-mode >= minimal" error:
  • Is cache mode (of any kind) needed to move files from one location to another? If so, why?
  • How should I address this error?

Thanks a bunch in advanced!

Jacko

2

Can you share the debug logs?

such a high number of --checkers and --transfers will surely get you throttled in OneDrive so you need to scale down. Search the forums for tips, but staying with default is usually enough.

Also, OneDrive is not a streaming upload remote, that means it needs the exact filesize before upload starts, it could be the network drive is giving a wrong value, this would be fixed with your idea of using a full cache mount, but your command is wrong as you end up copying to onedrive: and not x: drive that is where you mounted the vfs partition. Please note you will need enough storage locally to cache the files while they upload.

Is there any way you could run rclone directly in the network drive machine? It might simplify a lot of the headaches you are facing.

3

Thanks for your reply.

I need to clarify that my initial transfer was from my local C Drive to the mapped network drive, not from an existing network drive as I mistakenly mentioned in the initial post. Apologies for the confusion. The initial command was:

rclone copy C:\folder1\folder2 onedrive:folder3

not:

rclone copy "J:\folder1\folder2" onedrive:folder3

Moving forward...
I attempted another transfer to capture the debug logs. I cannot attach files to this reply as new users cannot upload attachments by the looks. Note link to google drive folder: rclone logs – Google Drive

  • rclone_debug_mount 1 first 600 lines.log (Please note this file was too large to attach, so I copied over the first 600 lines from this log, hence the name)
  • rclone_debug_sync 1

Based on our communication so far:

  • Noted about the --checkers and --transfers; these were not used in the logs I have sent.

  • Noted regarding copying to X: instead of onedrive: (good spot, thanks!). This was used in the logs I have sent.

Commands used:

To mount:

rclone mount onedrive: X: --network-mode --vfs-cache-mode full -vv > rclone_debug_mount.log 2>&1

To transfer:

rclone sync C:\SCR X:\ -vv > rclone_debug_sync.log 2>&1

From what I have seen via the logs

Some further questions regarding your query:

  • "Please note you will need enough storage locally to cache the files while they upload.":
    Noting the fact it's a 215GB transfer, do I need this much storage locally to cache the files while they upload, or is this done in batches? For example, batches of 1GB that are cached, and when the upload of that is finished, it moves to the next 1GB?

  • "Is there any way you could run rclone directly in the network drive machine?":
    For the logs sent, rclone was run on the same machine from which the folders were being transferred (Local C Drive). The transfer was also executed via the same machine from which the destination mapped network drive (X:) was mounted. If this is what you mean by running rclone directly on the machine?

Jacko

4

Not a problem,

Could you retry the original command (rclone copy C:\folder1\folder2 onedrive:folder3) and paste the logs? If it's local, would love to see exactly how the debug logs mark the error.

Now, you are using the mount command in a very inefficient way. If you already mounted the drive, there is no need to use rclone again, simply copy the file contents to the network drive and rclone mount will upload it in the background (and you can check the same on the logs)

Well, if you are using mount command (does not apply if you use sync directly) you will need enough space to whatever content you copy to the mount while rclone hold the file in the cache. It's up to you how much content you copy at the time.

Yes, the second assumption was if you were copying the contents from another computer via a shared network drive, so this does not apply.

So to recap, try again the sync command and share some debug logs, no need to copy everything just enough to recreate the problem, if it's a particular file, just copy that one, no need to copy the whole structure and we can go from there.

5

Hi Jose,

Thanks for the swift response.

Please find the debug logs attached for the initial command:

rclone copy "C:\SCR\SCR test transfer" "onedrive:\SCR test transfer copy" -vv > rclone_debug_sync.log 2>&1

https://drive.google.com/drive/folders/1-CdH77RHgzLew4fhmzroXJ5ngnnlQOFI?usp=sharing

I have made a separate file in this folder which lists all of the errors in the sync debug logs

"Now, you are using the mount command in a very inefficient way. If you already mounted the drive, there is no need to use rclone again, simply copy the file contents to the network drive and rclone mount will upload it in the background (and you can check the same on the logs)"

Noted. Thanks.

"Well, if you are using mount command (does not apply if you use sync directly) you will need enough space to whatever content you copy to the mount while rclone hold the file in the cache. It's up to you how much content you copy at the time."

I have 100GB left on the local drive, which I would like it to explain the issues I am having, however I have attempted to do this 10GB at a time, and get the same error.

It's worth noting that I originally encountered this error while transferring from the external network drive (J Drive), resulting in the same error, as mentioned on my original post on my thread. For context, I moved the content from the J Drive to my local drive so that I could then copy from my local drive to the mapped network drive to see if this would fix the issue, but I still got the same error.

I've also recreated this issue by doing another test transfer directly from the J Drive (as I did from the very start), and the same errors occur.

rclone copy "J:\Processing Read Write\SCR" "onedrive:\SCR test transfer copy from J Drive" -vv > rclone_debug_sync.log 2>&1

https://drive.google.com/drive/folders/17FHp9xNJPZTxDkQ3pu2Z4hoK1XzKxq32?usp=sharing

Note: I have made a separate file in this folder which lists all of the errors in the sync debug logs

My semi-educated guess here is that in both instances—transferring from the external network drive and from my local drive—the error is likely due to a caching issue. With insufficient room for the cache, only part of a file is cached, leading to the "corrupted on transfer: sizes differ" error.

I say semi-educated because like I said I tried this only doing 10GB at a time and got the same error, even when I had enough space on my local drive for the cache.

Thoughts?

Thanks,
Jacko

6

Ok checking the logs, this seems to be a known issue with Office files (most are xlsx for example)

Check:

Basically Microsoft is messing with the files causing the issue.

Hope this helps!

7

Thanks, re-attempt was done via..

rclone copy C:\folder1\folder2 X:\folder3 --ignore-checksum --ignore-size

after the below mount

rclone mount onedrive: X: --network-mode --vfs-cache-mode full

(Please note I acknowledge a second copy command isn't needed for rclone as its inefficient, just click and drag, but I figured I had to do it to be able to add the ignore checksum and ignore size functionality of rclone...

That seems to have addressed the initial size difference and read/write related errors, however upon execution of the suggested command line arguments in that link you sent...

--ignore-checksum --ignore-size

I now receive:

YYYY/MM/DD HH:MM:SS ERROR : sampledir/samplefile: Failed to copy: CreateFile \\?\X:\Sample\filepath\to\sampledir\samplefile.partial: The system cannot find the file specified.

...Where I am of the impression it makes a partial version of this file while it transfers all the data? Because in the above example, samplefile.partial exists in the destination, but not the original location, where the file is just samplefile

Also I receive error:

YYYY/MM/DD HH:MM:SS ERROR : sampledir/samplefile.pdf: Failed to copy: The request could not be performed because of an I/O device error.

Both of these errors seem to bottleneck the transfer once again...

Can you please provide your thoughts on this?

Jacko

8

I would try to copy directly and bypass mounting altogether to see if that fixes it.

Try:
rclone copy C:\folder1\folder2 onedrive:\folder3 --ignore-checksum --ignore-size

Mounting to copy files is really inefficient and you do not need to cache the data to upload as rclone does this automatically, you would only use mount in case you need to simulate a drive for other applications to read/write content from your cloud drive.