Single archive file balloons in size when copying to a remote

Help and Support
1

What is the problem you are having with rclone?

When copying a single archive file e.g. file.7z, it balloons from 130GB to 400GB. This in turn increases the time to transfer the file across to the remote. The size of the compressed archive is 130GB and there is a single file that is being copied.

What is your rclone version (output from rclone version)

rclone v1.53.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Microsoft Windows 10 Professional version 1909 Build 18636.1110

Which cloud storage system are you using? (eg Google Drive)

Microsoft OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone.exe --transfers=2 --checkers=2 --stats 10s --progress --bwlimit 8M --checksum --log-file=rclone_log.txt --log-level DEBUG copy c:\archive ODEncrypted:/

A log from the command with the -vv flag

2

are you seeing that from the rclone stats?
if so, given the errors, pacer messages and upload retries in the log file, the rclone stats will not reflect the file size but the amount of data rclone uploaded.

3

Thanks @asdffdsa. That makes sense.

Is there any reason why the transfer speed would be downgraded to a point that multiple retries are occurring? I'm unsure about the pacer messages. Is there anything that can be done to minimize these?

4

if this is a one time event, then no worries.
if this is a repetitive problem, then perhaps we can try a deep dive into the logs.

for what i have read in the forum, onedrive, does suffer from lots of reliability problems.
even though i have a 1GB onedrive, via office365, i never use it.

i would make sure to do this - https://rclone.org/onedrive/#getting-your-own-client-id-and-key
for a file of size 130GB, i would use the largest value for https://rclone.org/onedrive/#onedrive-chunk-size

5

The issue appears to occur with single large files.

Is there a reason why generating a unique client id and key would improve the reliability?

Is the recommendation to use a chunk size of 250M? If yes, what side effects would it have e.g. performance?

6

sorry, but if you have not generated your own client id and key, really no point on working on this any further.
i already posted the link about it, did you read it?

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.