Rclone size discrepancy confusion

Help and Support
1

Purpose:

Running rclone copy to download all of Google Drive to local hdd

Problem:

size discrepancy
- Dry run of rclone copy indicated 1.3 TiB (also rclone size says the same)
- Download stopped in middle as my 4TB disk ran out of space (this was unexpected i had more than 3TB free and i expected around 1.3TiB)
- Already downloaded data on local hdd is of size 3.4 TiB.
- rclone logs says transferred 946 GiB only

I understand that some of it might be caused by Google doc files size not being included in 1.3TiB but difference between 1.3TiB and 3.4TiB is too large and also rclone logs saying 946GiB adds to the confusion

rclone version:

rclone v1.68.0
- os/version: arch 23.1.1 (64 bit)
- os/kernel: 5.10.205-1-MANJARO (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.1
- go/linking: dynamic
- go/tags: none

Cloud storage:

Google drive.

File type of external hdd:

exFAT managed through FUSE

Command ran:

nohup rclone copy --checksum --fast-list --drive-chunk-size=128M --transfers 16 --checkers 16 --retries 5 --contimeout 10m --timeout 10m --stats 5m --stats-file-name-length 0 remote: "/run/media/myname/WD/MyDrive/FromRclone" --log-file="/run/media/myname/WD/MyDrive/FromRclone/Logs/rcloneCopyLogsJan2025.txt" --log-level INFO

Let me know what you think is causing the much more than expected size and how would you suggest to investigate.

2

I doubt it has anything to do with rclone.

Problem is "Size" vs "Size on disk".

exFAT formatted disk is probably a problem here. It is extremely wasteful format. For your disk size I expect allocation unit to be 256 or 512KiB (vs 4k for ext4). Such formatted disk is only really usable for storing big files. For small files you need A LOT of extra space as every file has to be stored in its own allocation units. 1KiB will occupy 1 unit - 512KiB, 700KiB file will occupy 2 units - 1024 KiB etc.

Either get bigger disk or format it using some filesystem better suited for this job.

1 Like
3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.