Rclone --fast-list behaviour with shared OneDrive (Business)

I like puppies.

What is the problem you are having with rclone?

Sorry for the long post !

I have rewritten the following multiple times, and am still not happy with it. Please bear with me.

Maybe starting with the heading “A log from the command that you were trying to run with the -vv flag” is more informative, concise and telling for experts, than reading my attempt at verbosely describing what I see and think happens.

Ok. Here goes the verbal description:

I've been using rclone to sync files to OneDrive Business (ODB) for years, but only recently discovered --fast-list. During testing, I added --fast-list as a parameter and found that rclone then tries to retransmit everything when the remote uses ODB shared folders. This happens not just the first time, but every single time. I believe this occurs because --fast-list does not enumerate shared folders (properly?), so it always appears as though they are missing, triggering a complete upload attempt each time. (I might be completely wrong!)

What do I mean by "ODB shared folders"? (I might not be using the correct terminology here.)
Access to OneDrive contents can be shared in several ways. One method allows user A to share a folder with user B (typically, by giving user B access to "a SharePoint" or an ODB folder). This only works within the same organisation. For user B and ODB, this appears as just another folder in user B's ODB storage. From rclone's perspective, however, it's different. rclone lsd remote: won't list shared folders, yet rclone can still read from and write to them.

Example: User A shares folder folder_a with User B. rclone can access this folder with rclone sync some_local_folder remote:/folder_a. If nothing changes locally or remotely, the next run will show INFO : There was nothing to transfer. Just as expected.
But the same command with --fast-list added (and --onedrive-delta as prerequisite), e.g. rclone sync some_local_folder remote:/folder_a --fast-list --onedrive-delta will detect all files as missing and [try to] reupload them.

I found this surprising, as my understanding is that --fast-list simply retrieves information from the remote in a different manner. This might reflect a gap in my understanding.

I searched for the problem on my end by:

  • Reading up on --fast-list here and here.
  • Searching the forum for similar experiences using the broad search term ‘Onedrive –fast-list’. There weren't many posts, which suggests this might still be an issue on my side
  • Stripping the command back to basics: rclone sync $source $dest --verbose --dry-run --onedrive-delta [--fast-list].
  • Testing the command on macOS (15.7.3 x86_64) and Ubuntu (24.04.3 LTS, aarch64) with no difference in results.
  • Checking for updates on both platforms using [sudo] rclone self-update.
  • Creating a folder with fewer files and running rclone sync (and rclone copy) multiple times to ensure this wasn't an odd --dry-run issue (similar to what can happen with rclone bisync).

What's the purpose of this lengthy post?
To raise awareness of this issue. I do not think it is a bug, but a constraint when using ODB shared folders. Perhaps a small note in the documentation would help others who encounter the same problem.

Would I like --fast-list to work with shared folders as well? Yes, but this is most likely an edge case and may not even be technically feasible.

Run the command 'rclone version' and share the full output of the command.

On Ubuntu:

rclone v1.72.1

  • os/version: ubuntu 24.04 (64 bit)
  • os/kernel: 6.8.0-1043-raspi (aarch64)
  • os/type: linux
  • os/arch: arm64 (ARMv8 compatible)
  • go/version: go1.25.5
  • go/linking: static
  • go/tags: none

On macOS:

rclone v1.72.1

  • os/version: darwin 15.7.3 (64 bit)
  • os/kernel: 24.6.0 (x86_64)
  • os/type: darwin
  • os/arch: amd64
  • go/version: go1.25.5
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Microsoft OneDrive Business of a Microsoft M365 EFC.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

```
rclone sync $source $remote:$shared_folder --verbose --stats=0 --onedrive-delta --fast-list [--dry-run]
```

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

```
[m1a]
type = sftp
ssh = ssh m1a
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

[m1af]
type = sftp
ssh = ssh -i ~/.ssh/ssh.internal.admin.pub admin-redacted-cl-m1@redacted
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

[m2a]
type = sftp
ssh = ssh m2a
shell_type = unix

[m3a]
type = sftp
ssh = ssh m3a
shell_type = unix

[odql]
type = onedrive
region = global
token = XXX
drive_id = XXX
drive_type = business

[odql_encrypted]
type = crypt
remote = odqlsc:Encrypted
filename_encryption = obfuscate
directory_name_encryption = true
password = XXX
password2 = XXX

[odqlsa]
type = onedrive
drive_id = XXX
drive_type = business
no_versions = true
token = XXX

[odqlsb]
type = onedrive
token = XXX
drive_id = XXX
drive_type = business
no_versions = true

[odqlsc]
type = onedrive
token = XXX
drive_id = XXX
drive_type = business
no_versions = true

[odqlsc_encrypted]
type = crypt
remote = odqlsc:Encrypted
filename_encryption = obfuscate
directory_name_encryption = true
password = XXX
password2 = XXX
no_versions = true

[odqlw]
type = onedrive
region = global
drive_id = XXX
drive_type = business
no_versions = true
token = XXX

[u1sd]
type = sftp
ssh = ssh u1sd
shell_type = unix

[u2sd]
type = sftp
ssh = ssh u2sd
shell_type = unix

[u3sd]
type = sftp
ssh = ssh u3sd
shell_type = unix

[u4sd]
type = sftp
ssh = ssh u4sd
shell_type = unix

[u5sd]
type = sftp
ssh = ssh u5sd
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

[u9a]
type = sftp
ssh = ssh u9a
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum
### Double check the config for sensitive info before posting publicly
```

A log from the command that you were trying to run with the -vv flag

I've left out the -vv flag for some commands, not because I think I am smarter, but for brevity. I will happily provide all commands with -vv if this helps.


+++ show contents:


sv-u5 ~ rclone lsd odqlsb:
          -1 2025-12-19 00:15:49         0 Attachments
          -1 2022-04-07 16:31:49         6 Repository    #  this folder has been shared with odql:

sv-u5 ~ rclone lsd odqlsb:Repository
          -1 2026-01-05 00:26:01       105 logs

sv-u5 ~ rclone lsd odql:
          -1 2024-05-23 23:51:53         1 Apps
          -1 2024-09-29 11:20:30         0 Attachments
          -1 2019-01-05 19:13:41         1 OneNote
-> shared folder 'Repository' _not_ displayed for odql:

sv-u5 ~ rclone lsd odql:Repository
          -1 2026-01-05 00:26:01       105 logs
-> shared folder 'Repository' _can_ be accessed _from odql:_


+++ sync:


sv-u5 ~ rclone sync /mnt/logs/ odqlsb:Repository/logs --verbose --stats=0 --onedrive-delta --dry-run
2026/01/05 13:13:48 INFO  : There was nothing to transfer

sv-u5 ~ rclone sync /mnt/logs/ odql:Repository/logs --verbose --stats=0 --onedrive-delta --dry-run
2026/01/05 13:13:58 INFO  : There was nothing to transfer
-> shared folder 'Repository' _can_ be synced _from odql:_


+++ add --fast-list:


sv-u5 ~ time rclone sync /mnt/logs/ odqlsb:Repository/logs --verbose --stats=0 --onedrive-delta --fast-list --dry-run
2026/01/05 13:20:17 INFO  : There was nothing to transfer

real	0m53.678s
user	0m1.842s
sys	0m0.152s


sv-u5 ~ time rclone sync /mnt/logs/ odql:Repository/logs --verbose --stats=0 --onedrive-delta --fast-list --dry-run
2026/01/05 13:22:27 NOTICE: audt3z9pw8lp: Skipped copy as --dry-run is set (size 279.591Ki)
2026/01/05 13:22:27 NOTICE: abvxj5h9m6x3: Skipped copy as --dry-run is set (size 2.931Mi)
2026/01/05 13:22:27 NOTICE: bd8nvmvh3qt0: Skipped copy as --dry-run is set (size 267.346Ki)
2026/01/05 13:22:27 NOTICE: bob2ei0ul0bq: Skipped copy as --dry-run is set (size 238.774Ki)
2026/01/05 13:22:27 NOTICE: al777ubxao59: Skipped copy as --dry-run is set (size 23.690Ki)
...
2026/01/05 13:22:27 NOTICE: zk39tu63u6ya: Skipped copy as --dry-run is set (size 214.284Ki)

real	0m1.077s
user	0m0.091s
sys	0m0.044s

-> --fast-list changes the way the contents are enumerated.
If used on a shared folder, it sees no files, and thus assumes everything needs to be [re]transmitted (not "re", actually).
Note how fast it is (1 sec vs. 53 sec).


+++ --fast-list and -vv


sv-u5 ~ time rclone sync /mnt/logs/ odqlsb:Repository/logs -vv --stats=0 --onedrive-delta --fast-list --dry-run
2026/01/05 13:43:24 DEBUG : rclone: Version "v1.72.1" starting with parameters ["rclone" "sync" "/mnt/logs/" "odqlsb:Repository/logs" "-vv" "--stats=0" "--onedrive-delta" "--fast-list" "--dry-run"]
2026/01/05 13:43:24 DEBUG : Creating backend with remote "/mnt/logs/"
2026/01/05 13:43:24 DEBUG : Using config file from "/home/service-deployment/.config/rclone/rclone.conf"
2026/01/05 13:43:24 DEBUG : fs cache: renaming cache item "/mnt/logs/" to be canonical "/mnt/logs"
2026/01/05 13:43:24 DEBUG : Creating backend with remote "odqlsb:Repository/logs"
2026/01/05 13:43:24 DEBUG : odqlsb: detected overridden config - adding "{HkZ8H}" suffix to name
2026/01/05 13:43:25 DEBUG : fs cache: renaming cache item "odqlsb:Repository/logs" to be canonical "odqlsb{HkZ8H}:Repository/logs"
2026/01/05 13:44:30 DEBUG : abvxj5h9m6x3: size = 3073088 OK
2026/01/05 13:44:30 DEBUG : audt3z9pw8lp: size = 286301 OK
2026/01/05 13:44:30 DEBUG : audt3z9pw8lp: Size and modification time the same (differ by -211.88527ms, within tolerance 1s)
2026/01/05 13:44:30 DEBUG : audt3z9pw8lp: Unchanged skipping
2026/01/05 13:44:30 DEBUG : bd8nvmvh3qt0: size = 273762 OK
2026/01/05 13:44:30 DEBUG : bd8nvmvh3qt0: Size and modification time the same (differ by -929.884651ms, within tolerance 1s)
2026/01/05 13:44:30 DEBUG : bd8nvmvh3qt0: Unchanged skipping
2026/01/05 13:44:30 DEBUG : bob2ei0ul0bq: size = 244505 OK
...
2026/01/05 13:44:30 DEBUG : u1dakk9ygmuw: Unchanged skipping
2026/01/05 13:44:30 DEBUG : s94nl0emkmjs: Unchanged skipping
2026/01/05 13:44:30 DEBUG : OneDrive root 'Repository/logs': Waiting for transfers to finish
2026/01/05 13:44:30 DEBUG : Waiting for deletions to finish
2026/01/05 13:44:30 INFO  : There was nothing to transfer
2026/01/05 13:44:30 DEBUG : 7 go routines active

real	1m5.654s
user	0m1.843s
sys	0m0.179s

->  still 'INFO  : There was nothing to transfer' as expected.


sv-u5 ~ time rclone sync /mnt/logs/ odql:Repository/logs -vv --stats=0 --onedrive-delta --fast-list --dry-run
2026/01/05 13:29:59 DEBUG : rclone: Version "v1.72.1" starting with parameters ["rclone" "sync" "/mnt/logs/" "odql:Repository/logs" "-vv" "--stats=0" "--onedrive-delta" "--fast-list" "--dry-run"]
2026/01/05 13:29:59 DEBUG : Creating backend with remote "/mnt/logs/"
2026/01/05 13:29:59 DEBUG : Using config file from "/home/service-deployment/.config/rclone/rclone.conf"
2026/01/05 13:29:59 DEBUG : fs cache: renaming cache item "/mnt/logs/" to be canonical "/mnt/logs"
2026/01/05 13:29:59 DEBUG : Creating backend with remote "odql:Repository/logs"
2026/01/05 13:29:59 DEBUG : odql: detected overridden config - adding "{HkZ8H}" suffix to name
2026/01/05 13:30:00 DEBUG : fs cache: renaming cache item "odql:Repository/logs" to be canonical "odql{HkZ8H}:Repository/logs"
2026/01/05 13:30:00 DEBUG : abvxj5h9m6x3: Need to transfer - File not found at Destination
2026/01/05 13:30:00 DEBUG : al777ubxao59: Need to transfer - File not found at Destination
2026/01/05 13:30:00 DEBUG : audt3z9pw8lp: Need to transfer - File not found at Destination
2026/01/05 13:30:00 DEBUG : bd8nvmvh3qt0: Need to transfer - File not found at Destination
...
2026/01/05 13:30:00 NOTICE: z9z4lbaghqtx: Skipped copy as --dry-run is set (size 550.203Ki)
2026/01/05 13:30:00 DEBUG : Waiting for deletions to finish
2026/01/05 13:30:00 DEBUG : 6 go routines active

real	0m0.809s
user	0m0.095s
sys	0m0.047s


+++

I wonder if this might be a problem with the Delta listing command from OneDrive. It’s the recommended method by Microsoft to list and keep track of changes to files between polls.

Personally I was having no issues with it until recently Microsoft started throttling me heavy on full refreshes, disabling delta fixed the throttling issue for me. I checked and validated rclone implementation and did not find any issues, rclone calls the API as expected. Wonder if this might be similar in the sense that it might be a Microsoft API issue.

Try searching online on your issue relating to delta, and try if you get the same behavior without delta in rclone. If so, it might be a Microsoft problem not rclone.

2 Likes

Huh! Removing –-onedrive-delta removed the problem, just as you suspected.

Thank you!

I’ve never used -–onedrive-delta before, but it is a hard requirement according to the rclone documentation on –-fast-list so I never even though about removing it.

… This must be enabled with the --onedrive-delta flag (or delta = true in the config file) as it can cause performance degradation. …

Without the --onedrive-delta flag, --fast-list doesn’t seem to make any difference. sigh

FWIW, I still get throttled by MS after about 5 mins of scans (that is, no file has yet been transferred; rclone is still wading through large directory structures to find what needs transferring). I’ve long ago stopped playing with –-checkers and --transfers, since that usually excarbates the problem and have yet to find a solution. I am the Admin of this M356 tenant, but that threshold is not something I found out how to configure. That explains the quest for any optimization I can find.

Thanks for taking the time to read and answer! Very much appreciated! Have a very nice day.

Try using –tpslimit 2 that should fix those throttling scenarios. Microsoft is very sensitive with the amount of queries per second. There’s a lot of documentation and tpslimit of 2 pretty much keeps you in the green. tpslimit of 3 would exceed it if you run it too long back to back.

2 Likes

Thanks! :orange_heart: I’ll have a look into that.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.