Rclone lsd is working but rclone copy not

AthLux · July 7, 2025, 6:10am

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I've copied the .rclone config file to another server and executed "rclone config reconnect ..".
rclone ls "onedrive-Matthias:/Pictures/Camera Roll/" --config /srv/piwigoStorage/matthias/.rclone.conf runs well on both server and "rclone copy .." run well on the old but don't work on the new server.

Run the command 'rclone version' and share the full output of the command.

on the non working server:
rclone v1.70.1

os/version: debian 11.11 (64 bit)
os/kernel: 5.10.0-35-amd64 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.24.4
go/linking: static
go/tags: none

on the working server:
rclone v1.69.2

os/version: debian 11.11 (64 bit)
os/kernel: 5.10.0-35-amd64 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.24.2
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Microsoft OneDrive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy -vv --transfers 50 --max-transfer 1b --cutoff-mode soft --no-traverse --config /srv/piwigoStorage/matthias/.rclone.conf 'one
drive-Matthias:/Pictures/Camera Roll/' /srv/piwigoStorage/matthias/cloudCache/oneDriveSync/M

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[Steuer]
type = dropbox
token = XXX

[onedrive-Matthias]
type = onedrive
token = XXX
drive_id = XXX
drive_type = personal
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the `-vv` flag

2025/07/07 07:30:15 DEBUG : rclone: Version "v1.70.1" starting with parameters ["rclone" "copy" "-vv" "--transfers" "50" "--max-transfer" "1b" "--cutoff-mode" "soft" "--no-traverse" "--config" "/srv/piwigoStorage/matthias/.rclone.conf" "onedrive-Matthias:/Pictures/Camera Roll/" "/srv/piwigoStorage/matthias/cloudCache/oneDriveSync/M"]
2025/07/07 07:30:15 DEBUG : Creating backend with remote "onedrive-Matthias:/Pictures/Camera Roll/"
2025/07/07 07:30:15 DEBUG : Using config file from "/srv/piwigoStorage/matthias/.rclone.conf"
2025/07/07 07:30:15 DEBUG : onedrive-Matthias: Loaded invalid token from config file - ignoring
2025/07/07 07:30:20 NOTICE: Time may be set wrong - time from "login.microsoftonline.com" is -17m33.544605451s different from this computer
2025/07/07 07:30:20 DEBUG : Saving config "token" in section "onedrive-Matthias" of the config file
2025/07/07 07:30:20 DEBUG : onedrive-Matthias: Saved new token in config file
2025/07/07 07:30:21 NOTICE: Time may be set wrong - time from "graph.microsoft.com" is -17m32.752756094s different from this computer
2025/07/07 07:30:21 DEBUG : fs cache: renaming cache item "onedrive-Matthias:/Pictures/Camera Roll/" to be canonical "onedrive-Matthias:Pictures/Camera Roll"
2025/07/07 07:30:21 DEBUG : Creating backend with remote "/srv/piwigoStorage/matthias/cloudCache/oneDriveSync/M"
2025/07/07 07:31:21 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                 0 / 0, -, Listed 7569
Elapsed time:        59.9s

2025/07/07 07:32:21 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                 0 / 0, -, Listed 7569
Elapsed time:      1m59.9s

:

kapitainsky · July 7, 2025, 12:05pm

Check your local server time.

Configure ntp or similar to keep it accurate all the time.

AthLux · July 7, 2025, 4:45pm

Thanks for this fast repsonse.
ntp installed and time is fine now. But the issues isn't solved:

rclone copy -vv --transfers 10 --max-transfer 1b --cutoff-mode soft --no-traverse --config /srv/piwigoStorage/matthias/.rclone.conf 'onedrive-Matthias:/Pictures/Camera Roll/' /srv/piwigoStorage/matthias/cloudCache/oneDriveSync/M
2025/07/07 18:37:46 DEBUG : rclone: Version "v1.70.1" starting with parameters ["rclone" "copy" "-vv" "--transfers" "10" "--max-transfer" "1b" "--cutoff-mode" "soft" "--no-traverse" "--config" "/srv/piwigoStorage/matthias/.rclone.conf" "onedrive-Matthias:/Pictures/Camera Roll/" "/srv/piwigoStorage/matthias/cloudCache/oneDriveSync/M"]
2025/07/07 18:37:46 DEBUG : Creating backend with remote "onedrive-Matthias:/Pictures/Camera Roll/"
2025/07/07 18:37:46 DEBUG : Using config file from "/srv/piwigoStorage/matthias/.rclone.conf"
2025/07/07 18:37:46 DEBUG : onedrive-Matthias: Loaded invalid token from config file - ignoring
2025/07/07 18:37:50 DEBUG : Saving config "token" in section "onedrive-Matthias" of the config file
2025/07/07 18:37:51 DEBUG : onedrive-Matthias: Saved new token in config file
2025/07/07 18:37:52 DEBUG : fs cache: renaming cache item "onedrive-Matthias:/Pictures/Camera Roll/" to be canonical "onedrive-Matthias:Pictures/Camera Roll"
2025/07/07 18:37:52 DEBUG : Creating backend with remote "/srv/piwigoStorage/matthias/cloudCache/oneDriveSync/M"
2025/07/07 18:38:52 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                 0 / 0, -, Listed 7570
Elapsed time:       1m0.0s

2025/07/07 18:39:52 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                 0 / 0, -, Listed 7570
Elapsed time:      1m59.9s

:

Maybe the "invalid token from config file - ignoring" is the issue? How could I solve that?

best regards
Matthias

kapitainsky · July 7, 2025, 7:24pm

Could you try the same with the latest rclone? v1.70.2? I think I have seen similar problems fixed in it.

Or even the latest beta.

AthLux · July 8, 2025, 7:56am

It is working on the original server with rclone v1.69.2
but not on the server received a copy of the .rclone.conf with rclone v1.70.1.
I've updated to v1.70.2 but it is still not working:

2025/07/08 09:58:05 DEBUG : rclone: Version "v1.70.2" starting with parameters ["rclone" "copy" "-vv" "--transfers" "10" "--max-transfer" "1b" "--cutoff-mode" "soft" "--no-traverse" "--config" "/srv/piwigoStorage/matthias/.rclone.conf" "onedrive-Matthias:/Pictures/Camera Roll/" "/srv/piwigoStorage/matthias/cloudCache/oneDriveSync/M"]
2025/07/08 09:58:05 DEBUG : Creating backend with remote "onedrive-Matthias:/Pictures/Camera Roll/"
2025/07/08 09:58:05 DEBUG : Using config file from "/srv/piwigoStorage/matthias/.rclone.conf"
2025/07/08 09:58:05 DEBUG : onedrive-Matthias: Loaded invalid token from config file - ignoring
2025/07/08 09:58:06 DEBUG : Saving config "token" in section "onedrive-Matthias" of the config file
2025/07/08 09:58:06 DEBUG : onedrive-Matthias: Saved new token in config file
2025/07/08 09:58:07 DEBUG : fs cache: renaming cache item "onedrive-Matthias:/Pictures/Camera Roll/" to be canonical "onedrive-Matthias:Pictures/Camera Roll"
2025/07/08 09:58:07 DEBUG : Creating backend with remote "/srv/piwigoStorage/matthias/cloudCache/oneDriveSync/M"
2025/07/08 09:59:07 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:        59.9s

2025/07/08 10:00:07 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                 0 / 0, -, Listed 7570
Elapsed time:       2m0.0s
:

kapitainsky · July 8, 2025, 10:00am

What is it for?

Also why do you use --no-traverse when it looks like your destination is local filesystem?

Can you simply run:

rclone copy -vv --no-traverse --config "/srv/piwigoStorage/matthias/.rclone.conf" "onedrive-Matthias:/Pictures/Camera Roll/" "/srv/piwigoStorage/matthias/cloudCache/oneDriveSync/M"

Does this command finish?

kapitainsky · July 8, 2025, 11:57am

I think it is related to:

github.com/rclone/rclone

deadlock in --no-traverse when all transfers skipped

opened 09:23AM - 02 Jul 25 UTC

closed 01:52PM - 04 Jul 25 UTC

ncw

bug

Issue made from https://github.com/rclone/rclone/pull/8647 by @jeremy #### Wha…t is the problem you are having with rclone? When every source object already exists at the destination and `--no-traverse` is set, march stalled/deadlocked after 100 entries because dstChan lagged srcChan. Introduced in v1.70.0 with callback-based syncing. I couldn't get a test case working reliably, but here's a script to reproduce: ```bash #!/usr/bin/env bash # Reproduces rclone v1.70+ deadlock with --no-traverse when all files exist # Usage: ./repro.sh [rclone-binary-path] # Use provided rclone binary or default to system rclone RCLONE="${1:-rclone}" echo "Using rclone: $RCLONE" echo "Version: $($RCLONE version | head -1)" echo # Create test directories src=$(mktemp -d) dst=$(mktemp -d) # Create 200 test files (enough to trigger the ~100 file deadlock) echo "Creating 200 test files..." for i in {1..200}; do echo "content$i" > "$src/file$i.txt" done # Initial copy to establish destination echo "Initial copy to destination..." $RCLONE copy "$src/" "$dst/" --quiet # Create file list filelist=$(mktemp) ls "$src" > "$filelist" # This command deadlocks in v1.70+ when all files exist at destination echo "Testing --no-traverse with all files existing (10s timeout)..." timeout 10s $RCLONE copy "$src/" "$dst/" \ --no-traverse \ --files-from "$filelist" \ --ignore-existing \ --progress if [ $? -eq 124 ]; then echo -e "\n❌ FAIL: Command timed out (deadlock detected)" exit 1 else echo -e "\n✅ PASS: Command completed successfully" fi # Cleanup rm -rf "$src" "$dst" "$filelist" ``` Results on 1.69.3, 1.70.2, and with this patch: ```bash Using rclone: rclone Version: rclone v1.69.3 Creating 200 test files... Initial copy to destination... Testing --no-traverse with all files existing (10s timeout)... Transferred: 0 B / 0 B, -, 0 B/s, ETA - Checks: 200 / 200, 100% Elapsed time: 0.0s ✅ PASS: Command completed successfully ``` ```bash Using rclone: rclone Version: rclone v1.70.2 Creating 200 test files... Initial copy to destination... Testing --no-traverse with all files existing (10s timeout)... Transferred: 0 B / 0 B, -, 0 B/s, ETA - Checks: 0 / 0, -, Listed 200 Elapsed time: 9.5s ❌ FAIL: Command timed out (deadlock detected) ``` ```bash Using rclone: ./rclone Version: rclone v1.71.0-DEV Creating 200 test files... Initial copy to destination... Testing --no-traverse with all files existing (10s timeout)... Transferred: 0 B / 0 B, -, 0 B/s, ETA - Checks: 200 / 200, 100%, Listed 200 Elapsed time: 0.0s ✅ PASS: Command completed successfully ``` #### What is your rclone version (output from `rclone version`) v1.70.2 #### The associated forum post URL from `https://forum.rclone.org` * https://forum.rclone.org/t/1-70-1-appears-to-hang-when-using-files-from-rclone-ls-no-traverse-and-ignore-existing/51720 * https://forum.rclone.org/t/deadlock-when-using-no-traverse-max-duration-and-files-from-with-many-files-100/51650 #### How to use GitHub * Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue. * Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue. * Subscribe to receive notifications on status change and new comments.

Please test with the latest beta. It should work now.

If you need it asap then you have to use beta or wait for rclone v1.71 or maybe v.1.70.3

AthLux · July 8, 2025, 1:00pm

Probably I misunderstood the meaning of this options but get the expected result in rclone v1.69.2.
--transfers 10 lead to start of 10 transfers in parallel. No additional transfer will be started because the max transfer size of 1 byte has been reached after this transfers.

Maybe rclone v1.70 verify this max transfer size before starting any transfer and will not transfer any file because all are bigger than 1 byte.

I will look for another solution for receiving limited amount of files by starting another thread in the forum.

I thank you very much for your support