Existing files not found by rclone

Suspected Bug
1

What is the problem you are having with rclone?

  1. Used rclone sync to upload a local directory to the remote.
  2. Usingrclone sync again copies all files instead of skiping previously copied files.
  3. Used rclone ls over the remote directory to troubleshoot. The directory is seen as empty by rclone but I can see all files from OneDrive web client.

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.1
- os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
- os/kernel: 10.0.19045.5011 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.23.1
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The sync command

./rclone sync --progress --fast-list --onedrive-chunk-size 65536000 --transfers 20 --checkers 20 --copy-links --log-level INFO --log-file rclone_log.txt "C:\Users\olive\Desktop\Oliver"  "OneDriveArchivo:/Compartidas/Japon 2024/Oliver"

The ls command

./rclone ls  "OneDriveArchivo:/Compartidas/Japon 2024/Oliver"

The rclone config contents with secrets removed.

[OneDrive]
type = onedrive
client_id = REMOVED
client_secret = REMOVED
token = {"access_token":"REMOVED","expiry":"2024-09-06T17:12:54.52663+02:00"}
drive_id = c7071e38386b99f5
drive_type = personal

[OneDriveArchivo]
type = onedrive
client_id = REMOVED
client_secret = REMOVED
token = {"access_token":"REMOVED","expiry":"2024-11-12T22:37:23.3939036+01:00"}
drive_id = REMOVED
drive_type = personal

A log from the command with the -vv flag

Beggining of the log of the sync (there are a lot of files there)

2024/11/12 21:57:08 DEBUG : rclone: Version "v1.68.1" starting with parameters ["C:\\Users\\olive\\rclone.exe" "sync" "--progress" "--fast-list" "--onedrive-chunk-size" "65536000" "--transfers" "20" "--checkers" "20" "--copy-links" "-vv" "--log-file" "rclone_log.txt" "C:\\Users\\olive\\Desktop\\Oliver" "OneDriveArchivo:/Compartidas/Japon 2024/Oliver"]
2024/11/12 21:57:09 DEBUG : Creating backend with remote "C:\\Users\\olive\\Desktop\\Oliver"
2024/11/12 21:57:09 DEBUG : Using config file from "C:\\Users\\olive\\AppData\\Roaming\\rclone\\rclone.conf"
2024/11/12 21:57:09 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2024/11/12 21:57:09 DEBUG : fs cache: renaming cache item "C:\\Users\\olive\\Desktop\\Oliver" to be canonical "local{12rtk}://?/C:/Users/olive/Desktop/Oliver"
2024/11/12 21:57:09 DEBUG : Creating backend with remote "OneDriveArchivo:/Compartidas/Japon 2024/Oliver"
2024/11/12 21:57:09 DEBUG : OneDriveArchivo: detected overridden config - adding "{VGGR9}" suffix to name
2024/11/12 21:57:09 DEBUG : fs cache: renaming cache item "OneDriveArchivo:/Compartidas/Japon 2024/Oliver" to be canonical "OneDriveArchivo{VGGR9}:Compartidas/Japon 2024/Oliver"
2024/11/12 21:57:11 DEBUG : 2024-09-18_IMG20240918101217.jpg: Need to transfer - File not found at Destination
2024/11/12 21:57:11 DEBUG : 2024-09-18_IMG20240918135715.jpg: Need to transfer - File not found at Destination
2024/11/12 21:57:11 DEBUG : 2024-09-18_IMG20240918145352.jpg: Need to transfer - File not found at Destination
2024/11/12 21:57:11 DEBUG : 2024-09-18_IMG20240918150932.jpg: Need to transfer - File not found at Destination
2024/11/12 21:57:11 DEBUG : 2024-09-18_IMG20240918164817.jpg: Need to transfer - File not found at Destination
2024/11/12 21:57:11 DEBUG : 2024-09-19_IMG20240919021805.jpg: Need to transfer - File not found at Destination
2024/11/12 21:57:11 DEBUG : 2024-09-19_IMG20240919053808.jpg: Need to transfer - File not found at Destination
2024/11/12 21:57:11 DEBUG : 2024-09-19_IMG20240919054013.jpg: Need to transfer - File not found at Destination
2024/11/12 21:57:11 DEBUG : 2024-09-19_IMG20240919101026.jpg: Need to transfer - File not found at Destination
2024/11/12 21:57:11 DEBUG : 2024-09-19_IMG20240919101208.jpg: Need to transfer - File not found at Destination
2024/11/12 21:57:11 DEBUG : 2024-09-19_IMG20240919101307.jpg: Need to transfer - File not found at Destination
2024/11/12 21:57:11 DEBUG : 2024-09-19_IMG20240919101446.jpg: Need to transfer - File not found at Destination

The log of the ls command

2024/11/12 21:55:24 DEBUG : rclone: Version "v1.68.1" starting with parameters ["C:\\Users\\olive\\rclone.exe" "ls" "-vv" "OneDriveArchivo:/Compartidas/Japon 2024/Oliver"]
2024/11/12 21:55:24 DEBUG : Creating backend with remote "OneDriveArchivo:/Compartidas/Japon 2024/Oliver"
2024/11/12 21:55:24 DEBUG : Using config file from "C:\\Users\\olive\\AppData\\Roaming\\rclone\\rclone.conf"
2024/11/12 21:55:25 DEBUG : fs cache: renaming cache item "OneDriveArchivo:/Compartidas/Japon 2024/Oliver" to be canonical "OneDriveArchivo:Compartidas/Japon 2024/Oliver"
2024/11/12 21:55:25 DEBUG : 4 go routines activ
2

hi,

perhaps for testing,
rclone copy and rclone ls a single file and post complete debug logs, not snippets.
and no flags, use default values.

1 Like
3

Try your sycn with --modify-window 1s flag. It is quite big subject I am not following in depth but here you can start your reading if interested in details:

Not sure if it is fixed in the latest beta - you can try.

1 Like
4

Hello, thanks for taking the time to answer

Here are the outputs when picking a single file

Sync, in this case the file is found

C:\Users\olive> ./rclone sync  -vv  "C:\Users\olive\Desktop\Oliver\2024-09-18_IMG20240918145352.jpg"  "OneDriveArchivo:/Compartidas/Japon 2024/Oliver"
2024/11/12 22:10:46 DEBUG : rclone: Version "v1.68.1" starting with parameters ["C:\\Users\\olive\\rclone.exe" "sync" "-vv" "C:\\Users\\olive\\Desktop\\Oliver\\2024-09-18_IMG20240918145352.jpg" "OneDriveArchivo:/Compartidas/Japon 2024/Oliver"]
2024/11/12 22:10:46 DEBUG : Creating backend with remote "C:\\Users\\olive\\Desktop\\Oliver\\2024-09-18_IMG20240918145352.jpg"
2024/11/12 22:10:46 DEBUG : Using config file from "C:\\Users\\olive\\AppData\\Roaming\\rclone\\rclone.conf"
2024/11/12 22:10:46 DEBUG : fs cache: adding new entry for parent of "C:\\Users\\olive\\Desktop\\Oliver\\2024-09-18_IMG20240918145352.jpg", "//?/C:/Users/olive/Desktop/Oliver"
2024/11/12 22:10:46 DEBUG : Creating backend with remote "OneDriveArchivo:/Compartidas/Japon 2024/Oliver" 
2024/11/12 22:10:47 DEBUG : fs cache: renaming cache item "OneDriveArchivo:/Compartidas/Japon 2024/Oliver" to be canonical "OneDriveArchivo:Compartidas/Japon 2024/Oliver"
2024/11/12 22:10:47 DEBUG : 2024-09-18_IMG20240918145352.jpg: Size and modification time the same (differ 
by 0s, within tolerance 1ms)
2024/11/12 22:10:47 DEBUG : 2024-09-18_IMG20240918145352.jpg: Unchanged skipping
2024/11/12 22:10:47 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         1.1s

2024/11/12 22:10:47 DEBUG : 4 go routines active

ls

PS C:\Users\olive> ./rclone ls  -vv  "OneDriveArchivo:/Compartidas/Japon 2024/Oliver/2024-09-18_IMG20240918145352.jpg"
2024/11/12 22:13:25 DEBUG : rclone: Version "v1.68.1" starting with parameters ["C:\\Users\\olive\\rclone.exe" "ls" "-vv" "OneDriveArchivo:/Compartidas/Japon 2024/Oliver/2024-09-18_IMG20240918145352.jpg"]        
2024/11/12 22:13:25 DEBUG : Creating backend with remote "OneDriveArchivo:/Compartidas/Japon 2024/Oliver/2024-09-18_IMG20240918145352.jpg"
2024/11/12 22:13:25 DEBUG : Using config file from "C:\\Users\\olive\\AppData\\Roaming\\rclone\\rclone.conf"
2024/11/12 22:13:27 DEBUG : fs cache: adding new entry for parent of "OneDriveArchivo:/Compartidas/Japon 2024/Oliver/2024-09-18_IMG20240918145352.jpg", "OneDriveArchivo:Compartidas/Japon 2024/Oliver"
2024/11/12 22:13:28 DEBUG : 4 go routines active
5

now, i see your issue, but not sure what is the cause

6

Tried but without luck. Thanks anyway :pray:

rclone ls is not even listing the files, that may be the root cause of the issue.

7

I see... this is real strange. But I can not reproduce it.

Could you try the same rclone ls using the latest rclone beta?

8

Here is the output for the beta:

./rclone ls  -vv  "OneDriveArchivo:/Compartidas/Japon 2024/Oliver"
2024/11/13 08:26:18 DEBUG : rclone: Version "v1.69.0-beta.8402.f639cd9c7" starting with parameters ["C:\\Users\\olive\\Desktop\\rclone-v1.69.0-beta.8402.f639cd9c7-windows-amd64\\rclone.exe" "ls" "-vv" "OneDriveArchivo:/Compartidas/Japon 2024/Oliver"]
2024/11/13 08:26:18 DEBUG : Creating backend with remote "OneDriveArchivo:/Compartidas/Japon 2024/Oliver"
2024/11/13 08:26:18 DEBUG : Using config file from "C:\\Users\\olive\\AppData\\Roaming\\rclone\\rclone.conf"
2024/11/13 08:26:19 DEBUG : fs cache: renaming cache item "OneDriveArchivo:/Compartidas/Japon 2024/Oliver" to be canonical "OneDriveArchivo:Compartidas/Japon 2024/Oliver"
2024/11/13 08:26:19 DEBUG : 4 go routines active
9

Thank you. Not that it solves the problem but at least we know that it is not something fixed in the latest beta (there are few OneDrive fixes there)

10

Files missing with rclone ls is definitely a problem.

Can you run your test ls with -vv --dump bodies and post the output. That will hopefully show what is going on.

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.