On a second migration, can't copy newest version of Excel files

Vivien_VOILLOT · March 30, 2023, 8:17am

What is the problem you are having with rclone?

I am migrating data form an AlFresco EDm hosted on webdav to Google Workspace Shared Drives.
The process I follow is to copy a first time a whole folder (and there are many of them) approximatively two weeks before its last migration (delta) for the users to have their data the morning of their training day.
What happens is if the first copy is almost perfect, the second one for at least one folder didn't take the newest versions of modified files that were previously uploaded.
e.g.
1- an Excel spreadsheet is lastly modified the 01-10
2- I ran the first copy at 03-15, it is uploaded (yay)
3- the file is then modified at the source, at 03-23
4- when I ran the second copy at 03-24, it is not even "seen" by Rclone.
I made checks of the source folder with ls, lsl, and lsf, the file is detected each time though.
A --dry-run copy confirms no excel files are detected, but only .pdfs files (27 files between the 03-15 and 24-03)

Run the command 'rclone version' and share the full output of the command.

rclone copy source:"G/01-ST/ST26" "ST26": --ignore-size --create-empty-src-dirs --log-file "c:\rclone\log-ST26.txt" --log-level INFO -P --transfers 16

Version

rclone v1.61.1

Which cloud storage system are you using? (eg Google Drive)

Webdav as source
Google Workspace as target

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy source:"G/01-ST/ST26" "ST26": --ignore-size --create-empty-src-dirs --log-file "c:\rclone\log-ST26.txt" --log-level INFO -P --transfers 16

I have as --dry-run the following :
rclone copy source:"G/01-ST/ST26" "ST26": --create-empty-src-dirs --log-file "c:\rclone\log-DRYRUN-ST26.txt" --log-level INFO -P --transfers 16 --dry-run --size-only
rclone copy source:"G/01-ST/ST26" "ST26": --create-empty-src-dirs --log-file "c:\rclone\log-DRYRUN-ST26.txt" --log-level INFO -P --transfers 16 --dry-run --refresh-times

The rclone config contents with secrets removed.

Abridged on the both remote we want to use
[source]
type = webdav
url = http://10.0.0.99:8080/alfresco/webdav/
vendor = other
user = [REDACTED]
pass = [REDACTED]

[ST26]
type = drive
client_id = rclone01manager@cool-splice-375710.iam.gserviceaccount.com
scope = drive
service_account_file = C:\rclone\cool-splice-375710-e859add01e00.json
team_drive = 0AJqak02Sv5k7Uk9PVA
root_folder_id =

A log from the command with the `-vv` flag

2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0AJ/FA0AJ053/CT2381593 - FA0AJ053.pdf: Skipped copy as --dry-run is set (size 30.934Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CC/FA0CC060/CT2381702 - FA0CC06 - 1.pdf: Skipped copy as --dry-run is set (size 31.125Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CC/FA0CC060/CT2381702 - FA0CC06 - 2.pdf: Skipped copy as --dry-run is set (size 31.125Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CC/FA0CC060/CT2381702 - FA0CC060.pdf: Skipped copy as --dry-run is set (size 31.125Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0AL/FA0AL113/CT2381893 - FA0AL113.pdf: Skipped copy as --dry-run is set (size 31.252Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CA/FA0CA085/CT2381591 - FA0CA085.pdf: Skipped copy as --dry-run is set (size 31.072Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CA/FA0CA088/CT2381707 - FA0CA088.pdf: Skipped copy as --dry-run is set (size 31.522Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CA/FA0CA095/CT2381998 - FA0CA095.pdf: Skipped copy as --dry-run is set (size 31.025Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CA/FA0CA097/CT2381952 - FA0CA097.pdf: Skipped copy as --dry-run is set (size 31.085Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0AS/FA0AS039/CT2381689 - FA0AS03 - 1.pdf: Skipped copy as --dry-run is set (size 31.170Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0AS/FA0AS039/CT2381689 - FA0AS039.pdf: Skipped copy as --dry-run is set (size 31.170Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CT/FA0CT019/CT2381682 - FA0CT019.pdf: Skipped copy as --dry-run is set (size 31.079Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CX/FA0CX128/CT2381993 - FA0CX128.pdf: Skipped copy as --dry-run is set (size 31.254Ki)
2023/03/30 10:13:55 NOTICE: 03-AF_en_cours/FA0B9A/07 FOURNISSEURS (EXTERNE)/CT2381649 - FA0B9A.pdf: Skipped copy as --dry-run is set (size 31.462Ki)
2023/03/30 10:13:55 NOTICE: 03-AF_en_cours/FA0B9A/07 FOURNISSEURS (EXTERNE)/CT2381665 - FA0B9A.pdf: Skipped copy as --dry-run is set (size 31.456Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CX/FA0CX025/CT2381717 - FA0CX025.pdf: Skipped copy as --dry-run is set (size 31.805Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CX/FA0CX025/CT2381897 - FA0CX025.pdf: Skipped copy as --dry-run is set (size 31.793Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CY/FA0CY018/CT2381792 - FA0CY018.pdf: Skipped copy as --dry-run is set (size 31.101Ki)
2023/03/30 10:13:55 NOTICE: 03-AF_en_cours/FA0B2A/07 FOURNISSEURS (EXTERNE)/CT2381732 - FA0B2A.pdf: Skipped copy as --dry-run is set (size 30.694Ki)
2023/03/30 10:13:55 NOTICE: 03-AF_en_cours/FA0B2A/07 FOURNISSEURS (EXTERNE)/CT2381687 - FA0B2A.pdf: Skipped copy as --dry-run is set (size 31.135Ki)
2023/03/30 10:13:55 NOTICE: 03-AF_en_cours/FA0B2A/07 FOURNISSEURS (EXTERNE)/CT2381866 - FA0B2 - 1.pdf: Skipped copy as --dry-run is set (size 31.277Ki)
2023/03/30 10:13:55 NOTICE: 03-AF_en_cours/FA0B2A/07 FOURNISSEURS (EXTERNE)/CT2381866 - FA0B2A.pdf: Skipped copy as --dry-run is set (size 31.277Ki)
2023/03/30 10:13:55 NOTICE: 03-AF_en_cours/FA0B2A/07 FOURNISSEURS (EXTERNE)/CT2381881 - FA0B2A.pdf: Skipped copy as --dry-run is set (size 30.900Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0AD/FA0AD189/CT2381793 - FA0AD189.pdf: Skipped copy as --dry-run is set (size 30.994Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CJ/FA0CJ003/CT2381681 - FA0CJ003.pdf: Skipped copy as --dry-run is set (size 30.818Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CJ/FA0CJ003/CT2381585 - FA0CJ003.pdf: Skipped copy as --dry-run is set (size 30.654Ki)
2023/03/30 10:13:55 NOTICE: 07-Inter_en_cours/FA0CJ/FA0CJ003/CT2381929 - FA0CJ003.pdf: Skipped copy as --dry-run is set (size 31.006Ki)
2023/03/30 10:13:55 NOTICE:
Transferred: 841.312 KiB / 841.312 KiB, 100%, 0 B/s, ETA -
Transferred: 27 / 27, 100%
Elapsed time: 2.4s

I should find here, e.g., "Liste extincteurs ST26.xlsx", that has been updated the 03-16 according the a lsl scan of the source.
The version of this file that I can find on the target Shared Drive bears the date 01-20.

Thanks

Animosity022 · March 30, 2023, 11:40am

Break it down to something a bit simpler and focus on one file to test with.

rclone lsl source:thatfile
rclone lsl destination:thatfile
If they are the same, make a change on the source and re-run the step 1
rclone copy source:thatfile destination:thatfile -vv --dry-run

Share all the output.

Vivien_VOILLOT · March 30, 2023, 12:51pm

Thanks.

So, here is what it returned :

1. rclone lsl source:thatfile
rclone lsl source:"G/01-ST/ST26.TODRIVE/12-Modeles/Liste extincteurs ST26.xlsx"
29237 2023-03-16 11:25:26.000000000 Liste extincteurs ST26.xlsx

2. rclone lsl destination:thatfile
rclone lsl ST26:"12-Modeles/Liste extincteurs ST26.xlsx"
29214 2023-01-20 11:16:12.000000000 Liste extincteurs ST26.xlsx

Bad luck for me, it has been updated the day after my first copy run (2023-03-15 PM).
And, at that step, it correctly seen and read by lsl commands.

3. If they are the same, make a change on the source and re-run the step 1
Skipped thus.

rclone copy source:thatfile destination:thatfile -vv --dry-run
rclone copy source:"G/01-ST/ST26.TODRIVE/12-Modeles/Liste extincteurs ST26.xlsx" ST26:"12-Modeles/Liste extincteurs ST26.xlsx" -vv --dry-run
2023/03/30 14:47:40 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "source:G/01-ST/ST26.TODRIVE/12-Modeles/Liste extincteurs ST26.xlsx" "ST26:12-Modeles/Liste extincteurs ST26.xlsx" "-vv" "--dry-run"]
2023/03/30 14:47:40 DEBUG : Creating backend with remote "source:G/01-ST/ST26.TODRIVE/12-Modeles/Liste extincteurs ST26.xlsx"
2023/03/30 14:47:40 DEBUG : Using config file from "C:\Users\Administrateur\AppData\Roaming\rclone\rclone.conf"
2023/03/30 14:47:40 DEBUG : found headers:
2023/03/30 14:47:40 DEBUG : fs cache: adding new entry for parent of "source:G/01-ST/ST26.TODRIVE/12-Modeles/Liste extincteurs ST26.xlsx", "source:G/01-ST/ST26.TODRIVE/12-Modeles"
2023/03/30 14:47:40 DEBUG : Creating backend with remote "ST26:12-Modeles/Liste extincteurs ST26.xlsx"
2023/03/30 14:47:42 DEBUG : fs cache: adding new entry for parent of "ST26:12-Modeles/Liste extincteurs ST26.xlsx", "ST26:12-Modeles"
2023/03/30 14:47:42 Failed to create file system for "ST26:12-Modeles/Liste extincteurs ST26.xlsx": is a file not a directory

If I run a :
rclone copy source:"G/01-ST/ST26.TODRIVE/12-Modeles/Liste extincteurs ST26.xlsx" ST26:"12-Modeles/Liste extincteurs ST26.xlsx" -vv --dry-run
So a "folderthatcontainsthatfile" rather than "thatfile", when it comes to my example file, I get this :
2023/03/30 14:50:27 NOTICE: Liste extincteurs ST26.xlsx: Skipped copy as --dry-run is set (size 28.552Ki)

It would imply it would copy it to replace the outdated actual one, but in fact it won't.

Animosity022 · March 30, 2023, 1:50pm

Oh sorry as I have a typo in my ask.

Rclone wants the destination to be a directory in rclone copy not a file.

rclone copy source:thatfile dest:

[felix@gemini ~]$ rclone copy /etc/hosts GD: -vv
2023/03/30 09:49:30 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2023/03/30 09:49:30 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "copy" "/etc/hosts" "GD:" "-vv"]
2023/03/30 09:49:30 DEBUG : Creating backend with remote "/etc/hosts"
2023/03/30 09:49:30 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2023/03/30 09:49:30 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2023/03/30 09:49:30 DEBUG : Creating backend with remote "GD:"
2023/03/30 09:49:30 DEBUG : GD: Loaded invalid token from config file - ignoring
2023/03/30 09:49:30 DEBUG : Saving config "token" in section "GD" of the config file
2023/03/30 09:49:30 DEBUG : Keeping previous permissions for config file: -rwxrwxr-x
2023/03/30 09:49:30 DEBUG : GD: Saved new token in config file
2023/03/30 09:49:31 DEBUG : hosts: Need to transfer - File not found at Destination
2023/03/30 09:49:32 DEBUG : hosts: md5 = 4cbac143722fd9acbf7440e3e37a2245 OK
2023/03/30 09:49:32 INFO  : hosts: Copied (new)
2023/03/30 09:49:32 INFO  :
Transferred:   	        386 B / 386 B, 100%, 385 B/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         2.1s

2023/03/30 09:49:32 DEBUG : 8 go routines active
[felix@gemini ~]$ vi /etc/hosts
[felix@gemini ~]$ sudo vi /etc/hosts
[felix@gemini ~]$
[felix@gemini ~]$
[felix@gemini ~]$ rclone copy /etc/hosts GD: -vv --dry-run
2023/03/30 09:49:51 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2023/03/30 09:49:51 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "copy" "/etc/hosts" "GD:" "-vv" "--dry-run"]
2023/03/30 09:49:51 DEBUG : Creating backend with remote "/etc/hosts"
2023/03/30 09:49:51 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2023/03/30 09:49:51 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2023/03/30 09:49:51 DEBUG : Creating backend with remote "GD:"
2023/03/30 09:49:52 DEBUG : hosts: Sizes differ (src 390 vs dst 386)
2023/03/30 09:49:52 NOTICE: hosts: Skipped copy as --dry-run is set (size 390)
2023/03/30 09:49:52 NOTICE:
Transferred:   	        390 B / 390 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.4s

2023/03/30 09:49:52 DEBUG : 6 go routines active
[felix@gemini ~]$ rclone copy /etc/hosts GD: -vv
2023/03/30 09:49:58 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2023/03/30 09:49:58 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "copy" "/etc/hosts" "GD:" "-vv"]
2023/03/30 09:49:58 DEBUG : Creating backend with remote "/etc/hosts"
2023/03/30 09:49:58 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2023/03/30 09:49:58 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2023/03/30 09:49:58 DEBUG : Creating backend with remote "GD:"
2023/03/30 09:49:59 DEBUG : hosts: Sizes differ (src 390 vs dst 386)
2023/03/30 09:50:00 DEBUG : hosts: md5 = 54a8094ba057c05f9d5901847bdb3f1e OK
2023/03/30 09:50:00 INFO  : hosts: Copied (replaced existing)
2023/03/30 09:50:00 INFO  :
Transferred:   	        390 B / 390 B, 100%, 389 B/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         1.7s

2023/03/30 09:50:00 DEBUG : 5 go routines active
[felix@gemini ~]$ rclone copy /etc/hosts GD: -vv
2023/03/30 09:50:01 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2023/03/30 09:50:01 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "copy" "/etc/hosts" "GD:" "-vv"]
2023/03/30 09:50:01 DEBUG : Creating backend with remote "/etc/hosts"
2023/03/30 09:50:01 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2023/03/30 09:50:01 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2023/03/30 09:50:01 DEBUG : Creating backend with remote "GD:"
2023/03/30 09:50:02 DEBUG : hosts: Size and modification time the same (differ by -534.225µs, within tolerance 1ms)
2023/03/30 09:50:02 DEBUG : hosts: Unchanged skipping
2023/03/30 09:50:02 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.4s

2023/03/30 09:50:02 DEBUG : 4 go routines active

There is a full example. Sorry as I wasn't quite awake on my post apparently when I informed on the steps.

Vivien_VOILLOT · March 30, 2023, 1:59pm

No problem

So, here is what I get if I want to copy in the intended destination folder :

rclone copy source:"G/01-ST/ST26.TODRIVE/12-Modeles/Liste extincteurs ST26.xlsx" ST26:"12-Modeles" -vv --dry-run
2023/03/30 15:58:19 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "source:G/01-ST/ST26.TODRIVE/12-Modeles/Liste extincteurs ST26.xlsx" "ST26:12-Modeles" "-vv" "--dry-run"]
2023/03/30 15:58:19 DEBUG : Creating backend with remote "source:G/01-ST/ST26.TODRIVE/12-Modeles/Liste extincteurs ST26.xlsx"
2023/03/30 15:58:19 DEBUG : Using config file from "C:\Users\Administrateur\AppData\Roaming\rclone\rclone.conf"
2023/03/30 15:58:19 DEBUG : found headers:
2023/03/30 15:58:19 DEBUG : fs cache: adding new entry for parent of "source:G/01-ST/ST26.TODRIVE/12-Modeles/Liste extincteurs ST26.xlsx", "source:G/01-ST/ST26.TODRIVE/12-Modeles"
2023/03/30 15:58:19 DEBUG : Creating backend with remote "ST26:12-Modeles"
2023/03/30 15:58:20 DEBUG : Liste extincteurs ST26.xlsx: Sizes differ (src 29237 vs dst 29214)
2023/03/30 15:58:20 NOTICE: Liste extincteurs ST26.xlsx: Skipped copy as --dry-run is set (size 28.552Ki)
2023/03/30 15:58:20 NOTICE:
Transferred: 28.552 KiB / 28.552 KiB, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 1.0s

2023/03/30 15:58:20 DEBUG : 9 go routines active

So it seems it would. And I gave it a try without the --dry-run and ... it worked O_o

Next :
In order to check my sanity, I wanted to try with other files in the same case. Three of them were in the very same folder :
Source : rclone lsl source:"G/01-ST/ST26.TODRIVE/12-Modeles" (extract for reading purposes) :
123571 2022-07-06 16:35:06.000000000 Organigramme couverture Référent SS.pdf
11673088 2023-03-16 17:12:31.000000000 Organigramme couverture Référent SS.pub
124310 2023-03-16 14:56:46.000000000 Organigramme couverture référent AC.pdf
Compared to :
Target : rclone lsl ST26:"12-Modeles" (also extract)
123571 2022-07-06 16:35:06.000000000 Organigramme couverture Référent SS.pdf
15125504 2022-11-16 07:53:39.000000000 Organigramme couverture Référent SS.pub
125022 2023-01-20 11:16:14.000000000 Organigramme couverture référent AC.pdf

I made the test for the first one : "Organigramme couverture Référent SS.pdf"
2023/03/30 16:08:07 DEBUG : Organigramme couverture Référent SS.pdf: Sizes identical
2023/03/30 16:08:07 DEBUG : Organigramme couverture Référent SS.pdf: Unchanged skipping

Then for the second "Organigramme couverture Référent SS.pub":
2023/03/30 16:08:30 DEBUG : Organigramme couverture Référent SS.pub: Sizes differ (src 11673088 vs dst 15125504)
2023/03/30 16:08:30 NOTICE: Organigramme couverture Référent SS.pub: Skipped copy as --dry-run is set (size 11.132Mi)

And finally for the last one "Organigramme couverture référent AC.pdf"
2023/03/30 16:11:01 DEBUG : Organigramme couverture référent AC.pdf: Sizes differ (src 124310 vs dst 125022)
2023/03/30 16:11:01 NOTICE: Organigramme couverture référent AC.pdf: Skipped copy as --dry-run is set (size 121.396Ki)

Ongoing Redaction

Animosity022 · March 30, 2023, 2:30pm

So that's saying the size does not match as it'll recopy.

Can you do the rclone lsl on each and validate the copy again?

Vivien_VOILLOT · March 30, 2023, 3:21pm

I have done it, removed the --dry-run and launched back the command : did the trick.
I should have set some bold in my previous post to mark it better

Anyway, I am now running a copy --dry-run on the whole source folder to target one. That was what I first did and wasn't ... effective.
So just looking if it still behaves the same (not working).
Two possibilities :

it works : ok, I launch it back and call it a day
it doesn't : and ... I would be schroedingerly suprised !
I would be because copying a subfolder seems to work and copying a file does work.
I wouldn't because it didn't worked this way the first time !

I wil let you know of the result.

system · April 29, 2023, 3:21pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.