With Rclone copy, HTML file content change not detected, tried checksum, update

[rafraf (raf)]

What is the problem you are having with rclone?

I have an html file that is being updated on source, but rclone copy does not work on destination file. The destination file content is different from origin, but script does not replace it. The filename stays the same but the contents are different.

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 6.5.0-41-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.9
- go/linking: static
- go/tags: none

2024/09/10 16:38:59 NOTICE: Successfully updated rclone from version v1.58.1 to version v1.68.0
root@9d920ac607e1:/workspace

Which cloud storage system are you using? (eg Google Drive)

dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --checksum --ignore-times $(pwd) dbx:/studio/ai/output/runpod_swarm/local/Grids/test3 -v --ignore-existing

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

2024/09/10 16:06:21 INFO  : 41/32.metadata.js: Copied (new)
2024/09/10 16:06:21 INFO  : 41/32.png: Copied (new)
2024/09/10 16:06:21 INFO  : 
Transferred:        1.000 MiB / 1.000 MiB, 100%, 512.094 KiB/s, ETA 0s
Checks:                14 / 14, 100%
Transferred:            2 / 2, 100%
Elapsed time:         4.4s

As a result, i have to delete the html in destination local path in order to completely recopy it. BUt i wish another flag would automatically recognise the difference in size for instance. Different time (update) doesnt work for some reason. checksum neither.

[Animosity022]

That's an old version, you'd want to update it.

Find the problematic file.

Run a copy with the one problematic file with -vv to produce a debug log.

The output you shared wouldn't give a reason as to why it was copied.

[rafraf (raf)]

updated:

root@507195a522c9:/workspace/StableSwarmUI/Output/local/Grids# rclone self-update
2024/09/10 22:10:33 NOTICE: Successfully updated rclone from version v1.58.1 to version v1.68.0

Ok ran the command with -vv:

root@507195a522c9:/workspace/StableSwarmUI/Output/local/Grids/0# rclone copy --checksum --ignore-times $(pwd) dbx:/studio/ai/output/runpod_swarm/local/Grids/0 -vv --ignore-existing
2024/09/10 22:12:22 DEBUG : rclone: Version "v1.68.0" starting with parameters ["rclone" "copy" "--checksum" "--ignore-times" "/workspace/StableSwarmUI/Output/local/Grids/0" "dbx:/studio/ai/output/runpod_swarm/local/Grids/0" "-vv" "--ignore-existing"]
2024/09/10 22:12:22 DEBUG : Creating backend with remote "/workspace/StableSwarmUI/Output/local/Grids/0"
2024/09/10 22:12:22 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/09/10 22:12:22 DEBUG : Creating backend with remote "dbx:/studio/ai/output/runpod_swarm/local/Grids/0"
2024/09/10 22:12:22 DEBUG : Dropbox root '': Using root namespace "2578308515"
2024/09/10 22:12:23 DEBUG : fs cache: renaming cache item "dbx:/studio/ai/output/runpod_swarm/local/Grids/0" to be canonical "dbx:studio/ai/output/runpod_swarm/local/Grids/0"
2024/09/10 22:12:23 DEBUG : bootstrap.min.css: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : data.js: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : image_metadata-log.ldb: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : image_metadata.ldb: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : bootstrap.bundle.min.js: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : proc.js: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : styles-user.css: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : styles.css: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : swarmui_gridgen_local.js: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : placeholder.png: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : jquery.min.js: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : jsgif.js: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : swarm_save_config.json: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : index.html: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : 22/21.metadata.js: Need to transfer - File not found at Destination
2024/09/10 22:12:23 DEBUG : 22/21.png: Need to transfer - File not found at Destination
2024/09/10 22:12:23 DEBUG : 11/21.metadata.js: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : 11/21.png: Destination exists, skipping
2024/09/10 22:12:23 DEBUG : Dropbox root 'studio/ai/output/runpod_swarm/local/Grids/0': Waiting for checks to finish
2024/09/10 22:12:23 DEBUG : Dropbox root 'studio/ai/output/runpod_swarm/local/Grids/0': Waiting for transfers to finish
2024/09/10 22:12:23 DEBUG : 22/21.png: Uploading chunk 1/1
2024/09/10 22:12:23 DEBUG : 22/21.metadata.js: Uploading chunk 1/1
2024/09/10 22:12:24 DEBUG : 22/21.png: Uploading chunk 2/1
2024/09/10 22:12:24 DEBUG : 22/21.metadata.js: Uploading chunk 2/1
2024/09/10 22:12:24 DEBUG : Dropbox root 'studio/ai/output/runpod_swarm/local/Grids/0': Adding "22/21.png" to batch
2024/09/10 22:12:25 DEBUG : Dropbox root 'studio/ai/output/runpod_swarm/local/Grids/0': Adding "22/21.metadata.js" to batch
2024/09/10 22:12:25 DEBUG : Dropbox root 'studio/ai/output/runpod_swarm/local/Grids/0': Batch idle for 500ms so committing
2024/09/10 22:12:25 DEBUG : Dropbox root 'studio/ai/output/runpod_swarm/local/Grids/0': Committing sync batch length 2 starting with: 22/21.png
2024/09/10 22:12:26 DEBUG : Dropbox root 'studio/ai/output/runpod_swarm/local/Grids/0': Committed sync batch length 2 starting with: 22/21.png
2024/09/10 22:12:26 DEBUG : 22/21.png: dropbox = 84cf55c6a2952d65259d50ce40d84e906edb04ecd8952b1d578add35c10a592c OK
2024/09/10 22:12:26 INFO  : 22/21.png: Copied (new)
2024/09/10 22:12:26 DEBUG : 22/21.metadata.js: dropbox = 02af5e7415745d05875a506b87725321e8c362a47f3e0a8eb1223848a0029881 OK
2024/09/10 22:12:26 INFO  : 22/21.metadata.js: Copied (new)
2024/09/10 22:12:26 INFO  : 
Transferred:        1.335 MiB / 1.335 MiB, 100%, 455.755 KiB/s, ETA 0s
Checks:                16 / 16, 100%
Transferred:            2 / 2, 100%
Elapsed time:         4.6s

2024/09/10 22:12:26 DEBUG : 9 go routines active
2024/09/10 22:12:26 INFO  : Dropbox root 'studio/ai/output/runpod_swarm/local/Grids/0': Committing uploads - please wait...
root@507195a522c9:/workspace/StableSwarmUI/Output/local/Grids/0# 

the index.html did not update. I can see from verbose that index.html was copied new but the web page produced does not change from previous state.

Here's a screenshot of the folder structure in which is supposed to be altered

If, however, i delete the destination index.html and rerun command, then web page does update correctly

[Animosity022]

@rafraf

2024/09/10 22:12:23 DEBUG : index.html: Destination exists, skipping

I only see that line and it was not copied because you have said to

      --ignore-existing                             Skip all files that exist on destination

If a file exists on the destination, you've asked rclone to skip it regardless of if it changes or not.