Rclone sync does not delete files in destination

muammar · September 11, 2021, 12:38pm

What is the problem you are having with rclone?

I am syncing to google drive with the —exclude flag. I correctly see files uploading to the destination with directories I want to be excluded. Still, if I delete a directory that is supposed to sync (not in the excluded directories), rclone does not seem to delete it. According to the documentation, rclone sync should make destination identical to source.

What is your rclone version (output from `rclone version`)

 % rclone version                                                                                                                                        !8126
rclone v1.53.3-DEV
- os/arch: linux/amd64
- go version: go1.15.9

I use Debian Unstable.

Which cloud storage system are you using? (eg Google Drive)

Google Drive.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync --config="/home/user/.config/rclone/rclone.conf" -vPL \
    /home drive:pneuma/home \
    --exclude "user/HDD1/**" --exclude "user/HDD2/**" \
    --exclude "user/HDD3/**" --exclude "user/.dropbox/**" \
    --exclude "user/.dropbox-dist/**"

The rclone config contents with secrets removed.


  % cat .config/rclone/rclone.conf                                                                                                                        !8128
[drive]
type = drive
client_id = SOMETHING
client_secret = SOMETHING
scope = drive
token = {"access_token":"SOMETHING","token_type":"Bearer","refresh_token":"1//SOMETHING"}

A log from the command with the `-vv` flag

Log is huge because it is syncing a /home/ directory.

Animosity022 · September 11, 2021, 12:43pm

With no log file, we can't see what you are seeing so we have no idea what's going on.

muammar · September 11, 2021, 12:55pm

Ok, let me try to dump the log in a file to share somewhere.

Animosity022 · September 11, 2021, 12:57pm

Also, you have a very old version of rclone and a dev build.

You should update that.

https://rclone.org/install/

muammar · September 11, 2021, 1:00pm

I just put the latest binary on /usr/local/bin and rerun the command with the —log-file flag. I will update this later.

muammar · September 12, 2021, 12:09pm

I read that rclone won’t delete files/directories in the destination if there are any errors during the process. After grepping ERROR on the log file, I see there are some errors happening:

 % grep ERROR logfile_rclone.log                                                                                                                         !8386
2021/09/11 09:07:52 ERROR : muammar/miniconda3/envs/jupyterhub/lib/libstdc++.so.6.0.21: Listing error: symlink: stat /home/muammar/miniconda3/envs/jupyterhub/lib/libstdc++.so.6.0.21: no such file or directory
2021/09/11 09:37:53 ERROR : muammar/config.jedi/pulse/3edf5731b82741cb8a30face205d51b9-runtime: Listing error: symlink: stat /home/muammar/config.jedi/pulse/3edf5731b82741cb8a30face205d51b9-runtime: no such file or directory
2021/09/11 19:43:57 ERROR : muammar/jupyterhub/jupyterhub.sqlite: corrupted on transfer: md5 hash differ "eb27c79319a327a37171d458750511f0" vs "cc07150cd2fdf3e9700e6529f0be63af"
2021/09/11 19:50:34 ERROR : muammar/miniconda3/envs/jupyterhub/lib/libstdc++.so.6.0.21: Listing error: symlink: stat /home/muammar/miniconda3/envs/jupyterhub/lib/libstdc++.so.6.0.21: no such file or directory
2021/09/11 20:19:17 ERROR : muammar/config.jedi/pulse/3edf5731b82741cb8a30face205d51b9-runtime: Listing error: symlink: stat /home/muammar/config.jedi/pulse/3edf5731b82741cb8a30face205d51b9-runtime: no such file or directory
2021/09/11 21:47:36 ERROR : muammar/miniconda3/pkgs/libgcc-7.2.0-h69d50b8_2/lib/libstdc++.so.6.0.21: Listing error: symlink: stat /home/muammar/miniconda3/pkgs/libgcc-7.2.0-h69d50b8_2/lib/libstdc++.so.6.0.21: no such file or directory
2021/09/11 22:03:42 ERROR : muammar/miniconda3/pkgs/python-3.8.11-h12debd9_0_cpython/compiler_compat/ld: Listing error: symlink: stat /home/muammar/miniconda3/pkgs/python-3.8.11-h12debd9_0_cpython/compiler_compat/ld: no such file or directory
2021/09/11 22:03:57 ERROR : muammar/miniconda3/pkgs/python-3.9.6-h12debd9_1/compiler_compat/ld: Listing error: symlink: stat /home/muammar/miniconda3/pkgs/python-3.9.6-h12debd9_1/compiler_compat/ld: no such file or directory
2021/09/11 22:37:52 ERROR : Google drive root 'BackupsandConfigs/Backups/pneuma/home': not deleting files as there were IO errors
2021/09/11 22:37:52 ERROR : Google drive root 'BackupsandConfigs/Backups/pneuma/home': not deleting directories as there were IO errors
2021/09/11 22:37:52 ERROR : Attempt 1/3 failed with 6 errors and: symlink: stat /home/muammar/miniconda3/pkgs/python-3.9.6-h12debd9_1/compiler_compat/ld: no such file or directory
2021/09/11 22:45:52 ERROR : muammar/miniconda3/envs/jupyterhub/lib/libstdc++.so.6.0.21: Listing error: symlink: stat /home/muammar/miniconda3/envs/jupyterhub/lib/libstdc++.so.6.0.21: no such file or directory
2021/09/11 23:15:24 ERROR : muammar/config.jedi/pulse/3edf5731b82741cb8a30face205d51b9-runtime: Listing error: symlink: stat /home/muammar/config.jedi/pulse/3edf5731b82741cb8a30face205d51b9-runtime: no such file or directory
2021/09/12 00:42:31 ERROR : muammar/miniconda3/pkgs/libgcc-7.2.0-h69d50b8_2/lib/libstdc++.so.6.0.21: Listing error: symlink: stat /home/muammar/miniconda3/pkgs/libgcc-7.2.0-h69d50b8_2/lib/libstdc++.so.6.0.21: no such file or directory
2021/09/12 00:58:00 ERROR : muammar/miniconda3/pkgs/python-3.8.11-h12debd9_0_cpython/compiler_compat/ld: Listing error: symlink: stat /home/muammar/miniconda3/pkgs/python-3.8.11-h12debd9_0_cpython/compiler_compat/ld: no such file or directory
2021/09/12 00:58:15 ERROR : muammar/miniconda3/pkgs/python-3.9.6-h12debd9_1/compiler_compat/ld: Listing error: symlink: stat /home/muammar/miniconda3/pkgs/python-3.9.6-h12debd9_1/compiler_compat/ld: no such file or directory
2021/09/12 01:31:29 ERROR : Google drive root 'BackupsandConfigs/Backups/pneuma/home': not deleting files as there were IO errors
2021/09/12 01:31:29 ERROR : Google drive root 'BackupsandConfigs/Backups/pneuma/home': not deleting directories as there were IO errors
2021/09/12 01:31:29 ERROR : Attempt 2/3 failed with 6 errors and: not deleting files as there were IO errors
2021/09/12 01:39:42 ERROR : muammar/miniconda3/envs/jupyterhub/lib/libstdc++.so.6.0.21: Listing error: symlink: stat /home/muammar/miniconda3/envs/jupyterhub/lib/libstdc++.so.6.0.21: no such file or directory
2021/09/12 02:08:56 ERROR : muammar/config.jedi/pulse/3edf5731b82741cb8a30face205d51b9-runtime: Listing error: symlink: stat /home/muammar/config.jedi/pulse/3edf5731b82741cb8a30face205d51b9-runtime: no such file or directory
2021/09/12 03:35:58 ERROR : muammar/miniconda3/pkgs/libgcc-7.2.0-h69d50b8_2/lib/libstdc++.so.6.0.21: Listing error: symlink: stat /home/muammar/miniconda3/pkgs/libgcc-7.2.0-h69d50b8_2/lib/libstdc++.so.6.0.21: no such file or directory
2021/09/12 03:51:21 ERROR : muammar/miniconda3/pkgs/python-3.8.11-h12debd9_0_cpython/compiler_compat/ld: Listing error: symlink: stat /home/muammar/miniconda3/pkgs/python-3.8.11-h12debd9_0_cpython/compiler_compat/ld: no such file or directory
2021/09/12 03:51:36 ERROR : muammar/miniconda3/pkgs/python-3.9.6-h12debd9_1/compiler_compat/ld: Listing error: symlink: stat /home/muammar/miniconda3/pkgs/python-3.9.6-h12debd9_1/compiler_compat/ld: no such file or directory
2021/09/12 04:24:57 ERROR : Google drive root 'BackupsandConfigs/Backups/pneuma/home': not deleting files as there were IO errors
2021/09/12 04:24:57 ERROR : Google drive root 'BackupsandConfigs/Backups/pneuma/home': not deleting directories as there were IO errors
2021/09/12 04:24:57 ERROR : Attempt 3/3 failed with 6 errors and: not deleting files as there were IO errors
(base)

Is there any flag to ignore those?

muammar · September 12, 2021, 12:10pm

I found on Github this:

--ignore-errors delete even if there are I/O errors

I will try this out and report back.

Animosity022 · September 12, 2021, 12:17pm

A full debug log would be needed to help out as noted before..

muammar · September 12, 2021, 12:36pm

Animosity022 · September 12, 2021, 2:31pm

That's a snippet of a log and not a full one.
That's not a debug log.

muammar · September 12, 2021, 2:48pm

That’s the log that rclone produces with —log-file /tmp/rclone_$(date +%d%b%Y).log.

Ok, I saw that default log level is NOTICE. I changed it to DEBUG. I will update later when the thing finishes.

jwink3101 · September 13, 2021, 7:58pm

Rclone won't do deleted when there are errors.

But is there a reason you are tying to back up trhings like miniconda and jupyter? Given that you seem content to ignore the errors, you cannot later rely on a restore and will have to reinstall anyway. Which is fine and easy to do. Why not just exclude them?

muammar · September 13, 2021, 8:13pm

This is a good point — why don’t you exclude the miniconda directory? I should do that because there isn’t any reason why I should back up those. I think I was just lazy letting that around. I will exclude it and try again because, to be honest, most of the errors came from there. I also realized there were many broken symlinks on some directories. I removed those, too.

muammar · September 14, 2021, 11:52am

After removing the broken symlinks, I had on some directories syncing to google drive,

sudo find -xtype l -delete

(be careful with command above), and excluding the miniconda3 directory I could see this line (light):

2021-09-13 17:23:10 INFO : muammar/notebooks/Gian_Muammar/02_VAE/02_VAE.reproduce_kldivergence/runs/events.out.tfevents.1631477376.pneuma.77987.0: Deleted

Thanks!

Animosity022 · September 14, 2021, 12:00pm

The reason I kept asking for a full debug log was to get that info out of you.

The older version doesn't give the same message as once you updated, the full log would show the:

version
better/improved notification

Like:

felix@gemini:~/test$ rclone copy /home/felix/test /home/felix/test2 -vv -L
2021/09/14 07:58:27 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2021/09/14 07:58:27 DEBUG : Setting --user-agent "animosityapp" from environment variable RCLONE_USER_AGENT="animosityapp"
2021/09/14 07:58:27 DEBUG : Setting --rc-user "felix" from environment variable RCLONE_RC_USER="felix"
2021/09/14 07:58:27 DEBUG : Setting --rc-pass "felix" from environment variable RCLONE_RC_PASS="felix"
2021/09/14 07:58:27 DEBUG : Setting default for drive-pacer-min-sleep="10ms" from environment variable RCLONE_DRIVE_PACER_MIN_SLEEP
2021/09/14 07:58:27 DEBUG : Setting default for drive-pacer-burst="1000" from environment variable RCLONE_DRIVE_PACER_BURST
2021/09/14 07:58:27 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "copy" "/home/felix/test" "/home/felix/test2" "-vv" "-L"]
2021/09/14 07:58:27 DEBUG : Creating backend with remote "/home/felix/test"
2021/09/14 07:58:27 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2021/09/14 07:58:27 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/09/14 07:58:27 DEBUG : fs cache: renaming cache item "/home/felix/test" to be canonical "local{12rtk}:/home/felix/test"
2021/09/14 07:58:27 DEBUG : Creating backend with remote "/home/felix/test2"
2021/09/14 07:58:27 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2021/09/14 07:58:27 DEBUG : fs cache: renaming cache item "/home/felix/test2" to be canonical "local{12rtk}:/home/felix/test2"
2021/09/14 07:58:27 ERROR : testing: Listing error: symlink: stat /home/felix/test/testing: no such file or directory
2021/09/14 07:58:27 DEBUG : Local file system at /home/felix/test2: Waiting for checks to finish
2021/09/14 07:58:27 DEBUG : Local file system at /home/felix/test2: Waiting for transfers to finish
2021/09/14 07:58:27 INFO  : There was nothing to transfer
2021/09/14 07:58:27 ERROR : Can't retry any of the errors - not attempting retries
2021/09/14 07:58:27 INFO  :
Transferred:   	          0 / 0 Byte, -, 0 Byte/s, ETA -
Errors:                 1 (no need to retry)
Elapsed time:         0.0s

2021/09/14 07:58:27 DEBUG : 3 go routines active
2021/09/14 07:58:27 Failed to copy: symlink: stat /home/felix/test/testing: no such file or directory

system · September 17, 2021, 12:00pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.