Random thoughts on sync/dry run/missing-on-src/check etc

Help and Support
1

Hello all

I got to this point just trying to optimize performance:

rclone v1.62.2

  • os/version: ubuntu 18.04 (64 bit)
  • os/kernel: 4.15.0-208-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: none

I am doing (effectivly) a sync
dropbox user folders -> box "user folders" and a user onedrive -> box "user folders")

yet another sanity check if there is maybe a better way to do this with sync dry run + using missing on src and missing on dat ? (I guess it will bomb cause of the folder heigharchy?)

because the deleted itsems go into a folder on the same heigharchy what I have been doing is this:

run these two in parallel (exclude.txt is the "team folders")
(I know the lsf sort is kinda redundant this more for my purposes to see large files)

rclone lsf --format \"sp\"  dropbox:/ --dropbox-impersonate '$email' --fast-list --recursive --files-only -exclude-from exclude.txt  | sort -nr | awk -F';' '{print \$2}' > '$email'-dropbox.log"
rclone lsf "box:/User Folders/'$fn'" --fast-list --recursive --files-only --exclude-from exclude.txt > '$email'-box.log'

grep -Fxv -f '$email'-box.log '$email'-dropbox.log > '$email'-diff.txt'
grep -Fxv -f '$email'-dropbox.log '$email'-box.log > '$email'-deleted.txt'

rclone copy dropbox:/ "box:/User Folders/'$fn'" --dropbox-impersonate '$email' --order-by size,desc --transfers 25 --log-file '$email'-transfer.log --no-update-modtime --fast-list --ignore-checksum  --size-only --checkers 30 --files-from-raw '$email'-diff.txt -vvvP'

rclone move "box:/User Folders/'$fn'" "box:/User Folders/'$fn'/Deleted"  --transfers 25 --log-file '$email'-transfer.log --no-update-modtime --fast-list --ignore-checksum  --size-only --checkers 30 --files-from-raw '$email'-deleted.txt -vvvP'

thanks

2

You appear to be doing the equivalent of rclone sync --backup-dir so moving all deleted files into the /Deleted folder.

--backup-dir will move all overwritten files too but it is probably close enough so that maybe you could use that?

So something like this?

rclone sync dropbox:/ "box:/User Folders/'$fn'" --dropbox-impersonate '$email' --order-by size,desc --transfers 25 --log-file '$email'-transfer.log --no-update-modtime --fast-list --ignore-checksum  --size-only --checkers 30 --backup-dir "box:/User Folders/'$fn'/Deleted"

Is the problem that the folder is on the same heirachy?

I think you should be able to exclude it from the sync. (this is a relatively recent change in rclone)

$ rclone sync src dst --backup-dir dst/Deleted
2023/05/11 11:56:35 ERROR : Fatal error received - not attempting retries
2023/05/11 11:56:35 Failed to sync: destination and parameter to --backup-dir mustn't overlap

vs

$ rclone sync src dst --backup-dir dst/Deleted --exclude "/Deleted/"
$ # all OK
1 Like
3

CERTAINLY wasn’t aware of that change will try it out

1 Like
4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.