cberni
(Cristiano Albiero Berni)
February 24, 2023, 7:50pm
1
opened 12:22PM - 14 Mar 22 UTC
<!--
We understand you are having a problem with rclone; we want to help you … with that!
**STOP and READ**
**YOUR POST WILL BE REMOVED IF IT IS LOW QUALITY**:
Please show the effort you've put into solving the problem and please be specific.
People are volunteering their time to help! Low effort posts are not likely to get good answers!
If you think you might have found a bug, try to replicate it with the latest beta (or stable).
The update instructions are available at https://rclone.org/commands/rclone_selfupdate/
If you can still replicate it or just got a question then please use the rclone forum:
https://forum.rclone.org/
for a quick response instead of filing an issue on this repo.
If nothing else helps, then please fill in the info below which helps us help you.
**DO NOT REDACT** any information except passwords/keys/personal info.
You should use 3 backticks to begin and end your paste to make it readable.
Make sure to include a log obtained with '-vv'.
You can also use '-vv --log-file bug.log' and a service such as https://pastebin.com or https://gist.github.com/
Thank you
The Rclone Developers
-->
#### The associated forum post URL from `https://forum.rclone.org`
https://forum.rclone.org/t/rclone-move-order-by-and-check-first-with-no-perfect-ordering/29586
#### What is the problem you are having with rclone?
rclone move with --checkers more than 1 is not deleting in perfect order --order-by modtime,ascending if the files are already in the remote backend
is there a way to have multiple checkers but only delete if order is perfect like the trasnfer?
#### What is your rclone version (output from `rclone version`)
rclone v1.57.0
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.13.0-1019-gcp (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none
#### Which cloud storage system are you using? (e.g. Google Drive)
google cloud storage
#### The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`)
rclone move /local remote:bucket --include-from include-file.txt --order-by modtime,ascending --max-delete 1078750 --cutoff-mode soft --check-first --delete-empty-src-dirs --no-traverse --fast-list -vv
#### A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`)
022/03/04 14:45:29 DEBUG : Setting --checkers "16" from environment variable RCLONE_CHECKERS="16"
2022/03/04 14:45:29 DEBUG : Setting --transfers "8" from environment variable RCLONE_TRANSFERS="8"
2022/03/04 14:45:29 DEBUG : Setting --config "/etc/rclone.conf" from environment variable RCLONE_CONFIG="/etc/rclone.conf"
2022/03/04 14:45:29 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "move" "/local" "remote:bucket" "--include-from" "/etc/include-file.txt" "--order-by" "modtime,ascending" "--max-delete" "1228331" "--cutoff-mode" "soft" "--check-first" "--delete-empty-src-dirs" "--no-traverse" "--fast-list" "-vv"]
2022/03/04 14:45:29 DEBUG : Creating backend with remote "/local"
2022/03/04 14:45:29 DEBUG : Using config file from "/etc/rclone.conf"
2022/03/04 14:45:29 DEBUG : Creating backend with remote "remote:bucket"
2022/03/04 14:45:29 INFO : GCS bucket bucket: Running all checks before starting transfers
#### How to use GitHub
* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.
What is the problem you are having with rclone?
rclone move is not respecting --order-by modtime,ascending with --check-first
the files are not moving on perfect ordering...
any ideia? is it a bug?
Run the command 'rclone version' and share the full output of the command.
rclone v1.57.0
os/version: ubuntu 20.04 (64 bit)
os/kernel: 5.13.0-1015-gcp (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.17.2
go/linking: static
go/tags: none
Which cloud storage system are you using? (eg Go…
rclone is an excellent tool that we use. but we are in need of a little urgent adjustment. I will explain below:
example: the files with its created time
local | remote
file16h | flle16h
file17h | file17h
file18h |
file19h |
file20h | file20h
command used: rclone move --check-first --order-by modtime,ascending
the files with created time 18h and 19h that are not in the remote backend yet will be copied respectind the --check-first
and the --order-by modtime,ascending
is applied perfectly.
the files with created time 16h, 17h and 20h that are ALREADY in the remote backend will just be deleted and are not respecting the perfect ordering. The file created after (file17h) can be deleted first than file before (file16h). file20h can be deleted before everybody.
We use this command with --max-delete
and lot of files. So the files that should be deleted first are the oldest ones.
The files should be moved as follows whether or not they already exist in remote backend: file16h > file17h > file18h > file19h > file20h
We write a code bellow that could help. What would be the best way to try to improve this?
committed 05:43PM - 25 Jul 22 UTC
ncw
(Nick Craig-Wood)
March 8, 2023, 1:24pm
4
I've implemented the basic idea from the code above but I re-wrote it as I thought of a way of doing it much more neatly. I've fixed the transfer stats, made tests and written docs too
Please give this a go
v1.62.0-beta.6784.3b28ea0ba.fix-6033-delete-perfect-ordering on branch fix-6033-delete-perfect-ordering (uploaded in 15-30 mins)
1 Like