Retry on a few selected subdirectories

What is the problem you are having with rclone?

Hello,

I am periodically copying a large directory (+1million objects) from one cloud provider to another, and there are always a few errors. I created a script to parse out the errors from the log file, and then just copy the failed files one by one using the "--files-from" and the "--no-traverse" options. This works very well on files, but fails when the error relates to a directory.
Is there a clever way I can tell rclone to only copy a set of directories from a file without traversing anything else? (looking for something like a "--dirs-from" option I suppose).

I guess I could create a script that uses a loop and iterates through the failed directories and try to copy them one by one. There is usually somewhere between 40-200 only so could be feasible I suppose.
Any other thoughts or suggestions on how to best retry individual directories in a large copy job?

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.2

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 6.5.0-1015-azure (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.6
  • go/linking: static
  • go/tags: none

-->

Which cloud storage system are you using? (eg Google Drive)

Dropbox as Source Azure Blob as destination

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone: Version "v1.65.2" starting with parameters ["/usr/bin/rclonev1.65.2" "--progress" "--max-depth=-1" "--checkers=28" "--no-traverse" "--files-from=retry-files.txt" "--retries=1" "--low-level-retries=1" "
--azureblob-archive-tier-delete=true" "--dropbox-impersonate" "**REDACTED**" "--config" "/home/dbsyncuser/.config/rclone/rclone.conf" "--transfers=16" "--log-file=/home/dbsyncuser/logs//**REDACTED**-retry-2024-03-05.log" "--log-level" "DEBUG" "copy" "**REDACTED**:**REDACTED**" "**REDACTED**:**REDACTED**"]

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

source:
type = dropbox
client_id = XXX
client_secret = XXX
token = XXX

dest:
type = azureblob
account = XXX
key = XXX
access_tier = Archive
archive_tier_delete = true
chunk_size = 100M

A log from the command that you were trying to run with the -vv flag

2024/03/05 08:01:24 DEBUG : Creating backend with remote "**REDACTED**:**REDACTED**"
2024/03/05 08:01:24 DEBUG : Using config file from "/home/dbsyncuser/.config/rclone/rclone.conf"
2024/03/05 08:01:24 DEBUG : **REDACTED**: detected overridden config - adding "{jndPw}" suffix to name
2024/03/05 08:01:25 DEBUG : fs cache: renaming cache item "**REDACTED**:**REDACTED**" to be canonical "**REDACTED**{jndPw}:**REDACTED**"
2024/03/05 08:01:25 DEBUG : Creating backend with remote "**REDACTED**:**REDACTED**"
2024/03/05 08:01:25 DEBUG : **REDACTED**: detected overridden config - adding "{-GSw4}" suffix to name
2024/03/05 08:01:25 DEBUG : fs cache: renaming cache item "**REDACTED**:**REDACTED**" to be canonical "**REDACTED**{-GSw4}:**REDACTED**"
2024/03/05 08:01:28 ERROR : Dropbox root '**REDACTED**': error reading source root directory: is a directory not a file
2024/03/05 08:01:28 DEBUG : Azure container **REDACTED**: Waiting for checks to finish
2024/03/05 08:01:28 DEBUG : Azure container **REDACTED**: Waiting for transfers to finish
2024/03/05 08:01:28 ERROR : Attempt 1/1 failed with 1 errors and: is a directory not a file
2024/03/05 08:01:28 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.8s

2024/03/05 08:01:28 DEBUG : 7 go routines active
2024/03/05 08:01:28 INFO  : Dropbox root '**REDACTED**': Committing uploads - please wait...
2024/03/05 08:01:28 Failed to copy: is a directory not a file

I too would love to see a --dirs-from option for several reasons!

It doesn't really exist at the moment, but for your purposes you can get pretty close with filter rules. (--filter-from rather than --files-from.)

For example, say you want to include only directories A and B. Your file could look like:

+ /A/**
+ /B/**
- **

(The last line tells it to exclude everything else.)

And then for your command you'd add:

--filter-from /path/to/your/filters.txt

Thanks a lot!

I should easily be able to modify my script to produce a filter-file instead.

And expanding on this, I might even be able to handle both files and directories this way instead. Reading a bit on the filter-syntax, I am thinking I could strip the last character in the path that errors, and replace it with ** which should then match both for a file and a directory.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.