How to copy _SUCCESS files using rclone

Help and Support
1

What is the problem you are having with rclone?

I have a local directory /A, with many sub directories /1, /2, /3 ..., each with many data files and _SUCCESS mark files. I want to copy this local directory /A to the remote object store. During the copying process, I want the _SUCCESS file in the sub directory to be copied after all data files in the sub directory copied.
I have the following questions:

  1. how to copy _SUCCESS file in the end after all other files copied?
  2. What is the behavior of copying subfolders？Is it first rclone ls to get a flattern file list, and then transfer it concurrently in order?

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy config-a:a/b/c/ config-b:a/b/c/  -P --transfers=12 --dry-run --order-by size,descending
  • I tried to use order-by size,descending to make _SUCCESS the last one to copy. But affected by parallel tranfers, _SUCCESS file will be transferred with some data files in the final, and finished faster than the normal big data files.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1
- os/version: darwin 12.5.1 (64 bit)
- os/kernel: 21.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.20.6
- go/linking: dynamic
- go/tags: cmount
2

I think only way is to do this in two steps:

  1. copy src: dst: --exclude="_SUCCESS"
  2. only when (1) successful copy src: dst: --include="_SUCCESS"
1 Like
3

Thanks! But I have many sub-directories to copy. Once all data files in a sub-direcotry copied, a _SUCCESS file should be touched. Because object storage does not have a concept of directory, it seems a little hard to control the copy behavior per sub-directory.

4

You never mentioned what is your destination remote.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.