I'm looking for an improvement to "copy" on a specified set of objects/files that provides standardized (parsable) results from which could be inferred on whether each object was copied, skipped, or failed. Ideally, I'm looking for a solution that combines "copy --files-from" and "check --combined". I would like to specify a list of objects to copy with options "--files-from" and "--combined", and get a standardized output like option "--combined" does for a "check".
Current Limitations:
- rclone copy doesn't provide a standardized result. The only way to tell the status from each object is 1) by the stdout comments (not standardized/parsable), 2) by the return code which only works per file if only one file is provided, or 3) by querying the files with check before and after the fact.
- The --files-from option does not provide any result for skipped files (for both copy and check). (Is this a bug? I was expecting the file that was skipped to be mentioned in the stdout comments for copy, and report the mismatch in the check.)
- rclone check doesn't really run on individual files. If a single object or subset of objects is specified, it touches (though it skips over the actual compare) every object in a bucket. If a single object is specified and it is missing on the source, it treats the object as a bucket or folder. Also ran into a bug where rclone check on one object would hang while touching all objects in a bucket where there were many objects (1000) and at least one object (not the one specified) was being modified.
- rclone copy's "--error-on-no-transfer" option is only useful for getting result details if it is run a single object
Without this feature improvement, it looks like I need to choose between two options:
1- Copy each file individually with option "--error-on-no-transfer". Then run rclone check with "--combined" and "--files-from" options to double-check and refine the results (to know if an object was skipped because the file doesn't exist on source or because it already exists at the destination).
2- Copy with option "--files-from". Then run rclone check on the entire bucket with "--combined".
With the first option, I miss out on parallel processing that "--files-from" provides. With the second option, the results for each object in the copy are delayed until the very end, and files not in the --files-from list are checked unnecessarily.