Chunker remote works perfectly but misses resume functionality. It means that if your many chunks upload is interrupted and you start again everything have to be re-uploaded. Worse still is that previously uploaded chunks will be left behind... and stay there forever invisible for chunker remote unless you manually clean them in underlying remote.
It happens because all chunks during upload are stored with random suffix - only renamed when all file is uploaded (making all operation sort of atomic).
There is no rclone command to do it at the moment. So it has to be done by hand:
Let's say this is my setup:
box <-- crypt <-- chunker
I use custom chunk file format - so it might have to be adjusted for default or other formats:
[chunker]
type = chunker
name_format = *.rcc###
to keep files length as short as possible.
When chunker uploads "chunked" file it creates files like:
path/to/bigfile.ext.rcc001_ybc34a
path/to/bigfile.ext.rcc002_ybc34a
...
path/to/bigfile.ext.rcc099_ybc34a
and only when all file is finished it renames them to:
path/to/bigfile.ext.rcc001
path/to/bigfile.ext.rcc002
...
path/to/bigfile.ext.rcc099
so to find orphans (do this when there are no active uploads) I run:
rclone lsf --files-only --format p --recursive crypt: --include "{{(.*).rcc\d{3}_\S{6}}}" > orphaned_chunks.list
Please note you have to use remote below chunker (crypt in this case) as these files are invisible in chunker itself.
Inspect orphaned_chunks.list to make sure it contains only orphaned chunks - ones ending with rccXXX_abcdef in this example,
and delete them in one bulk operation:
rclone delete crypt: --files-from orphaned_chunks.list
as always with destructive operations it is advisable to run delete first with -vv --dry-run and triple check what is going to happen.
