Hello,
I'm using rclone to sync data to Openstack swift, so i use the option --swift-chunk-size.
Work great.
I've just have a problem when it's used with backup-dir option, i have the errors:
Failed to copy: Too Large Object
Not deleting source as copy failed: Too Large Object
Couldn't move into backup dir: Too Large Object
How can we can use chunks for the backup-dir too ? (i've not fond in the doc)
What version of rclone are you running?
What command are you running?
Can you run the same thing with debug -vv and share the log?
I'm running the v1.51.0.
My command is:
rclone sync $SRC $DEST_CURRENT --exclude-from $SRC_IGNORED --copy-links --backup-dir $DEST_ARCHIVES --suffix "_$TIMESTAMP.bak" --retries 1 --swift-chunk-size 4.5G --config=$DEST_CONFIG
Can you run the command with -vv and share the debug log?
For the initial upload of the file (Working fine):
2020/05/09 14:50:24 DEBUG : FW/10Gio.dat: Uploading segment file "FW/10Gio.dat/1589028624.117945697/10737418240/00000000" into "current_segments"
* FW/10Gio.dat: 10% /10G, 16.992M/s, 9m1s
* FW/10Gio.dat: 20% /10G, 17.860M/s, 7m37s
* FW/10Gio.dat: 31% /10G, 18.860M/s, 6m9s
* FW/10Gio.dat: 44% /10G, 23.081M/s, 4m5s
2020/05/09 14:54:51 DEBUG : FW/10Gio.dat: Uploading segment file "FW/10Gio.dat/1589028624.117945697/10737418240/00000001" into "current_segments"
* FW/10Gio.dat: 57% /10G, 22.687M/s, 3m12s
* FW/10Gio.dat: 69% /10G, 22.147M/s, 2m22s
* FW/10Gio.dat: 80% /10G, 19.204M/s, 1m42s
* FW/10Gio.dat: 92% /10G, 19.092M/s, 38s
2020/05/09 14:59:00 DEBUG : FW/10Gio.dat: Uploading segments into "current_segments" seems done (EOF)
2020/05/09 14:59:01 DEBUG : FW/10Gio.dat: Returning empty Md5sum for swift large object
2020/05/09 14:59:01 INFO : FW/10Gio.dat: Copied (new)
When i delete the file and expect to copy in backup-dir:
2020/05/09 15:02:47 ERROR : FW/10Gio.dat: Failed to copy: Too Large Object
2020/05/09 15:02:47 ERROR : FW/10Gio.dat: Not deleting source as copy failed: Too Large Object
2020/05/09 15:02:47 ERROR : FW/10Gio.dat: Couldn't move into backup dir: Too Large Object
2020/05/09 15:02:47 ERROR : Swift container current: not deleting directories as there were IO errors
2020/05/09 15:02:47 ERROR : Attempt 1/1 failed with 2 errors and: failed to delete 1 files
I'm guessing the backup move isn't chunking it. So it's over 5GB and fails. Might be a bug.
Can you try to just rclone move a file as well with debug? Just like to isolate it and do it on the remote like:
rclone move remote:bigfile remote:bigfile.new -vv
or something along those lines.
A move doesn't work:
rclone move myopenstack:current/FW/10Gio.dat myopenstack:deleted_files/FW/10Gio.dat -vv --config=myopenstack.config
2020/05/09 15:23:55 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "move" "myopenstack:current/FW/10Gio.dat" "myopenstack:deleted_files/FW/10Gio.dat" "-vv" "--config=myopenstack.config"]
2020/05/09 15:23:55 DEBUG : Using config file from "/root/.rclone/myopenstack.config"
2020/05/09 15:23:58 DEBUG : 10Gio.dat: Need to transfer - File not found at Destination
2020/05/09 15:23:59 ERROR : 10Gio.dat: Failed to copy: Too Large Object
2020/05/09 15:23:59 ERROR : 10Gio.dat: Not deleting source as copy failed: Too Large Object
2020/05/09 15:23:59 ERROR : Attempt 1/3 failed with 1 errors and: Too Large Object
2020/05/09 15:23:59 DEBUG : 10Gio.dat: Need to transfer - File not found at Destination
2020/05/09 15:23:59 ERROR : 10Gio.dat: Failed to copy: Too Large Object
2020/05/09 15:23:59 ERROR : 10Gio.dat: Not deleting source as copy failed: Too Large Object
2020/05/09 15:23:59 ERROR : Attempt 2/3 failed with 1 errors and: Too Large Object
2020/05/09 15:23:59 DEBUG : 10Gio.dat: Need to transfer - File not found at Destination
2020/05/09 15:24:00 ERROR : 10Gio.dat: Failed to copy: Too Large Object
2020/05/09 15:24:00 ERROR : 10Gio.dat: Not deleting source as copy failed: Too Large Object
2020/05/09 15:24:00 ERROR : Attempt 3/3 failed with 1 errors and: Too Large Object
2020/05/09 15:24:00 Failed to move: Too Large Object
And the same logs appears if we move inside the same container:
2020/05/09 15:28:19 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "move" "myopenstack:current/FW/10Gio.dat" "myopenstack:current/FW/10Gio5.dat" "-vv" "--config=myopenstack.config"]
2020/05/09 15:28:19 DEBUG : Using config file from "/root/.rclone/myopenstack.config"
2020/05/09 15:28:22 DEBUG : 10Gio.dat: Need to transfer - File not found at Destination
2020/05/09 15:28:23 ERROR : 10Gio.dat: Failed to copy: Too Large Object
2020/05/09 15:28:23 ERROR : 10Gio.dat: Not deleting source as copy failed: Too Large Object
2020/05/09 15:28:23 ERROR : Attempt 1/3 failed with 1 errors and: Too Large Object
2020/05/09 15:28:23 DEBUG : 10Gio.dat: Need to transfer - File not found at Destination
2020/05/09 15:28:23 ERROR : 10Gio.dat: Failed to copy: Too Large Object
2020/05/09 15:28:23 ERROR : 10Gio.dat: Not deleting source as copy failed: Too Large Object
2020/05/09 15:28:23 ERROR : Attempt 2/3 failed with 1 errors and: Too Large Object
2020/05/09 15:28:23 DEBUG : 10Gio.dat: Need to transfer - File not found at Destination
2020/05/09 15:28:24 ERROR : 10Gio.dat: Failed to copy: Too Large Object
2020/05/09 15:28:24 ERROR : 10Gio.dat: Not deleting source as copy failed: Too Large Object
2020/05/09 15:28:24 ERROR : Attempt 3/3 failed with 1 errors and: Too Large Object
2020/05/09 15:28:24 Failed to move: Too Large Object
Got it.
The issue is already logged here:
You can follow that issue for once it gets tested / resolved.
How can make to move from "client side" and not from server side ?
I don't think you have the option to disable it.
For move command, you could make another remote with a different name and move that way.
Not much you can do with the backup-dir though.
We can disable it via --disable move,copy
But it's not possible to use backup-dir without server-side copy or in an other destination:
2020/05/09 16:00:16 Failed to sync: can't use --backup-dir on a remote which doesn't support server side move or copy
2020/05/09 15:51:20 Failed to sync: parameter to --backup-dir has to be on the same remote as destination
Does that work for Swift? Did you test it?
Yes its working:
2020/05/09 16:26:33 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "move" "myopenstack:current/FW/10Gio.dat" "myopenstack:current/FW/10Gio5.dat" "-vv" "--config=myopenstack.config" "--disable" "move,copy"]
2020/05/09 16:26:33 DEBUG : Using config file from "/root/.rclone/myopenstack.config"
2020/05/09 16:26:33 DEBUG : Reset feature "move"
2020/05/09 16:26:33 DEBUG : Reset feature "copy"
2020/05/09 16:26:34 DEBUG : Reset feature "move"
2020/05/09 16:26:34 DEBUG : Reset feature "copy"
2020/05/09 16:26:35 DEBUG : 10Gio.dat: Need to transfer - File not found at Destination
2020/05/09 16:26:36 DEBUG : 10Gio.dat: Uploading segment file "FW/10Gio5.dat/10Gio.dat/1589034396.096224295/10737418240/00000000" into "current_segments"
But we can't use backup-dir without server side copy or move enabled
Sweet!
Learn something new every day as I thought those were for Google, but they appear to be global. I'll remember that as well.
To make server side move or copy for files bigger than 5GB we'll need #515 fixed 
I'd love some help with this if someone wanted to volunteer?
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.