What is the problem you are having with rclone?
We are trying to copy data from an io.ReadCloser from a golang script to an OpenStack swift object store.
Sometimes it does work and sometimes we are running into an error. The copying takes around 12h-14h.
We are using the following method:
operations.Rcat(ctx, ftarget, dstFileName, pipe, time.Now(), nil)
which gets the cmd.StdoutPipe() from the backup command.
2024/06/12 09:26:25 DEBUG : XXX: Uploading segments into "XXX" seems done (EOF)
2024/06/12 09:26:26 DEBUG : XXX: Sizes differ (src 281129647622 vs dst 28800318982)
2024/06/12 09:26:26 ERROR : XXX: corrupted on transfer
Sometimes it runs through successfully and sometimes it fails.
Run the command 'rclone version' and share the full output of the command.
github.com/rclone/rclone v1.66.0
go 1.22.2
docker image: alpine:latest
Which cloud storage system are you using? (eg Google Drive)
We are using OpenStack Swift
The command you were trying to run (eg rclone copy /tmp remote:tmp)
We are using the rclone directly imported as golang library and are using the following method from the operations package:
// Rcat reads data from the Reader until EOF and uploads it to a file on remote
//
// in is closed at the end of the transfer
func Rcat(ctx context.Context, fdst fs.Fs, dstFileName string, in io.ReadCloser, modTime time.Time, meta fs.Metadata) (dst fs.Object, err error) {
return rcatSrc(ctx, fdst, dstFileName, in, modTime, meta, nil)
}
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[XXX]
type = swift
env_auth = true
user = XXX
key = XXX
auth = https://XXX:443/v3
domain = XXX
tenant = XXX
tenant_domain = XXX
auth_version = 3
endpoint_type = internal
A log from the command that you were trying to run with the -vv flag
Logs from failed runs.
2024/06/12 09:22:50 DEBUG : XXX: Uploading segment file "XXX/-1/00000052" into "XXX"
2024/06/12 09:26:25 DEBUG : XXX: Uploading segments into "XXX_segments" seems done (EOF)
2024/06/12 09:26:26 DEBUG : XXX: Sizes differ (src 281129647622 vs dst 28800318982)
2024/06/12 09:26:26 ERROR : XXX: corrupted on transfer
another log from another try:
2024/06/12 12:31:04 DEBUG : XXX: Uploading segments into "XXX" seems done (EOF)
2024/06/12 12:31:04 DEBUG : XXX: Sizes differ (src 318285130095 vs dst 98168056175)
2024/06/12 12:31:04 ERROR : XXX: corrupted on transfer
