Using --s3-disable-checksum will stop the checksum being calculated which is not what you want. The checksum is calculated by default if you use a post rclone 1.40.
So if you want those files to have checksums then deleting them and reuploading them with v1.42 and without the --s3-disable-checksum flag.
I suggest you just try one and see what happens.
So if I identify a file that fails to cryptcheck, delete it, re-upload is using rclone 1.42 without the --s3-disable-checksum flag the file should now cryptcheck successfully?
The easiest way to identify those files would be to use rclone md5sum and look for the files with missing md5sum.
Then re-uploading them without the --s3-disable-checksum should do the trick.
Hi @ncw,
I identified a file that was failing to cryptcheck and deleted it using the following command.
rclone delete encryptedremote:/path/to/file.xxx
I then re-uploaded the file using rclone 1.44 but it's still missing a md5sum and failing to cryptcheck.
Hmm, try deleting it and uploading it again with --s3-chunk-size 1000T - does that work?
I dropped it down to 10M and got it to run successfully since the file is just over 5 MB.
0 differences found
It panic with a size of 1000T.
panic: runtime error: makeslice: len out of range
goroutine 61 [running]:
github.com/ncw/rclone/vendor/github.com/aws/aws-sdk-go/service/s3/s3manager.(*uploader).init.func1(0xc0003b88e0, 0x0)
/Users/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go:487 +0x43
sync.(*Pool).Get(0xc0003b88e0, 0x1f2d620, 0xc0002ee1e0)
/Users/travis/.gimme/versions/go1.11.1.darwin.amd64/src/sync/pool.go:151 +0xa6
github.com/ncw/rclone/vendor/github.com/aws/aws-sdk-go/service/s3/s3manager.(*uploader).nextReader(0xc0003b8870, 0xc0001e3f70, 0x0, 0xc00046e000, 0xc0002ee380, 0x2b8c000, 0x0, 0x88, 0x2b8c000)
/Users/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go:546 +0x6c
github.com/ncw/rclone/vendor/github.com/aws/aws-sdk-go/service/s3/s3manager.(*uploader).upload(0xc0003b8870, 0x0, 0x0, 0x0)
/Users/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go:463 +0x1e8
github.com/ncw/rclone/vendor/github.com/aws/aws-sdk-go/service/s3/s3manager.Uploader.UploadWithContext(0x3e80000000000, 0x2, 0x0, 0x2710, 0x1f5d640, 0xc0001e3d58, 0x0, 0x0, 0x0, 0x2f524b0, ...)
/Users/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go:381 +0x1a9
github.com/ncw/rclone/vendor/github.com/aws/aws-sdk-go/service/s3/s3manager.Uploader.Upload(0x3e80000000000, 0x2, 0x0, 0x2710, 0x1f5d640, 0xc0001e3d58, 0x0, 0x0, 0x0, 0xc00031fa00, ...)
/Users/travis/gopath/src/github.com/ncw/rclone/vendor/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go:352 +0xe4
github.com/ncw/rclone/backend/s3.(*Object).Update.func2(0xc00007cf60, 0xc0002ee360, 0x20)
/Users/travis/gopath/src/github.com/ncw/rclone/backend/s3/s3.go:1595 +0xd9
github.com/ncw/rclone/lib/pacer.(*Pacer).call(0xc00007cf60, 0xc0002ee360, 0x1, 0xc00073a001, 0xc0002ee360)
/Users/travis/gopath/src/github.com/ncw/rclone/lib/pacer/pacer.go:383 +0x16a
github.com/ncw/rclone/lib/pacer.(*Pacer).CallNoRetry(0xc00007cf60, 0xc0002ee360, 0x1b, 0xc000485e50)
/Users/travis/gopath/src/github.com/ncw/rclone/lib/pacer/pacer.go:415 +0x3e
github.com/ncw/rclone/backend/s3.(*Object).Update(0xc0007966c0, 0x1f2d620, 0xc0002ee1e0, 0x1f41420, 0xc0002ee200, 0xc000170d10, 0x1, 0x1, 0x0, 0x18)
/Users/travis/gopath/src/github.com/ncw/rclone/backend/s3/s3.go:1594 +0x52c
github.com/ncw/rclone/backend/s3.(*Fs).Put(0xc000274b40, 0x1f2d620, 0xc0002ee1e0, 0x1f41420, 0xc0002ee200, 0xc000170d10, 0x1, 0x1, 0x20, 0x1cce020, ...)
/Users/travis/gopath/src/github.com/ncw/rclone/backend/s3/s3.go:1160 +0x101
github.com/ncw/rclone/fs.Fs.Put-fm(0x1f2d620, 0xc0002ee1e0, 0x1f41420, 0xc0002ee200, 0xc000170d10, 0x1, 0x1, 0x30, 0x28, 0xc0013e8bb8, ...)
/Users/travis/gopath/src/github.com/ncw/rclone/backend/cache/cache.go:1502 +0x88
github.com/ncw/rclone/backend/crypt.(*Fs).put(0xc0003d85a0, 0x1f2d600, 0xc0001223c0, 0x2f266c0, 0xc000592c00, 0xc000170d10, 0x1, 0x1, 0xc0013e8c98, 0x5, ...)
/Users/travis/gopath/src/github.com/ncw/rclone/backend/crypt/crypt.go:357 +0x194
github.com/ncw/rclone/backend/crypt.(*Fs).Put(0xc0003d85a0, 0x1f2d600, 0xc0001223c0, 0x2f266c0, 0xc000592c00, 0xc000170d10, 0x1, 0x1, 0xc000558e60, 0x8, ...)
/Users/travis/gopath/src/github.com/ncw/rclone/backend/crypt/crypt.go:389 +0xd5
github.com/ncw/rclone/fs/operations.Copy(0x1f44ce0, 0xc0003d85a0, 0x0, 0x0, 0xc00088da0d, 0x2e, 0x1f455e0, 0xc000592c00, 0x883a665efedb94b9, 0x873ee640744c8ac5, ...)
/Users/travis/gopath/src/github.com/ncw/rclone/fs/operations/operations.go:294 +0x1299
github.com/ncw/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc000459180, 0xc00032a300, 0x1f44ce0, 0xc0003d85a0, 0xc000459258)
/Users/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:280 +0x23c
created by github.com/ncw/rclone/fs/sync.(*syncCopyMove).startTransfers
/Users/travis/gopath/src/github.com/ncw/rclone/fs/sync/sync.go:306 +0x9f
That confirms my theory.
What is happening is that rclone can't read the MD5 sum of the file when it is uploading it (which it needs to know in advance for chunked files).
I think what I need to do here is introduce --s3-upload-cutoff and just do single part uploads below that cutoff. I'm pretty sure these will be quicker for files below 100M or so anyway. Also I think the default --s3-chunk-size is probably too small.
I made an issue about this: s3: bring back single part uploads and `--s3-upload-cutoff` Ā· Issue #2772 Ā· rclone/rclone Ā· GitHub
That will need a different fix that I'll implement in the above issue.