S3 minio - files be deleted unreasonably

s123unny · September 28, 2020, 4:35am

What is the problem you are having with rclone?

Rclone deleted files unreasonably. I use rclone to sync a backup everyday and I found that files are deleted on one day and be copied again on the later day. I only have logs on the INFO level and do not know how to reproduce it. However, it has happened two times.
Thank you in advance.

What is your rclone version (output from `rclone version`)

rclone v1.53.1

os/arch: linux/amd64
go version: go1.15

Which cloud storage system are you using? (eg Google Drive)

s3 minio

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone --s3-disable-checksum --bwlimit 500M --exclude-from=rclone_exclude.txt --log-file=crontablog --log-level INFO sync production:videos canvas:videos
rclone --s3-disable-checksum --bwlimit 500M --exclude-from=rclone_exclude.txt --log-file=crontablog --log-level INFO --max-delete=50 sync production:videos canvas:videos

P.S. I added --max-delete after the first time files being deleted.

A log from the command with the `-vv` flag

First log which deleted 80% files.

2020/09/20 04:37:01 INFO  :
Transferred:   	  176.064G / 176.064 GBytes, 100%, 114.719 MBytes/s, ETA 0s
Errors:                 5 (retrying may help)
Checks:            291190 / 301199, 97%
Transferred:          846 / 846, 100%
Elapsed time:      37m0.2s
Checking:
 *               medium/XiG3hqM8GtRugk82DwoMqAGK: checking
 *               medium/XiKVm92vsnJ2bLTNu4ecFJH1: checking
 *               medium/XiPsKtYbkEtWi63gJoeHTbkd: checking
 *               medium/XiRyzTzZkzgrdTv5dWg7V7Er: checking
 *               medium/XifPSGVWPQ7Bpn3QPTFdtjUJ: checking
 *               medium/XizPmS8JPivQ6jkoweGXs5wi: checking
 *               medium/Xj7UFBd45QbFoaYscyB724GQ: checking
 *               medium/XjT7DgycyvZa8RChMctJBkSy: checking

2020/09/20 04:37:34 INFO  : medium/recordings/109-1/B21-U1230/0/2020-09-17-10-20-00_109-1_B21-U1230_0_M10-501.mp4: Copied (new)
2020/09/20 04:38:01 INFO  :
Transferred:   	  176.224G / 176.224 GBytes, 100%, 110.600 MBytes/s, ETA 0s
Errors:                 5 (retrying may help)
Checks:            298788 / 307323, 97%
Transferred:          847 / 847, 100%
Elapsed time:      38m0.2s
Checking:
 *               medium/o7Qu7noyX75iRaPgdXdUMpV4: checking
 *               medium/o7ZDC4SgM9cFhAUbC19dr9VM: checking
 *               medium/o7bhkwqi3xbRXyNbqejHtsCG: checking
 *               medium/o7ciSKVewyah27oE9Xk2J5zG: checking
 *               medium/o7nyUqnD3jkTTsjCHzmxV1y9: checking
 *               medium/o7pm1FJM4cgr4NbXyKXmrkih: checking
 *               medium/o7tsLHxVQEAvcCQ6Kgj64CiB: checking
 *               medium/o85ZFEbvuK6ywVFjLC3NF7CD: checking

2020/09/20 04:39:01 INFO  :
Transferred:   	  176.224G / 176.224 GBytes, 100%, 106.677 MBytes/s, ETA 0s
Errors:                 5 (retrying may help)
Checks:            303139 / 307323, 99%
Transferred:          847 / 847, 100%
Elapsed time:      39m0.2s
Checking:
 *               medium/x38dkQAM46qkcNXBMhD5vXXX: checking
 *               medium/x3J9whkfyLd9s7hTVpVgV331: checking
 *               medium/x3YXmhWJ8rcRkDgre6GUJEU5: checking
 *               medium/x41biLc4PdhCdiorL5esY8QK: checking
 *               medium/x447rdYVyJnCQR8wBizgP1UW: checking
 *               medium/x49Eyg1LgUbqw5YtDiQcCjjK: checking
 *               medium/x4Cy72k1iQZM11ZhDK9NGZQp: checking
 *               medium/x4WacL26HLbTMfMgcpcxNuxX: checking

2020/09/20 04:39:24 ERROR : S3 bucket videos: not deleting files as there were IO errors
2020/09/20 04:39:24 ERROR : S3 bucket videos: not deleting directories as there were IO errors
2020/09/20 04:39:24 ERROR : Attempt 2/3 failed with 6 errors and: march failed with 3 error(s): first error: SerializationError: failed to unmarshal error message
	status code: 504, request id: , host id:
caused by: UnmarshalError: failed to unmarshal error message
	00000000  3c 68 74 6d 6c 3e 0d 0a  3c 68 65 61 64 3e 3c 74  |<html>..<head><t|
00000010  69 74 6c 65 3e 35 30 34  20 47 61 74 65 77 61 79  |itle>504 Gateway|
00000020  20 54 69 6d 65 2d 6f 75  74 3c 2f 74 69 74 6c 65  | Time-out</title|
00000030  3e 3c 2f 68 65 61 64 3e  0d 0a 3c 62 6f 64 79 20  |></head>..<body |
00000040  62 67 63 6f 6c 6f 72 3d  22 77 68 69 74 65 22 3e  |bgcolor="white">|
00000050  0d 0a 3c 63 65 6e 74 65  72 3e 3c 68 31 3e 35 30  |..<center><h1>50|
00000060  34 20 47 61 74 65 77 61  79 20 54 69 6d 65 2d 6f  |4 Gateway Time-o|
00000070  75 74 3c 2f 68 31 3e 3c  2f 63 65 6e 74 65 72 3e  |ut</h1></center>|
00000080  0d 0a 3c 68 72 3e 3c 63  65 6e 74 65 72 3e 6e 67  |..<hr><center>ng|
00000090  69 6e 78 3c 2f 63 65 6e  74 65 72 3e 0d 0a 3c 2f  |inx</center>..</|
000000a0  62 6f 64 79 3e 0d 0a 3c  2f 68 74 6d 6c 3e 0d 0a  |body>..</html>..|

caused by: expected element type <Error> but have <html>
2020/09/20 04:40:01 INFO  :
Transferred:   	  176.224G / 176.224 GBytes, 100%, 105.285 MBytes/s, ETA 0s
Checks:            307323 / 307323, 100%
Transferred:          847 / 847, 100%
Elapsed time:      40m0.2s

2020/09/20 04:41:01 INFO  :
Transferred:   	  176.224G / 176.224 GBytes, 100%, 105.285 MBytes/s, ETA 0s
Checks:            307323 / 307323, 100%
Transferred:          847 / 847, 100%
Elapsed time:      41m0.2s

2020/09/20 04:41:59 INFO  : 7jpiANW4zxVWEmyiWJRVdq2C.lock: Copied (new)
2020/09/20 04:41:59 INFO  : 8wRDGmPMLHz3xnVMSF98yAh6.lock: Copied (new)
2020/09/20 04:42:01 INFO  : 6uowbTSV3Rc1mhdoSbSJqpDb.lock: Updated modification time in destination
2020/09/20 04:42:01 INFO  : 6qPqcnDguND2xfdUUc8r9p1e.lock: Updated modification time in destination
2020/09/20 04:42:01 INFO  : 7gqzmRkhZcbH747Qt65JwyJR.lock: Updated modification time in destination
2020/09/20 04:42:01 INFO  : 7iY39GbdoRmZUn22cHPbttdp.lock: Updated modification time in destination
2020/09/20 04:42:01 INFO  :
Transferred:   	  176.224G / 176.224 GBytes, 100%, 105.146 MBytes/s, ETA 0s
Checks:            311524 / 312321, 100%
Transferred:          849 / 849, 100%
Elapsed time:      42m0.2s
Checking:
 *                      8khzhuMVRLsZ5GQ8Jome71du: checking
 *                      8kp4Um1M61uj9LC9syHdq4zJ: checking
 *                      8kxuRv5cRHMJ4xZajjcFgAy1: checking
 *                      8kyy31x9VvK9cnB1dvePDWBY: checking
 *                      8m9qQQYGpwSn8gCwZbSn22Xh: checking

2020/09/20 04:42:27 INFO  : uploads/youtube_video/thumbnail/8492/0.jpg: Deleted
2020/09/20 04:42:27 INFO  : uploads/youtube_video/thumbnail/1300/thumb_16_9_0.jpg: Deleted
2020/09/20 04:42:27 INFO  : uploads/uploaded_video/thumbnail/10390/8FpyJ9ruZLKbEvcARjDSDNjs1575443260.png: Deleted
2020/09/20 04:42:27 INFO  : medium/1QiuFiCWCuAZeW97dF2fj1TF: Deleted
2020/09/20 04:42:27 INFO  : uploads/uploaded_video/thumbnail/15750/thumb_16_9_ZCitBbFgvYT4vsunkUaLE4vL1584517909.png: Deleted
2020/09/20 04:42:28 INFO  : uploads/uploaded_video/thumbnail/29082/tKYTrjDFZuSn4V8hARjhboZb1588593911.png: Deleted
2020/09/20 04:42:28 INFO  : uploads/uploaded_video/thumbnail/24073/thumb_16_9_jXzLzZRtaoZvXF3ixmWVQvuk1586923152.png: Deleted
2020/09/20 04:42:28 INFO  : uploads/uploaded_video/thumbnail/5107/thumb_16_9_YMigKNnh9gzhQn9edJqfAD3U1550590475.png: Deleted
2020/09/20 04:42:28 INFO  : DPdSo7VKpAbx8LpvNkAi9ayU: Deleted
2020/09/20 04:42:28 INFO  : medium/6EXyverddvMTdno58bSjxPMW: Deleted
2020/09/20 04:42:28 INFO  : ultra_high/51DMgeKQ71DH76qN1wEjw5Rs: Deleted
2020/09/20 04:42:28 INFO  : bXh9PCMrTK8TbEG1Uq2m39Uh: Deleted
2020/09/20 04:42:28 INFO  : medium/dHft2wE96kRih4usDAmx3zbv: Deleted
2020/09/20 04:42:28 INFO  : uploads/uploaded_video/thumbnail/25208/6sYC5FtsCcCQicAccyPT8EeH1587310705.png: Deleted
2020/09/20 04:42:28 INFO  : medium/Jxgb9o4ksaT311ytg6oPmNjz: Deleted
2020/09/20 04:42:29 INFO  : uploads/uploaded_video/thumbnail/35238/thumb_16_9_ToHGJRXPQudgzz3NciiMmZUJ1590484582.png: Deleted
2020/09/20 04:42:29 INFO  : aW1Uru8WkcFVekuyLT5rRxPb: Deleted
2020/09/20 04:42:29 INFO  : recordings/108-2/B01-101A2/01/2020-03-19-13-20-00_108-2_B01-101A2_01_M10-501.mp4: Deleted
2020/09/20 04:42:29 INFO  : uploads/uploaded_video/thumbnail/41169/thumb_16_9_RGfwf5iXaYfEWiSC3K9mvNfh1597655724.png: Deleted
2020/09/20 04:42:29 INFO  : uploads/uploaded_video/thumbnail/10942/bQsRU98KzPNefWU7MBJBheiy1576650335.png: Deleted
...
2020/09/20 06:10:00 INFO  : UdRf8SnEWSvRu7sde9L2GosD: Deleted
2020/09/20 06:10:01 INFO  : high/vT7GPqonGaPvpdtjQPHoXBxe: Deleted
2020/09/20 06:10:01 INFO  : vytF9PYZJfQxa7AcSaoz3viV: Deleted
2020/09/20 06:10:01 ERROR : Attempt 3/3 succeeded
2020/09/20 06:10:01 INFO  :
Transferred:   	  176.224G / 176.224 GBytes, 100%, 25.889 MBytes/s, ETA 0s
Checks:            522323 / 522323, 100%
Deleted:           210002
Transferred:          849 / 849, 100%
Elapsed time:    2h9m59.4s

Second log with --max-delete:

2020/09/26 04:28:24 ERROR : S3 bucket videos: not deleting files as there were IO errors
2020/09/26 04:28:24 ERROR : S3 bucket videos: not deleting directories as there were IO errors
2020/09/26 04:28:24 ERROR : Attempt 1/3 failed with 5 errors and: failed to open source object: NoSuchKey: The specified key does not exist.
	status code: 404, request id: 16381FCFFD8C63AF, host id:
2020/09/26 04:29:01 INFO  :
Transferred:   	  258.771G / 258.771 GBytes, 100%, 163.827 MBytes/s, ETA 0s
Checks:            219934 / 219934, 100%
Transferred:          976 / 976, 100%
Elapsed time:      29m0.0s

2020/09/26 04:29:24 ERROR : : error reading source directory: SerializationError: failed to unmarshal error message
	status code: 504, request id: , host id:
caused by: UnmarshalError: failed to unmarshal error message
	00000000  3c 68 74 6d 6c 3e 0d 0a  3c 68 65 61 64 3e 3c 74  |<html>..<head><t|
00000010  69 74 6c 65 3e 35 30 34  20 47 61 74 65 77 61 79  |itle>504 Gateway|
00000020  20 54 69 6d 65 2d 6f 75  74 3c 2f 74 69 74 6c 65  | Time-out</title|
00000030  3e 3c 2f 68 65 61 64 3e  0d 0a 3c 62 6f 64 79 20  |></head>..<body |
00000040  62 67 63 6f 6c 6f 72 3d  22 77 68 69 74 65 22 3e  |bgcolor="white">|
00000050  0d 0a 3c 63 65 6e 74 65  72 3e 3c 68 31 3e 35 30  |..<center><h1>50|
00000060  34 20 47 61 74 65 77 61  79 20 54 69 6d 65 2d 6f  |4 Gateway Time-o|
00000070  75 74 3c 2f 68 31 3e 3c  2f 63 65 6e 74 65 72 3e  |ut</h1></center>|
00000080  0d 0a 3c 68 72 3e 3c 63  65 6e 74 65 72 3e 6e 67  |..<hr><center>ng|
00000090  69 6e 78 3c 2f 63 65 6e  74 65 72 3e 0d 0a 3c 2f  |inx</center>..</|
000000a0  62 6f 64 79 3e 0d 0a 3c  2f 68 74 6d 6c 3e 0d 0a  |body>..</html>..|

caused by: expected element type <Error> but have <html>
2020/09/26 04:29:24 ERROR : S3 bucket videos: not deleting files as there were IO errors
2020/09/26 04:29:24 ERROR : S3 bucket videos: not deleting directories as there were IO errors
2020/09/26 04:29:24 ERROR : Attempt 2/3 failed with 1 errors and: SerializationError: failed to unmarshal error message
	status code: 504, request id: , host id:
caused by: UnmarshalError: failed to unmarshal error message
	00000000  3c 68 74 6d 6c 3e 0d 0a  3c 68 65 61 64 3e 3c 74  |<html>..<head><t|
00000010  69 74 6c 65 3e 35 30 34  20 47 61 74 65 77 61 79  |itle>504 Gateway|
00000020  20 54 69 6d 65 2d 6f 75  74 3c 2f 74 69 74 6c 65  | Time-out</title|
00000030  3e 3c 2f 68 65 61 64 3e  0d 0a 3c 62 6f 64 79 20  |></head>..<body |
00000040  62 67 63 6f 6c 6f 72 3d  22 77 68 69 74 65 22 3e  |bgcolor="white">|
00000050  0d 0a 3c 63 65 6e 74 65  72 3e 3c 68 31 3e 35 30  |..<center><h1>50|
00000060  34 20 47 61 74 65 77 61  79 20 54 69 6d 65 2d 6f  |4 Gateway Time-o|
00000070  75 74 3c 2f 68 31 3e 3c  2f 63 65 6e 74 65 72 3e  |ut</h1></center>|
00000080  0d 0a 3c 68 72 3e 3c 63  65 6e 74 65 72 3e 6e 67  |..<hr><center>ng|
00000090  69 6e 78 3c 2f 63 65 6e  74 65 72 3e 0d 0a 3c 2f  |inx</center>..</|
000000a0  62 6f 64 79 3e 0d 0a 3c  2f 68 74 6d 6c 3e 0d 0a  |body>..</html>..|

caused by: expected element type <Error> but have <html>
2020/09/26 04:30:01 INFO  :
Transferred:   	  258.771G / 258.771 GBytes, 100%, 163.827 MBytes/s, ETA 0s
Checks:            219934 / 219934, 100%
Transferred:          976 / 976, 100%
Elapsed time:      30m0.0s

2020/09/26 04:30:31 INFO  : uploads/uploaded_video/thumbnail/15621/thumb_16_9_1L9bT4Y6RNnaD4Jh9bE14QfH1584501286.png: Deleted
2020/09/26 04:30:31 INFO  : uploads/uploaded_video/thumbnail/5387/thumb_16_9_dAu71eqPNYyMVAoqN7nGDwkf1552009824.png: Deleted
2020/09/26 04:30:32 INFO  : low/ETkjGWavFurpAwTH9QcbDZXn: Deleted
2020/09/26 04:30:32 INFO  : ultra_high/1wyHQbjw3Ht6kMTNzCnid3tv: Deleted
2020/09/26 04:30:32 INFO  : p1uz2DpmNUbxZHNTp7BwPiBs: Deleted
2020/09/26 04:30:32 INFO  : 3rCejGBQTDBD692bgGokhyQM: Deleted
2020/09/26 04:30:32 INFO  : uploads/uploaded_video/file/2696/thumb_16_9_單元_5_03_轉動慣量.png: Deleted
2020/09/26 04:30:32 INFO  : RG2un1smvcPVryassC5cRVUn: Deleted
2020/09/26 04:30:32 INFO  : high/vp9ovCGT6zCyGxjNEoi4S71i: Deleted
2020/09/26 04:30:32 INFO  : uploads/uploaded_video/thumbnail/22183/aARoMr5FwHaGJPHGhD5NpwWJ1586400132.png: Deleted
2020/09/26 04:30:32 INFO  : low/T8JdymHCx5qHApPperfymBPW: Deleted
2020/09/26 04:30:32 INFO  : uploads/youtube_video/thumbnail/2721/thumb_16_9_0.jpg: Deleted
2020/09/26 04:30:32 INFO  : uploads/uploaded_video/thumbnail/4464/LFU6eXWBFaMXiV7vGMsU8zyw1548222958.png: Deleted
2020/09/26 04:30:32 INFO  : high/YHgjLWpFmmJUeUiXpYtmVFFQ: Deleted
2020/09/26 04:30:32 INFO  : faw6xGn5Q4nLb8bWURmtf8bu: Deleted
2020/09/26 04:30:32 INFO  : uploads/uploaded_video/file/826/thumb_C語言入門_5c29af2404af4c94b0b4545154462f46.png: Deleted
2020/09/26 04:30:32 INFO  : Uj1fjtNgz3y9WHgPsWmbt6ig: Deleted
2020/09/26 04:30:32 INFO  : low/QpeE5gwq9LK6Z88k5Q2B5AEV: Deleted
2020/09/26 04:30:32 INFO  : high/G4hGDRbgrKTJrkwbZ5WrBnF4: Deleted
2020/09/26 04:30:32 INFO  : VbiAKBrDypvj89pFwDyth1ds: Deleted
2020/09/26 04:30:32 INFO  : ultra_high/7gqzmRkhZcbH747Qt65JwyJR: Deleted
2020/09/26 04:30:32 INFO  : uploads/youtube_video/thumbnail/6589/thumb_16_9_0.jpg: Deleted
2020/09/26 04:30:32 INFO  : ultra_high/V7gVdggHVvneVGQfjuN61k9o: Deleted
2020/09/26 04:30:32 INFO  : medium/e4jyM6uTmiN1fW45FqkANYM9: Deleted
2020/09/26 04:30:32 INFO  : low/fCA3Q8Fqde1PyGEF44j6X5LY: Deleted
2020/09/26 04:30:32 INFO  : high/JFQ4yoeMdUh1QjrFwsN18gym: Deleted
2020/09/26 04:30:32 INFO  : medium/kAoyMxQQ6q2bck59qjGoGu1q: Deleted
2020/09/26 04:30:32 INFO  : uploads/uploaded_video/thumbnail/40981/thumb_16_9_gpUayMir6yfYvcYzNgmtE4FT1596813253.png: Deleted
2020/09/26 04:30:32 INFO  : low/MFGUwLBhPxSQG4r3zD9M84rK: Deleted
2020/09/26 04:30:32 INFO  : uploads/uploaded_video/thumbnail/33604/oGZKpdWxgSeZEqqHxgpyP9Up1589958703.png: Deleted
2020/09/26 04:30:32 INFO  : uploads/uploaded_video/thumbnail/17051/RCk9dtoGEYH7fe7K6XQfUddT1584954328.png: Deleted
2020/09/26 04:30:32 INFO  : RRz8f1Gf2uWYKtzGPGYnue6f: Deleted
2020/09/26 04:30:32 INFO  : uploads/uploaded_video/thumbnail/31081/i2afDpdNAq2P9ocgHCgGEiu31589191467.png: Deleted
2020/09/26 04:30:32 INFO  : uploads/uploaded_video/thumbnail/6888/9VJtZ6amt1wNsZ5SWMoL1vtc1564716876.png: Deleted
2020/09/26 04:30:32 INFO  : erdygexAujUvLuBLEzzcsTG6: Deleted
2020/09/26 04:30:32 INFO  : medium/NFd7YB9LiibvZhgEeZ3Ziua3: Deleted
2020/09/26 04:30:33 INFO  : medium/wUoSXHPZQB3NrJ9ojnackMZ5: Deleted
2020/09/26 04:30:33 INFO  : high/3s9nxBhuPLyKZra3Zk5DAfz7: Deleted
2020/09/26 04:30:33 INFO  : iQhZQe2YnjYTyx5pz79gr9Qd: Deleted
2020/09/26 04:30:33 INFO  : low/68k14u623Pa4E2Zqw2YkMXby: Deleted
2020/09/26 04:30:33 INFO  : uploads/youtube_video/thumbnail/41068/0.jpg: Deleted
2020/09/26 04:30:33 INFO  : high/BAiHiTyhq3nBzarxgGQ1H6jQ: Deleted
2020/09/26 04:30:33 INFO  : high/VK5HWeU3dNNBHP1LDQUBiCAN: Deleted
2020/09/26 04:30:33 INFO  : uploads/uploaded_video/file/2477/thumb_16_9_傳遞參數至父類別建構式_dd608a53ef0b49758e052c40fcd2e5e0.png: Deleted
2020/09/26 04:30:33 INFO  : low/KUSQakTNEdJx7bpb5SPA7LCi: Deleted
2020/09/26 04:30:33 INFO  : uploads/uploaded_video/thumbnail/40412/dYHUkgYs7hwxU4GuQvo4x5jQ1593758591.png: Deleted
2020/09/26 04:30:33 INFO  : high/FSHK34rEgn6YGzj4f97fgt3S: Deleted
2020/09/26 04:30:33 ERROR : Got fatal error on delete: --max-delete threshold reached
2020/09/26 04:30:33 INFO  : high/FsbYK6xRuunnFcqohmUkSSPz: Deleted
2020/09/26 04:30:33 ERROR : Got fatal error on delete: --max-delete threshold reached
2020/09/26 04:30:33 INFO  : medium/wByff5e3a465icGWkbUJMb2g: Deleted
2020/09/26 04:30:33 ERROR : Got fatal error on delete: --max-delete threshold reached
2020/09/26 04:30:33 INFO  : high/FJBgHaPrYbgLq2fjmCBYnnSD: Deleted
2020/09/26 04:30:33 ERROR : Got fatal error on delete: --max-delete threshold reached
2020/09/26 04:30:33 ERROR : Cancelling sync due to fatal error: failed to delete 4 files
2020/09/26 04:30:33 ERROR : S3 bucket videos: not deleting directories as there were IO errors
2020/09/26 04:30:33 ERROR : Fatal error received - not attempting retries
2020/09/26 04:30:33 INFO  :
Transferred:   	  258.771G / 258.771 GBytes, 100%, 163.627 MBytes/s, ETA 0s
Errors:                 5 (fatal error encountered)
Checks:            219988 / 219988, 100%
Deleted:               54
Transferred:          976 / 976, 100%
Elapsed time:     30m32.0s

2020/09/26 04:30:33 Failed to sync with 5 errors: last error was: failed to delete 4 files

ncw · September 28, 2020, 11:49am

Are you reading from s3 minio as well as writing to it?

If so this error may be the cause of the problems

2020/09/20 04:39:24 ERROR : Attempt 2/3 failed with 6 errors and: march failed with 3 error(s): first error: SerializationError: failed to unmarshal error message
	status code: 504, request id: , host id:
caused by: UnmarshalError: failed to unmarshal error message
	00000000  3c 68 74 6d 6c 3e 0d 0a  3c 68 65 61 64 3e 3c 74  |<html>..<head><t|
00000010  69 74 6c 65 3e 35 30 34  20 47 61 74 65 77 61 79  |itle>504 Gateway|
00000020  20 54 69 6d 65 2d 6f 75  74 3c 2f 74 69 74 6c 65  | Time-out</title|
00000030  3e 3c 2f 68 65 61 64 3e  0d 0a 3c 62 6f 64 79 20  |></head>..<body |
00000040  62 67 63 6f 6c 6f 72 3d  22 77 68 69 74 65 22 3e  |bgcolor="white">|
00000050  0d 0a 3c 63 65 6e 74 65  72 3e 3c 68 31 3e 35 30  |..<center><h1>50|
00000060  34 20 47 61 74 65 77 61  79 20 54 69 6d 65 2d 6f  |4 Gateway Time-o|
00000070  75 74 3c 2f 68 31 3e 3c  2f 63 65 6e 74 65 72 3e  |ut</h1></center>|
00000080  0d 0a 3c 68 72 3e 3c 63  65 6e 74 65 72 3e 6e 67  |..<hr><center>ng|
00000090  69 6e 78 3c 2f 63 65 6e  74 65 72 3e 0d 0a 3c 2f  |inx</center>..</|
000000a0  62 6f 64 79 3e 0d 0a 3c  2f 68 74 6d 6c 3e 0d 0a  |body>..</html>..|

caused by: expected element type <Error> but have <html>

This is a gateway timeout error from minio. I don't know whether it is from the source or destination.

That will could cause bits of the source to appear to be missing. Now rclone should note that the source listing was incomplete and not delete stuff but it looks like that detection didn't work for some reason. I guess that is the root of the issue. I'll examine the source to see if I can see a reason why that might be.

Maybe you are accessing the minio cluster faster than it can cope with.

Note you've limited it to 500 MByte/s if you want 500 MBit/s then you want --bwlimit 62M

Can you send me a complete log with -vv of a sync with this error message please? If you want to email it to nick@craig-wood.com (with a link to this issue) or PM me a link that would help enormously to work out what is going on.

ncw · September 29, 2020, 3:14pm

Thanks for the log... I think solving the gateway timeout would be the best thing to do.

Try lowering --checkers 1 --transfers 1 so as not to hit it so hard.

s123unny · October 5, 2020, 3:18am

Can you look at another log? (in the direct message)
I think the gateway timeout is not related as the deletion just occur.
Moreover, those files should not be deleted and do exist in the source bucket.

ncw · October 5, 2020, 2:16pm

I checked out the log.

I don't think those files are deleted because of that listing failing as you said "I think the gateway timeout is not related as the deletion just occur."

Could you run rclone check just to double check? Maybe run it a few times to see if the intermittent timeout affects it.

s123unny · October 6, 2020, 8:04am

I get different result when running rclone check. Very confusing
Some result said lots of files are missing... this may cause deletion?

2020/10/06 10:49:49 NOTICE: S3 bucket videos: 93 files missing
2020/10/06 10:49:49 NOTICE: S3 bucket videos: 210006 files missing
2020/10/06 10:49:49 NOTICE: S3 bucket videos: 210099 differences found
2020/10/06 10:49:49 NOTICE: S3 bucket videos: 210099 errors while checking
2020/10/06 10:49:49 NOTICE: S3 bucket videos: 15907 matching files

2020/10/06 10:59:49 NOTICE: S3 bucket videos: 302 files missing
2020/10/06 10:59:49 NOTICE: S3 bucket videos: 175523 files missing
2020/10/06 10:59:49 NOTICE: S3 bucket videos: 175825 differences found
2020/10/06 10:59:49 NOTICE: S3 bucket videos: 175825 errors while checking
2020/10/06 10:59:49 NOTICE: S3 bucket videos: 50390 matching files

2020/10/06 14:01:18 NOTICE: S3 bucket videos: 1244 files missing
2020/10/06 14:01:18 NOTICE: S3 bucket videos: 386 files missing
2020/10/06 14:01:18 NOTICE: S3 bucket videos: 1631 differences found
2020/10/06 14:01:18 NOTICE: S3 bucket videos: 1631 errors while checking
2020/10/06 14:01:18 NOTICE: S3 bucket videos: 189546 matching files

ncw · October 6, 2020, 4:20pm

I agree...

Yes this will be cause of the deletions.

Can you try adding --fast-list to the rclone check command and see if it is more consistent?

Expect a long pause at the start while rclone reads the file listing.

s123unny · October 8, 2020, 8:25am

Maybe the issue is related to the backend. Since I run rclone size on both buckets, one is consistent but not another.
I am going to check on my minio setting and get back with my finding if any. Really appreciate for your response!

ncw · October 8, 2020, 5:20pm

It is likely that sometimes the minio backend is not returning the full listing. That would be my guess.

Try the --fast-list flag which does the listings in a different way - it may help.

system · December 8, 2020, 1:20pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

S3 minio - files be deleted unreasonably

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

A log from the command with the -vv flag

What is your rclone version (output from `rclone version`)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

A log from the command with the `-vv` flag