Dreamhost Two large buckets - lots of files. Resource intensive? Time?

Help and Support
1

I’m testing out rclone now to backup my website. It’s an art discussion forum, so lots of attachments and files…pretty big. I have one bucket that has 70535 files and another that has 145992 files…The second one is about 85 gigabytes and the first one is about 3 gigabytes.

I know the first sync is going to take a while. But I’m wondering if future sync’s are going to take a long time or use a lot of resources? I’ve attempted other s3 based sync tools before and gave up on them because they were way too intensive for nightly backups of such a large collection of files. Is this going to be the right way to go with this?

2

Ooops, first bucket hung and eventually timed out at 16191 files. Tried to restart it and it seemed like it was taking its time. I’m guessing s3 is probably not going to work out for a solution like mine unless I’m missing some kind of optimization step?

3

Those aren’t unreasonable sizes to backup I would have thought it would work just fine!

Future syncs will be quicker. If you want to speed them up futher, then use --checksum or --size-only with S3. Also consider using the latest beta with the --fast-list flag which will use more memory (holding the entire directory listing) but less transactions.

rclone is designed to be re-run when things go wrong. Just keep running it and it will get there in the end. See point above for pause before it starts syncing.

4

I just installed the beta and it’s working better with --checksum and --fast-list. I set verbose and got some interesting output…anyone got any ideas? I’m going to paste a few examples:

    status code: 403, request id: tx00000000000000032bde4-00595da824-16f60feb-default, host id:

2017/07/05 23:01:56 ERROR : utilities/Thumbs/Challenges/TN_joao paulo_Jul09.png: Failed to copy: SignatureDoesNotMatch:
status code: 403, request id: tx00000000000000032be43-00595da824-16f60fe6-default, host id:
2017/07/05 23:01:56 ERROR : utilities/Thumbs/Challenges/TN_joao paulo_Mar09.png: Failed to copy: SignatureDoesNotMatch:
status code: 403, request id: tx00000000000000032be20-00595da824-16f3ed89-default, host id:
2017/07/05 23:01:56 ERROR : utilities/Thumbs/Challenges/TN_joao paulo_Nov08.png: Failed to copy: SignatureDoesNotMatch:
status code: 403, request id: tx00000000000000032be23-00595da824-16f3ed89-default, host id:
2017/07/05 23:01:57 ERROR : utilities/Thumbs/Challenges/TN_joao paulo_Oct08.png: Failed to copy: SignatureDoesNotMatch:
status code: 403, request id: tx00000000000000032bdcd-00595da825-16f60ffa-default, host id:
2017/07/05 23:01:57 INFO : S3 bucket guildsite: Waiting for checks to finish
2017/07/05 23:01:57 INFO : S3 bucket guildsite: Waiting for transfers to finish
2017/07/05 23:01:57 ERROR : utilities/Thumbs/Challenges/TN_joao paulo_Sep08.png: Failed to copy: SignatureDoesNotMatch:
status code: 403, request id: tx00000000000000032be31-00595da825-16f60ff5-default, host id:
2017/07/05 23:01:57 ERROR : utilities/Thumbs/Challenges/TN_tilt‎_LiteMay11.png: Failed to copy: SignatureDoesNotMatch:
status code: 403, request id: tx00000000000000032bdd1-00595da825-16f60ffa-default, host id:
2017/07/05 23:01:57 ERROR : utilities/Thumbs/Challenges/TN_walterful‎_LiteMay11.png: Failed to copy: SignatureDoesNotMatch:
status code: 403, request id: tx00000000000000032be2b-00595da825-16f3ed89-default, host id:
2017/07/05 23:01:57 ERROR : utilities/Thumbs/Challenges/TN_yu gnomi_Aug08.png: Failed to copy: SignatureDoesNotMatch:
status code: 403, request id: tx00000000000000032bdf9-00595da825-16f60feb-default, host id:
2017/07/05 23:02:29 INFO :
Transferred: 111.521 MBytes (317.210 kBytes/s)
Errors: 91
Checks: 211329
Transferred: 0
Elapsed time: 6m0s
Transferring:

  • …humbs/Challenges/TN_ Diamond_LiteMay10.png: 100% done, 11.053 kBytes/s, ETA: 0s
  • …umbs/Challenges/TN_A R Frost_LiteApr10.png: 100% done, 8.455 kBytes/s, ETA: 0s
  • …umbs/Challenges/TN_Greason Wolfe_Apr10.png: 100% done, 7.195 kBytes/s, ETA: 0s

2017/07/05 23:03:29 INFO :
Transferred: 111.521 MBytes (271.896 kBytes/s)
Errors: 91
Checks: 211329
Transferred: 0
Elapsed time: 7m0s
Transferring:

  • …humbs/Challenges/TN_ Diamond_LiteMay10.png: 100% done, 206 Bytes/s, ETA: 0s
  • …umbs/Challenges/TN_A R Frost_LiteApr10.png: 100% done, 158 Bytes/s, ETA: 0s
  • …umbs/Challenges/TN_Greason Wolfe_Apr10.png: 100% done, 134 Bytes/s, ETA: 0s

2017/07/05 23:04:00 ERROR : utilities/Thumbs/Challenges/TN_A R Frost_LiteApr10.png: Failed to copy: SignatureDoesNotMatch:
status code: 403, request id: tx000000000000000382dee-00595da8a0-16f60ff0-default, host id:
2017/07/05 23:04:01 ERROR : utilities/Thumbs/Challenges/TN_ Diamond_LiteMay10.png: Failed to copy: SignatureDoesNotMatch:
status code: 403, request id: tx0000000000000001289d9-00595da8a0-16fcfc16-default, host id:
2017/07/05 23:04:01 ERROR : utilities/Thumbs/Challenges/TN_Greason Wolfe_Apr10.png: Failed to copy: SignatureDoesNotMatch:
status code: 403, request id: tx000000000000000382e02-00595da8a1-16f60ff0-default, host id:
2017/07/05 23:04:01 ERROR : S3 bucket guildsite: not deleting files as there were IO errors
2017/07/05 23:04:01 ERROR : Attempt 3/3 failed with 94 errors and: SignatureDoesNotMatch:
status code: 403, request id: tx000000000000000382e02-00595da8a1-16f60ff0-default, host id:
2017/07/05 23:04:01 Failed to sync: SignatureDoesNotMatch:
status code: 403, request id: tx000000000000000382e02-00595da8a1-16f60ff0-default, host id: