I see. It's probably a partial upload, given the lights went out a few days ago (thankfully using ZFS as the root filesystem), but still it's pretty weird to see to be fair.
I'll try to copy now without --dry-run and see how things go, I'll keep you posted.
Thanks!
- do the same with small folder and see if all looks ok
and better to test like this:
copy "EncFS-test:data/media/tv/Shiki" "EncFS-NEW:data/media/tv/Shiki"
If all ok you do not have to delete anything
you do
copy "EncFS-test:data/media/tv/Shiki" "EncFS-NEW:"
so all in wrong place
you can also press BIG START button and mount EncFS-NEW at the same time so you can check every few hours how things look like
DEBUG all to file - and:
tail -f logfile | grep —ignore-case ERROR so you will have early warning system in place:)
Okay, it seems the copy operation went along smoothly (thanks for catching my oversight, by the way).
The latest log file is as follows:
Fortunately it seems there were no errors aside from the EOF one you mentioned earlier, but aside from that perhaps I missed a step, as running
Gave me
tail -f /home/toasterdev/first_run.log | grep —-ignore-case ERROR
grep: ERROR: No such file or directory
In any case, I think now with the mount active and being able to check how it goes along, if everything seems alright, I should be able to begin the transfer operation.
maybe:
tail -f /home/toasterdev/first_run.log | grep —-ignore-case "ERROR"
tail -f /home/toasterdev/first_run.log | grep —-ignore-case "ERROR"
grep: ERROR: No such file or directory
No dice either, though the file is present.
ls -lah first_run.log
-rw-r----- 1 toasterdev toasterdev 243K Jul 7 13:18 first_run.log
this is the problem - some auto-correction it is two times minus sign. but when I copy paste from this forum:
Thanks, you're right.
It seems to be processing something, but it indeed is running now, though I suspect I'll have to try it again with a new transfer to see its effects.
Thank you!
This command instead of showing all crap will only display lines containing error keyword
Perfect! I'll set it up with a beeper then just to be sure.
Honestly, thanks for everything!
Okay, status report (Sorry for the wait, a pipe burst open and cleanup was a mess).
So, files do indeed copy up to a point, but there are in particular about 47 of them so far that fail due to (presumably) corruption, as I'm not able to find them inside the mount, but even if I set --retries to 1 for some reason the copy operation stops after these failures:
2023/07/11 01:31:39 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2018-16.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402018_t0043": failed to authenticate decrypted block - bad password?
2023/07/11 01:31:41 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2014-10.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402014_t0039": failed to authenticate decrypted block - bad password?
2023/07/11 01:32:30 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2017-02.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402017_t0018": failed to authenticate decrypted block - bad password?
2023/07/11 01:32:37 ERROR : FILE REDACTED : Failed to copy: failed to authenticate decrypted block - bad password?
2023/07/11 01:32:38 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2018-14.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402018_t0040": failed to authenticate decrypted block - bad password?
2023/07/11 01:32:43 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2003-15.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402003_t0007": failed to authenticate decrypted block - bad password?
2023/07/11 01:32:57 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2015-15.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402015_t0035": failed to authenticate decrypted block - bad password?
2023/07/11 01:33:20 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2011-05.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402011_t0026": failed to authenticate decrypted block - bad password?
2023/07/11 01:33:34 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2009-13.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402009_t0024": failed to authenticate decrypted block - bad password?
2023/07/11 01:34:48 ERROR : FILE REDACTED : Failed to copy: failed to authenticate decrypted block - bad password?
2023/07/11 01:35:41 ERROR : FILE REDACTED : Failed to copy: failed to authenticate decrypted block - bad password?
2023/07/11 01:35:42 ERROR : FILE REDACTED : Failed to copy: failed to authenticate decrypted block - bad password?
2023/07/11 01:44:15 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2017-02.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402017_t0018": failed to authenticate decrypted block - bad password?
2023/07/11 01:44:30 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2014-10.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402014_t0039": failed to authenticate decrypted block - bad password?
2023/07/11 01:44:47 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2018-16.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402018_t0043": failed to authenticate decrypted block - bad password?
2023/07/11 01:45:39 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2003-15.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402003_t0007": failed to authenticate decrypted block - bad password?
2023/07/11 01:45:48 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2018-14.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402018_t0040": failed to authenticate decrypted block - bad password?
2023/07/11 01:46:06 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2015-15.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402015_t0035": failed to authenticate decrypted block - bad password?
2023/07/11 01:46:07 ERROR : FILE REDACTED : Failed to copy: failed to open source object: can't open: metadata is too big
2023/07/11 01:46:07 ERROR : FILE REDACTED : Failed to copy: failed to open source object: can't open: metadata is too big
2023/07/11 01:47:20 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2009-13.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402009_t0024": failed to authenticate decrypted block - bad password?
2023/07/11 01:48:44 ERROR : FILE REDACTED : Failed to copy: failed to authenticate decrypted block - bad password?
2023/07/11 01:50:08 ERROR : FILE REDACTED : Failed to copy: failed to authenticate decrypted block - bad password?
2023/07/11 01:51:02 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2011-05.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402011_t0026": failed to authenticate decrypted block - bad password?
2023/07/11 01:53:51 ERROR : FILE REDACTED : Failed to copy: failed to authenticate decrypted block - bad password?
2023/07/11 01:55:16 ERROR : FILE REDACTED : Failed to copy: failed to authenticate decrypted block - bad password?
2023/07/11 01:56:31 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2014-10.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402014_t0039": failed to authenticate decrypted block - bad password?
2023/07/11 01:56:33 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2018-16.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402018_t0043": failed to authenticate decrypted block - bad password?
2023/07/11 01:56:33 ERROR : FILE REDACTED : Failed to copy: failed to authenticate decrypted block - bad password?
2023/07/11 01:56:36 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2017-02.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402017_t0018": failed to authenticate decrypted block - bad password?
2023/07/11 01:57:46 ERROR : FILE REDACTED : Failed to copy: failed to authenticate decrypted block - bad password?
2023/07/11 01:57:52 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2003-15.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402003_t0007": failed to authenticate decrypted block - bad password?
2023/07/11 01:57:53 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2018-14.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402018_t0040": failed to authenticate decrypted block - bad password?
2023/07/11 01:58:23 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2015-15.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402015_t0035": failed to authenticate decrypted block - bad password?
2023/07/11 02:07:53 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2009-13.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402009_t0024": failed to authenticate decrypted block - bad password?
2023/07/11 02:08:39 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2018-16.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402018_t0043": failed to authenticate decrypted block - bad password?
2023/07/11 02:08:44 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2014-10.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402014_t0039": failed to authenticate decrypted block - bad password?
2023/07/11 02:09:05 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2011-05.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402011_t0026": failed to authenticate decrypted block - bad password?
2023/07/11 02:09:06 ERROR : FILE REDACTED : Failed to copy: failed to open source object: can't open: metadata is too big
2023/07/11 02:09:38 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2003-15.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402003_t0007": failed to authenticate decrypted block - bad password?
2023/07/11 02:10:13 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2015-15.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402015_t0035": failed to authenticate decrypted block - bad password?
2023/07/11 02:10:31 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2018-14.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402018_t0040": failed to authenticate decrypted block - bad password?
2023/07/11 02:10:40 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2017-02.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402017_t0018": failed to authenticate decrypted block - bad password?
2023/07/11 02:19:27 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2009-13.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402009_t0024": failed to authenticate decrypted block - bad password?
2023/07/11 02:19:31 ERROR : FILE REDACTED : Failed to copy: Post "https://pod-040-2018-16.backblaze.com/b2api/v1/b2_upload_file/a1a8361055873b1e89920e15/c004_v0402018_t0043": failed to authenticate decrypted block - bad password?
2023/07/11 03:15:51 ERROR : FILE REDACTED : Failed to copy: failed to authenticate decrypted block - bad password?
2023/07/11 03:18:04 ERROR : FILE REDACTED : Failed to copy: failed to authenticate decrypted block - bad password?
2023/07/11 03:18:04 ERROR : Attempt 1/1 failed with 47 errors and: failed to authenticate decrypted block - bad password?
2023/07/11 03:18:04 INFO :
Transferred: 140.794 GiB / 140.794 GiB, 100%, 19.827 MiB/s, ETA 0s
Errors: 47 (retrying may help)
Checks: 6193 / 6193, 100%
Elapsed time: 1h57m14.6s
2023/07/11 03:18:04 DEBUG : 9 go routines active
2023/07/11 03:18:04 Failed to copy with 47 errors: last error was: failed to authenticate decrypted block - bad password?
(Thanks by the way for the advice with the tail command; it did end up picking up the rc stats as well, but it led me down the rabbit hole of Regex, so aside from a bit of insomnia it's been quite the trip!)
I've tried repeating the transfer about five times now to no avail. Would you happen to know of a way of either deleting the problematic files in bulk or some setting to skip them?
Thanks for the help!
This looks like some temporary error - but as you tried retrying then maybe these are corrupted files.
It is only 47 of them so should be faster to delete them one by one instead of writing some clever script.
I see.
So I did just that, and the good news is that the transfer went smoothly. The bad news is that the remote seems to only see up to 6193 checks, of which 1,333.5 GB were transferred successfully, but the bucket consists of about 4,908.7 GB, which makes a sizeable difference.
This time around there were no errors in the log, so why would the transfer stop prematurely?
can you run:
rclone size EncFS-test:
rclone size EncFS-NEW:
Sure!
sudo rclone --config=/opt/rclone/rclone.conf size EncFS-test:
Total objects: 6.193k (6193)
Total size: 1.207 TiB (1326653028037 Byte)
sudo rclone --config=/opt/rclone/rclone.conf size EncFS-NEW:
Total objects: 6.193k (6193)
Total size: 1.207 TiB (1326653028037 Byte)
so all is OK.
now if your bucket shows 5T space used it means that you either have some other data.
Or have enabled backblaze versioning.
You can clean old versions by running:
rclone cleanup Backblaze:bucket_name
To disable versioning permanently you have to change it in B2 settings.
I see.
I already ran the command and disabled versioning on the bucket, but even though I've only used it with rclone, there is still a bit of a discrepancy between the buckets:
RCloneEncFS
Created:June 20, 2023
File Lifecycle:Keep only the last version
Current Files:12,848
Current Size:3,160 GB
RCloneEncFS-NEW
Created:July 7, 2023
File Lifecycle:Keep only the last version
Current Files:6,225
Current Size:1,333.5 GB
Even running rclone ls shows almost a doubling of the listed paths (there are two logs linked, one in each word).
I could see the test.txt being a difference, but the amount seems a bit too different for that to be the case. Is there something I may be missing?
It is some mess you created, e.g.:
Your original sync commands:
rclone sync ToasterDEV-GDrive: Backblaze:/RCloneEncFS &
rclone sync Lizita63-OneDrive:/Union Backblaze:/RCloneEncFS &
what sync does? it makes destination the same as source. Deleting stuff from destination if not present in source.
And you sync two sources to the same destination? Result is something but for sure not what you wanted.
The even more bizarre
rclone sync Union:/: Backblaze:/RCloneEncFS &
rclone move Union:/ Backblaze:/RCloneEncFS &
I assume Union:/: is typo as not sure what it means.
and & at the end of every line? did you run all these commands at the same time?
What is possible is that it is not only mess but also some data is encrypted and some not - so you can't access all using crypt.
Okay, that explains it. It seems I misunderstood how sync worked, as I thought it made source and destination the same in every aspect. Going forward I'll be sure to either use copy or move with a --dry-run first.
You're right, it is a typo.
No, I ran each of them one at a time but in the background so I could disown them and keep the rc web gui running to keep an eye on them in case my SSH session got disconnected.
Oh well, honestly it could be worse. Fortunately the files were not mission critical, so at least this will serve me as a lesson on how NOT to handle file operations with rclone. Then I guess I'll go ahead and mount the new bucket to begin checking for the missing files and looking into handling replacements.
Thanks for all the help! You've been really kind and timely, please tell me if there's anything I might be useful for.
Yes now I am 99% sure that problem is with syncing multiple chunked parts of the union to the same destination. So there are many missing chunks - as sometimes subsequent sync was deleting chunks synced before. Some files survived - but I would not be sure if they are 100% correct.
I think that key problem is that you over engineered it.
It could be as simple as:
rclone sync Union: Backblaze:RCloneEncFS
And all would be perfectly correct.
By making things complex without reason you create more chances for mistakes.
Good that files were not "mission critical" and that you have right attitude to learn from it:)