So I used to use rclone wtih gdrive for my plex server, but recently moved to mega as they have 4tb / €19.99/mo instead of the 1tb / €9.99 i was paying google.
Anyway, all was working well after the transition up until a few days ago.
I use this rclone mount for backups as well, this can illustrate the problem most clearly. So when I run
However, when I check through the mega web client, my daily backups seem to have been going through, so my file list should be:
backup-configs-ndo2-Jun-20.tar.gz TO backup-configs-ndo2-Jun-26.tar.gz
I also do deletions in my backup script with rclone delete ... --min-age 6d - these obviously have not been working because it keeps thinking my oldest file is the Jun-16 one which is no longer there…
Is there some sort of cache maybe that is not being refreshed / deleted where it will not or cannot fetch the new file list? This isnt supperrr critical as my backups are still being pushed to mega, however this obviously is not the correct behavior… Any idea whats going?
So yes, in short - I am missing files when listed through rclone.
I do not use the rclone backend cache - I just thought maybe it kept another smaller cache of contents of the remotes to make browsing files quicker.
The amount of files, according to the above command, is 1244.
Its hard to do a sync, because I dont have the contents of even any folder on my mega remote on the server having rclone troubles - and I dont want to sync from my remote to my server as that is wayyy too much content / too large of files.
EDIT: I ran it as a dry-run from my remote to my server. The output was as follows:
rclone sync mega:ndoX_backup/daily . -vv --dry-run
2018/06/26 21:28:57 DEBUG : rclone: Version "v1.41-075-ga193ccdb-drive_v2_downloadβ" starting with parameters ["rclone" "sync" "mega:ndoX_backup/daily" "." "-vv" "--dry-run"]
2018/06/26 21:28:57 DEBUG : Using config file from "/home/ndo/.config/rclone/rclone.conf"
2018/06/26 21:28:57 INFO : mega root 'ndoX_backup/daily': Modify window not supported
2018/06/26 21:28:57 NOTICE: backup-configs-ndo2-Jun-16-18.tar.gz: Not copying as --dry-run
2018/06/26 21:28:57 NOTICE: backup-configs-ndo2-Jun-19-18.tar.gz: Not copying as --dry-run
2018/06/26 21:28:57 NOTICE: backup-configs-ndo2-Jun-20-18.tar.gz: Not copying as --dry-run
2018/06/26 21:28:57 NOTICE: backup-configs-ndo2-Jun-21-18.tar.gz: Not copying as --dry-run
2018/06/26 21:28:57 NOTICE: backup-configs-ndo2-Jun-22-18.tar.gz: Not copying as --dry-run
2018/06/26 21:28:57 NOTICE: backup-configs-ndo2-Jun-18-18.tar.gz: Not copying as --dry-run
2018/06/26 21:28:57 INFO : Local file system at /home/ndo/backups: Waiting for checks to finish
2018/06/26 21:28:57 NOTICE: backup-configs-ndo2-Jun-17-18.tar.gz: Not copying as --dry-run
2018/06/26 21:28:57 INFO : Local file system at /home/ndo/backups: Waiting for transfers to finish
2018/06/26 21:28:57 INFO : Waiting for deletions to finish
2018/06/26 21:28:57 NOTICE: backup-Feb-26-18.log: Not deleting as --dry-run
2018/06/26 21:28:57 NOTICE: backup-Feb-27-18.log: Not deleting as --dry-run
2018/06/26 21:28:57 INFO :
Transferred: 0 Bytes (0 Bytes/s)
Errors: 0
Checks: 2
Transferred: 7
Elapsed time: 800ms
2018/06/26 21:28:57 DEBUG : 5 go routines active
2018/06/26 21:28:57 DEBUG : rclone: Version "v1.41-075-ga193ccdb-drive_v2_downloadβ" finishing with parameters ["rclone" "sync" "mega:ndoX_backup/daily" "." "-vv" "--dry-run"]
As you can see it grabbed the incorrect file list again. I’ll include a screenshot of that folder (/ndoX_backup/daily/) from the mega webclient:
This must be some kind of problem at mega - some kind of caching problem. I don’t think rclone will be making up those file names by itself.
is it possible that the web interface you are looking at is not the same mega account that rclone is syncing with? (Best to check the obvious things before we go too far down the rabbit hole!)
Ah, I have an idea. Are there two identically named directories (it might be in any of the parents)? mega can have duplicate directories. You can use rclone dedupe to remove duplicates (despite the docs rclone dedupe should work with mega too).
Hmm I dont think so, below is a screenshot of my dir structure - inside the bottom most folders (tv, music, movies) are just folders of the names of TV shows (Adventure Time, The Wire, etc.) and the same for music albums and movies…
They don’t look duplicated do they… I think a duplicated directory would fit the symptoms perfectly though… As in rclone is backup up to a different directory than you are looking at in the web interface.
Maybe you could try copying a single file with an unusual name into that directory with rclone
Then see if it appears in the rclone listing rclone ls mega:ndoX_backup/daily and in the web interface. If it doesn’t appear in the web interface try using the search to find it!
Another oddity - so as I mentioned I also use this mega remote for storing my plex media. I just noticed that my couchpotato / sickrage auto-downloaded media successfully gets uploaded via my own custom rclone script and is visible via the mega web UI anddd in plex, but not the rclone cli. I just tried to play a newly added media file and it worked perfectly, but via rclone ls mega:plex_enc/tv/... the episodes only go up to Jun 21. Right around when the daily backups stopped appearing via the rclone cli
This is sounding like a mega problem rather than an rclone problem…
Can you use the inspector in the browser to work out which API endpoint the browser is using? I just tried it and rclone and the browser are using g.api.mega.co.nz - I wonder if your browser is using a different endpoint.
I also found https://g.api.mega.co.nz/ as the api endpoint in the secureboot.js file loaded by the web app in my browser. Is there a better/more accurate place to look?
EDIT: I also just downloaded the latest stable version of rclone on a new ubuntu desktop here at work (18.04) and added my mega remote again and got the same results, only the backups from Jun16-Jun22 are shown.
So it does look like it is a problem on Megas part, huh?
I was looking that the network transactions in the console but that is probably good enough.
Hmm, maybe! The mega API is undocumented so the use of it was reverse engineered from their C++ command line client. It might be worth trying listings with that and if you get the same problem or not.
The daily automated integration tests have not shown any problem with mega in general.
Can you try running a list with -vv --mega-debug and see if it prints anything unusual?
You can also run with -vv --dump bodies if you want to see what mega returns, but it is incomprehensible as mega name all their JSON blobs with single characters and the data is all encrypted!
If the MEGAcmd client works then I’ll try to think of some ways of debugging this problem.
Hmm okay very strange, I pulled the latest .deb of megaCMD from their site, logged in and run mega-ls with the same path (ndoX_backup/daily) and got the following output:
$(ndo@ndo4)-(~/Downloads)-(12:33 Do Jun 28)->
$(jobs:0)-(33 files, 375Mb)-> mega-ls ndoX_backup/daily
backup-configs-ndo2-Jun-23-18.tar.gz
backup-configs-ndo2-Jun-24-18.tar.gz
backup-configs-ndo2-Jun-25-18.tar.gz
backup-configs-ndo2-Jun-26-18.tar.gz
backup-configs-ndo2-Jun-27-18.tar.gz
backup-configs-ndo2-Jun-28-18.tar.gz
potato.txt
So that seems to be correct…
running rclone ls with -vv --mega-debug outputs:
$(ndo@ndo4)-(~/Documents/newtelco)-(12:27 Do Jun 28)->
$(jobs:0)-(35 files, 50Mb)-> rclone -vv --mega-debug ls mega:ndoX_backup/daily
2018/06/28 12:27:50 DEBUG : rclone: Version "v1.42" starting with parameters ["rclone" "-vv" "--mega-debug" "ls" "mega:ndoX_backup/daily"]
2018/06/28 12:27:50 DEBUG : Using config file from "/home/ndo/.config/rclone/rclone.conf"
3614397493 backup-configs-ndo2-Jun-16-18.tar.gz
3615225405 backup-configs-ndo2-Jun-17-18.tar.gz
3711552307 backup-configs-ndo2-Jun-18-18.tar.gz
3712047377 backup-configs-ndo2-Jun-19-18.tar.gz
3713137151 backup-configs-ndo2-Jun-20-18.tar.gz
3713579560 backup-configs-ndo2-Jun-21-18.tar.gz
3713823950 backup-configs-ndo2-Jun-22-18.tar.gz
2018/06/28 12:27:50 DEBUG : 5 go routines active
2018/06/28 12:27:50 DEBUG : rclone: Version "v1.42" finishing with parameters ["rclone" "-vv" "--mega-debug" "ls" "mega:ndoX_backup/daily"]
with --dump bodies just outputted a bunch of sinlge letter variables and encrypted values - as you said.
Just a note that I am seeing these exact results as this as well. Not just with rclone, but other command line mega tools as well. (Though the official MEGAcmd does return accurate results – I just can’t install it on my server as it requires root)
I could understand a bug in rclone missing out some directory entries - that kind of thing has happened before. But making up directory entries? That is very strange.
There don’t appear to be any file in common with the two listings which again makes me think of duplicate directories.
I wonder… Is the directory rclone is listing in the trash?
Can you use the MegaCmd to do a recursive listing of the files and try to find the backup-configs-ndo2-Jun-16-18.tar.gz file? It must be in your mega drive somewhere? I don’t know whether you can list the trash too, if you can then please check there too (or check on the website).
I think those files must be there somewhere!
Interesting. Which command line mega tools did you try?
It doesn’t look like to me that it is made up files or files in the trash, or even another directory. It’s appears as though the results returned are cached from a couple days ago – new results / changes are not showing.
I’ve been talking with Mega support on this and they pointed out that their official SDK seems to be working, it’s the same one used by their mobile apps (and the above mentioned MEGAcmd). I have verified this – that it is working properly with MEGAcmd and their mobile app, but rclone (and others like the old megacmd [github.com/t3rm1n4l/megacmd] and megatools [github… megous/megatools]) are not working.
rclone and megatools are not using our SDK. Therefore, they are unable to
receive updates to the account state.
What we do on our side to reduce loading times for properly implemented
clients is to take snapshots of accounts that have exceeded a certain
threshold size (such as yours). When you log in, you get that snapshot
(which is very fast) plus the delta updates that have happened since
the snapshot was taken (which is also very fast). Your client is then
supposed to apply those delta updates to get a correct view of the
remote state. If the client fails to do so, the view remains outdated.
We could now simply send rclone a freshly generated copy of your account
state every time it asks. However, if the account is large, this will
take a while and, worse, cause infrastructure load at our end that we
are keen to avoid.
Furthermore, we don’t really see rclone as particularly useful if it
doesn’t get updates in real time, but has to poll for it?
We strongly urge the authors of rclone to do things properly. Our SDK
hides all the complexity in relation to delta update processing and
gives the app developer convenient notifications of actual state
notifications at a fairly high level of abstraction.
If you could please make a new issue on github with a summary of the problem and that statement from mega, that would be really helpful.
There is obviously some bit of the API that rclone is missing or some bit of it which is broken. rclone should be applying those updates but it doesn’t appear to be. This would be a lot easier to debug if mega’s API wasn’t so cryptic!
Note that megacmd is based off the same library https://github.com/t3rm1n4l/go-mega as rclone. Note also that I took over maintaining the library because the original author had lost interest and I needed to add a lot more features for rclone support.
Would it be possible for one or the other of you to give me access to your account temporarily? I know that is a big ask but this problem will be a lot easier to fix if I have access to an account which isn’t working. All I’ll be doing is doing directory listings - I won’t upload or download any files. If you are OK with that then please email details to nick@craig-wood.com - along with the paths of some files you can see in the web interface but not via rclone.
Okay so somewhat good news, right? At least we have the problem somewhat nailed down!
Seeing as your already somewhat familiar with my account / account structure I’ll give you access…
Give me a little bit to remove some of the more personal things and change the PW, etc. and I will send the details to the above provided e-mail address.
