With Mega: "undecrypted material" or "undecrypted folder" after sync

JazzTp · May 8, 2020, 7:45pm

Problem:

This is happening with Mega:

I have a tree, I give pupils a link "a" to folder "A".
rclone sync => changes to files or folders nested into folder "A"
3.1. Access with link "a" without being logged in => the modified/new parts appear as "undecrypted folder" or "undecrypted material"
3.2. If I access logging into my Mega account => everything fine.
3.3. Once logged in, if I create new links to the newly created or modified parts => everything fine also without being logged in, BUT THEN I'd have a PROLIFERATION of links and I'd have to log in anyway each time I need to sync, in which case better:
Not use rclone, just log in and modify anything manually through the web interface => link "a" still works on anything.

Question (I'm new to Mega and rclone, I'm probably missing some concepts)

Does rclone create incompatible keys each time it updates files/folders in a previously uploaded tree?
Could rclone, when syncing, retrieve from Mega the key that was previously used for that tree, or could it store the key into the (encrypted) config file when rclone itself created it?

I'm filing this as a suspected bug because it rules out updating via sync, which pretty much means I'd have to use the web interface anyway and not rclone.

I'd have another doubt

If I may, while I'm on it: an upload which took about five hours at 300-800 kBps only took about 9 minutes CPU user time, is rclone really encrypting locally?

rclone version

rclone v1.51.0
os/arch: linux/amd64
go version: go1.13.7

(installed rclone from the GitHub releases page, apparently this platform doesn't allow me to include the URL, I guess it's unnecessary anyway)

OS

Linux Ubuntu 18.04.4 LTS, 64 bit

Cloud storage system

Mega

The command

time rclone -vv sync . nb_mega:ESCUELA/TROMPETA/

A log from the command with the `-vv` flag

lemon@PC:~/Documents/TEACHING/000_NUBE_en2partes/ORIGINAL$ time rclone -vv sync . nb_mega:ESCUELA/TROMPETA/
2020/05/08 16:37:42 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "-vv" "sync" "." "nb_mega:ESCUELA/TROMPETA/"]
Enter configuration password:
password:
2020/05/08 16:37:55 DEBUG : Using config file from "/home/lemon/.config/rclone/rclone.conf"
2020/05/08 16:37:58 DEBUG : doSync.sh: Sizes identical
2020/05/08 16:37:58 DEBUG : doSync.sh: Unchanged skipping
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/01__TROMPETA_FOBA_I__resonancia_lenguaEnFormaDeArco_digitación.mp4: Sizes identical
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/01__TROMPETA_FOBA_I__resonancia_lenguaEnFormaDeArco_digitación.mp4: Unchanged skipping
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/03__TROMPETA_FOBA_I__agarre__envíenSusVideos__mojar__olaDeMar__embocadura__ATENCIÓNlenguaNoDemasiadoAltaAdelante__cantarConElTeclado-afinaciónConTrp-Voz.mp4: Sizes identical
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/03__TROMPETA_FOBA_I__agarre__envíenSusVideos__mojar__olaDeMar__embocadura__ATENCIÓNlenguaNoDemasiadoAltaAdelante__cantarConElTeclado-afinaciónConTrp-Voz.mp4: Unchanged skipping
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/04__TROMPETA_FOBA_I__conceptos_teclado_escalaCromatica_afinarConLaVoz.mp4: Sizes identical
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/04__TROMPETA_FOBA_I__conceptos_teclado_escalaCromatica_afinarConLaVoz.mp4: Unchanged skipping
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/CONTENIDOS.html: Sizes identical
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/CONTENIDOS.html: Unchanged skipping
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/CONTENIDOS.md: Sizes identical
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/CONTENIDOS.md: Unchanged skipping
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/CONTENIDOS.pdf: Sizes identical
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/CONTENIDOS.pdf: Unchanged skipping
2020/05/08 16:37:58 DEBUG : PARTES/DUO___lenguajeSincopado___Qué_se_le_va_a_hacer__puedenElegirLaTonalidad/Qué_se_le_va_a_hacer__concertBbMayor__trompeta-saxTenor.pdf: Sizes identical
2020/05/08 16:37:58 DEBUG : PARTES/DUO___lenguajeSincopado___Qué_se_le_va_a_hacer__puedenElegirLaTonalidad/Qué_se_le_va_a_hacer__concertBbMayor__trompeta-saxTenor.pdf: Unchanged skipping
2020/05/08 16:37:58 DEBUG : PARTES/DUO___lenguajeSincopado___Qué_se_le_va_a_hacer__puedenElegirLaTonalidad/Qué_se_le_va_a_hacer__concertBbMayor__trompeta-trombón.pdf: Sizes identical
2020/05/08 16:37:58 DEBUG : PARTES/DUO___lenguajeSincopado___Qué_se_le_va_a_hacer__puedenElegirLaTonalidad/Qué_se_le_va_a_hacer__concertBbMayor__trompeta-trombón.pdf: Unchanged skipping
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/02__TROMPETA_FOBA_I__ataquesConLenguaAnclada__disociar-cuerdasVocalesLaringeFaringe-cinturónAbdominal__soporteColumnaDeAire__checklist.mp4: Sizes identical
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/02__TROMPETA_FOBA_I__ataquesConLenguaAnclada__disociar-cuerdasVocalesLaringeFaringe-cinturónAbdominal__soporteColumnaDeAire__checklist.mp4: Unchanged skipping
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/CONTENIDOS.odt: Sizes identical
2020/05/08 16:37:58 DEBUG : FOBA_I__VIDEOS/CONTENIDOS.odt: Unchanged skipping
2020/05/08 16:37:58 DEBUG : PARTES/DUO___lenguajeSincopado___Qué_se_le_va_a_hacer__puedenElegirLaTonalidad/Qué_se_le_va_a_hacer__concertCMayor__trompeta-saxTenor.pdf: Sizes identical
2020/05/08 16:37:58 DEBUG : PARTES/DUO___lenguajeSincopado___Qué_se_le_va_a_hacer__puedenElegirLaTonalidad/Qué_se_le_va_a_hacer__concertCMayor__trompeta-saxTenor.pdf: Unchanged skipping
2020/05/08 16:37:58 DEBUG : PARTES/DUO___lenguajeSincopado___Qué_se_le_va_a_hacer__puedenElegirLaTonalidad/Qué_se_le_va_a_hacer__concertCMayor__trompeta-trombón.pdf: Sizes identical
2020/05/08 16:37:58 DEBUG : PARTES/DUO___lenguajeSincopado___Qué_se_le_va_a_hacer__puedenElegirLaTonalidad/Qué_se_le_va_a_hacer__concertCMayor__trompeta-trombón.pdf: Unchanged skipping
2020/05/08 16:37:58 DEBUG : PARTES/SOLFEO_SINCOPADO/Introduccion_al_solfeo_sincopado_v01.001.pdf: Sizes identical
2020/05/08 16:37:58 DEBUG : PARTES/SOLFEO_SINCOPADO/Introduccion_al_solfeo_sincopado_v01.001.pdf: Unchanged skipping
2020/05/08 16:37:58 DEBUG : PARTES/SOLFEO_SINCOPADO/Solfeo_sincopado_–_3_4__SectionA__v01.000.pdf: Sizes identical
2020/05/08 16:37:58 DEBUG : PARTES/SOLFEO_SINCOPADO/Solfeo_sincopado_–_3_4__SectionA__v01.000.pdf: Unchanged skipping
2020/05/08 16:37:58 DEBUG : PARTES/SOLFEO_SINCOPADO/Solfeo_sincopado_–_3_4__SectionB__v01.000.pdf: Sizes identical
2020/05/08 16:37:58 DEBUG : PARTES/SOLFEO_SINCOPADO/Solfeo_sincopado_–_3_4__SectionB__v01.000.pdf: Unchanged skipping
2020/05/08 16:37:58 DEBUG : PARTES/V7alt_Im7___IIm9b5_V7alt_Im7/IIm9b5_V7alt_Im7__passingLines.pdf: Sizes identical
2020/05/08 16:37:58 DEBUG : PARTES/V7alt_Im7___IIm9b5_V7alt_Im7/IIm9b5_V7alt_Im7__passingLines.pdf: Unchanged skipping
2020/05/08 16:37:58 DEBUG : PARTES/V7alt_Im7___IIm9b5_V7alt_Im7/V7_alt__Im7___v01.007.pdf: Sizes identical
2020/05/08 16:37:58 DEBUG : PARTES/V7alt_Im7___IIm9b5_V7alt_Im7/V7_alt__Im7___v01.007.pdf: Unchanged skipping
2020/05/08 16:37:58 DEBUG : PARTES/V7alt_Im7___IIm9b5_V7alt_Im7/V7_alt_alone_v01.001.pdf: Sizes identical
2020/05/08 16:37:58 DEBUG : PARTES/V7alt_Im7___IIm9b5_V7alt_Im7/V7_alt_alone_v01.001.pdf: Unchanged skipping
2020/05/08 16:37:58 INFO  : mega root 'ESCUELA/TROMPETA': Waiting for checks to finish
2020/05/08 16:37:58 INFO  : mega root 'ESCUELA/TROMPETA': Waiting for transfers to finish
2020/05/08 16:38:58 INFO  : 
Transferred:   	       174 / 174 Bytes, 100%, 2 Bytes/s, ETA 0s
Checks:                19 / 19, 100%
Transferred:            0 / 1, 0%
Elapsed time:        59.9s
Transferring:
 *                         logForRcloneForum.txt:100% /174, 0/s, 0s

2020/05/08 16:39:16 INFO  : logForRcloneForum.txt: Copied (new)
2020/05/08 16:39:16 INFO  : Waiting for deletions to finish
2020/05/08 16:39:16 INFO  : 
Transferred:   	       174 / 174 Bytes, 100%, 2 Bytes/s, ETA 0s
Checks:                19 / 19, 100%
Transferred:            1 / 1, 100%
Elapsed time:      1m18.6s

2020/05/08 16:39:16 DEBUG : 13 go routines active
2020/05/08 16:39:16 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "-vv" "sync" "." "nb_mega:ESCUELA/TROMPETA/"]

real	1m34,651s
user	0m0,602s
sys	0m0,021s
lemon@PC:~/Documents/TEACHING/000_NUBE_en2partes/ORIGINAL$

Animosity022 · May 8, 2020, 9:36pm

Can you share your rclone.conf without any keys or passwords? I'm assuming the sync command is what you are using that was posted? That will validate if you are loading to the encrypted remote or not.

It should be pretty easy to tell though as if you are seeing encrypted names and such, it's encrypted.

I can't quite figure out what you are asking here. The way rclone works is you have a remote and you can copy/sync files to that remote. If you run a sync, you can rune a rclone ls on that remote and check the files that are uploaded there. It keeps your remote configuration in your rclone.conf and that has the authentication that you used to authorize that remote. If it needs to refresh a key, it does that all by itself.

As for creating links and doing things in mega, perhaps someone else can chime in as I'm not familiar with mega from a user perspective as I can just explain how rclone works with a remote.

JazzTp · May 8, 2020, 10:59pm

[nb_mega]
type = mega
user = mymail@domain.xxx
pass = looksLikeAHashWithh_someUnder-scoreandsome-dashCharsWithinIt

Yes

I'm sorry, I tried to be as clear as possible, through those points 1 to 4, but most evidently I wasn't.

The new/replaced material is there after the sync.

Let's say I spent countless hours preparing material for my pupils in an arts school – @#&! this lockdown – and want to be able to give them 2-3 links to folders into which I'll be updating/adding material as time goes by, e.g. one tree for VIDEOS_1st_YEAR, one for SYNCOPATED_SOLFEGE, etc..

It turns out that using rclone sync to push the new material KILLS the readibility of that material via the few links I have already issued, those links to folders no longer permit to read all the contents, I should login and issue new links to all the newly written files, one single folder ends up needing more than one link to read its contents.

IF AND ONLY IF I'm still unclear:

I can spend more words to describe the same, by I doubt you have the time to read through this lot of lines:

After the sync operation:

If I access Mega through the web interface and log in, I can read all of it.
BUT if I simulate being one person to whom I want to share my material and I navigate with the web browser, without logging in, to a previously issued link to a parent folder, the beginning of a tree, or the "root" of the tree, as you prefer, I see as "undecrypted material" or "undecrypted folder" anything that was added or replaced into that tree by the sync operation.

What rclone sync has just written is not readable via the previously issued link, although unchanged files still are readable via the previously issued link.

Meaning: not the whole tree is now readable via one single link to a parent folder.

I'd have to issue new links specific to anything which has been added/replaced by rclone in that tree...
... Which pretty much would make using rclone pointless, and Mega itself, if the goal is to share a tree – material I prepared for pupils – without having to issue specific links to any subfolder or file I might update/add in time.

I guess this is not the intended way rclone should work, I must be experimenting an ongoing problem.

Shouldn't rclone sync be able to add/replace part of the contents of a tree AND keep the uploaded (newly added or modified) files readable via a previously issued link to a parent folder?

Animosity022 · May 8, 2020, 11:44pm

You aren't using encryption since you are not using a crypt remote.

That is written up here:

https://rclone.org/crypt/

Sync makes the source the same in the destination

If you don't want to sync, you can use copy instead. Copy makes the source mirror the destination without deleting any content on the destination.

I'm not sure as you'd have to share a debug log and if something is going wrong, we can see. You can create a debug log by using -vv on your command sharing the log if you want to troubleshoot it.

JazzTp · May 9, 2020, 2:53am

Thank you.
As I'm not encrypting, that is not the source of the problem.

Not encrypting locally => one more reason to expect the default 'rclone -vv sync ...' behavior to act like 'rsync -avu --delete ...' except connected to the cloud storage service, which is: what it uploads, is just uploaded as if I did it via the web interface, leaving to the server the encryption task (Mega encrypts anything by default anyway) => maintained compatibility with a previously issue link to a parent folder.

But it is not happening that way.

Reading that page it's clear I lack info because it's not clear to me.

Yes, I'd like to sync, just like 'rsync -avu --delete ...', and then be able to open the whole tree with one link, the one I already had to the tree root folder.

You could test this yourself:

create a free Mega.nz account
upload a small tree you have on your local hard disk, with nested folders and a few files (via rclone copy or rclone sync or via the Mega web interface, the way you like)
with a web browser, log into Mega and get the link to the root folder of your backed up tree
logout from Mega
on your local hard disk, change / add something
perform an rclone -vv sync ...
with a browser, still without being logged in Mega (or if you use Firefox you can create a separate private tab with ctrl-shift-p), navigate to the same link you already got:

you should now find out that the files you have added or changed locally and which were synced upwards are now appearing as "undecrypted folders" or "undecrypted material".

=> Bottom line, you can no longer give out a single link permitting to read the whole tree on Mega.

Each time you sync, you would have a proliferation of links, if you wanted to let somebody access all the contents parts of that tree.

I did use -vv and did paste the console output into a previous post, as you had previously suggested.

Never mind, don't bother, I'm certainly missing something, which must be making my comm channel obfuscated.

Thank you for your time.

Animosity022 · May 9, 2020, 3:03am

When someone edits a post, I'd have to scroll back up and figure out you changed so I didn't see it.

Good luck.

JazzTp · May 9, 2020, 5:19am

Sorry, your answer made me realize that you hadn't actually previously asked me to add the output of rclone -vv ..., I actually included it from my very first post here, because I came here with the post already in the clipboard, I had already prepared it according to the guidelines given by the GitHub rclone repository when creating an issue... I was originally going to create an issue on GitHub, then I read the recommendations and decided to come to the forum. And it's no problem at all if you forgot about that or didn't even read it the first time, I forget plenty of things all the time (and remember plenty of others), and we must all be "parsing" thousands of lines per day of news, tech stuff, chat messages, etc..

Unimportant further detail: when starting an issue in GitHub for rclone, I got this input "form", which I followed, then posted here instead of creating the isuse on GitHub:

<!--

Welcome :-) We understand you are having a problem with rclone; we want to help you with that!

If you've just got a question or aren't sure if you've found a bug then please use the rclone forum:

    https://forum.rclone.org/

instead of filing an issue for a quick response.

If you think you might have found a bug, please can you try to replicate it with the latest beta?

    https://beta.rclone.org/
    
If you can still replicate it with the latest beta, then please fill in the info below which makes our lives much easier.  A log with -vv will make our day :-)

Thank you

The Rclone Developers

-->

#### What is the problem you are having with rclone?



#### What is your rclone version (output from `rclone version`)



#### Which OS you are using and how many bits (eg Windows 7, 64 bit)



####  Which cloud storage system are you using? (eg Google Drive)



#### The command you were trying to run (eg `rclone copy /tmp remote:tmp`)



#### A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)

Bye, thanks again. I'll check rclone in a few months again.

Animosity022 · May 9, 2020, 10:06am

Your first show has debug output but doesn't show there are any issues. If you want to create a debug and replicate the issue, we can help out.

I read it the first time. I was mentioned we don't get updates when you edit the post.

Right as we're trying to figure out if you actually have an issue as we can't seem to get there yet. If you want to add more information, we can help out.

JazzTp · May 10, 2020, 12:00am

Thank you.
You mean you went through these steps and you didn't get to the same problem?

JazzTp:

You could test this yourself:

create a free Mega.nz account

upload a small tree you have on your local hard disk, with nested folders and a few files (via rclone copy or rclone sync or via the Mega web interface, the way you like)

with a web browser, log into Mega and get the link to the root folder of your backed up tree

logout from Mega

on your local hard disk, change / add something

perform an rclone -vv sync ...

with a browser, still without being logged in Mega (or if you use Firefox you can create a separate private tab with ctrl-shift-p), navigate to the same link you already got:you should now find out that the files you have added or changed locally and which were synced upwards are now appearing as "undecrypted folders" or "undecrypted material".

=> Bottom line, you can no longer give out a single link permitting to read the whole tree on Mega.

ncw · May 10, 2020, 3:58pm

The rclone implementation of the mega backend is based reverse engineering the mega sync program which is a giant, badly documented C++ program.

There is no public documentation for the mega API alas.

This does lead to incompatibilities sometimes and we will do our best to fix them.

Rclone does encrypt data when it sends it to mega - that is part of the protocol and you don't need the crypt backend for that.

I'm not sure what is causing this - I suspect it is something to do with the multiple encryption keys necessary for sharing things on Mega.

I can replicate this

Removing the link and remaking it causes it to work again

You can use rclone with the MegaCMD that mega provide - see this comment - that is guaranteed 100% compatible and you can use all of rclone's features. You can't run two at once though.

So I expect rclone is missing some small detail in the mega protocol which is causing this

JazzTp · May 11, 2020, 1:26am

Thanks for replying, that's it!

Oh I see, well you already did a great job, then.

Interesting, I hadn't tested.

Also interesting, thank you. In the meantime, I've installed the MEGAsync app, and I must admit it's very well conceived. AppArmor isn't issuing any weird complaint as far as I can tell. But I tend to prefer the command line if it is an option, so I might be giving rclone+MegaCMD a try one of these days.

Again, thank you.

ncw · May 12, 2020, 11:25am

I'd like to make that easy at some point, so make it part of the config user interface.

Thank you too and apologies for the problems.

system · July 12, 2020, 7:25am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.