Mounting MEGA drive to Google colab takes to long

Help and Support
1

Hi, since my computer is weak I want to load some files to google colab and use provided computational power there. I have succesfully configured my MEGA cloud by running rclone config. The problem is that mounting takes very long time. Just to be clear its about 12GB of images.. (something like 12 000 png files). Is there some way to do it quicker?
And also, do I need to mount again my MEGA cloud to google colab when starting my computer again? If so, is there some workaround?
I have tried to mount folder with one pdf inside, but it also running for 3minutes already :confused:

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Enter name for new remote.
name> MEGA

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
29 / Mega
\ (mega)
Storage> 29

Option user.
User name.
Enter a value.
user> xxxxxxxxxxxxx@gmail.com

Option pass.
Password.
Choose an alternative below.
y) Yes, type in my own password
g) Generate random password
y/g> y
Enter the password:
password:xxxxxxx
Confirm the password:
password:xxxxxxx

Edit advanced config?
y) Yes
n) No (default)
y/n> n

Configuration complete.
Options:

  • type: mega
  • user: xxxxxxxxxxx@gmail.com
  • pass: *** ENCRYPTED ***
    Keep this "MEGA" remote?
    y) Yes this is OK (default)
    e) Edit this remote
    d) Delete this remote
    y/e/d> y

Current remotes:

Name Type
==== ====
MEGA mega

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

!sudo rclone --vfs-cache-mode writes mount MEGA:/workingdir /content/mega

2

hello and welcome to the forum,

what version of rclone - rclone version
latest stable is v1.62.2

when rclone mount starts,
it scans every file in the rclone cache and decides whats need to be done with the file.
only after that does the mount become active.

of course without a rclone debug log, that is just an educated guess.
so need to post a rclone debug log.

3

COMMANDS
!rclone config show

!rclone lsf MEGA:

#Fix fatal error

!apt-get updaten

!apt-get install -y fuse3

!sudo rclone --vfs-cache-mode writes mount MEGA:/workingdir /content/mega --log-level DEBUG

LOG BEFORE DEBUG
type = mega
user = [honzamares408@gmail.com]
pass = Ml5TrJJupFzNfD9U-rPmHq6hT0nympqTtAXfG-efMFlyoI4Q
Download/
Uploads/
Welcome to MEGA.pdf
workingdir/
E: Invalid operation updaten
Reading package lists... Done
Building dependency tree
Reading state information... Done
fuse3 is already the newest version (3.9.0-2).
The following package was automatically installed and is no longer required: libfuse2
Use 'apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.

DEBUG LOG
2023/04/26 06:15:56 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "--vfs-cache-mode" "writes" "mount" "MEGA:/workingdir" "/content/mega" "--log-level" "DEBUG"]
2023/04/26 06:15:56 DEBUG : Creating backend with remote "MEGA:/workingdir"
2023/04/26 06:15:56 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/04/26 06:16:07 DEBUG : fs cache: renaming cache item "MEGA:/workingdir" to be canonical "MEGA:workingdir"
2023/04/26 06:16:07 INFO : mega root 'workingdir': poll-interval is not supported by this remote
2023/04/26 06:16:07 DEBUG : vfs cache: root is "/root/.cache/rclone"
2023/04/26 06:16:07 DEBUG : vfs cache: data root is "/root/.cache/rclone/vfs/MEGA/workingdir"
2023/04/26 06:16:07 DEBUG : vfs cache: metadata root is "/root/.cache/rclone/vfsMeta/MEGA/workingdir"
2023/04/26 06:16:07 DEBUG : Creating backend with remote "/root/.cache/rclone/vfs/MEGA/workingdir"
2023/04/26 06:16:07 DEBUG : Creating backend with remote "/root/.cache/rclone/vfsMeta/MEGA/workingdir"
2023/04/26 06:16:07 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/04/26 06:16:07 DEBUG : mega root 'workingdir': Mounting on "/content/mega"
2023/04/26 06:16:07 DEBUG : : Root:
2023/04/26 06:16:07 DEBUG : : >Root: node=/, err=
2023/04/26 06:16:28 DEBUG : /: Attr:
2023/04/26 06:16:28 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2023/04/26 06:17:07 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/04/26 06:17:28 DEBUG : /: Attr:
2023/04/26 06:17:28 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2023/04/26 06:18:07 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/04/26 06:19:07 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/04/26 06:20:07 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/04/26 06:21:07 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/04/26 06:22:07 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/04/26 06:23:07 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/04/26 06:24:07 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/04/26 06:25:07 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/04/26 06:26:07 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/04/26 06:27:07 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/04/26 06:27:34 DEBUG : /: Attr:
2023/04/26 06:27:34 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=
2023/04/26 06:28:07 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)

4

The mega backend loads all the info about all the files on startup. That is just the way the mega protocol works, so if you have lots of files it will take a long time to startup.

5

But why is it also taking long when trying to mount only one folder with one pdf file inside?

6

It still has to load all the files in your mega account.

You'll need to use a different cloud provider to improve this I think.

7

You think that MEGA is slow in this regard? At first I wanted to use Onedrive but it was really slow when uploading files from PC to cloud. Mega was much faster in this regard.

8

At initial startup, mega is slow if you have lots of files in your drive. Once it has started it is fine.

9

Okay but when I want to use Google Colab, I need to do initial start every time after computer is powered off? Meaning I need to wait for few hours every time until MEGA cloud is mounted?

10

mega is slow on startup.
onedrive is slow for file transfer.

given the very small amounf of data, might find a comprise solution.
such as aws, b2, dropbox, gdrive, wasabi

what do you mean by my computer? your local phyiscal machine or the colab virtual machine?
as long as the colab virtual machine is runing, that does not affect your local machine.

11

I can confirm that MEGA has a problem during login of an account with millions of files. I opened a problem report with them because I literally could not login to my account using their login web page. It would never finish, even after multiple days. They were unable to fix the problem, and I had to cancel the account and abandon MEGA. Some months later, just because I was curious, I could still attempt the login. It still would not complete. So, I imagine that they themselves were unable to delete the files or the account.

12

Just to be sure, how are you verifying whether the mount is ready or not?

The mount runs in foreground by default and you will need to add --daemon to make it run in the background.

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.