Rclone using a lot of data and not uploading files effectively

Setup

I used the crypt remote and folder/remote solids is dependent upon folder/remote mega_sync.

So, anything in solids remote is going to be encrypted and uploaded in mega_sync
because it's connected to crypt remote solids.

So, the result should be, I get a file mirroring my entire Documents folder in Mega

This is happening, but except for some problems with syncing between Documents Folder and solids folder. solids folder seems to be making multiples of some files.

solids---> mega_sync is working as intended.

The main problem, I think is with, rclone sync '/home/bob/Documents/' '/home/bob/Downloads/solids/'

Under no circumstances should this file create mutiple copies of the same file, but this seems to have happened here.

What is the problem you are having with rclone?

Also, these are encrypted files. I think you would be able to tell if you look at my config but I don't know how to effectively explain myself :frowning:

rclone seems to be stuck at the last largest file.

edit: This is no longer the problem as after taking an insane amount of bandwidth, it seemed to have uploaded the file successfully. But I added a 4mb file to the mix and rclone is still drinking data bandwidth like it's water and it's not uploading the 4mb file.

edit 2: After eating through an insane amount of data, it was successful in uploading the 4 mb file, but it seems to have uploaded a lot more, because the whole folder size is double what it is. It should be 750 mbs it's 1.489... GB rn in Mega

edit 3: rclone is making duplicates
image
They are supposed to be the same file. I cut and paste it with a different name to check how it works.

image
It's supposed to be like this, there should only be one file. Because I cut and paste the thing and most importantly this is the sub folder of the source folder, so the solids remote should mirror this as I using command rclone sync '/home/bob/Documents/' '/home/bob/Downloads/solids/'

Run the command 'rclone version' and share the full output of the command.

(base) bob@bob:~
$ rclone version
rclone v1.63.1
- os/version: debian 11.7 (64 bit)
- os/kernel: [redacted] (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.6
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Mega, but I think this problem seems to be occur on other sites too

The command you were trying to run (eg rclone copy /tmp remote:tmp)

echo "Syncing now!"
rclone sync '/home/bob/Documents/' '/home/bob/Downloads/solids/' -P -vv --log-file=rclonesync_upload_to_rcloneforum.txt & echo "Mounting mega_sync" & rclone mount mega_sync: /home/bob/Downloads/mega_sync --vfs-cache-mode writes -P -vv --log-file=mounting_folder_connected_to_mega_i_e_mega_sync_upload_to_rcloneforum.txt & echo "Mounting solids" & rclone mount solids: /home/bob/Downloads/solids/ --vfs-cache-mode writes --allow-non-empty -P -vv --log-file=mounting_folder_where_crypt_remote_mounts_upload_to_rcloneforum.txt

In essence, I am trying to run 3 separate commands, these are
[1] rclone sync '/home/bob/Documents/' '/home/bob/Downloads/solids/' -P -vv --log-file=rclonesync_upload_to_rcloneforum.txt (haven't uploaded the log, is it necessary?)

[2] rclone mount mega_sync: /home/bob/Downloads/mega_sync --vfs-cache-mode writes -P -vv --log-file=mounting_folder_connected_to_mega_i_e_mega_sync_upload_to_rcloneforum.txt

[3] rclone mount solids: /home/bob/Downloads/solids/ --vfs-cache-mode writes --allow-non-empty -P -vv --log-file=mounting_folder_where_crypt_remote_mounts_upload_to_rcloneforum.txt

The rclone config contents with secrets removed.

$ rclone config
Current remotes:

Name                 Type
====                 ====
mega_sync            mega
solids               crypt

A log from the command with the -vv flag

Log of second command [2] rclone mount mega_sync: /home/bob/Downloads/mega_sync --vfs-cache-mode writes -P -vv --log-file=mounting_folder_connected_to_mega_i_e_mega_sync_upload_to_rcloneforum.txt

Log of Third Command [3] rclone mount solids: /home/bob/Downloads/solids/ --vfs-cache-mode writes --allow-non-empty -P -vv --log-file=mounting_folder_where_crypt_remote_mounts_upload_to_rcloneforum.txt

hi,
am i correct that you are using rclone sync to copy files to rclone mount?
why not just rclone sync direct to mega?

I am sorry, but I am actually encrypting it. Should I have mentioned it?

does not matter but the questions remains - why rclone sync into rclone mount ?

why not something simple, such as
rclone sync /home/bob/Documents solids:Documents

I am sorry, I am not very good at this :frowning:

I installed this thing today
ughh... but I would be glad if you could help me solve this problem.

you might be making this too complicated.

if you want to sync from local to mega, i shared an example command.

but I want it encrypted too. so I used the crypt remote and folder/remote solids is dependent upon folder/remote mega_sync.

So, anything in solids remote is going to be encrypted and uploaded in mega_sync
because it's connected to crypt remote solids.

So, the result should be, I get a file mirroring my entire Documents folder in Mega

This is happening, but except for some problems with syncing between Documents Folder and solids folder. solids folder seems to be making multiples of some files.

solids---> mega_sync is working as intended.

The main problem, I think is with, rclone sync '/home/bob/Documents/' '/home/bob/Downloads/solids/'

Under no circumstances should this file create mutiple copies of the same file, but this seems to have happened here.

edit: Thanks for sticking with me and trying to help :')

IMO, you are making this way too complicated.
sorry, nothing you have written explains why you need to use rclone sync and rclone mount at the same time.

  1. kill all the rclone mount commands that might be running. do not run rclone mount
  2. i already posted an example command.

Actually, I cut the document from Documents and then pasted it in Desktop (not relevant)

First the name was SSP full.pdf I changed it to SSP complete.pdf and pasted it back in.

Now, the document folder contains SSP complete.pdf only.

But after syncing, it didn't delete SSP full.pdf and then only keep SSP complete.pdf but it kept both, even though at this time, Document has only one file SSP complete.pdf and it should be syncing from mirror source.

Is there a better way to do what I am doing, this part, two folders part we discussed previously?

not sure how else to re-explain.
now, i think we are starting to go around in circles.

do not use rclone mount
kill all rclone mount commands
just use rclone sync

yes it is relevant, impossible to understand what you have done in the past.

just start over, delete the relevant folders in mega

hey, I will do that, thank you for your help! I got what you are saying now.

1 Like

you are most welcome.
sorry if i was direct but glad you got what i was saying now.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.