Copy HDD to network share (OneDrive)

Help and Support
1

Hi.
I want to move my files from external HDD (WD Elements 1TB, mounted as /Volumes/Elements) to the network share (Microsoft OneDrive).
After moving around 100GB (my HDD is full in 80% - 800GB) rclone is hanging.
I'm not sure if I'm using the proper commands.

Steps:

  1. rclone configuration (add network share: share_onedrive)
  2. mount network share
    rclone --vfs-cache-mode writes mount share_onedrive: /Users/jkacz/share_onedrive_folder/
  3. simultaneously in new terminal: sync HDD to OneDrive.
    rclone sync --progress /Volumes/Elements/ /Users/user/share_onedrive_folder/

I want to use OneDrive to use the files now, and create a backup to HDD every month.
I'll try to use the command similar to the step number 3:
*rclone sync --progress /Users/user/share_onedrive_folder/ /Volumes/Elements/ *

My question is: is this a proper way to sync the files?
Currently the process takes hours and hours (I thought copying should be much faster).
Also sending the files to the cloud is using only 20% of bandwith and it is bouncing (like 300kb/s for two seconds, 3MB/s for two seconds and again...)

Thank you in advance for your help.

System info:
MacOS Catalina (10.15.7)
rclone v1.53.3
os/arch: darwin/amd64
go version: go1.15.5

2

hello and welcome to the forum,

IMHO, not the way to use rclone.
you are using rclone twice

  1. rclone mount
  2. rclone sync to that rclone mount

no need for rclone mount for such a simple copy/sync operation.
try rclone sync /Volumes/Elements/ share_onedrive:

and if for some reason, you want to double use rclone sync to a rclone mount, then try not using vfs cache, and you will not run out of hard drive space

3

@asdffdsa right, thanks!
Now it is working properly, it's not hanging. I wanted to have a preview of the files, so I used two commands, but it seems to be not the right way.

I have a few questions more:

  1. I have a visualisation of the progress:
    Transferred: 706.071M / 208.485 GBytes, 0%, 3.243 MBytes/s, ETA 18h13m29s
    The amount of the total data is wrong. I assume it will grow automatically to the right size in time, right? (It's because the whole HDD was not scanned at the beginning if I understand correctly)
    Edit: --check-first solved the problem

  2. The ETA: ETA 18h13m29s - this is the time to send the data (currently 208.485 GBytes) to the OneDrive, right? And it is including the network bandwith limitations, right?

  3. Currently I stopped rclone, and I want to see the files.
    I'm using the command rclone mount, but I'm getting the error: Fatal error: Directory is not empty:.
    I can remove the content of the directory, or use --allow-non-empty parameter, but it will cause downloading the files from the beginning. Is it possible to mount it and sync the files with the folder (in the folder there are some files from OneDrive, but some new are missing, and I want to mount it and sync only the differences).

  4. I can see, that the files are send simultaneosuly (few at the same time). Is it ok for the health of the HDD? Maybe I should limit amount of files to 1, to force rsync to read the files sequentially, as they were written on HDD?
    Edit: I think it has a difference - I've added --transfers=1 parameter

  5. I can see the progress of copying the file, and some files have been copied in 100%, but I'm checking it on my phone, and they are not available.
    I understand, that there is some delay between they will be visible, and it is because of the OneDrive architecture, but if they were copied in 100%, they will be visible in a while for sure, right?

4

your approach is overly complex, hard to give advice on it and all these problems that you have as a result of that.

--allow-non-empty should not be used, unless you really 100% understand what it does, and if you understood what it does, i am sure you would not use it.

5

@asdffdsa Why do you think it's complex? I just want to sync HDD with remote drive.
Do you think I should do something in a different way? Rsync [Edit: Rclone] is a new tool for me. It's complex for the beginner, sorry.

6

i think you mean rclone

if you want to sync from local to onedrive, then all you need is

and use this when testing, as sync can be dangerous
https://rclone.org/docs/#n-dry-run

7

Of course, rclone :slight_smile:
And yes, I'm using the command right now. But all the questions are still valid.
The important is the question number 5 - when I used two commands (mound and sync) I had the moved files on the phone app in the same time. When I'm using one command, they are "sent" in 100%, but I cannot see them in my phone. I don't understand it.

8

are you still double rcloning?
if not, what is the command you are running?

when you first posted, you were asked for information.
if you could post that missing info, then we could better help you.

9

Currently I'm using only the command:
rclone sync /Volumes/Elements/ share_onedrive:
In the command line I can see that files are transferred, but they are not (on the phone app they are not visible, right now after 20 min, it looks like no files were copied).
I'm thinking, maybe there is a limitation of max data transfer in OneDrive? But if yes, then why rclone is showing that everything is ok and files are transferred? Strange :confused:

10

without a debug log and the rclone config file, i cannot see your system, cannot know what is going on.

cannot do anything about a phone app...
if you do a rclone ls share_onedrive:, do you see the files that you thnk were uploaded?

11

It's hard to say, I have a thousands of files. Can I sync (sync) and show (ls) only one folder from the share_onedrive?

12

if you want to see the folders names, rclone lsd share_onedrive:
append a folder name to the end of share_onedrive:
then rclone ls share_onedrive:foldername command.

13

I'm trying to debug it, but right now rclone is sending large files. I need to wait, I don't want to disturb. I will reply after it process to the smaller files.

14

in the mean time, you can still run another rclone command in another command prompt.
so run the rclone lsd or rclone ls, that will not affect that rclone sync

15

Now it looks good, sorry for the confusion. I uploaded 8,8GB file and it was visible immediately on the mobile app.
So I think I don't need anthing more. I will us this command. Thanks for your help @asdffdsa.

16

sure,
rclone can be a bit overwhelming at first, but you seem on the way to be a fellow rcloner

1 Like
17

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.