Sync between two mounted Cloud Drives

PaOne_G · February 20, 2021, 4:03pm

Hello Everyone,

I'm a Newbie & basically an absolute noob seeking help as trying to figure out things scouring through previous topics is not really solving my issue.

I have mounted 2 Cloud Drives i.e. OneDrive(empty-backup) & GDrive(with Data). I'd been attempting to copy/sync a bunch of folders(note with subfolders) one at a time from GDrive to the empty Onedrive but it doesn't seem to work as I'm probably doing something wrong.

Here is the command

rclone sync(or copy) GDrive:/folder1 OneDrive:/

OUTPUT:

Usage:
  rclone sync source:path dest:path [flags]

Flags:
  --create-empty-src-dirs   Create empty source dirs on destination after sync
  -h, --help                    help for sync

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

Basically I'm trying to retain the exact hierarchy and folder structure of the Gdrive with the data.

Any assistance with a solution to this is very much appreciated.

Thank you in advance.

Should there be additional info needed, please let me know.

asdffdsa · February 20, 2021, 4:07pm

hello and welcome to the forum,

when you posted, there was a template of questions for you to answer, so that we can help you.

PaOne_G · March 8, 2021, 6:07pm

Is this accurate?

What is the problem you are having with rclone? - None

What is your rclone version (output from rclone version)
rclone v1.54.0

os/arch: windows/amd64
go version: go1.15.7

Which OS you are using and how many bits- Windows 10, 64 bit

Which cloud storage system are you using? Google Drive & OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync(or copy) GDrive:/folder1 OneDrive:/

A log from the command with the -vv flag
Usage:
rclone sync source:path dest:path [flags]

Flags:
--create-empty-src-dirs Create empty source dirs on destination after sync
-h, --help help for sync

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

asdffdsa · March 8, 2021, 6:11pm

we need to see the the exact command with a debug output
for a debug output, add -vv to the command.

PaOne_G · March 8, 2021, 7:57pm

Thank you for your prompt response

Here is the Output:

C:\Users\Price\Downloads\RCLONEX>rclone sync C1:/sort this folder MindSpace: -vv

2021/03/09 01:25:31 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "sync" "C1:/sort" "this" "folder" "MindSpace:" "-vv"]
Usage:
rclone sync source:path dest:path [flags]

Flags:
--create-empty-src-dirs Create empty source dirs on destination after sync
-h, --help help for sync

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.
Command sync needs 2 arguments maximum: you provided 4 non flag arguments: ["C1:/sort" "this" "folder" "MindSpace:"]

C:\Users\Price\Downloads\RCLONEX>

asdffdsa · March 8, 2021, 8:58pm

if a path has a space character in it, then it must be quotes so try
rclone sync "C1:sort this folder" MindSpace: -vv

https://rclone.org/docs/#windows

PaOne_G · March 8, 2021, 9:31pm

C:\Users\Price\Downloads\RCLONEX>rclone sync "C1:/sort this folder" MindSpace: -vv

2021/03/09 02:51:58 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "sync" "C1:/sort this folder" "MindSpace:" "-vv"]
2021/03/09 02:51:58 DEBUG : Using config file from "C:\Users\Price\.config\rclone\rclone.conf"
2021/03/09 02:51:58 DEBUG : Creating backend with remote "C1:/sort this folder"
2021/03/09 02:52:00 DEBUG : fs cache: renaming cache item "C1:/sort this folder" to be canonical "C1:sort this folder"
2021/03/09 02:52:00 DEBUG : Creating backend with remote "MindSpace:"
2021/03/09 02:52:01 ERROR : : error reading source directory: directory not found
2021/03/09 02:52:01 DEBUG : One drive root '': Waiting for checks to finish
2021/03/09 02:52:01 DEBUG : One drive root '': Waiting for transfers to finish
2021/03/09 02:52:01 ERROR : One drive root '': not deleting files as there were IO errors
2021/03/09 02:52:01 ERROR : One drive root '': not deleting directories as there were IO errors
2021/03/09 02:52:01 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2021/03/09 02:52:02 ERROR : : error reading source directory: directory not found
2021/03/09 02:52:02 DEBUG : One drive root '': Waiting for checks to finish
2021/03/09 02:52:02 DEBUG : One drive root '': Waiting for transfers to finish
2021/03/09 02:52:02 ERROR : One drive root '': not deleting files as there were IO errors
2021/03/09 02:52:02 ERROR : One drive root '': not deleting directories as there were IO errors
2021/03/09 02:52:02 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2021/03/09 02:52:03 ERROR : : error reading source directory: directory not found
2021/03/09 02:52:03 DEBUG : One drive root '': Waiting for checks to finish
2021/03/09 02:52:03 DEBUG : One drive root '': Waiting for transfers to finish
2021/03/09 02:52:03 ERROR : One drive root '': not deleting files as there were IO errors
2021/03/09 02:52:03 ERROR : One drive root '': not deleting directories as there were IO errors
2021/03/09 02:52:03 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2021/03/09 02:52:03 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 4.8s

2021/03/09 02:52:03 DEBUG : 5 go routines active
2021/03/09 02:52:03 Failed to sync: directory not found

Above is the output after executing the said command, I've mounted the remote drive a couple of times and executed the sync but the same error is returned.

The Remote drive "MindSpace" is absolutely empty (no directories or files)with nothing in it.

Not sure what's going on here

asdffdsa · March 8, 2021, 9:35pm

source directory: directory not found
this is about the source, not the dest.

try rclone lsd C1: -vv

if that does not work, then post the config file, redact id/passwords.

PaOne_G · March 8, 2021, 9:38pm

 2021/03/09 03:07:46 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "lsd" "C1:" "-vv"]
2021/03/09 03:07:46 DEBUG : Using config file from "C:\\Users\\Price\\.config\\rclone\\rclone.conf"
2021/03/09 03:07:46 DEBUG : Creating backend with remote "C1:"
          -1 2020-04-24 12:34:15        -1 3TB Courses／Ebooks
          -1 2020-06-15 04:12:10        -1 Computers courses + Programming + Hacking
          -1 2020-06-13 23:02:23        -1 Courses (From Contributor Drive)
          -1 2020-04-19 16:51:48        -1 Education
          -1 2020-04-30 13:07:59        -1 Fitness Guide
          -1 2020-02-19 22:30:32        -1 Practice Eye Contact
          -1 2020-06-14 17:48:53        -1 Unsorted
          -1 2020-05-23 23:35:51        -1 sort this folder
2021/03/09 03:07:46 DEBUG : 4 go routines active

asdffdsa · March 8, 2021, 9:48pm

looks like C1:sort this folder does exist in the source

try rclone ls "C1:sort this folder" -vv

PaOne_G · March 8, 2021, 10:22pm

Not sure why I see it but Rclone doesnt see the directory.
I've deleted the remote, remounted it and tried again.

C:\Users\Price\Downloads\RCLONEX>rclone ls "C1:sort this folder" -vv
2021/03/09 03:50:02 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "ls" "C1:sort this folder" "-vv"]
2021/03/09 03:50:02 DEBUG : Using config file from "C:\\Users\\Price\\.config\\rclone\\rclone.conf"
2021/03/09 03:50:02 DEBUG : Creating backend with remote "C1:sort this folder"
2021/03/09 03:50:04 DEBUG : 4 go routines active
2021/03/09 03:50:04 Failed to ls: directory not found

asdffdsa · March 8, 2021, 10:24pm

try rclone ls -vv on

a folder without a space character
a diferent folder with a space character.

PaOne_G · March 8, 2021, 10:34pm

Other directories seem to show, not sharing the output cause it's a lot lengthy.

Problem seems to be only with directories with a space.
When I try to simply copy, it keeps saying duplicate found ignoring even when the destination is absolutely empty.

C:\Users\Price\Downloads\RCLONEX>rclone copy C1: MindSpace: -vv
2021/03/09 04:08:35 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "copy" "C1:" "MindSpace:" "-vv"]
2021/03/09 04:08:35 DEBUG : Using config file from "C:\\Users\\Price\\.config\\rclone\\rclone.conf"
2021/03/09 04:08:35 DEBUG : Creating backend with remote "C1:"
2021/03/09 04:08:35 DEBUG : Creating backend with remote "MindSpace:"
2021/03/09 04:08:36 DEBUG : Untitled.txt: Size and modification time the same (differ by -497ms, within tolerance 1s)
2021/03/09 04:08:36 DEBUG : Untitled.txt: Unchanged skipping
2021/03/09 04:08:39 NOTICE: Dangling shortcut "The Complete 2018 Mind Mapping Step-By-Step Mastery Course" detected
2021/03/09 04:08:39 DEBUG : Education/[FreeCoursesOnline.Me] [MindValley] Superbrain By Jim Kwik [FCO].tar: Starting multipart upload
2021/03/09 04:08:39 NOTICE: Education/Banking Data Interpretation/Maths DI 1.mp4: Duplicate object found in source - ignoring
2021/03/09 04:08:39 NOTICE: Education/Banking Data Interpretation/Maths DI 10: Duplicate object found in source - ignoring
2021/03/09 04:08:39 NOTICE: Education/Banking Data Interpretation/Maths DI 11.mp4: Duplicate object found in source - ignoring
2021/03/09 04:08:39 NOTICE: Education/Banking Data Interpretation/Maths DI 12: Duplicate object found in source - ignoring
2021/03/09 04:08:39 NOTICE: Education/Banking Data Interpretation/Maths DI 13: Duplicate object found in source - ignoring
2021/03/09 04:08:39 NOTICE: Education/Banking Data Interpretation/Maths DI 14: Duplicate object found in source - ignoring
2021/03/09 04:08:39 NOTICE: Education/Banking Data Interpretation/Maths DI 15: Duplicate object found in source - ignoring

asdffdsa · March 8, 2021, 10:47pm

this is about the source, not the dest

gdrive allows for duplicates, rclone cannot handle them and will not work well.
you need to remove the duplicates.
Google drive

not sure what to tell you, as i never use a space character in a path.
but as a test

create a folder with a space
copy a file to that folder
list the files in that folder

c:\data\rclone\scripts\rclone.exe mkdir "gdrive-a1b2:path with space" -vv 
DEBUG : rclone: Version "v1.54.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "mkdir" "gdrive-a1b2:path with space" "-vv"]
DEBUG : Creating backend with remote "gdrive-a1b2:path with space"
DEBUG : Using RCLONE_CONFIG_PASS password.
DEBUG : Using config file from "c:\\data\\rclone\\scripts\\rclone.conf"
DEBUG : Google drive root 'path with space': Making directory
DEBUG : 4 go routines active

c:\data\rclone\scripts\rclone.exe copy C:\data\rclone\scripts\rr\other\test.txt "gdrive-a1b2:path with space" -vv 
DEBUG : rclone: Version "v1.54.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "copy" "C:\\data\\rclone\\scripts\\rr\\other\\test.txt" "gdrive-a1b2:path with space" "-vv"]
DEBUG : Creating backend with remote "C:\\data\\rclone\\scripts\\rr\\other\\test.txt"
DEBUG : Using RCLONE_CONFIG_PASS password.
DEBUG : Using config file from "c:\\data\\rclone\\scripts\\rclone.conf"
DEBUG : fs cache: adding new entry for parent of "C:\\data\\rclone\\scripts\\rr\\other\\test.txt", "//?/C:/data/rclone/scripts/rr/other"
DEBUG : Creating backend with remote "gdrive-a1b2:path with space"
DEBUG : test.txt: Need to transfer - File not found at Destination
DEBUG : test.txt: MD5 = b59c67bf196a4758191e42f76670ceba OK
INFO  : test.txt: Copied (new)
INFO  : 
Transferred:   	         4 / 4 Bytes, 100%, 2 Bytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         2.9s

c:\data\rclone\scripts\rclone.exe ls "gdrive-a1b2:path with space" -vv 
DEBUG : rclone: Version "v1.54.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "ls" "gdrive-a1b2:path with space" "-vv"]
DEBUG : Using RCLONE_CONFIG_PASS password.
DEBUG : Using config file from "c:\\data\\rclone\\scripts\\rclone.conf"
DEBUG : Creating backend with remote "gdrive-a1b2:path with space"
        4 test.txt

PaOne_G · March 8, 2021, 11:04pm

asdffdsa:


c:\data\rclone\scripts\rclone.exe copy C:\data\rclone\scripts\rr\other\test.txt "gdrive-a1b2:path with space" -vv 
DEBUG : rclone: Version "v1.54.0" starting with parameters ["c:\\data\\rclone\\s

Nothing much in there, except this

[C1]
type = drive
scope = drive
token =
team_drive = 0AGrE5Xhf10CaUk9PVA
root_folder_id =

asdffdsa · March 8, 2021, 11:18pm

you marked ths topic as solved, does that mean you got it working now?

timevortex · March 9, 2021, 3:07am

I see a forward slash after C1: that is likely the problem. It should be "C1:sort this folder" rather than "C1:/sort this folder".

asdffdsa · March 9, 2021, 3:00pm

yes, without the slash is the correct syntax.
https://rclone.org/docs/#remote-path-to-dir

but if you look at the debug log here, Sync between two mounted Cloud Drives - #7 by PaOne_G
rclone removes the slash for you.
fs cache: renaming cache item "C1:/sort this folder" to be canonical "C1:sort this folder"

timevortex · March 9, 2021, 3:47pm

Ah I missed that. Neat!

PaOne_G · March 10, 2021, 6:36am

For some reason the google mounted remote drive with the directory with space does not seem to exist but I tried with another remote (OneDrive) everything seems to work as expected.

C:\Users\Price\Downloads\RCLONEX>rclone ls "Apple: A" -vv
2021/03/09 06:50:34 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "ls" "Apple: A" "-vv"]
2021/03/09 06:50:34 DEBUG : Using config file from "C:\\Users\\Price\\.config\\rclone\\rclone.conf"
2021/03/09 06:50:34 DEBUG : Creating backend with remote "Apple: A"
        0 New Microsoft Office Word Document.docx
2021/03/09 06:50:35 DEBUG : 3 go routines active