Hi, I'm planning to move around 10 TB from Google Drive to microsoft onedrive using a raspberry pi 3 B. If anyone has similar experience, would it be better to use a ubuntu server OS or the default raspbian OS? As the Raspberry would never shutdown it needs to be a reliable OS.
hello,
in another recent post, i worked with a fellow rcloner to get rclone to work with just 256MB
the problem will be the lack of RAM, which i think it just 1GB
about the 10GB, how many files total, what is the average size of each file?
i'd say around 200-300 files. Each is a zipped file taking around 50-90 GB. It's alot of 8K stock footage that I want to backup
good, that should be easy to do with 1GB.
if you have lots of small files, gdrive has lots of limits and throttling, that would be a slow pain.
i am not a gdrive expert, but given you can only upload 750GB per 24 hour period, optimizing the command seems pointless.
if you upload at full speed, rclone would hit that limit quickly and error out.
you might want to use --bwlimit=8.5M, i think that is the correct value, you can search the forum.
so i would use the a very simple rclone command, use defaults.
rclone copy source dest --bwlimit=8.5M
and you might want to use a log file
Thanks for the help. Another problem popped up though. When using this command on my windows 10 computer it works fine
rclone copy "GoogleDrive:\Test Move to Onedrive" "OneDrive:\UploadLocation" -P
But when using the exact same command on the Raspberry Pi running Armv7 Linux version
I get
2021-03-05 22:53:18 ERROR : : error reading source directory: directory not found
Is the linux flags different from windows 10? Don't know why it can't find the directory. When configuring the 2 drives I set them as GoogleDrive and OneDrive. Same as on my windows 10 computer
on the pi?
did you copy the rclone.conf file from the windows machine to the pi
or
did you re-create a new config file on the pi?
you need to post the command with a debug log, add -vv to the command.
post the debug log, use three backticks before and after the log so it will look like
2021/03/02 09:32:00 DEBUG : rclone: Version "v1.53.4" starting with parameters ["C:\\data\\rclone\\scripts\\rclone.exe" "copy" "C:\\Users\\user01\\AppData\\Local\\Temp\\kdbx\\zip\\kdbx.20210302.093158.7z" "wasabi01:en07/kdbx/zip/backup" "--stats=0" "--fast-list" "--log-level=DEBUG" "--log-file=C:\\data\\rclone\\logs\\kdbx_quick_wasabi\\20210302.093158\\rclone.log"]
2021/03/02 09:32:00 DEBUG : Creating backend with remote "C:\\Users\\user01\\AppData\\Local\\Temp\\kdbx\\zip\\kdbx.20210302.093158.7z"
2021/03/02 09:32:00 DEBUG : Using RCLONE_CONFIG_PASS password.
2021/03/02 09:32:00 DEBUG : Using config file from "C:\\data\\rclone\\scripts\\rclone.conf"
2021/03/02 09:32:00 DEBUG : fs cache: adding new entry for parent of "C:\\Users\\user01\\AppData\\Local\\Temp\\kdbx\\zip\\kdbx.20210302.093158.7z", "//?/C:/Users/user01/AppData/Local/Temp/kdbx/zip"
2021/03/02 09:32:00 DEBUG : Creating backend with remote "wasabi01:en07/kdbx/zip/backup"
2021/03/02 09:32:00 DEBUG : kdbx.20210302.093158.7z: Need to transfer - File not found at Destination
2021/03/02 09:32:00 INFO : S3 bucket en07 path kdbx/zip/backup: Bucket "en07" created with ACL "private"
I went through the entire process of setting up the remotes again on the pi, I did however use the same client id and client secret that I use on the windows pc. Should I have copy the config files over to the pi from the windows pc?
Here's the debug
2021/03/05 23:11:46 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "copy" "GoogleDrive:\\Test Move to Onedrive" "OneDrive:\\UploadLocation" "-P" "-vv"]
2021/03/05 23:11:46 DEBUG : Using config file from "/home/pi/.config/rclone/rclone.conf"
2021/03/05 23:11:46 DEBUG : Creating backend with remote "GoogleDrive:\\Test Move to Onedrive"
2021/03/05 23:11:46 DEBUG : Google drive root '\Test Move to Onedrive': root_folder_id = "0AFzQGTaH610nUk9PVA" - save this in the config to speed up startup
2021/03/05 23:11:47 DEBUG : Creating backend with remote "OneDrive:\\UploadLocation"
2021-03-05 23:11:48 ERROR : : error reading source directory: directory not found
2021-03-05 23:11:48 DEBUG : One drive root '\UploadLocation': Waiting for checks to finish
2021-03-05 23:11:48 DEBUG : One drive root '\UploadLocation': Waiting for transfers to finish
2021-03-05 23:11:48 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2021-03-05 23:11:48 ERROR : : error reading source directory: directory not found
2021-03-05 23:11:48 DEBUG : One drive root '\UploadLocation': Waiting for checks to finish
2021-03-05 23:11:48 DEBUG : One drive root '\UploadLocation': Waiting for transfers to finish
2021-03-05 23:11:48 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2021-03-05 23:11:49 ERROR : : error reading source directory: directory not found
2021-03-05 23:11:49 DEBUG : One drive root '\UploadLocation': Waiting for checks to finish
2021-03-05 23:11:49 DEBUG : One drive root '\UploadLocation': Waiting for transfers to finish
2021-03-05 23:11:49 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 3.1s
2021/03/05 23:11:49 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 3.1s
2021/03/05 23:11:49 DEBUG : 7 go routines active
2021/03/05 23:11:49 Failed to copy: directory not found
do a rclone listremotes - make sure you have the correct remote
whatever the remote is, easier to test with rclone lsd nameoftheremote: -vv
listremotes give me this
pi@edison:~ $ rclone listremotes
GoogleDrive:
OneDrive:
rclone lsd GoogleDrive: -vv gives this
pi@edison:~ $ rclone lsd GoogleDrive: -vv
2021/03/05 23:32:06 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "lsd" "GoogleDrive:" "-vv"]
2021/03/05 23:32:06 DEBUG : Using config file from "/home/pi/.config/rclone/rclone.conf"
2021/03/05 23:32:06 DEBUG : Creating backend with remote "GoogleDrive:"
2021/03/05 23:32:07 DEBUG : Google drive root '': root_folder_id = "0AFzQGTaH610nUk9PVA" - save this in the config to speed up startup
-1 2021-02-20 09:26:50 -1 Download
-1 2021-03-04 22:21:11 -1 Test Move to Onedrive
-1 2021-02-20 09:26:50 -1 UploadLocation
2021/03/05 23:32:07 DEBUG : 3 go routines active
rclone lsd OneDrive: -vv gives this
pi@edison:~ $ rclone lsd OneDrive: -vv
2021/03/05 23:32:39 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "lsd" "OneDrive:" "-vv"]
2021/03/05 23:32:39 DEBUG : Using config file from "/home/pi/.config/rclone/rclone.conf"
2021/03/05 23:32:39 DEBUG : Creating backend with remote "OneDrive:"
-1 2021-03-05 00:00:09 1 UploadLocation
2021/03/05 23:32:40 DEBUG : 5 go routines active
In the previous reply with the debug it says source directory not found
Does that mean the problem is cause somewhere in the
"GoogleDrive:\Test Move to Onedrive"
and not the
"OneDrive:\UploadLocation"
yes, the problem is with the source, with is GoogleDrive:
try over again,
rclone mkdir GoogleDrive:TestMoveToOnedrive -vv
rclone lsd GoogleDrive:TestMoveToOnedrive -vv
rclone copy GoogleDrive:TestMoveToOnedrive OneDrive:UploadLocation -P -vv
Hi thank you very much. Seems like the rclone mkdir command worked out
2021/03/05 23:59:53 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "mkdir" "GoogleDrive:TestMoveToOnedrive" "-vv"]
2021/03/05 23:59:53 DEBUG : Creating backend with remote "GoogleDrive:TestMoveToOnedrive"
2021/03/05 23:59:53 DEBUG : Using config file from "/home/pi/.config/rclone/rclone.conf"
2021/03/05 23:59:54 DEBUG : Google drive root 'TestMoveToOnedrive': root_folder_id = "0AFzQGTaH610nUk9PVA" - save this in the config to speed up startup
2021/03/05 23:59:54 DEBUG : Google drive root 'TestMoveToOnedrive': Making directory
2021/03/05 23:59:55 DEBUG : 4 go routines active
pi@edison:~ $ rclone lsd GoogleDrive:TestMoveToOnedrive -vv
2021/03/06 00:00:19 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "lsd" "GoogleDrive:TestMoveToOnedrive" "-vv"]
2021/03/06 00:00:19 DEBUG : Using config file from "/home/pi/.config/rclone/rclone.conf"
2021/03/06 00:00:19 DEBUG : Creating backend with remote "GoogleDrive:TestMoveToOnedrive"
2021/03/06 00:00:19 DEBUG : Google drive root 'TestMoveToOnedrive': root_folder_id = "0AFzQGTaH610nUk9PVA" - save this in the config to speed up startup
2021/03/06 00:00:21 DEBUG : 4 go routines active
pi@edison:~ $ rclone copy GoogleDrive:TestMoveToOnedrive OneDrive:UploadLocation -P -vv
2021/03/06 00:00:29 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "copy" "GoogleDrive:TestMoveToOnedrive" "OneDrive:UploadLocation" "-P" "-vv"]
2021/03/06 00:00:29 DEBUG : Using config file from "/home/pi/.config/rclone/rclone.conf"
2021/03/06 00:00:29 DEBUG : Creating backend with remote "GoogleDrive:TestMoveToOnedrive"
2021/03/06 00:00:30 DEBUG : Google drive root 'TestMoveToOnedrive': root_folder_id = "0AFzQGTaH610nUk9PVA" - save this in the config to speed up startup
2021/03/06 00:00:31 DEBUG : Creating backend with remote "OneDrive:UploadLocation"
2021-03-06 00:00:32 DEBUG : One drive root 'UploadLocation': Waiting for checks to finish
2021-03-06 00:00:32 DEBUG : One drive root 'UploadLocation': Waiting for transfers to finish
2021-03-06 00:00:32 INFO : There was nothing to transfer
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time: 3.1s
2021/03/06 00:00:32 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time: 3.1s
2021/03/06 00:00:32 DEBUG : 7 go routines active
I have a question though, was it because the original had GoogleDrive:\ and the " \ " was causing the errors, or because "Test Move To Drive" has spaces or combination of both?
good, you got it working.
i never uses spaces in path names, they are evil and create unintended consequences.
it should not matter with rclone and gdrive
about that slash, i did not mention that the correct syntax is not to use it.
with gdrive, rclone should work in either case.
some backends like ftp,sftp it does matter
https://rclone.org/docs/#syntax-of-remote-paths
i have tried and it work good as mini data server.
hello welcome to the forum,
which Pi device are you using?
how much memory does it have?
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.