Multiple Url In Text File & Wanted To Use "rclone copyurl" For Parallel Uploads To Remote

ok, now that is working, let's implement @Animosity022 suggestion.
test using two lines.

rclone copyurl -a --stats-one-line -v 'https://pepper.devilz.workers.dev/0:/www.TamilBlasters.re%20-%20The%20Family%20Man%20(2021)%20S02%20(E01-09)%20%5BHindi%20-%201080p%20HD%20AVC%20-%20x264%20-%20UNTOUCHED%20-%20DDP5.1%20(640kbps)%20-%201.8.2GB%20-%20ESubs%5D/EP-01%20-%20www.TamilBlasters.re.mkv' onedrive:  
rclone copyurl -a --stats-one-line -v 'https://pepper.devilz.workers.dev/0:/www.TamilBlasters.re%20-%20The%20Family%20Man%20(2021)%20S02%20(E01-09)%20%5BHindi%20-%201080p%20HD%20AVC%20-%20x264%20-%20UNTOUCHED%20-%20DDP5.1%20(640kbps)%20-%201.8.2GB%20-%20ESubs%5D/EP-01%20-%20www.TamilBlasters.re.mkv' onedrive:  

@asdffdsa Okay. As now I have updated my .sh file on linux machine for only 2 files. And followed @Animosity022 suggestion.

It shows status update for past 1 minute. As @Animosity022 suggested --progress is real time updates.


ubuntu@ubuntu:~$ chmod +x uploader.sh
ubuntu@ubuntu:~$ ./uploader.sh
2022/02/03 19:31:28 INFO  :       224 MiB / 2.876 GiB, 8%, 4.696 MiB/s, ETA 9m39s
2022/02/03 19:32:28 INFO  :   504.323 MiB / 2.876 GiB, 17%, 4.294 MiB/s, ETA 9m28s
2022/02/03 19:33:28 INFO  :       770 MiB / 2.876 GiB, 26%, 4.319 MiB/s, ETA 8m23s


Depending on what you want to, you can dump it all into a log file as well and get the adds/removes/deletes.

Add in something like:

--log-file /home/ubuntu/upload.log

and you'll get a file that is appended to that you can go back to and check.

So my logfile looks like:

2022/02/03 14:09:00 INFO  :     6.199 GiB / 6.199 GiB, 100%, 28.051 MiB/s, ETA -
2022/02/03 14:09:02 INFO  : Somefile.mkv: Copied (new)
2022/02/03 14:09:02 INFO  : Somefile.mkv: Deleted
2022/02/03 14:09:02 INFO  : Blah: Removing directory
2022/02/03 14:09:02 INFO  :     6.199 GiB / 6.199 GiB, 100%, 26.297 MiB/s, ETA 0s
2022/02/03 14:09:02 INFO  : Dropbox root 'SecureT/smu5ej34ujbdoip1cm3mlk92q4': Commiting uploads - please wait...

only need to do that one time.

I am assuming remote uploads in this methods are not parallel. I mean all 9 url starts getting remote uploading at same time.

Currently it is going one by one url. Just thinking is it possible..?

Ohhh. Okay. My bad.

Just a off-topic doubt. If I may any suggestion. When I run speedtest on my linux server (ubuntu) which is hosted as virtual machine on oracle cloud. I receive below speeds.

image

But when I use rclone copyurl I receive somewhere between 5 MB/s - 6 MB/s.

Any suggestion how I can fix..?

need to test speeds.
--- rclone copyurl to local
--- rclone copy from local to onedrive

In this situation local indicates ubuntu linux server / windows based physical pc..?

not sure what you mean, as both tests use local

Low speeds happening on when running rclone copyurl on linux instance. So, that is what is local..?

and if you wget the file, what is the speed?

Tried for below case :-

  • Used same rclone config on both linux server hosted on oracle cloud & windows 10 based PC.

  • Used same url for upload to onedrive.


https://gdriveindex.mrdeveloper.workers.dev/1:/Kapil.Sharma.Im.Not.Done.Yet.2022.1080p.NF.10bit.DDP.5.1.x265.%5BHashMiner%5D/Kapil.Sharma.Im.Not.Done.Yet.2022.1080p.NF.10bit.DDP.5.1.x265.[HashMiner].mkv

  • Speeds are same between 5 - 6 MB/s.

If I use wget I get constant of 13 MB/s.

based on that, what do you expect from rclone copyurl.

migth try to create a http remote and use rclone copy which supports multi-thread downloads.

Okay. I have created http remote now on rclone config.

How do use this url to download to local & then copy to destination which is onedrive.

Please correct me if I am understanding wrong concept of rclone copy.

What to be entered in this..?


Storage> 43
Option url.
URL of http host to connect to.
E.g. "https://example.com", or "https://user:pass@example.com" to use a username and password.
Enter a string value. Press Enter for the default ("").
url>

maybe try this 1st

I am using onedrive business as my remote in rclone. And wanted to mount it as a network drive always whenever system restarted or shutdown.

For this I have created task scheduler too on windows.

rclone mount command :-

rclone mount onedrive: E: --vfs-cache-mode full --network-mode
The service rclone has been started.

But if I close terminal or shutdown / restart system. Task scheduler is not working.

Any way to get it work.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.