Upload to OneDrive Stuck at 100% and Won't Complete

Help and Support
1

What is the problem you are having with rclone?

When uploading files to OneDrive using rclone, the first file gets to 100% upload, but then the upload speed drops to 0, and the file is unable to be transferred. On OneDrive itself, the directory the file is going to be uploaded to is created, and OneDrive indicates that there are files (items) in the directory, but when I open the directory, there are no files inside.

I have been successful in uploading files to OneDrive for about 9 months, but have only encountered this issue a few days ago. I have not changed anything in my setup from when I was able to upload files successfully.

I have tried re-configuring the remote, that did not work. I have setup a new remote to send the files to, that did not work. I have tried using a stronger internet connection, that did not help (the setup is using a Raspberry Pi 4 with a SIM card hat, I tried uploading files using a strong wifi connection instead of the hat).

When running the command with the -vv flag for this post, I did notice that the speed was abysmally slow (less than 100 B/s)

Any help would be appreciated.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2

  • os/version: raspbian 11.2
  • os/kernel: 5.4.75-v7l+ (armv7l)
  • os/type: linux
  • os/arch: arm (ARMv7 compatible)
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Microsoft OneDrive (for Business)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --progress /home/pi/tarpin/tarpin_connection_hist.log jslab_onedrive:RAFT_Data/$raft_num/tarpin_logs

The rclone config contents with secrets removed.

[jslab_onedrive]
type = onedrive
token = {"access_token":"redacted","token_type":"Bearer","refresh_token":"redacted","expiry":"2023-04-13T11:05:58.28490397-07:00"}
drive_id = redacted
drive_type = business

A log from the command with the -vv flag

2023/04/13 10:23:39 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "copy" "--progress" "-vv" "/home/pi/tarpin/tarpin_connection_hist.log" "jslab_onedrive:RAFT_Data/RAFT1/tarpin_logs"]
2023/04/13 10:23:39 DEBUG : Creating backend with remote "/home/pi/tarpin/tarpin_connection_hist.log"
2023/04/13 10:23:39 DEBUG : Using config file from "/home/pi/.config/rclone/rclone.conf"
2023/04/13 10:23:39 DEBUG : fs cache: adding new entry for parent of "/home/pi/tarpin/tarpin_connection_hist.log", "/home/pi/tarpin"
2023/04/13 10:23:39 DEBUG : Creating backend with remote "jslab_onedrive:RAFT_Data/RAFT1/tarpin_logs"
2023-04-13 10:23:44 DEBUG : tarpin_connection_hist.log: Need to transfer - File not found at Destination
2023-04-13 10:23:44 DEBUG : tarpin_connection_hist.log: Starting multipart upload
2023-04-13 10:23:46 DEBUG : tarpin_connection_hist.log: Uploading segment 0/369 size 369
2023-04-13 10:25:49 DEBUG : pacer: low level retry 1/10 (error Put "https://redacted-my.sharepoint.com/personal/redacted/_api/v2.0/drives/b!MlK-YgXOikS6zNJyXjBUUzh0w5hrbYpLqQfI1fWHZkmBQ0Xbqv2fQbaOLEmfsHnf/items/01X7S6X646CXZMOR4JMZHIZIANLOKRXVA5/uploadSession?guid='redacted'&overwrite=True&rename=False&dc=0&tempauth=redacted": read tcp 21.172.205.172:33572->13.107.136.8:443: read: connection reset by peer)
2023-04-13 10:25:49 DEBUG : pacer: Rate limited, increasing sleep to 20ms
Transferred:            369 B / 369 B, 100%, 0 B/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:      2m27.5s
Transferring:
 *                    tarpin_connection_hist.log:100% /369, 0/s, 0s
2

hello and welcome to the forum,

not sure why the issues started a few days ago.

fwiw, test on another machine, could be an issue with the pi4

onedrive is telling rclone to slow down.

this might help a bit, might not help at all.
You may choose to create and use your own Client ID, in case the default one does not work well for you. For example, you might see throttling.

3

Hi Nick,

Looks like an internet issue or OneDrive rejecting your put request.

Is this command working correctly (showing the content of the folder):

rclone lsf "jslab_onedrive:RAFT_Data/RAFT1/tarpin_logs" -vv

and this:

rclone touch "jslab_onedrive:RAFT_Data/RAFT1/tarpin_logs/testfile" -vv

Do you have another machine (e.g Windows Desktop) that you can try from?

When did you update to 1.62.2? Do you see the same with 1.61.1?
(Could be a variation of this issue)

4

Is this command working correctly (showing the content of the folder):

rclone lsf "jslab_onedrive:RAFT_Data/RAFT1/tarpin_logs" -vv

and this:

rclone touch "jslab_onedrive:RAFT_Data/RAFT1/tarpin_logs/testfile" -vv

Trying both of these commands, they both work, I was able to see the contents of the folder 'tarpin_logs' (which was empty) and was able to create 'testfile'. Tried the first command again after creating 'testfile' and was able to see that 'testfile' was present.

Do you have another machine (e.g Windows Desktop) that you can try from?

I do have a machine with windows desktop I can try, will update post once I try it out.

EDIT: Tried it with my windows machine, and I was able to upload a test text file without issue

When did you update to 1.62.2? Do you see the same with 1.61.1?

I updated to 1.62.2 a few days ago after the first Pi4 failed to upload, seeing if updating would fix the issue, which it did not. Not sure if the issue was in 1.61.1 as I had updated from 1.57.x. I have a total of three Pi4s I am using. After the first Pi4 failed, I was able to upload files with a different Pi4, but several days later it is now facing the same issue as the first Pi4 (the third Pi4 is also facing this issue)

I am going to try @asdffdsa 's idea of creating and using my own Client ID to see if that will help. It could very well be OneDrive throttling me. I did just check in OneDrive and it seems that ~tmpfiles are being created in OneDrive. Could be a source of OneDrive deciding to throttle as there are a lot of these files being created.

EDIT: Due to certain security concerns, I will not be able to create an app through Azure that would allow me to create my own client ID and secret.

5

Hey friend

I'd like you to know that I've already faced many problems with Onedrive and if I hadn't broken my head I would have already abandoned the boat. With that in mind, I would like to make a few suggestions:

  • Try to limit transfers and checkers to a maximum of 4

  • Add flags --user-agent "ISV|rclone.org|rclone/v1.62.2" --tpslimit 2

If None of the Alternatives work, I recommend that you create a new Remote with the Local type and then use the Web version of Rclone to synchronize the two locations. I couldn't identify it further, it seems that Onedrive's limitations disappear when transferring through the web version of Rclone. Example my Internet is 300 mb's of Downloads and 150 of Upload and when making transfers through the terminal the maximum I reach is 8 mb that when transferring, already for the web version it transfers at the full speed contracted by my ISP apart from that I am not limited by Onedrive I already transferred 1TB from Google to Onedrive in just over 2 days using this method

6

Thanks, perfect follow up test!

Interesting, I wonder if this is a pattern or just coincidence.

Does the upload fail consistently on you Pi4?
Is the Windows machine and the Pi4 on the same (local) network?

Good to know, then it doesn't seem to be caused by the rclone update. No need to test with 1.61.1.

Are all three Pi4s uploading to the same OneDrive account at the same time (and almost continuously 24/7) - or are the uploads more like nightly backups executed one at a time by each Pi4?

I am asking to understand how many concurrent transfers you are trying to push to the same OneDrive account.

Are you doing a lot of downloads from the same OneDrive account?

No problem, it often makes no difference for OneDrive.

I just noted that there is a 2 minutes and 3 sec delay between rclone starting the upload of the 369 bytes and the connection being reset by the peer. This is a long time and looks like some kind of network issue where packets are lost causing connections to timeout - or something like that.

Do you see a stable connection if performing this command from the Pi4 while trying the above copy:

ping onedrive.microsoft.com
7

Interesting, I wonder if this is a pattern or just coincidence.

Does the upload fail consistently on you Pi4?
Is the Windows machine and the Pi4 on the same (local) network?

The upload does fail consistently on the Pi4. The Pis and tehe windows machine are on different networks.

Are all three Pi4s uploading to the same OneDrive account at the same time (and almost continuously 24/7) - or are the uploads more like nightly backups executed one at a time by each Pi4?

I am asking to understand how many concurrent transfers you are trying to push to the same OneDrive account.

Are you doing a lot of downloads from the same OneDrive account?

The Pis are uploading to the same OneDrive account. While the Pis are running the code to execute the file upload at the same time via cronjobs, the files that get uploaded are on the Pi at random times in the day (files are uploaded to the Pi via USB connection from user). In this way, multiple Pis are not typically uploading files to the same OneDrive account at the same time, but it could happen.

Not doing any downloads from the OneDrive account.

Do you see a stable connection if performing this command from the Pi4 while trying the above copy

ping onedrive.microsoft.com

I get a stable connection and pretty quick ping times. I have tried the command multiple times at different times of the day just to be sure, and have not had an issue yet.

One thing I have not gotten around to trying that I will do at some point, is to try using a different cloud storage (Google Drive or DropBox, etc) and see if I am able to use those to see if it is a OneDrive specific issue or a Pi specific issue.

8

Perfect, this pretty much rules out any OneDrive rate limiting/throttling.

Great, this very much indicates an issue with the Pis network connection to OneDrive.

Looks like the basic network connection is fine.

Consistent network failures/oddities on a seemingly stable connection are typically caused by some intermediary component e.g. proxy or firewall where some network packets are being modified/stopped.

This also matches this observation:

It is however somewhat puzzling that the issue continued even after switching from the SIM card to the wifi connection.
Is the SIM card and the wifi connection using the same backbone/ISP?

It could also be caused by some (updated/added) intermediary component in the Pis network setup/stack.
Are you using some kind of VPN?

Can you test and reproduce when connecting the Pi to the same network as used by the successful Windows machine?

That is a good idea, you could just use a free google account and try the failing transfer something like this:

rclone copy --progress -vv /home/pi/tarpin/tarpin_connection_hist.log googledrive:some/test/folder

If this fails then we know it has something to do with the network, a successful test can however not be used to conclude that it is related to OneDrive.

Finally I would like to see the communication between the Pi and OneDrive, can you please post the output from this:

rclone copy --dump headers /home/pi/tarpin/tarpin_connection_hist.log jslab_onedrive:RAFT_Data/RAFT1/tarpin_logs

I am looking for lost/malformed requests/responses.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.