Rclone on RPi can lsf and mkdir on remote OneDrive but cannot copy

Help and Support
1

What is the problem you are having with rclone?

I am using rclone to upload research-related data (stored in .HDF5 files ~10-20MB) from a Raspberry Pi 4 to our lab's Sharepoint site through OneDrive. The files are generated and uploaded using Python scripts, which use rclone's "mkdir" to create a directory on the remote, then "copy" to copy the HDF5 file into that directory. This has worked seamlessly for years, but since Apr 24th the "copy" command started timing out and failing. Strangely, the "mkdir" command still works, and on occasion (~1/5 attempts) the file is successfully copied. Some other relevant information:

  • The issue arose simultaneously on the 4 different Raspberry Pi machines we use for our research.
  • I get the same issue with 1) the python-rclone Python library, 2) with rclone commands given through python using os.system, and 3) inputting rclone commands directly to the terminal.
  • I tried updating the binary, reconfiguring the remote, and uninstalling and re-installing rclone on the machines but the issue persists.
  • Other Windows and MacOS computers on the same network successfully use rclone to copy files to the same OneDrive directory, with the same credentials.
  • I get the same issue when I move to different parts of the building (i.e. connecting to different routers on the same network).
  • I've gotten the same issue transferring other file types, including smaller (~80kB) .wav files

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.0

  • os/version: raspbian 10.12
  • os/kernel: 5.10.17-v7l+ (armv7l)
  • os/type: linux
  • os/arch: arm
  • go/version: go1.17.8
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --progress temp_data_path sharepoint:final_data_path

The rclone config contents with secrets removed.

[sharepoint]
type=onedrive
token = ****
drive_id = ****
drive_type = documentLibrary

A log from the command with the -vv flag

rclone copy -vv ./rclone_test sharepoint:
2022/04/28 17:39:30 DEBUG : rclone: Version "v1.58.0" starting with parameters ["rclone" "copy" "-vv" "./rclone_test" "sharepoint:"]
2022/04/28 17:39:30 DEBUG : Creating backend with remote "./rclone_test"
2022/04/28 17:39:30 DEBUG : Using config file from "/home/pi/.config/rclone/rclone.conf"
2022/04/28 17:39:30 DEBUG : fs cache: renaming cache item "./rclone_test" to be canonical "/home/pi/Desktop/yesterday_data/rclone_test"
2022/04/28 17:39:30 DEBUG : Creating backend with remote "sharepoint:"
2022/04/28 17:39:30 DEBUG : One drive root '': Token expired but no uploads in progress - doing nothing
2022/04/28 17:39:30 DEBUG : sharepoint: Loaded invalid token from config file - ignoring
2022/04/28 17:39:31 DEBUG : Saving config "token" in section "sharepoint" of the config file
2022/04/28 17:39:31 DEBUG : sharepoint: Saved new token in config file
2022/04/28 17:39:31 DEBUG : One drive root '': Waiting for checks to finish
2022/04/28 17:39:31 DEBUG : One drive root '': Waiting for transfers to finish
2022/04/28 17:39:31 DEBUG : ms6242_2022-04-28_block1.hdf5: Starting multipart upload
2022/04/28 17:39:32 DEBUG : ms6242_2022-04-28_block1.hdf5: Uploading segment 0/11589152 size 10485760
2022/04/28 17:39:42 DEBUG : pacer: low level retry 1/10 (error Put "https://uottawa.sharepoint.com/sites/*****/_api/v2.0/drives/*****/items/*****/uploadSession?guid='*****'&overwrite=True&rename=False&dc=0&tempauth=*****": dial tcp: lookup uottawa.sharepoint.com on *****: dial tcp *****: i/o timeout)
2022/04/28 17:39:42 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022/04/28 17:39:52 DEBUG : pacer: low level retry 2/10 (error Put "https://uottawa.sharepoint.com/sites/*****/_api/v2.0/drives/*****/items/*****/uploadSession?guid='*****'&overwrite=True&rename=False&dc=0&tempauth=****": dial tcp: lookup uottawa.sharepoint.com on *****: dial tcp *****: i/o timeout)
2022/04/28 17:39:52 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2022/04/28 17:40:02 DEBUG : pacer: low level retry 3/10 (error Put "https://uottawa.sharepoint.com/sites/*****/_api/v2.0/drives/*****/items/*****/uploadSession?guid='*****'&overwrite=True&rename=False&dc=0&tempauth=****": dial tcp: lookup uottawa.sharepoint.com on *****: dial tcp *****: i/o timeout)

Here I terminated the process after a few minutes, but these error messages continue indefinitely.

2

hello and welcome to the forum,

tl;dr - the issue only happens on the pi, correct?

could be a dns issue or network issue.

3

Hi, yes essentially I only get the issue when trying to copy files between the Pi and the remote (in both directions). Do you know what I should do to test whether it's a dns/network issue? Since we're at a university sometimes they make changes to the network without warning users and they don't exactly have RPi users in mind when making those changes. Thanks!

4

not a linix network expert.

are you using pi-hole or any sort of customized dns servers.
i would test using google dns servers.

5

Whatever is in those **** is the DNS server that isn't working.

6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.