Transfer stuck using Samba share on rclone mounted Onedrive

peppe · October 18, 2024, 2:09pm

Hi all,

i have a strange issue.

What is the problem you are having with rclone?

OneDrive is mounted locally
this mountpoint is shared via SMB
The issue is when i transfer file from this mounted Onedrive via SMB the transfer hanghing randomly.

If I copy the file directly from Onedrive mount point to a local folder the transfer go well, after the file was in vfs cache the transfer via SMB works fine.

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.1
os/version: debian 12.7 (64 bit)
os/kernel: 6.8.12-2-pve (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.23.1
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

root@fileserver:~# cat /etc/systemd/system/rclone.service
[Install]
WantedBy=default.target

[Unit]
Description=crypt (rclone)
AssertPathIsDirectory=/mnt/OneDrive

[Service]
Type=simple
User=root
ExecStart=/usr/bin/rclone mount \
--config=/root/.config/rclone/rclone.conf \
--allow-non-empty \
--umask 0000 \
--dir-perms 0777 \
--tpslimit 1 \
--file-perms 0777 \
--buffer-size 0 \
--vfs-cache-max-size 100G \
--poll-interval 60s \
--vfs-cache-max-age 3h \
--cache-dir=/2TBdisk/vfs-cache \
--vfs-cache-mode full \
--allow-other \
--log-level DEBUG \
--vv
--rc \
--log-file /opt/rclone.log 5T_OneDrive: /mnt/OneDrive/

ExecStop=systemctl stop smbd.service
ExecStop=sleep 3
ExecStop=/bin/fusermount -u /mnt/OneDrive/
ExecStop=systemctl start smbd.service
Restart=always
RestartSec=10

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[5T_OneDrive]
type = onedrive
client_id = XXX
client_secret = XXX
drive_type = business
token = XXX
drive_id = XXX
chunk_size = 50Mi
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the `-vv` flag

2024/10/18 15:52:34 DEBUG : vfs cache: looking for range={Pos:283115520 Size:131072} in [{Pos:0 Size:276070400}] - present false
2024/10/18 15:52:34 DEBUG : &{Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst (rw)}: Read: len=131072, offset=276168704
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst(0xc000a12240): _readAt: size=131072, off=276168704
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.RangeSeek from -1 to 283115520 length -1
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at -1 length 32768 chunkOffset 283115520 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.openRange at 283115520 length 134217728
2024/10/18 15:52:34 DEBUG : vfs cache: looking for range={Pos:276168704 Size:131072} in [{Pos:0 Size:276103168}] - present false
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276103168 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276135936 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276168704 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276201472 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst(0xc000a12240): >_readAt: n=131072, err=<nil>
2024/10/18 15:52:34 DEBUG : &{Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst (rw)}: >Read: read=131072, err=<nil>
2024/10/18 15:52:34 DEBUG : &{Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst (rw)}: Read: len=131072, offset=276299776
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst(0xc000a12240): _readAt: size=131072, off=276299776
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276234240 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : vfs cache: looking for range={Pos:276299776 Size:131072} in [{Pos:0 Size:276234240}] - present false
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276267008 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276299776 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst(0xc000a12240): >_readAt: n=131072, err=<nil>
2024/10/18 15:52:34 DEBUG : &{Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst (rw)}: >Read: read=131072, err=<nil>
2024/10/18 15:52:34 DEBUG : &{Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst (rw)}: Read: len=131072, offset=276430848
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst(0xc000a12240): _readAt: size=131072, off=276430848
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276332544 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : vfs cache: looking for range={Pos:276430848 Size:131072} in [{Pos:0 Size:276332544}] - present false
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276365312 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276398080 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276430848 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst(0xc000a12240): >_readAt: n=131072, err=<nil>
2024/10/18 15:52:34 DEBUG : &{Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst (rw)}: >Read: read=131072, err=<nil>
2024/10/18 15:52:34 DEBUG : &{Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst (rw)}: Read: len=131072, offset=276561920
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst(0xc000a12240): _readAt: size=131072, off=276561920
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276463616 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : vfs cache: looking for range={Pos:276561920 Size:131072} in [{Pos:0 Size:276463616}] - present false
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276496384 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276529152 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst(0xc000a12240): >_readAt: n=131072, err=<nil>
2024/10/18 15:52:34 DEBUG : &{Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst (rw)}: >Read: read=131072, err=<nil>
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276561920 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : &{Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst (rw)}: Read: len=131072, offset=276692992
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst(0xc000a12240): _readAt: size=131072, off=276692992
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276594688 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : vfs cache: looking for range={Pos:276692992 Size:131072} in [{Pos:0 Size:276594688}] - present false
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276627456 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276660224 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276692992 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276725760 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst(0xc000a12240): >_readAt: n=131072, err=<nil>
2024/10/18 15:52:34 DEBUG : &{Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst (rw)}: >Read: read=131072, err=<nil>
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276758528 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276791296 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 276824064 length 32768 chunkOffset 142606336 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.openRange at 276824064 length 268435456
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst(0xc000a12240): >_readAt: n=131072, err=<nil>
2024/10/18 15:52:34 DEBUG : &{Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst (rw)}: >Read: read=131072, err=<nil>
2024/10/18 15:52:34 DEBUG : &{Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst (rw)}: Read: len=131072, offset=284164096
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst(0xc000a12240): _readAt: size=131072, off=284164096
2024/10/18 15:52:34 DEBUG : vfs cache: looking for range={Pos:284164096 Size:131072} in [{Pos:0 Size:276824064}] - present false
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.RangeSeek from -1 to 284164096 length -1
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at -1 length 32768 chunkOffset 284164096 chunkSize 134217728
2024/10/18 15:52:34 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.openRange at 284164096 length 134217728
2024/10/18 15:52:35 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 145784832 length 32768 chunkOffset 145752064 chunkSize 134217728
2024/10/18 15:52:35 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 145817600 length 32768 chunkOffset 145752064 chunkSize 134217728
2024/10/18 15:52:35 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 145850368 length 32768 chunkOffset 145752064 chunkSize 134217728
2024/10/18 15:52:35 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 145883136 length 32768 chunkOffset 145752064 chunkSize 134217728
2024/10/18 15:52:36 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 146833408 length 32768 chunkOffset 146800640 chunkSize 134217728
2024/10/18 15:52:36 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 146866176 length 32768 chunkOffset 146800640 chunkSize 134217728
2024/10/18 15:52:36 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 146898944 length 32768 chunkOffset 146800640 chunkSize 134217728
2024/10/18 15:52:36 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 146931712 length 32768 chunkOffset 146800640 chunkSize 134217728
2024/10/18 15:52:37 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 147881984 length 32768 chunkOffset 147849216 chunkSize 134217728
2024/10/18 15:52:37 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 147914752 length 32768 chunkOffset 147849216 chunkSize 134217728
2024/10/18 15:52:37 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 147947520 length 32768 chunkOffset 147849216 chunkSize 134217728
2024/10/18 15:52:37 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 147980288 length 32768 chunkOffset 147849216 chunkSize 134217728
2024/10/18 15:52:37 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: vfs cache: stopping download thread as it timed out
2024/10/18 15:52:37 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 148930560 length 32768 chunkOffset 148897792 chunkSize 134217728
2024/10/18 15:52:37 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 148963328 length 32768 chunkOffset 148897792 chunkSize 134217728
2024/10/18 15:52:37 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 148996096 length 32768 chunkOffset 148897792 chunkSize 134217728
2024/10/18 15:52:37 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: ChunkedReader.Read at 149028864 length 32768 chunkOffset 148897792 chunkSize 134217728
2024/10/18 15:52:38 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: vfs cache: stopping download thread as it timed out
2024/10/18 15:52:39 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: vfs cache: stopping download thread as it timed out
2024/10/18 15:52:40 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: vfs cache: stopping download thread as it timed out
2024/10/18 15:52:41 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: vfs cache: stopping download thread as it timed out
2024/10/18 15:52:42 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: vfs cache: stopping download thread as it timed out
2024/10/18 15:52:42 DEBUG : Proxmox_dump/vzdump-lxc-102-2024_10_15-03_00_02.tar.zst: vfs cache: stopping download thread as it timed out

asdffdsa · October 18, 2024, 2:33pm

welcome to the forum,

that allows for over-mounting. should not use unless 1000% sure you need it.
tho, odds are that is not the issue.

other then that, not seeing any issue with the mount command.

tho, i do not recommend using it in production, rclone has the smb remote.
might try testing with that.

check out my howto guide about smb

just curious, why use complex rclone mount, instead of simple rclone copy|move

peppe · October 18, 2024, 4:36pm

I use mount because i need some file available at any time.

peppe · October 18, 2024, 4:38pm

maybe i solved by using these configs:

ExecStart=/usr/bin/rclone mount \
--config=/root/.config/rclone/rclone.conf \
--allow-non-empty \
--umask 0000 \
--dir-perms 0777 \
--tpslimit 1 \
--file-perms 0777 \
--buffer-size 64M \
--vfs-cache-max-size 100G \
--poll-interval 60s \
--vfs-read-chunk-size 128M \
--vfs-read-chunk-size-limit 1G \
--vfs-cache-max-age 3h \
--cache-dir=/2TBdisk/vfs-cache \
--vfs-cache-mode full \
--allow-other \
--log-level DEBUG \
--rc \
--log-file /opt/rclone.log 5T_OneDrive: /mnt/OneDrive/

now it seems works fine.

asdffdsa · October 18, 2024, 4:44pm

ok, i get that.

fwiw, copying large backup files using mount is double work, double copy, using the cache as a middle-man.
faster and more reliable to rclone copy|move direct to cloud.
or run rclone on the server containing the source files.
that is what i do.

system · November 17, 2024, 4:44pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.