Rclone mount G drive (exported via nfsv4) does not finish uploading all files

What is the problem you are having with rclone?

Using rclone mount on Ubuntu box, but exporting it out as NFSv4 mount to centos workstation. User is saving out e.g. four mov files (~150M to 300M each) directly from the app to the NFS share. Consistently, one of the files does not get uploaded to Gdrive. It will eventually get uploaded if I restart the rclone process. I last tried with options --vfs-write-back=10s and --low-level-retries=20, with no success.
In this case, file "FILE0457000H Business 30_Addressable_No_Price_30_Captions.mov" did not get uploaded (until I restarted rclone mount).

Not sure if I'm missing something in the mount options?

What is your rclone version (output from rclone version)

rclone v1.55.1

  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.3
  • go/linking: static
  • go/tags: none

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04.2 LTS
5.8.0-59-generic x86_64

Which cloud storage system are you using? (eg Google Drive)

Google Drive (Shared -- formerly known as Teams)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Running rclone mount as a service, using the template from the wiki; here I've only pasted the start|stop part.


ExecStart=/usr/bin/rclone mount sharedGdrive: /export/rclonefs/studio \
--allow-other \
--cache-dir=/export/d200/r_cache \
--umask=002 \
--vfs-cache-mode=writes \
--vfs-cache-poll-interval=30s \
--vfs-write-back=10s \
--low-level-retries=20 \
--retries=10 \
--retries-sleep=1s \
--drive-chunk-size=128M \
--log-file=/var/log/rclone/rclonefs.log \
--log-level=DEBUG
ExecStop=/bin/fusermount -uz /export/rclonefs/studio

The rclone config contents with secrets removed.

[sharedGdrive]
type = drive
client_id = redacted.apps.googleusercontent.com
client_secret = redacted
scope = drive
token = {redacted}
team_drive = redacted

A log from the command with the -vv flag

pastebin: rclone-071621 - Pastebin.com
had to prune to get it under 500k and had to redact client project specifics

snippet:


2021/07/16 20:32:17 INFO  : REDACTED/internal_review/2021_07_16/830pm/FILE0457000H Business 30_Addressable_No_Price_30_Captions.mov: vfs cache: queuing for upload in 10s
2021/07/16 20:32:17 DEBUG : REDACTED/internal_review/2021_07_16/830pm/FILE0457000H Business 30_Addressable_No_Price_30_Captions.mov: vfs cache: cancelling writeback (uploading false) 0xc00a68f030 item 4
2021/07/16 20:32:23 INFO  : REDACTED/internal_review/2021_07_16/830pm/FILE0459000H Business 30_Spot_No-Price_30_Captions.mov: vfs cache: queuing for upload in 10s
2021/07/16 20:32:23 DEBUG : REDACTED/internal_review/2021_07_16/830pm/FILE0459000H Business 30_Spot_No-Price_30_Captions.mov: vfs cache: cancelling writeback (uploading false) 0xc0032d45b0 item 5
2021/07/16 20:32:25 INFO  : REDACTED/internal_review/2021_07_16/830pm/FILE0457000H Business 30_Addressable_No_Price_30_Captions.mov: vfs cache: queuing for upload in 10s
2021/07/16 20:32:35 DEBUG : REDACTED/internal_review/2021_07_16/830pm/FILE0457000H Business 30_Addressable_No_Price_30_Captions.mov: vfs cache: starting upload
2021/07/16 20:32:40 INFO  : REDACTED/internal_review/2021_07_16/830pm/FILE0457000H Business 30_Addressable_No_Price_30_Captions.mov: vfs cache: upload succeeded try #1
2021/07/16 20:33:58 INFO  : REDACTED/internal_review/2021_07_16/830pm/FILE0459000H Business 30_Spot_No-Price_30_Captions.mov: vfs cache: queuing for upload in 10s
2021/07/16 20:33:58 DEBUG : REDACTED/internal_review/2021_07_16/830pm/FILE0459000H Business 30_Spot_No-Price_30_Captions.mov: vfs cache: cancelling writeback (uploading false) 0xc004460b60 item 6
2021/07/16 20:39:42 INFO  : REDACTED/internal_review/2021_07_16/830pm/New Folder/FILE0459000H Business 30_Spot_No-Price_30_Captions.mov: vfs cache: queuing for upload in 10s
2021/07/16 20:39:42 DEBUG : REDACTED/internal_review/2021_07_16/830pm/New Folder/FILE0459000H Business 30_Spot_No-Price_30_Captions.mov: vfs cache: cancelling writeback (uploading false) 0xc004460230 item 7

thanks!

In case anyone ends up googling this.

Update: Finally working as expected. Added "--no-modtime" and that seemed to do the trick. All files exported out of the app were uploaded to Google Shared Drive.

Here is the last snippet that worked for me:

ExecStart=/usr/bin/rclone mount sharedGdrive: /export/rclonefs/studio \
--allow-other \
--cache-dir=/export/d200/r_cache \
--umask=002 \
--vfs-cache-mode=writes \
--vfs-cache-poll-interval=30s \
--vfs-write-back=10s \
--low-level-retries=20 \
--retries=20 \
--retries-sleep=1s \
--no-modtime \
--log-file=/var/log/rclone/rclonefs.log
ExecStop=/bin/fusermount -uz /export/rclonefs/studio

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.