Strange behaviour in changing --vfs-read-chunk-size for google drive download quota limits prevention

Help and Support
1

What is the problem you are having with rclone?

I am using Rclone to mount a shared drive where I have a lot of videos (around 1TB cumulative, given by thousands of videos of max 3GB size). All the video need to be elaborated by a python script, that's analyze the videos frame by frame. In the first attempt I got for 2 times download quota limit exceed without any chance to complete the elaboration. I looked in the web trying to find a way and I discovered all the parameters that can be used. However I am not sure I am going in the right direction since the log file using the parameters ( --vfs-read-chunk-size) and not using it seems the same. Could you please help me?

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 6.2.0-1012-aws (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.1
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

command without parameters:

rclone mount --log-file=log.txt --log-level=DEBUG --drive-team-drive=XXXXXXXXX --daemon drive_shared:'' /home/ubuntu/gdrive_input/data

command with read parameters

rclone mount --log-file=log5.txt --log-level=DEBUG --drive-team-drive=XXXXXXXXX --daemon --buffer-size 1G --vfs-read-chunk-size 2G --vfs-read-chunk-size-limit 8G drive_shared:'' /home/ubuntu/gdrive_input/data

Please run 'rclone config redacted' and share the full output. If you get command not found, please

type = drive
client_id = XXXXX
client_secret = YYYYY
scope = drive
token = {"access_token":"____","token_type":"Bearer","refresh_token":"____","expiry":"____"}
team_drive = ____
root_folder_id =

A log from the command that you were trying to run with the -vv flag

Log for command without chunk parameters

2023/09/23 12:54:08 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "mount" "--log-file=log_2.txt" "--log-level=DEBUG" "--drive-team-drive=------------" "--daemon" "drive_shared:" "/home/ubuntu/gdrive_input/data"]
2023/09/23 12:54:08 DEBUG : Creating backend with remote "drive_shared:"
2023/09/23 12:54:08 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/09/23 12:54:08 DEBUG : drive_shared: detected overridden config - adding "{Idu2x}" suffix to name
2023/09/23 12:54:08 DEBUG : fs cache: renaming cache item "drive_shared:" to be canonical "drive_shared{Idu2x}:"
2023/09/23 12:54:08 DEBUG : rclone: Version "v1.64.0" starting with parameters ["/usr/bin/rclone" "mount" "--log-file=log_2.txt" "--log-level=DEBUG" "--drive-team-drive=0AJqFC9xznQASUk9PVA" "--daemon" "drive_shared:" "/home/ubuntu/gdrive_input/data"]
2023/09/23 12:54:08 DEBUG : Creating backend with remote "drive_shared:"
2023/09/23 12:54:08 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/09/23 12:54:08 DEBUG : drive_shared: detected overridden config - adding "{Idu2x}" suffix to name
2023/09/23 12:54:08 DEBUG : fs cache: renaming cache item "drive_shared:" to be canonical "drive_shared{Idu2x}:"
2023/09/23 12:54:08 DEBUG : Google drive root '': Mounting on "/home/ubuntu/gdrive_input/data"
2023/09/23 12:54:08 DEBUG : : Root: 
2023/09/23 12:54:08 DEBUG : : >Root: node=/, err=<nil>
2023/09/23 12:54:09 DEBUG : rclone: Version "v1.64.0" finishing with parameters ["/usr/bin/rclone" "mount" "--log-file=log_2.txt" "--log-level=DEBUG" "--drive-team-drive=0AJqFC9xznQASUk9PVA" "--daemon" "drive_shared:" "/home/ubuntu/gdrive_input/data"]
2023/09/23 12:54:13 DEBUG : /: Lookup: name="Olimpico"
2023/09/23 12:54:13 DEBUG : /: >Lookup: node=Olimpico/, err=<nil>
2023/09/23 12:54:13 DEBUG : Olimpico/: Attr: 
2023/09/23 12:54:13 DEBUG : Olimpico/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2023/09/23 12:54:13 DEBUG : Olimpico/: Lookup: name="Girato_integrale"
2023/09/23 12:54:14 DEBUG : Olimpico/: >Lookup: node=Olimpico/Girato_integrale/, err=<nil>
2023/09/23 12:54:14 DEBUG : Olimpico/Girato_integrale/: Attr: 
2023/09/23 12:54:14 DEBUG : Olimpico/Girato_integrale/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2023/09/23 12:54:14 DEBUG : Olimpico/Girato_integrale/: Lookup: name="CameraIsotta"
2023/09/23 12:54:14 DEBUG : Olimpico/Girato_integrale/: >Lookup: node=Olimpico/Girato_integrale/CameraIsotta/, err=<nil>
2023/09/23 12:54:14 DEBUG : Olimpico/Girato_integrale/CameraIsotta/: Attr: 
2023/09/23 12:54:14 DEBUG : Olimpico/Girato_integrale/CameraIsotta/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2023/09/23 12:54:14 DEBUG : Olimpico/Girato_integrale/CameraIsotta/: Lookup: name="C0489.MP4"
2023/09/23 12:54:15 DEBUG : Olimpico/Girato_integrale/CameraIsotta/: >Lookup: node=Olimpico/Girato_integrale/CameraIsotta/C0489.MP4, err=<nil>
2023/09/23 12:54:15 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: Attr: 
2023/09/23 12:54:15 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: >Attr: a=valid=1s ino=0 size=25170414 mode=-rw-r--r--, err=<nil>
2023/09/23 12:54:15 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: Open: flags=OpenReadOnly
2023/09/23 12:54:15 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: Open: flags=O_RDONLY
2023/09/23 12:54:15 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: >Open: fd=Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r), err=<nil>
2023/09/23 12:54:15 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: >Open: fh=&{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}, err=<nil>
2023/09/23 12:54:15 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=65536, offset=0
2023/09/23 12:54:15 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.openRange at 0 length 134217728
2023/09/23 12:54:17 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 0 length 4096 chunkOffset 0 chunkSize 134217728
2023/09/23 12:54:17 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 4096 length 8192 chunkOffset 0 chunkSize 134217728
2023/09/23 12:54:17 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 12288 length 16384 chunkOffset 0 chunkSize 134217728
2023/09/23 12:54:17 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 28672 length 32768 chunkOffset 0 chunkSize 134217728
2023/09/23 12:54:17 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 61440 length 65536 chunkOffset 0 chunkSize 134217728
2023/09/23 12:54:17 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 126976 length 131072 chunkOffset 0 chunkSize 134217728
2023/09/23 12:54:17 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=65536, err=<nil>
2023/09/23 12:54:17 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: Attr: 
2023/09/23 12:54:17 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: >Attr: a=valid=1s ino=0 size=25170414 mode=-rw-r--r--, err=<nil>
2023/09/23 12:54:17 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=8192, offset=25165824
2023/09/23 12:54:17 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 258048 length 262144 chunkOffset 0 chunkSize 134217728
2023/09/23 12:54:17 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ReadFileHandle.seek from 65536 to 25165824 (fs.RangeSeeker)
2023/09/23 12:54:17 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.RangeSeek from 520192 to 25165824 length -1
2023/09/23 12:54:17 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at -1 length 4096 chunkOffset 25165824 chunkSize 134217728
2023/09/23 12:54:17 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.openRange at 25165824 length 134217728
2023/09/23 12:54:18 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 25169920 length 8192 chunkOffset 25165824 chunkSize 134217728
2023/09/23 12:54:18 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=4590, err=<nil>
2023/09/23 12:54:18 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: Attr: 
2023/09/23 12:54:18 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: >Attr: a=valid=1s ino=0 size=25170414 mode=-rw-r--r--, err=<nil>
2023/09/23 12:54:18 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=65536
2023/09/23 12:54:18 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ReadFileHandle.seek from 25170414 to 65536 (fs.RangeSeeker)
2023/09/23 12:54:18 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.RangeSeek from 25170414 to 65536 length -1
2023/09/23 12:54:18 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at -1 length 4096 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:18 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.openRange at 65536 length 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 69632 length 8192 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 77824 length 16384 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 94208 length 32768 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 126976 length 65536 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 192512 length 131072 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 323584 length 262144 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=196608
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 585728 length 524288 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=327680
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=458752
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 1110016 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=589824
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=720896
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=851968
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=983040
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 2158592 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=1114112
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=1245184
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=1376256
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=1507328
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=1638400
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=1769472
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=1900544
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2031616
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 3207168 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2162688
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2293760
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2424832
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2555904
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2686976
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2818048
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 4255744 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 5304320 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 6352896 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 7401472 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 8450048 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 9498624 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 10547200 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 11595776 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 12644352 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 13692928 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 14741504 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 15790080 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 16838656 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 17887232 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2949120
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3080192
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 18935808 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3211264
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3342336
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3473408
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3604480
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3735552
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3997696
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: waiting for in-sequence read to 3997696 for 20ms
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3866624
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=4128768
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 19984384 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=4259840
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=4390912
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=4521984
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=4653056
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=4784128
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=4915200
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=5046272
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=5177344
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=5308416
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 21032960 length 1048576 chunkOffset 65536 chunkSize 134217728
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: Attr: 
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: >Attr: a=valid=1s ino=0 size=25170414 mode=-rw-r--r--, err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Flush: 
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Flush: err=<nil>
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Release: 
2023/09/23 12:54:19 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ReadFileHandle.Release closing
2023/09/23 12:54:19 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Release: err=<nil>
2

And here the log with the chunk size parameters

Log for command with chunk parameters

2023/09/23 12:49:47 Can't set -v and --log-level
2023/09/23 12:50:13 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "mount" "--log-file=log5.txt" "--log-level=DEBUG" "--drive-team-drive=------------" "--daemon" "--buffer-size" "1G" "--vfs-read-chunk-size" "2G" "--vfs-read-chunk-size-limit" "8G" "drive_shared:" "/home/ubuntu/gdrive_input/data"]
2023/09/23 12:50:13 DEBUG : Creating backend with remote "drive_shared:"
2023/09/23 12:50:13 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/09/23 12:50:13 DEBUG : drive_shared: detected overridden config - adding "{Idu2x}" suffix to name
2023/09/23 12:50:13 DEBUG : fs cache: renaming cache item "drive_shared:" to be canonical "drive_shared{Idu2x}:"
2023/09/23 12:50:13 DEBUG : rclone: Version "v1.64.0" starting with parameters ["/usr/bin/rclone" "mount" "--log-file=log5.txt" "--log-level=DEBUG" "--drive-team-drive=0AJqFC9xznQASUk9PVA" "--daemon" "--buffer-size" "1G" "--vfs-read-chunk-size" "2G" "--vfs-read-chunk-size-limit" "8G" "drive_shared:" "/home/ubuntu/gdrive_input/data"]
2023/09/23 12:50:13 DEBUG : Creating backend with remote "drive_shared:"
2023/09/23 12:50:13 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/09/23 12:50:13 DEBUG : drive_shared: detected overridden config - adding "{Idu2x}" suffix to name
2023/09/23 12:50:13 DEBUG : fs cache: renaming cache item "drive_shared:" to be canonical "drive_shared{Idu2x}:"
2023/09/23 12:50:13 DEBUG : Google drive root '': Mounting on "/home/ubuntu/gdrive_input/data"
2023/09/23 12:50:13 DEBUG : : Root: 
2023/09/23 12:50:13 DEBUG : : >Root: node=/, err=<nil>
2023/09/23 12:50:13 DEBUG : rclone: Version "v1.64.0" finishing with parameters ["/usr/bin/rclone" "mount" "--log-file=log5.txt" "--log-level=DEBUG" "--drive-team-drive=0AJqFC9xznQASUk9PVA" "--daemon" "--buffer-size" "1G" "--vfs-read-chunk-size" "2G" "--vfs-read-chunk-size-limit" "8G" "drive_shared:" "/home/ubuntu/gdrive_input/data"]
2023/09/23 12:50:19 DEBUG : /: Lookup: name="Olimpico"
2023/09/23 12:50:19 DEBUG : /: >Lookup: node=Olimpico/, err=<nil>
2023/09/23 12:50:19 DEBUG : Olimpico/: Attr: 
2023/09/23 12:50:19 DEBUG : Olimpico/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2023/09/23 12:50:19 DEBUG : Olimpico/: Lookup: name="Girato_integrale"
2023/09/23 12:50:20 DEBUG : Olimpico/: >Lookup: node=Olimpico/Girato_integrale/, err=<nil>
2023/09/23 12:50:20 DEBUG : Olimpico/Girato_integrale/: Attr: 
2023/09/23 12:50:20 DEBUG : Olimpico/Girato_integrale/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2023/09/23 12:50:20 DEBUG : Olimpico/Girato_integrale/: Lookup: name="CameraIsotta"
2023/09/23 12:50:20 DEBUG : Olimpico/Girato_integrale/: >Lookup: node=Olimpico/Girato_integrale/CameraIsotta/, err=<nil>
2023/09/23 12:50:20 DEBUG : Olimpico/Girato_integrale/CameraIsotta/: Attr: 
2023/09/23 12:50:20 DEBUG : Olimpico/Girato_integrale/CameraIsotta/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2023/09/23 12:50:20 DEBUG : Olimpico/Girato_integrale/CameraIsotta/: Lookup: name="C0489.MP4"
2023/09/23 12:50:21 DEBUG : Olimpico/Girato_integrale/CameraIsotta/: >Lookup: node=Olimpico/Girato_integrale/CameraIsotta/C0489.MP4, err=<nil>
2023/09/23 12:50:21 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: Attr: 
2023/09/23 12:50:21 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: >Attr: a=valid=1s ino=0 size=25170414 mode=-rw-r--r--, err=<nil>
2023/09/23 12:50:21 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: Open: flags=OpenReadOnly
2023/09/23 12:50:21 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: Open: flags=O_RDONLY
2023/09/23 12:50:21 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: >Open: fd=Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r), err=<nil>
2023/09/23 12:50:21 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: >Open: fh=&{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}, err=<nil>
2023/09/23 12:50:21 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=65536, offset=0
2023/09/23 12:50:21 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.openRange at 0 length 2147483648
2023/09/23 12:50:22 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 0 length 4096 chunkOffset 0 chunkSize 2147483648
2023/09/23 12:50:22 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 4096 length 8192 chunkOffset 0 chunkSize 2147483648
2023/09/23 12:50:22 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 12288 length 16384 chunkOffset 0 chunkSize 2147483648
2023/09/23 12:50:22 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 28672 length 32768 chunkOffset 0 chunkSize 2147483648
2023/09/23 12:50:22 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 61440 length 65536 chunkOffset 0 chunkSize 2147483648
2023/09/23 12:50:22 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 126976 length 131072 chunkOffset 0 chunkSize 2147483648
2023/09/23 12:50:22 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=65536, err=<nil>
2023/09/23 12:50:22 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: Attr: 
2023/09/23 12:50:22 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: >Attr: a=valid=1s ino=0 size=25170414 mode=-rw-r--r--, err=<nil>
2023/09/23 12:50:22 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=8192, offset=25165824
2023/09/23 12:50:22 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 258048 length 262144 chunkOffset 0 chunkSize 2147483648
2023/09/23 12:50:22 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ReadFileHandle.seek from 65536 to 25165824 (fs.RangeSeeker)
2023/09/23 12:50:22 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.RangeSeek from 520192 to 25165824 length -1
2023/09/23 12:50:22 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at -1 length 4096 chunkOffset 25165824 chunkSize 2147483648
2023/09/23 12:50:22 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.openRange at 25165824 length 2147483648
2023/09/23 12:50:23 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 25169920 length 8192 chunkOffset 25165824 chunkSize 2147483648
2023/09/23 12:50:23 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=4590, err=<nil>
2023/09/23 12:50:23 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: Attr: 
2023/09/23 12:50:23 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: >Attr: a=valid=1s ino=0 size=25170414 mode=-rw-r--r--, err=<nil>
2023/09/23 12:50:23 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=65536
2023/09/23 12:50:23 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ReadFileHandle.seek from 25170414 to 65536 (fs.RangeSeeker)
2023/09/23 12:50:23 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.RangeSeek from 25170414 to 65536 length -1
2023/09/23 12:50:23 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at -1 length 4096 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:23 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.openRange at 65536 length 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 69632 length 8192 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 77824 length 16384 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 94208 length 32768 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 126976 length 65536 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 192512 length 131072 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 323584 length 262144 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=196608
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 585728 length 524288 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=327680
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=458752
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 1110016 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=589824
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=720896
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=851968
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=983040
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 2158592 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=1114112
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=1245184
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=1376256
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=1507328
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=1638400
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=1769472
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=1900544
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2031616
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 3207168 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2162688
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2293760
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2424832
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2555904
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2686976
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2818048
2023/09/23 12:50:24 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 4255744 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 5304320 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 6352896 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 7401472 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 8450048 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 9498624 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 10547200 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 11595776 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 12644352 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:24 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 13692928 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:25 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 14741504 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:25 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 15790080 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:25 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 16838656 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:25 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 17887232 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=2949120
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3080192
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3211264
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3342336
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3473408
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3604480
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3735552
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3866624
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=3997696
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=4128768
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=4259840
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=4390912
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=4521984
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=4653056
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=4784128
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 18935808 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=5046272
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=4915200
2023/09/23 12:50:25 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: waiting for in-sequence read to 5046272 for 20ms
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=5177344
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Read: len=131072, offset=5308416
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Read: read=131072, err=<nil>
2023/09/23 12:50:25 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: Attr: 
2023/09/23 12:50:25 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: >Attr: a=valid=1s ino=0 size=25170414 mode=-rw-r--r--, err=<nil>
2023/09/23 12:50:25 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 19984384 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:25 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 21032960 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Flush: 
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Flush: err=<nil>
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: Release: 
2023/09/23 12:50:25 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ReadFileHandle.Release closing
2023/09/23 12:50:25 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 22081536 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:25 DEBUG : Olimpico/Girato_integrale/CameraIsotta/C0489.MP4: ChunkedReader.Read at 23130112 length 1048576 chunkOffset 65536 chunkSize 2147483648
2023/09/23 12:50:25 DEBUG : &{Olimpico/Girato_integrale/CameraIsotta/C0489.MP4 (r)}: >Release: err=<nil>
2023/09/23 12:50:57 DEBUG : /: Attr: 
2023/09/23 12:50:57 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2023/09/23 12:50:57 DEBUG : : Statfs: 
2023/09/23 12:50:57 DEBUG : Google drive root '': read info from Shared Drive "Triathlon Cesenatico 2023"
2023/09/23 12:50:57 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2023/09/23 12:50:57 DEBUG : /home/ubuntu/gdrive_input/data: Unmounted externally. Just exit now.
2023/09/23 12:50:57 DEBUG : rclone: Version "v1.64.0" finishing with parameters ["/usr/bin/rclone" "mount" "--log-file=log5.txt" "--log-level=DEBUG" "--drive-team-drive=0AJqFC9xznQASUk9PVA" "--daemon" "--buffer-size" "1G" "--vfs-read-chunk-size" "2G" "--vfs-read-chunk-size-limit" "8G" "drive_shared:" "/home/ubuntu/gdrive_input/data"]
3

welcome to the forum,

does your script read every single byte of the entire video file or does it work like ffmpeg, reads a small part of the entire file.

and might want to use a cache mode, such as --vfs-cache-mode=full
if you need to re-analyze a video, it would be in the local cache, no need to read direct from gdrive.

4

There's no flag that really does this. If you have to download the whole file, that goes against your quota. Google doesn't really share/document/explain what the download quota is and shared drives are even a bit more unique.

Read chunk size really would potentially reduce API hits, but Google's daily API quota is really impossible to hit as it's a billion or something.

So if you have a 1GB file, and a read size of say 256M, it takes 4 requests to get the whole file, 1GB range, 1 request, etc.

5

Unfortunately yes. I need to go from the start to the end frame by frame.
I tried with the cache flag as well. But I was not able to understand from API calls any differences.

6

That was my expectation. There is something I am missing, because the 2 logs appears to be the same from api calls even if I placed --vfs-read-chunk-size 2GB. Am I reading the log file in a wrong way?

7

They are different as you can see the chunksize from one to the other.

8

IMO given your workload you are missing the key flags:

--vfs-cache-mode full --vfs-cache-max-size 200G

Their lack (and default --vfs-cache-mode off) most likely causes your script reading the same data again and again from the remote exhausting you quotas in the process.

Adjust --vfs-cache-max-size to the amount of disk space you can spare for cache - more is better.

Also remove --buffer-size 1G flag - it seems to be useless based on people experience.

9

Maybe / maybe not.

You only want that flag if you are closing and re-opening the file. If you are opening it once and sequentially reading, this would add waste the process. I didn't see any re-open in your log, but it was a short log.

10

Right. I was probably misunderstanding the way how RClone logs. Looking at elements LENGHT and ChunkOffset I was thinking every of these rows in the log were representing the API calls. But I am understanding right now these are not API calls but, maybe, RClone elaboration of API returned values. Did I got properly?
There is a way to see the API calls RClone did to gdrive?

11

I'll share a log of the complete process in few hours. The log comes from a portion of the process I reduced to create a log faster

12

This is happening to me as well while I am running a rebuild of my plex library.

ERROR : IO error: vfs reader: failed to write to cache file: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded

Options for my mount:

# Global
RCLONE_CHECKERS=8
RCLONE_FAST_LIST=true
RCLONE_LOG_LEVEL=INFO
RCLONE_TRANSFERS=4
RCLONE_USE_MMAP=true

# Mount
RCLONE_ALLOW_OTHER=true
RCLONE_ATTR_TIMEOUT=5m
RCLONE_CACHE_DIR="/var/lib/rclone/cache"
RCLONE_DIR_CACHE_TIME=768h
RCLONE_POLL_INTERVAL=5m
RCLONE_VFS_CACHE_MAX_AGE="384h"
RCLONE_VFS_CACHE_MAX_SIZE="500G"
RCLONE_VFS_CACHE_MODE="full"
RCLONE_VFS_CACHE_POLL_INTERVAL="10m"
RCLONE_VFS_READ_AHEAD="64M"
RCLONE_VFS_WRITE_BACK="1m"
RCLONE_WRITE_BACK_CACHE=true

I don't have video preview thumbnails generation active, so it should only be reading each file enough to do the analysis but I've hit the download quota 2 days in a row and there is no way I am hitting the 10TB download limit.

Are the quotas possibly reduced when google drive is in the read-only state (due to over-quota storage)?

[gcd]
type = drive
scope = drive.readonly
token = XXX
team_drive =
root_folder_id = XXX
use_trash = false
skip_gdocs = true
chunk_size = 256Mi
acknowledge_abuse = true
client_id = XXX
client_secret = XXX
13

Google doesn't document any download limits and if you are using a team drive, those are a bit worse from what I've seen.

There's nothing to do from the rclone side as you have to download less and wait it out as Google will not tell you no matter how hard you ask what limit you are breaking nor what the actual limit is.

Read only or write doesn't matter either as downloads and uploads 'seem' to be different quotas as the only documented quota is the 750GB per user upload limit.

14

I may possibly be underestimating just how much I've downloaded during these scan as i think I got through about 30,000 files before it capped out. Even if I only downloaded 64Mb of each file that is ~2TB.

I'll drop the read ahead and buffer size down and try again tonight.

With transfers at 4 and buffer size at the default of 16Mb, does that mean it would download 4x16Mb + 64Mb (read ahead) for each file, regardless of how much of the actual file is read?

15

The flags really do not impact anything as best as I've tested previously. Google does not document nor will they give you any information on your quota so it's legit a guessing game to understand why you got it.

16

Guys, perhaps you know if the quota limit Google drive issue persists also using a RClone copy call?
What if for instance I copy an entire folder to a different cloud bucket (e.g. AWS S3)?

17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.