Getting Video Duration of Remote Files

Andrew_Jensen · December 28, 2021, 7:30pm

What is the problem you are having with rclone?

Trying to get the video duration time (in seconds or milliseconds) of videos on google drive without having to download them first because I have to verify the video duration of around 2TB of them, but local machine only has 50 GB storage, and downloading and getting duration one by one and deleting each will exceed my network quota. So, would like to know what is the most efficient way given the constraints.

What is your rclone version (output from `rclone version`)

rclone v1.57.0

os/version: ubuntu 20.04 (64 bit)
os/kernel: 5.11.0-1022-oracle (aarch64)
os/type: linux
os/arch: arm64
go/version: go1.17.2
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone ls remote:/

asdffdsa · December 28, 2021, 7:40pm

hello and welcome to the forum,

that should be possible.

if you were to download the entire file, by what method do you obtain the video duration?

Andrew_Jensen · December 28, 2021, 8:07pm

I would possibly use python script or bash script:

ffprobe -v quiet -of csv=p=0 -show_entries format=duration SingleFileName.mp4

asdffdsa · December 28, 2021, 8:12pm

you can use a command such as this, for example,

rclone mount remote: /path/to/a/local/folder --vfs-cache-mode=full

this will have remote: appear as local storage.

and a command like this, will download just the data ffprobe asks for, not the entire file.
ffprobe -v quiet -of csv=p=0 -show_entries format=duration /path/to/a/local/folder/SingleFileName.mp4

for example, in my remote:, i have a video file of size 6.6GB
when i run ffprobe on the media file, rclone downloads just 18MB

d:\data\m\ffmpeg\bin\ffprobe.exe -v quiet -of csv=p=0 -show_entries format=duration b:\rclone\mount\remote\file.mkv 
8902.763000

Andrew_Jensen · December 28, 2021, 8:56pm

In my system, I mounted using the same command above, and ran the ffprobe and I used sudo nethogs -v 3 to check the traffic, and it shows rclone downloading the full file:

For example: for one file, it is downloading 436.562 MB.

  10095 ubuntu   rclone enp0s3      0.584     436.562 MB

asdffdsa · December 28, 2021, 9:04pm

something is not correct on your end.

rclone only downloads what it is asked to.

add -vv to your rclone mount command, run it, run the ffprobe and post the entire rclone debug log.

and notice the actual size on disk, just a fraction of the entire file size

Andrew_Jensen · December 28, 2021, 9:48pm

Hi, the log is too big for this text area. It says it has max char limitation. I was trying to figure out how to share it. It is possible to attach as a file?

asdffdsa · December 28, 2021, 9:55pm

ok, post the top 30 lines of the debug log.

please enclose it with three backtickes so it is formatted like this.

rclone version
rclone v1.57.0
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.10.60.1-microsoft-standard-WSL2 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

Andrew_Jensen · December 28, 2021, 9:56pm

2021/12/28 21:42:05 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "mount" "remote:" "./gdrive" "--vfs-cache-mode=full" "-vv"]
2021/12/28 21:42:05 DEBUG : Creating backend with remote "remote:"
2021/12/28 21:42:05 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2021/12/28 21:42:06 DEBUG : Google drive root '': 'root_folder_id = 0AKfe2XZobcMzUk9PVA' - save this in the config to speed up startup
2021/12/28 21:42:06 DEBUG : vfs cache: root is "/home/ubuntu/.cache/rclone"
2021/12/28 21:42:06 DEBUG : vfs cache: data root is "/home/ubuntu/.cache/rclone/vfs/remote"
2021/12/28 21:42:06 DEBUG : vfs cache: metadata root is "/home/ubuntu/.cache/rclone/vfsMeta/remote"
2021/12/28 21:42:06 DEBUG : Creating backend with remote "/home/ubuntu/.cache/rclone/vfs/remote/"
2021/12/28 21:42:06 DEBUG : Creating backend with remote "/home/ubuntu/.cache/rclone/vfsMeta/remote/"
2021/12/28 21:42:06 DEBUG : Google drive root '': Mounting on "./gdrive"
2021/12/28 21:42:06 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2021/12/28 21:42:06 DEBUG : : Root: 
2021/12/28 21:42:06 DEBUG : : >Root: node=/, err=<nil>
2021/12/28 21:42:06 DEBUG : /: Lookup: name=".Trash"
2021/12/28 21:42:06 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/12/28 21:42:06 DEBUG : /: Attr: 
2021/12/28 21:42:06 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/28 21:42:06 DEBUG : /: Lookup: name=".xdg-volume-info"
2021/12/28 21:42:06 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/12/28 21:42:06 DEBUG : /: Lookup: name="autorun.inf"
2021/12/28 21:42:06 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/12/28 21:42:06 DEBUG : /: Lookup: name=".Trash-1001"
2021/12/28 21:42:06 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/12/28 21:42:06 DEBUG : /: ReadDirAll: 
2021/12/28 21:42:06 DEBUG : /: >ReadDirAll: item=18, err=<nil>
2021/12/28 21:42:06 DEBUG : /: Attr: 
2021/12/28 21:42:06 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/28 21:42:06 DEBUG : /: ReadDirAll: 
2021/12/28 21:42:06 DEBUG : /: >ReadDirAll: item=18, err=<nil>
2021/12/28 21:42:06 DEBUG : /: Attr: 
2021/12/28 21:42:06 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/28 21:42:06 DEBUG : /: ReadDirAll: 
2021/12/28 21:42:06 DEBUG : /: >ReadDirAll: item=18, err=<nil>
2021/12/28 21:42:06 DEBUG : /: Attr: 
2021/12/28 21:42:06 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/28 21:42:06 DEBUG : /: ReadDirAll: 
2021/12/28 21:42:06 DEBUG : /: >ReadDirAll: item=18, err=<nil>
2021/12/28 21:42:06 DEBUG : /: Attr: 
2021/12/28 21:42:06 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/28 21:42:06 DEBUG : /: ReadDirAll: 
2021/12/28 21:42:06 DEBUG : /: >ReadDirAll: item=18, err=<nil>
2021/12/28 21:42:06 DEBUG : /: Attr: 
2021/12/28 21:42:06 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/28 21:42:06 DEBUG : /: ReadDirAll: 
2021/12/28 21:42:06 DEBUG : /: >ReadDirAll: item=18, err=<nil>
2021/12/28 21:42:06 DEBUG : /: Attr: 
2021/12/28 21:42:06 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/28 21:42:06 DEBUG : /: ReadDirAll: 
2021/12/28 21:42:06 DEBUG : /: >ReadDirAll: item=18, err=<nil>
2021/12/28 21:42:06 DEBUG : /: Attr: 
2021/12/28 21:42:06 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/28 21:42:06 DEBUG : /: ReadDirAll: 
2021/12/28 21:42:06 DEBUG : /: >ReadDirAll: item=18, err=<nil>
2021/12/28 21:42:06 DEBUG : /: Attr: 
2021/12/28 21:42:06 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/28 21:42:06 DEBUG : /: ReadDirAll: 
2021/12/28 21:42:06 DEBUG : /: >ReadDirAll: item=18, err=<nil>
2021/12/28 21:42:06 DEBUG : /: Attr: 
2021/12/28 21:42:06 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/28 21:42:06 DEBUG : /: ReadDirAll: 
2021/12/28 21:42:06 DEBUG : /: >ReadDirAll: item=18, err=<nil>
2021/12/28 21:42:06 DEBUG : /: Attr:

Andrew_Jensen · December 28, 2021, 9:59pm

This part might have more info after the command: ffprobe -v quiet -of csv=p=0 -show_entries format=duration ~/ws/gdrive/v1.mp4


2021/12/28 21:42:06 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2021/12/28 21:42:06 DEBUG : /: ReadDirAll: 
2021/12/28 21:42:06 DEBUG : /: >ReadDirAll: item=18, err=<nil>
2021/12/28 21:42:06 DEBUG : /: Lookup: name="autorun.inf"
2021/12/28 21:42:06 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/12/28 21:42:38 DEBUG : /: Lookup: name="v1.mp4"
2021/12/28 21:42:38 DEBUG : /: >Lookup: node=v1.mp4, err=<nil>
2021/12/28 21:42:38 DEBUG : v1.mp4: Attr: 
2021/12/28 21:42:38 DEBUG : v1.mp4: >Attr: a=valid=1s ino=0 size=439181679 mode=-rw-rw-r--, err=<nil>
2021/12/28 21:42:38 DEBUG : v1.mp4: Open: flags=OpenReadOnly+0x20000
2021/12/28 21:42:38 DEBUG : v1.mp4: Open: flags=O_RDONLY|0x20000
2021/12/28 21:42:38 DEBUG : v1.mp4: newRWFileHandle: 
2021/12/28 21:42:38 DEBUG : v1.mp4: >newRWFileHandle: err=<nil>
2021/12/28 21:42:38 DEBUG : v1.mp4: >Open: fd=v1.mp4 (rw), err=<nil>
2021/12/28 21:42:38 DEBUG : v1.mp4: >Open: fh=&{v1.mp4 (rw)}, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: Read: len=65536, offset=0
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): _readAt: size=65536, off=0
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): openPending: 
2021/12/28 21:42:38 DEBUG : v1.mp4: vfs cache: checking remote fingerprint "439181679,2021-12-28 21:36:36.588 +0000 UTC,9e8f84598c2762fef9c4ada9d276ca1a" against cached fingerprint ""
2021/12/28 21:42:38 DEBUG : v1.mp4: vfs cache: truncate to size=439181679
2021/12/28 21:42:38 DEBUG : : Added virtual directory entry vAddFile: "v1.mp4"
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): >openPending: err=<nil>
2021/12/28 21:42:38 DEBUG : vfs cache: looking for range={Pos:0 Size:65536} in [] - present false
2021/12/28 21:42:38 DEBUG : v1.mp4: ChunkedReader.RangeSeek from -1 to 0 length -1
2021/12/28 21:42:38 DEBUG : v1.mp4: ChunkedReader.Read at -1 length 4096 chunkOffset 0 chunkSize 134217728
2021/12/28 21:42:38 DEBUG : v1.mp4: ChunkedReader.openRange at 0 length 134217728
2021/12/28 21:42:38 DEBUG : v1.mp4: ChunkedReader.Read at 4096 length 8192 chunkOffset 0 chunkSize 134217728
2021/12/28 21:42:38 DEBUG : v1.mp4: ChunkedReader.Read at 12288 length 16384 chunkOffset 0 chunkSize 134217728
2021/12/28 21:42:38 DEBUG : v1.mp4: ChunkedReader.Read at 28672 length 32768 chunkOffset 0 chunkSize 134217728
2021/12/28 21:42:38 DEBUG : v1.mp4: ChunkedReader.Read at 61440 length 65536 chunkOffset 0 chunkSize 134217728
2021/12/28 21:42:38 DEBUG : v1.mp4: ChunkedReader.Read at 126976 length 131072 chunkOffset 0 chunkSize 134217728
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): >_readAt: n=65536, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: >Read: read=65536, err=<nil>
2021/12/28 21:42:38 DEBUG : v1.mp4: Attr: 
2021/12/28 21:42:38 DEBUG : v1.mp4: >Attr: a=valid=1s ino=0 size=439181679 mode=-rw-rw-r--, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: Read: len=131072, offset=65536
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): _readAt: size=131072, off=65536
2021/12/28 21:42:38 DEBUG : vfs cache: looking for range={Pos:65536 Size:131072} in [{Pos:0 Size:126976}] - present false
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: Read: len=131072, offset=196608
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): _readAt: size=131072, off=196608
2021/12/28 21:42:38 DEBUG : vfs cache: looking for range={Pos:196608 Size:131072} in [{Pos:0 Size:126976}] - present false
2021/12/28 21:42:38 DEBUG : v1.mp4: ChunkedReader.Read at 258048 length 262144 chunkOffset 0 chunkSize 134217728
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): >_readAt: n=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: >Read: read=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: Read: len=131072, offset=327680
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): _readAt: size=131072, off=327680
2021/12/28 21:42:38 DEBUG : vfs cache: looking for range={Pos:327680 Size:131072} in [{Pos:0 Size:258048}] - present false
2021/12/28 21:42:38 DEBUG : v1.mp4: ChunkedReader.Read at 520192 length 524288 chunkOffset 0 chunkSize 134217728
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): >_readAt: n=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: >Read: read=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): >_readAt: n=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: >Read: read=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: Read: len=131072, offset=458752
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): _readAt: size=131072, off=458752
2021/12/28 21:42:38 DEBUG : vfs cache: looking for range={Pos:458752 Size:131072} in [{Pos:0 Size:520192}] - present false
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: Read: len=131072, offset=589824
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): _readAt: size=131072, off=589824
2021/12/28 21:42:38 DEBUG : vfs cache: looking for range={Pos:589824 Size:131072} in [{Pos:0 Size:520192}] - present false
2021/12/28 21:42:38 DEBUG : v1.mp4: ChunkedReader.Read at 1044480 length 1048576 chunkOffset 0 chunkSize 134217728
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): >_readAt: n=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: >Read: read=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): >_readAt: n=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: >Read: read=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: Read: len=131072, offset=720896
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): _readAt: size=131072, off=720896
2021/12/28 21:42:38 DEBUG : vfs cache: looking for range={Pos:720896 Size:131072} in [{Pos:0 Size:1044480}] - present true
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): >_readAt: n=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: >Read: read=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: Read: len=131072, offset=851968
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): _readAt: size=131072, off=851968
2021/12/28 21:42:38 DEBUG : vfs cache: looking for range={Pos:851968 Size:131072} in [{Pos:0 Size:1044480}] - present true
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): >_readAt: n=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: >Read: read=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: Read: len=131072, offset=983040
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): _readAt: size=131072, off=983040
2021/12/28 21:42:38 DEBUG : vfs cache: looking for range={Pos:983040 Size:131072} in [{Pos:0 Size:1044480}] - present false
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: Read: len=131072, offset=1114112
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): _readAt: size=131072, off=1114112
2021/12/28 21:42:38 DEBUG : vfs cache: looking for range={Pos:1114112 Size:131072} in [{Pos:0 Size:1044480}] - present false
2021/12/28 21:42:38 DEBUG : v1.mp4: ChunkedReader.Read at 2093056 length 1048576 chunkOffset 0 chunkSize 134217728
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): >_readAt: n=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: >Read: read=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): >_readAt: n=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: >Read: read=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: Read: len=131072, offset=1245184
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): _readAt: size=131072, off=1245184
2021/12/28 21:42:38 DEBUG : vfs cache: looking for range={Pos:1245184 Size:131072} in [{Pos:0 Size:2093056}] - present true
2021/12/28 21:42:38 DEBUG : v1.mp4(0x40008e2a80): >_readAt: n=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : &{v1.mp4 (rw)}: >Read: read=131072, err=<nil>
2021/12/28 21:42:38 DEBUG : v1.mp4: Attr:

asdffdsa · December 28, 2021, 10:13pm

not sure exactly what is going on, as in my testing on windows, ffprobe, downloaded just a small amount of the total file.

now that we have the debug log, perhaps @Animosity022 has an idea?

Animosity022 · December 29, 2021, 1:45pm

I can’t test myself but I’d imagine the command being run is grabbing the full file.

Andrew_Jensen · December 29, 2021, 5:18pm

Okay, I was trying to avoid using Google Drive API, but looks like in the system I'm running, it's better to use the API to only get the video metadata otherwise it will download whole file. Thank you for looking into this issue. I will try to directly use the Google Drive Python API.

Animosity022 · December 29, 2021, 8:36pm

That’s not right.

You can medianfo a file or just ffprobe it and it’ll give you the info. No cache needed and takes 1-3 seconds per file roughly.

system · January 28, 2022, 8:36pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Getting Video Duration of Remote Files

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

What is your rclone version (output from `rclone version`)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)