anyway to intercept rclone uploaded chunks

Help and Support
1

What is the problem you are having with rclone?

i am uploading files to google drive with rclone and i want to generate some hashes , also i may read this chunks of data to broadcast to multiple storages as backup.
So, my doubt is to listen for chunks of data rclone processing for file in uploading through stdout or some medium, is it possible?

Run the command 'rclone version' and share the full output of the command.

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy ... remote:...

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

Paste  log here
2

???

or can you provide some example what you want to do and what does not work?

3

lets say a file named a.mp4 is being uploaded to drive simply by rclone copy a.mp4 drive:a.mp4
what i want is to read each chunk of data before rclone send it to drive through http api, so

Rclone ---> prints the chunk data of file in stream to stdout ( can be different medium like a unix socket ) ---> Then upload that chunk to drive.

And i want to read those chunk from stdout of rclone in my script, so my doubt was if rclone prints those chunks in debug level before sending or anyhow possible those chunks of data before rclone sends them?

4

not possible as I am aware - but lets see what others will say.

Any particular reason you need it? What is the problem you try to solve?

5

i am generating hashes for different files and doing it seperately is inefficient on huge input of data.
rclone already reading that data while uploading so if it able to print it to stdout or pass it thru a medium , then the hashing files would be on the go, also it will be flexible to integrate and modify different hashing logic when you have access to the uploaded chunk of data during rclone copy/move/sync

6

Ok clear. I do not think anything like this exist. Let's call it very exotic requirement. All is possible. But I think to get it you have dive into rclone code and DYI

7

Yeh, seems like it doesn't exists for now..
my go skills aren't that good to cope it myself so I can try working it around by reading file in python and passing the bytes to rclone subprocess , since rclone rcat allows stdin data
My intention for creating this post is to get better solution to this workaround cuz python is slow as well
Or if intercepting the chunks of underlying source during transfer is found to be good enough to get implemented then it's nice

8

good thing is it is all open source - you need it you can do it - you reallllly need it you just pay somebody to do it

9

Right! , well thanks for your help mate

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.