Data flow diagram

Help and Support
1

Can we have data flow diagram .

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

Paste  log here
1 Like
2

Are you asking something about rclone?

3

Hello,

Yes, I am asking about rclone. How the data flow happens.

Example : If i have source as: AWS bucket and Destination as : GCP bucket and rclone is installed on-premise. How the dataflow will happen ?

Thanks in advance!

Regards,
Raghava Rao

4

On the machine running rclone, you'd download from the source and upload to the destination using your machine's bandwidth with two different cloud providers.

5

Thank You so much for the clarification.
I have couple of query before implementing.

Query 1 : Does it use machine memory(cache) because use we are planning to move huge amount of data.

Query 2: While copying data, if machine(rclone machine) crash. What will happen to the data in progress ?

Query 3: Do we have any document which guide me how to run copy job through batches.

Regards,
Raghava

6

Sure, rclone uses memory but it's very dependent on what you are running, how many transfers are going, buffer size, etc. Depending on the memory on the server, you can tweak these things to run. I don't know what a huge amount of data as size is all relative.

Open transfers would be aborted if the machine went off and they'd have to be reuploaded. There is nothing stored on the client.

You can check out:

and

7

Thank You for the quick response.

Come to Query 1 again. its bothering me about machine hardware requirement (i.e. memory processes and hard disk) . Do we have any machine hardware requirement matrix.

Regards,
Raghava

8

You can run on very tiny machines or very large machines. It's really dependent on what parameters you use and the expectations of how fast you want things to go.

The smaller the machine, the lower the settings and longer things take. People run on raspberry Pis and on huge machines.

9

Thank You so much.

11

Hello,

Can you please help me to understand Transferred(1%)and Transferring(0%) while i start the copy.

Bandwith limited as 102KB. But its using more then that.

D:\rclone>rclone copy AWS:rclone_migration12_aws GCP:rclone_migration12_gcp --bwlimit 0.1m -P -v
2020/08/12 13:05:55 INFO : Starting bandwidth limiter at 102.399kBytes/s
Transferred: 8.996M / 1.416 GBytes, 1%, 106.797 kBytes/s, ETA 3h50m20s
Checks: 10 / 10, 100%
Transferred: 0 / 1, 0%
Elapsed time: 1m26.2s
Transferring:

  •                   test_file.zip:  0% /1.416G, 97.908k/s, 4h11m14s

Regards,
Raghava Rao

12

It's how much of the file it transferred and you limited a lot with bwlimit.

The bwlimit is for file transfers and you'll see some more with the list operations and such.

13

Thank You for the reply.

My question is .. which one is download and which is upload ?

2020/08/12 13:05:55 INFO : Starting bandwidth limiter at 102.399kBytes/s
Transferred: 8.996M / 1.416 GBytes, 1%, 106.797 kBytes/s, ETA 3h50m20s
Checks: 10 / 10, 100%
Transferred: 0 / 1, 0%
Elapsed time: 1m26.2s
Transferring:

  •            **test_file.zip:  0% /1.416G, 97.908k/s, 4h11m14s**

We can see two copy %

14

Both as you are moving from the source to destination so it has to download from the source and upload to the destination.

15

So, You mean to say from the output.
Transferred: 8.996M / 1.416 GBytes, 1%, 106.797 kBytes/s, ETA 3h50m20s --> Download

Transferring:

  • test_file.zip: 0% /1.416G, 97.908k/s, 4h11m14 --> Upload

Is that right ?

16

No.

That's just overall stats for the whole process.

That's the one file that's being transferred.

When. you copy or move a file, you have to download from the source and upload to the destination.

The top one aggregates a bit more and the bottom is the per file being moved. You are only moving 1 file so it really doesn't matter which you look at.

17

Thank You for your quick support.

18

Hi Animosity022,

Thank You for all your help and support on clearing all my query's.

My query on data flow again. As you said, rlcone machine will be download from source and upload to destination.

When it download, where it will save in the machine --> it uses cache or HDD

Lets take example of 10GB single file

Regards,
Raghava Rao

19

It doesn't save anything on the local machine as it just uses memory to move it.

20

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.