Need help getting status output how I want and/or is there anywhere I can see examples of various verbosity and logging outputs?

Help and Support
1

What I Want To Do

I'm trying to get rclone output to match what I want. I know I could play around with various combinations but that's very time consuming. I was hoping someone could help me -- or point me to some documented examples of different switches and their result that would help me get to what I want faster.

This is my rclone command:

rclone copy -v "$SOURCE" "$DESTINATION" --files-from "$TO_TRANSFER" --no-traverse

I'm getting output like this:

transferring files from source '/source' to destination 'gp:destination'
<6>INFO  : folder1/folder2/folder3/IMG_9008.jpg: Copied (new)
<6>INFO  : folder1/folder2/folder3/IMG_9009.jpg: Copied (new)
<6>INFO  : folder1/folder2/folder3/IMG_9011.jpg: Copied (new)
<6>INFO  : folder1/folder2/folder3/IMG_9012.jpg: Copied (new)
<6>INFO  : folder1/folder2/folder3/IMG_9010.jpg: Copied (new)
<6>INFO  : folder1/folder2/folder3/IMG_9014.jpg: Copied (new)
<6>INFO  : folder1/folder2/folder3/IMG_9013.jpg: Copied (new)
<6>INFO  : folder1/folder2/folder3/IMG_9016.jpg: Copied (new)
<6>INFO  : folder1/folder2/folder3/IMG_9015.jpg: Copied (new)
<6>INFO  : folder1/folder2/folder3/IMG_9018.jpg: Copied (new)
<6>INFO  : folder1/folder2/folder3/IMG_9019.jpg: Copied (new)
<6>INFO  : 
Transferred:       57.881 MiB / 28.554 GiB, 0%, 474.285 KiB/s, ETA 17h30m3s
Transferred:           11 / 5519, 0%
Elapsed time:       1m0.0s
Transferring:
 * folder1…e, Zurich/IMG_9017.jpg:100% /3.000Mi, 107.388Ki/s, 0s
 * folder1…e, Zurich/IMG_9020.jpg:100% /2.671Mi, 123.835Ki/s, 0s
 * folder1…e, Zurich/IMG_9021.jpg:100% /3.156Mi, 155.911Ki/s, 0s
 * folder1…e, Zurich/IMG_9022.jpg:100% /3.258Mi, 183.294Ki/s, 0s

I know I can get rid of that status information with --stats=0 but I'm trying to see how to get the other lines to match a specific tab separated output:

  • date/time
  • path/to/file
  • action like copied, replaced, updated, etc...
  • status like success or error
  • how long it took to transfer this file

Why I Am Asking

In case it matters, I'll explain why I am asking.

I've got a script I'm working on to copy 2 TB of photos from my computer to Google Photos. Due to Google GCP API limits, I can't do all the photos in one shot. It will transfer so many files and then start erroring. So I'll stop the script, wait a few days (week) and start over.

The issue is that when I start getting GCP errors, I'm not sure if a file fully transferred or not. So there may be an incomplete transfer. Google Photos has a few limitations:

So, instead, I'm thinking of doing this:

  1. Dumping a list of all files that need to be transferred
  2. Using that file with --files-from in my copy command
  3. When I interrupt the transfer, before I restart it, I compare the output of the copy command with my initial list of files and remove any files that rclone says successfully copied
  4. Then starting the copy command again with the updated --files-from
2

Log files are not customizable beyond the amount of information included defined by --log-level flag. Unless you want to fiddle with source code and create custom rclone version for your task.

IMO you should not really see any errors if your copy command is properly tuned for Google Drive. By using few extra flags you can adjust speed to stay below transactions quotas. I am not Google user but you can find many examples on this forum how to do it. Start with something simple like --tpslimit 10

And make sure you have your own client_id configured.

Then only thing to control is daily Gdrive upload quota but this can be taken care by --max-transfer flag.

3

Humm. Then I wonder if stats output could be sent to a different file or use a different file stream so it can be redirected to a different file? Just some way to differentiate between the other output?

I will check TPSLIMIT. Thank you!

4

Have a look at this thread:

In general I think nobody thought about using logs as an input for further processing. It could be definitely done better - but it has to wait for a brave soul to tackle it and submit PR:) Joy of open source:)

5

Thanks. I'll see if I can figure something out.

And yes, while I do love rclone, it's logging/output could use some more standardization. But it's still the best tool out there. :slight_smile:

1 Like
6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.