How can I audit the files I am copying with rclone?

Help and Support
1

What is the problem you are having with rclone?

I would like to count the number of files that have been copied remotely.

Run the command 'rclone version' and share the full output of the command.

rclone 1.60.1

  • os/version: fedora 37 (64 bit)
  • os/kernel: 6.0.11-300.fc37.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.3
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone ls gdrive:path/to/files | wc -l

Hello, everyone. Sorry if I didn't follow this template 100%. I'm not sure if I am asking this in the correct place.
Anyway, what I am trying to do is audit the files I am copying using rclone. For example, let's say I have a local path like:

/files/mydata

Let's assume that path has two folders each with two files (four files total).

What is a good way to count what's been copied remotely so that I can compare to what I have locally?

I was thinking of doing a simple LS but pipe it to a line counter but I'm not sure how accurate that is or how it deals with directories.

Any ideas on a better way to audit what is uploaded vs. what I have locally? Keeping in mind I have over 1 millions files I probably would need to break down the audits into smaller folders.

Thanks for any suggestions.

2

Hi cbmeeks,

You could use rclone lsl something like this:

rclone lsl /files/mydata/path/to/files | wc -l
rclone lsl gdrive:path/to/files | wc -l

Edit: You may have to add --stats=0 to disable stats from interfering, not sure.

Here are some other possibilities:

rclone sync --dry-run /files/mydata/path/to/files gdrive:path/to/files

rclone check /files/mydata/path/to/files gdrive:path/to/files
rclone check --download /files/mydata/path/to/files gdrive:path/to/files

rclone md5sum gdrive:path/to/files
rclone md5sum --download /files/mydata/path/to/files

You can see an overview of all the available commands here: https://rclone.org/commands/

3

Thanks! I looked at the docs but obviously not close enough.

I will do more research next time.

4

No problem, the docs can be overwhelming, especially when you don't know exactly what you are looking for.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.