Have a list of files under multiple folders on crypt remote - Need to get total size of just those files combined

Help and Support
1

What is the problem you are having with rclone?

Not exactly an rclone problem. I used rclone to copy multiple files under multiple folders from my local folder to a crypted remote.
Total files I noticed during the upload progress, using "--progress", was 28 GiB.
I did a dry run pass and then ran the actual copy command.
After the upload completed, I ran the copy command again multiple times to make sure anything which might have failed got uploaded. In the end made sure that nothing was left to copy.
I also ran the crypt check command (--one-way) to verify the files.

I had multiple folders under this particular local folder, which had multiple files under each to upload. There were some files already on the remote folder under each of these folders.
I generated a file containing the names for all the local files and then deleted the files from local. When I noticed the free space, it had cleared over 100GiB. Now I am a little concerned on the 72GiB+ which did not seem to have been uploaded because as I mentioned above it just showed total upload as 28 GiB.

I have a list of files (and their respective sha1 sum) which were present locally during the upload and before the local delete.

Is there a way I could automate looking up if each of those files are present in the remote?
Also, how can I get the total size of all those files combined on the remote?

What is your rclone version (output from rclone version)

rclone v1.56.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

  • os/version: Microsoft Windows 10 (64 bit)
  • os/kernel: 10.0.18363.1679 (x86_64)
  • os/type: windows
  • os/arch: amd64

Which cloud storage system are you using? (eg Google Drive)

crypted remote - Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy . crypt-remote-name:path/to/directory/ --progress --dry-run
rclone copy . crypt-remote-name:path/to/directory/ --progress
rclone cryptcheck . crypt-remote-name:path/to/directory/ --one-way --progress

The rclone config contents with secrets removed.

N/A

A log from the command with the -vv flag

N/A
2

hello and welcome to the forum,

if you want to understand what rclone is doing, need to use --log-level and perhaps --log-file

if you have a list of files, let's call it filelist.txt

perhaps try rclone cryptcheck --include-from=filelist.txt

perhaps try rclone size --include-from=filelist.txt?

1 Like
3

@asdffdsa That was quick and Thank you. Using rclone size --include-from=filelist.txt helped me get the total file size. I am guessing most of those files were uploaded during the last run many months back. Will use --log-file from now on.

The cryptcheck with --include-from did not work for me as it required a source and destination and since I had already deleted the files from my (local) source it could not be run.

Additional question, Is there a way I can have rclone log each run to a timestamped file name?
I could not find anything similar during my quick search on rclone docs.

4

it depends on what script language you want to use?
i use python to script rclone, but for a simple dos batch file that will create a filename such as
rclone.20210725.120101.log

for /f "tokens=2 delims==" %%I in ('wmic os get localdatetime /format:list') do set datetime=%%I
set logfile=rclone.%datetime:~0,4%%datetime:~4,2%%datetime:~6,2%.%datetime:~8,2%%datetime:~10,2%%datetime:~12,2%.log

rclone lsd . --log-file=%logfile% --log-level=DEBUG
1 Like
5

So, looks like I will need to script it/pass the timestamped file name to rclone myself.

Thanks for sharing your python script.

6

thanks was a windows/dos old-style batch file, not python...

7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.