Add directories to create to transfer count

Feature
1

I'm not positive, but I think messages like these after all transfers complete are due to including the --create-empty-src-dirs flag:

2022/05/31 10:42:03 INFO  : 2022/05/31 10:42:03 -     1.107 TiB / 1.107 TiB, 100%, 9.092 KiB/s, ETA 0s
2022/05/31 10:42:03 DEBUG : home/user/5.20.11/NRD/backport/src/groovy/com/.svn/tmp/props: Making directory
2022/05/31 10:42:04 DEBUG : Temporary_Items/TeamRoom1_backup/Users/user/AppData/LocalLow/Sun/Java/Deployment/cache/6.0/58: Making directory
2022/05/31 10:42:06 DEBUG : home/user/New Laptop Movement/Documents/ABG/Rand.gov/branches/.svn/tmp/text-base: Making directory
2022/05/31 10:42:07 DEBUG : home/user/4.6.11 backup/nrd_public/grails-app/services/gov/.svn/props: Making directory
2022/05/31 10:42:07 DEBUG : home/user/New Laptop Movement/Documents/ABG/Coaches/branches/Milestone_Oct_06_2006/BRAC/javascript/.svn/props: Making directory
2022/05/31 10:42:08 INFO  : 2022/05/31 10:42:08 -     1.107 TiB / 1.107 TiB, 100%, 9.092 KiB/s, ETA 0s

On large datasets (this is 1.25 TB), there can be a great many empty directories to create, but there's no information in the stats about how long this might take. The idea here is to include the count of directories that need to be created as part of the transfer count and decrement them as the directories are being created. Presumably that would lead to a non-zero ETA time as well.

2

They are, yes

You are right of course

How long does it take for you? I don't think we create the directories in parallel and we could do that quite easily.

3

For this dataset, it took about 6.5 hours. Here are the first and last Making directory lines.

2022/05/31 10:31:15 DEBUG : home/user1/New Laptop Movement/...: Making directory
2022/05/31 16:59:16 DEBUG : home/user2/5.20.11/...: Making directory
4

That is quite a long time!

What file system is rclone creating the directories on?

Any idea how many directories?

Running the directory creation in parallel seems like a winner here.

5

The target in this case is SharePoint, and there were about 34,000 directories.

6

An easy fix would be for rclone to count the number of directories created in the stats block so you'd at least see that counting up. You are using one line stats so they wouldn't show here though.

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.