What is the problem you are having with rclone?
I am trying to create a csv file that contains a list of all the files and folders with size and modification date information of our Dropbox and our Shared Google Drive (drive).
The goal is to get an idea of the folder structure with files, specifically mentioning the size of each file and path and also important, listing the a reliable date of the file to get an idea of how old the files are. To me lsl looked like a perfect option to use, it's recursive and I would be able to redirect everything into a readable csv file so my colleagues could get a good idea of the folder structure and file information. But when we opened the csv file we noticed a couple of very old dates in the list. So modification date is not reliable, I saw other posts mentioning --format options rclone lsf but it does not include a upload date parameter. Is there a way to let rclone list upload date of a file next to modification date of a file?
Run the command 'rclone version' and share the full output of the command.
rclone v1.62.2
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.15.90.1-microsoft-standard-WSL2 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Dropbox & Google Drive (Shared Drive)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
For instance row 12 says, modification date
Dropbox
rclone lsl n:/redacted/redacted > Dropbox-files-folders-structure.csv
more specifically
rclone lsl n:/redacted/redacted | sed -r 's/ *([^ ]*) ([^ ]*) ([^ ]*) (.*)/\1,\2,\3,\4/' > Dropbox-files-folders-structure.csv
Drive
rclone -v lsl gd-shrd-exp:/ | sed -r 's/ *([^ ]*) ([^ ]*) ([^ ]*) (.*)/\1,\2,\3,\4/' > drive-files-folders-structure.csv
Also realising now I can use --format to ouput to a csv file with custom delimiters, after I spend time struggling with sed's regex. This post took a bit of time to make and half way realised the weird dates where no fault of rclone.
The rclone config contents with secrets removed.
[n]
type = dropbox
token = {"access_token":"redacted","token_type":"bearer","refresh_token":"redacted","expiry":"2023-05-24T15:10:06.828684444+02:00"
A log from the command with the -vv flag
rclone -vv 2>dropbox-lsl-stderr.log lsl n:/redacted/redacted| sed -r 's/ *([^ ]*) ([^ ]*) ([^ ]*) (.*)/\1,\2,\3,\4/' > Dropbox-files-folders-structure.csv
2023/05/24 11:24:41 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "-vv" "lsl" "n:/redacted/redacted"]
2023/05/24 11:24:41 DEBUG : Creating backend with remote "n:/redacted/redacted"
2023/05/24 11:24:41 DEBUG : Using config file from "/home/redacted/.config/rclone/rclone.conf"
2023/05/24 11:24:43 DEBUG : Dropbox root '': Using root namespace "4675797712"
2023/05/24 11:24:44 DEBUG : fs cache: renaming cache item "n:/redacted/redacted" to be canonical "n:redacted/redacted"
2023/05/24 11:25:15 NOTICE: too_many_requests/...: Too many requests or write operations. Trying again in 300 seconds.
2023/05/24 11:25:15 DEBUG : pacer: low level retry 1/10 (error too_many_requests/...)
2023/05/24 11:25:15 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2023/05/24 11:25:15 DEBUG : pacer: Reducing sleep to 3m45s
2023/05/24 11:25:15 DEBUG : pacer: Reducing sleep to 2m48.75s
2023/05/24 11:25:15 DEBUG : pacer: Reducing sleep to 2m6.5625s
2023/05/24 11:25:15 DEBUG : pacer: Reducing sleep to 1m34.921875s
2023/05/24 11:25:15 DEBUG : pacer: Reducing sleep to 1m11.19140625s
2023/05/24 11:25:15 DEBUG : pacer: Reducing sleep to 53.393554687s
2023/05/24 11:25:15 DEBUG : pacer: Reducing sleep to 40.045166015s
2023/05/24 11:25:15 DEBUG : pacer: Reducing sleep to 30.033874511s
2023/05/24 11:30:15 DEBUG : pacer: Reducing sleep to 22.525405883s
2023/05/24 11:30:45 DEBUG : pacer: Reducing sleep to 16.894054412s
2023/05/24 11:31:08 DEBUG : pacer: Reducing sleep to 12.670540809s
2023/05/24 11:31:24 DEBUG : pacer: Reducing sleep to 9.502905606s
2023/05/24 11:31:37 DEBUG : pacer: Reducing sleep to 7.127179204s
2023/05/24 11:31:47 DEBUG : pacer: Reducing sleep to 5.345384403s
2023/05/24 11:31:54 DEBUG : pacer: Reducing sleep to 4.009038302s
2023/05/24 11:31:59 DEBUG : pacer: Reducing sleep to 3.006778726s
2023/05/24 11:32:03 DEBUG : pacer: Reducing sleep to 2.255084044s
2023/05/24 11:32:06 DEBUG : pacer: Reducing sleep to 1.691313033s
2023/05/24 11:32:08 DEBUG : pacer: Reducing sleep to 1.268484774s
2023/05/24 11:32:10 DEBUG : pacer: Reducing sleep to 951.36358ms
2023/05/24 11:32:11 DEBUG : pacer: Reducing sleep to 713.522685ms
2023/05/24 11:32:12 DEBUG : pacer: Reducing sleep to 535.142013ms
2023/05/24 11:32:13 DEBUG : pacer: Reducing sleep to 401.356509ms
2023/05/24 11:32:14 DEBUG : pacer: Reducing sleep to 301.017381ms
2023/05/24 11:32:14 DEBUG : pacer: Reducing sleep to 225.763035ms
2023/05/24 11:32:14 DEBUG : pacer: Reducing sleep to 169.322276ms
2023/05/24 11:32:15 DEBUG : pacer: Reducing sleep to 126.991707ms
2023/05/24 11:32:15 DEBUG : pacer: Reducing sleep to 95.24378ms
2023/05/24 11:32:15 DEBUG : pacer: Reducing sleep to 71.432835ms
2023/05/24 11:32:15 DEBUG : pacer: Reducing sleep to 53.574626ms
2023/05/24 11:32:15 DEBUG : pacer: Reducing sleep to 40.180969ms
2023/05/24 11:32:15 DEBUG : pacer: Reducing sleep to 30.135726ms
2023/05/24 11:32:15 DEBUG : pacer: Reducing sleep to 22.601794ms
2023/05/24 11:32:16 DEBUG : pacer: Reducing sleep to 16.951345ms
2023/05/24 11:32:16 DEBUG : pacer: Reducing sleep to 12.713508ms
2023/05/24 11:32:16 DEBUG : pacer: Reducing sleep to 10ms
2023/05/24 11:35:56 NOTICE: too_many_requests/...: Too many requests or write operations. Trying again in 300 seconds.
2023/05/24 11:35:56 DEBUG : pacer: low level retry 1/10 (error too_many_requests/...)
2023/05/24 11:35:56 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2023/05/24 11:35:56 DEBUG : pacer: Reducing sleep to 3m45s
2023/05/24 11:35:56 NOTICE: too_many_requests/...: Too many requests or write operations. Trying again in 300 seconds.
2023/05/24 11:35:56 DEBUG : pacer: low level retry 1/10 (error too_many_requests/...)
2023/05/24 11:35:56 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2023/05/24 11:35:56 DEBUG : pacer: Reducing sleep to 3m45s
2023/05/24 11:35:56 DEBUG : pacer: Reducing sleep to 2m48.75s
2023/05/24 11:35:56 DEBUG : pacer: Reducing sleep to 2m6.5625s
2023/05/24 11:35:56 DEBUG : pacer: Reducing sleep to 1m34.921875s
2023/05/24 11:35:56 DEBUG : pacer: Reducing sleep to 1m11.19140625s
2023/05/24 11:35:56 DEBUG : pacer: Reducing sleep to 53.393554687s
2023/05/24 11:40:56 DEBUG : pacer: Reducing sleep to 40.045166015s
2023/05/24 11:41:50 DEBUG : pacer: Reducing sleep to 30.033874511s
2023/05/24 11:42:30 DEBUG : pacer: Reducing sleep to 22.525405883s
2023/05/24 11:43:00 DEBUG : pacer: Reducing sleep to 16.894054412s
2023/05/24 11:43:22 DEBUG : pacer: Reducing sleep to 12.670540809s
2023/05/24 11:43:40 DEBUG : pacer: Reducing sleep to 9.502905606s
2023/05/24 11:43:52 DEBUG : pacer: Reducing sleep to 7.127179204s
2023/05/24 11:44:01 DEBUG : pacer: Reducing sleep to 5.345384403s
2023/05/24 11:44:08 DEBUG : pacer: Reducing sleep to 4.009038302s
2023/05/24 11:44:14 DEBUG : pacer: Reducing sleep to 3.006778726s
2023/05/24 11:44:18 DEBUG : pacer: Reducing sleep to 2.255084044s
2023/05/24 11:44:21 DEBUG : pacer: Reducing sleep to 1.691313033s
2023/05/24 11:44:23 DEBUG : pacer: Reducing sleep to 1.268484774s
2023/05/24 11:44:25 DEBUG : pacer: Reducing sleep to 951.36358ms
2023/05/24 11:44:26 DEBUG : pacer: Reducing sleep to 713.522685ms
2023/05/24 11:44:27 DEBUG : pacer: Reducing sleep to 535.142013ms
2023/05/24 11:44:28 DEBUG : pacer: Reducing sleep to 401.356509ms
2023/05/24 11:44:28 DEBUG : pacer: Reducing sleep to 301.017381ms
2023/05/24 11:44:29 DEBUG : pacer: Reducing sleep to 225.763035ms
2023/05/24 11:44:29 DEBUG : pacer: Reducing sleep to 169.322276ms
2023/05/24 11:44:29 DEBUG : pacer: Reducing sleep to 126.991707ms
2023/05/24 11:44:29 DEBUG : pacer: Reducing sleep to 95.24378ms
2023/05/24 11:44:30 DEBUG : pacer: Reducing sleep to 71.432835ms
2023/05/24 11:44:30 DEBUG : pacer: Reducing sleep to 53.574626ms
2023/05/24 11:44:30 DEBUG : pacer: Reducing sleep to 40.180969ms
2023/05/24 11:44:30 DEBUG : pacer: Reducing sleep to 30.135726ms
2023/05/24 11:44:30 DEBUG : pacer: Reducing sleep to 22.601794ms
2023/05/24 11:44:30 DEBUG : pacer: Reducing sleep to 16.951345ms
2023/05/24 11:44:30 DEBUG : pacer: Reducing sleep to 12.713508ms
2023/05/24 11:44:30 DEBUG : pacer: Reducing sleep to 10ms
2023/05/24 11:48:06 NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 300 seconds.
2023/05/24 11:48:06 DEBUG : pacer: low level retry 1/10 (error too_many_requests/.)
2023/05/24 11:48:06 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2023/05/24 11:48:07 NOTICE: too_many_requests/: Too many requests or write operations. Trying again in 300 seconds.
2023/05/24 11:48:07 DEBUG : pacer: low level retry 1/10 (error too_many_requests/)
2023/05/24 11:48:07 DEBUG : pacer: Reducing sleep to 3m45s
2023/05/24 11:48:07 NOTICE: too_many_requests/: Too many requests or write operations. Trying again in 300 seconds.
2023/05/24 11:48:07 DEBUG : pacer: low level retry 2/10 (error too_many_requests/)
2023/05/24 11:48:07 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2023/05/24 11:48:07 DEBUG : pacer: Reducing sleep to 3m45s
2023/05/24 11:48:07 DEBUG : pacer: Reducing sleep to 2m48.75s
2023/05/24 11:48:07 DEBUG : pacer: Reducing sleep to 2m6.5625s
2023/05/24 11:48:07 DEBUG : pacer: Reducing sleep to 1m34.921875s
2023/05/24 11:48:07 DEBUG : pacer: Reducing sleep to 1m11.19140625s
2023/05/24 11:49:18 INFO : Signal received: interrupt
2023/05/24 11:49:18 INFO : Dropbox root 'redacted/redacted': Committing uploads - please wait...
2023/05/24 11:49:18 INFO : Exiting...