I'm just learning rclone, so maybe I'm missing something here. I have a directory structure in my s3 bucket that looks like:
s3://
--my/files/of/interest/
|--file1.txt
|--...
--other/files
--more/files
When I run rclone copy remote://my/files/of/interest/file1.txt ., I get file1.txt in my drive. Nice!
But when I try to get the whole directory, rclone copy remote://my/files/of/interest/, or rclone copy remote://my/files/of/interest (tried both), I get an error:
Failed to sync: error reading source directory "other": can only list from root
It feels like I'm running into a permission boundary in a part of my bucket that I'm not actually trying to access. I don't have or need permissions to view the other contents of the bucket, but since I'm only interested in the one directory, I don't see why rclone needs to look into that part of the file system.
I'm using rclone v1.36 on Ubuntu 18.04, trying to copy from an AWS S3 bucket
Logs:
$ rclone sync -vv aws://mx-automotive-derived/redfish_datasets/kitti/kitti_drive_0001/images/camera_00/ ./camera_00/
2020/01/08 17:45:08 DEBUG : rclone: Version "v1.36" starting with parameters ["rclone" "sync" "-vv" "aws://mx-automotive-derived/redfish_datasets/kitti/kitti_drive_0001/images/camera_00/" "./camera_00/"]
2020/01/08 17:45:09 INFO : Local file system at /home/zbowen/playground/images/camera_00: Modify window is 1ns
2020/01/08 17:45:09 INFO : Local file system at /home/zbowen/playground/images/camera_00: Waiting for checks to finish
2020/01/08 17:45:09 INFO : Local file system at /home/zbowen/playground/images/camera_00: Waiting for transfers to finish
2020/01/08 17:45:09 ERROR : Local file system at /home/zbowen/playground/images/camera_00: not deleting files as there were IO errors
2020/01/08 17:45:09 ERROR : Attempt 1/3 failed with 0 errors and: error reading source directory "mx-automotive-samsung-usa": can only list from root
2020/01/08 17:45:09 INFO : Local file system at /home/zbowen/playground/images/camera_00: Waiting for checks to finish
2020/01/08 17:45:09 INFO : Local file system at /home/zbowen/playground/images/camera_00: Waiting for transfers to finish
2020/01/08 17:45:09 ERROR : Local file system at /home/zbowen/playground/images/camera_00: not deleting files as there were IO errors
2020/01/08 17:45:09 ERROR : Attempt 2/3 failed with 0 errors and: error reading source directory "mx-automotive-samsung-usa": can only list from root
2020/01/08 17:45:09 INFO : Local file system at /home/zbowen/playground/images/camera_00: Waiting for checks to finish
2020/01/08 17:45:09 INFO : Local file system at /home/zbowen/playground/images/camera_00: Waiting for transfers to finish
2020/01/08 17:45:09 ERROR : Local file system at /home/zbowen/playground/images/camera_00: not deleting files as there were IO errors
2020/01/08 17:45:09 ERROR : Attempt 3/3 failed with 0 errors and: error reading source directory "mx-automotive-samsung-usa": can only list from root
2020/01/08 17:45:09 Failed to sync: error reading source directory "mx-automotive-samsung-usa": can only list from root
Also:
$ rclone lsd aws://
-1 2019-12-04 16:23:08 -1 mx-automotive-derived
-1 2019-10-30 10:46:18 -1 mx-automotive-samsung-usa
-1 2019-10-30 10:46:18 -1 mx-automotive-terraform-state
-1 2019-10-29 10:57:12 -1 mx-automotive-test
-1 2019-11-06 14:16:55 -1 sia-drive-data
-1 2019-11-05 15:14:12 -1 sia-remote-state
so you can see that it's failing in a bucket that I'm not trying to read from...