When uploading files to Google Drive from parallel threads we face an issue where multiple directories with the same name are created. For example for an upload which just completed
# rclone ls gdrive: | grep GGG000S 1094 GGG000S-20180823_122050/GGG000S.img.meta.enc 63 GGG000S-20180823_122050/GGG000S.img.meta.md5 1094 GGG000S-20180823_122050/GGG000S.img.meta.enc 63 GGG000S-20180823_122050/GGG000S.img.meta.md5 # rclone ls gdrive:GGG000S-20180823_122050/ 227352447 GGG000S.img.000001 1089 GGG000S.img.enc 82 GGG000S.img.map 62 GGG000S.img.md5 20480 GGG000S.img.meta.000001 12 firstupload 4096 job16.log
So there are two GGG000S-20180823_122050 created with some of the files in the first directory and some in the other directory but with the same name.
One thing which seems to work is
- rclone mkdir
- sleep 60
- rclone copy gdrive:
- Continue with regular uploads
This behavior seems specific to Google Drive and for e.g Google Clould does have this issue. Also it does not seem to be because of the parallel uploads since we have seen this behavior even if we first upload a file before uploading the others in parallel.
Is this something which is a known issue with Google Drive ?
Our rclone copy commands are issued as
rclone copy /tmp/GGG000S-20180823_122050/GGG000S.img.000001 gdrive:/GGG000S-20180823_122050/