Sync GCS bucket : "ERROR : : Entry doesn't belong in directory "" (same as directory) - ignoring"

Help and Support
1

What is the problem you are having with rclone?

Hello, I cannot sync a gcs bucket with a s3 bucket when I run rclone sync command, it returns ERROR : : Entry doesn't belong in directory "" (same as directory) - ignoring . The files beginning with a slash / are not synced. I tried solutions proposed here and here but it wasn't helpful.
Here is a sample of the content of the source bucket if it can help :

❯ rclone ls gcs:bucket_gcs
 33278413 /dhp/dhp_2020-09-26
 33342758 /gdv/gdv_2020-10-13
  1282303 /gdv/gdv_2021-02-01
  1282303 /gdv/gdv_2021-02-02
     1738 uploads/1601317336-test.csv
     1738 uploads/1601321871-test.csv
     1738 uploads/1601323175-test.csv
     1738 uploads/1601323353-test.csv

Thank you in advance for your answer.

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.10.16.3-microsoft-standard-WSL2 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.9
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

I'm using Google Cloud Storage as source and S3 as destination.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

 rclone sync -P gcs:bucket_gcs s3:bucket_s3/projects/project_name/bucket_gcs

The rclone config contents with secrets removed.

[gcs]
type = google cloud storage
bucket_acl = private

[s3]
type = s3
provider = AWS
access_key_id = XXXXXXX
secret_access_key = XXXXXXX
region = us-east-2

A log from the command with the -vv flag

❯ rclone sync -P gcs:bucket_gcs s3:bucket_s3/projects/project_name/bucket_gcs -vv
2022/05/10 10:36:20 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "sync" "-P" "gcs:bucket_gcs" "s3:bucket_s3/projects/project_name/bucket_gcs" "-vv"]
2022/05/10 10:36:20 DEBUG : Creating backend with remote "gcs:bucket_gcs"
2022/05/10 10:36:20 DEBUG : Using config file from "/home/nicolefort/.config/rclone/rclone.conf"
2022/05/10 10:36:20 DEBUG : Creating backend with remote "s3:bucket_s3/projects/project_name/bucket_gcs"
2022-05-10 10:36:21 ERROR : : Entry doesn't belong in directory "" (same as directory) - ignoring
2022-05-10 10:36:21 DEBUG : S3 bucket bucket_s3 path projects/project_name/bucket_gcs: Waiting for checks to finish
2022-05-10 10:36:22 DEBUG : uploads/1601062437-PROD_events.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601062437-PROD_events.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601317208-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(1) (1).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601317208-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(1) (1).csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601317336-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601317336-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601323353-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601323353-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601321871-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601321871-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601323175-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601323175-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601323863-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601323863-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601323591-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601323591-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601324483-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601324483-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601325571-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601325571-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601325179-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601325179-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601325728-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601325728-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601325998-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601325998-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601326209-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601326209-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601326523-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601326523-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601327140-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601327140-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601327337-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601327337-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601327899-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601327899-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601328271-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601328271-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601328779-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601328779-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601329986-test.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601329986-test.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601330148-test_long.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601330148-test_long.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601368690-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(2).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601368690-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(2).csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601368310-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(2).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601368310-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(2).csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601368956-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(2).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601368956-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(2).csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601369221-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(2).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601369221-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(2).csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601369240-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(2).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601369240-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(2).csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601369657-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(2).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601369657-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(2).csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601369852-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(2).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601369852-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(2).csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601371428-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601371428-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601372442-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(3).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601372442-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(3).csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601373205-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(3).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601373205-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(3).csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601373610-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(3).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601373610-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(3).csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601375745-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(3).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601375745-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(3).csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601376658-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(3).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601376658-P04T_25_ULAND92_100_24092020_093203_TPR_7015517(3).csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1601377262-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1601377262-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Unchanged skipping
2022-05-10 10:36:22 DEBUG : uploads/1602009788-P04T_25_ULAND92_100_29092020_070630_TPR_7022008.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:22 DEBUG : uploads/1602009788-P04T_25_ULAND92_100_29092020_070630_TPR_7022008.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1602011724-P04T_25_ULAND92_100_06102020_125557_TPR_7035027.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1602011724-P04T_25_ULAND92_100_06102020_125557_TPR_7035027.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1602010023-P04T_25_ULAND92_100_29092020_070630_TPR_7022008.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1602010023-P04T_25_ULAND92_100_29092020_070630_TPR_7022008.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1602015467-P04T_25_ULAND92_100_06102020_125557_TPR_7035027.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1602015467-P04T_25_ULAND92_100_06102020_125557_TPR_7035027.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1602243522-P04T_25_ULAND92_100_08102020_083211_TPR_7037945.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1602243522-P04T_25_ULAND92_100_08102020_083211_TPR_7037945.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1602259949-TOTO: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1602259949-TOTO: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1602521805-P04T_25_ULAND92_100_08102020_083211_TPR_7037945.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1602521805-P04T_25_ULAND92_100_08102020_083211_TPR_7037945.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1602522766-P04T_25_ULAND92_100_08102020_083211_TPR_7037945.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1602522766-P04T_25_ULAND92_100_08102020_083211_TPR_7037945.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1602595631-P04T_25_ULAND92_100_08102020_083211_TPR_7037945.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1602595631-P04T_25_ULAND92_100_08102020_083211_TPR_7037945.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1612213555-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1612213555-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1612214369-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1612214369-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1612214657-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1612214657-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1612215539-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1612215539-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1612279527-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1612279527-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1612215143-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1612215143-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1619390436-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1619390436-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1619390875-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1619390875-P04T_25_ULAND92_100_24092020_093203_TPR_7015517.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1619390900-P04T_25_ULAND92_100_29092020_070630_TPR_7022008.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1619390900-P04T_25_ULAND92_100_29092020_070630_TPR_7022008.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1620048235-hubspot-crm-exports-heka-advanced-ai-mtl-2021-04-29-1.xlsx: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1620048235-hubspot-crm-exports-heka-advanced-ai-mtl-2021-04-29-1.xlsx: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1620051032-55_None_complete.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1620051032-55_None_complete.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1620051047-55_None_complete.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1620051047-55_None_complete.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1620051067-P04T_25_ULAND92_100_29092020_070630_TPR_7022008 (1).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1620051067-P04T_25_ULAND92_100_29092020_070630_TPR_7022008 (1).csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1620051162-P04T_25_ULAND92_100_29092020_070630_TPR_7022008 (1).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1620051162-P04T_25_ULAND92_100_29092020_070630_TPR_7022008 (1).csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1621168568-P04T_25_UROUX92_100_14052021_193550_TPR_7221438.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1621168568-P04T_25_UROUX92_100_14052021_193550_TPR_7221438.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1621171250-P04T_25_UROUX92_100_14052021_193550_TPR_7221438.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1621171250-P04T_25_UROUX92_100_14052021_193550_TPR_7221438.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1622694606-P04T_25_ULAND92_100_23092020_103655_TPR_7013634.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1622694606-P04T_25_ULAND92_100_23092020_103655_TPR_7013634.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1622695354-P04T_25_ULAND92_100_23092020_103655_TPR_7013634.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1622695354-P04T_25_ULAND92_100_23092020_103655_TPR_7013634.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1622695406-P04T_25_ULAND92_100_23092020_103655_TPR_7013634.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1622695406-P04T_25_ULAND92_100_23092020_103655_TPR_7013634.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1622695797-P04T_25_ULAND92_100_23092020_103655_TPR_7013634.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1622695797-P04T_25_ULAND92_100_23092020_103655_TPR_7013634.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1622696035-P04T_25_ULAND92_100_23092020_103655_TPR_7013634.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1622696035-P04T_25_ULAND92_100_23092020_103655_TPR_7013634.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1622696953-P04T_25_ULAND92_100_23092020_103655_TPR_7013634.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1622696953-P04T_25_ULAND92_100_23092020_103655_TPR_7013634.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1622697110-P04T_25_ULAND92_100_23092020_103655_TPR_7013634.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1622697110-P04T_25_ULAND92_100_23092020_103655_TPR_7013634.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1623966613-arbres.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1623966613-arbres.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1623967595-arbres.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1623967595-arbres.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1623967809-arbres.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1623967809-arbres.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1626256769-arbres-publics (1).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1626256769-arbres-publics (1).csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1626257098-arbres-publics (1).csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1626257098-arbres-publics (1).csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1626257566-arbres.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1626257566-arbres.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1623967180-arbres.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1623967180-arbres.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1626257859-arbres.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1626257859-arbres.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : uploads/1626258170-arbres.csv: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022-05-10 10:36:23 DEBUG : uploads/1626258170-arbres.csv: Unchanged skipping
2022-05-10 10:36:23 DEBUG : S3 bucket bucket_s3 path projects/project_name/bucket_gcs: Waiting for transfers to finish
2022-05-10 10:36:23 DEBUG : Waiting for deletions to finish
2022-05-10 10:36:23 INFO  : There was nothing to transfer
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                77 / 77, 100%
Elapsed time:         2.8s
2022/05/10 10:36:23 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                77 / 77, 100%
Elapsed time:         2.8s

2022/05/10 10:36:23 DEBUG : 24 go routines active```
2

Bug in the private bucket maybe with the "/" as well?

rclone lsd gcs:bucket_gcs -vv --dump bodies

and share the output.

3

Hello, here is the output :

❯ rclone lsd gcs:bucket_gcs -vv --dump bodies
2022/05/10 13:23:20 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "lsd" "gcs:bucket_gcs" "-vv" "--dump" "bodies"]
2022/05/10 13:23:20 DEBUG : Creating backend with remote "gcs:bucket_gcs"
2022/05/10 13:23:20 DEBUG : Using config file from "/home/nicolefort/.config/rclone/rclone.conf"
2022/05/10 13:23:20 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/05/10 13:23:21 ERROR : : Entry doesn't belong in directory "" (same as directory) - ignoring
           0 2022-05-10 13:23:21        -1 uploads
2022/05/10 13:23:21 DEBUG : 4 go routines active
4

I think that's similar to the other post so would @ncw to chime in on what he wants as he fixed the other item.

5

That is almost certainly a directory marker of some kind. That is a 0 sized object ending in a /.

Rclone does its best to ignore them but it hasn't here.

Despite the ERROR message it looks like the sync completed OK. In fact ignoring a directory marker is usually the right thing to do.

This should have printed an HTTP transaction?

6

Thanks for your answer.

The sync completed OK but the /dhp/* and /gdv/* files under this directory marker are not in the destination s3 bucket :

❯ rclone ls s3:bucket_s3/projects/project_name/bucket_gcs
     1738 uploads/1601317336-test.csv
     1738 uploads/1601321871-test.csv
     1738 uploads/1601323175-test.csv
     1738 uploads/1601323353-test.csv

Indeed but there is no HTTP transaction in the output :confused: . Apparently similar to this.

7

Oh, I see the problem now

It is those leading / that is causing the problem.

I don't think there is a great workaround for those at the moment - is it possible to remove the leading /?

8

Unfortunately on my side it would be difficult to remove the leading / without a significant revamp of the code using these bucket files. But if the / character can be replaced by another character in the s3 bucket when syncing, that would do the job for me.

9

Ultimately that is the best way to fix this problem. Rclone has an encoding scheme where it converts characters the upstream can't handle into unicode equivalents.

So what rclone needs to do is convert the / into ／ so then the directory name would be ／gdv

10

OK, but I thought the conversion of / into ／ was already handled by rclone according to this. Sorry the encoding system is not very clear for me.

11

It is handled for some cases, yes, but not this one.

What are you planning to do with the data? Are you planning to sync it back to GCS and would you want the initial / preserved in that case?

12

Actually I'm doing this sync to have a backup of all my buckets in s3 in case of a major GCP failure (which is rather unlikely to happen, hopefully). In this case, eventually I'd like to sync the buckets back into GCS.

  • The ideal case being the file names are exactly the same than before such that no adaptation is needed.
  • A less ideal case would be to have corrupted file names (because of / conversion for example) requiring subsequent code adaptation, but at least data is not lost.
  • The worst scenario would be that (some) files are absent in the s3 bucket, which would prevent a complete backup of my system.
13

With the encoding scheme I have in mind, what would happen is that the file names would be converted into an rclone standard format (which could be saved to disk) and back again to object storage format so a copy between two object storage systems would mean no change in names.

This might be something a bit easier to implement in the short term - I could do a flag to drop leading / on keys for example.

This is essentially the same issue as this one I think

14

That would be great, every problem of this kind would be prevented

That would be very useful for me !

15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.