(Note: I've replaced the : in some URLs with COLON to pass the anti-spam test for new users)
What is the problem you are having with rclone?
I'm using a series of Docker containers, each of which copy in files from DigitalOcean spaces, process them, then copy them back (with some new files created).
The outbound copying is working fine - the first container creates the files and uploads them.
But subsequent containers find nothing to copy. They don't give any errors - just report nothing to do.
Most bizarly, when using --dump bodies
with rclone ls
(or with copy
) it appears the files are being listed in the body, but still not recognised.
Much of the debugging I've already done came from suggestions in this thread:
Run the command 'rclone version' and share the full output of the command.
I've tried this with two versions of rclone, by default I'm using the one in the ubuntu repository which is a bit older, but I've also tried it with the latest 1.62.2.
Annoyingly the examples below were with 1.45, but I can re-create them with 1.62 if needed (the problem persists):
root@b673a8d6c98a:/src# rclone --version
rclone v1.45
- os/arch: linux/amd64
- go version: go1.11.6
root@b673a8d6c98a:/src# rclone --version
rclone v1.62.2
- os/version: debian 10.11 (64 bit)
- os/kernel: 5.4.0-148-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
DigitalOcean
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
Among others:
rclone ls imas:
The rclone config contents with secrets removed.
root@b673a8d6c98a:/src# rclone config show
[imas]
type = s3
provider = DigitalOcean
env_auth = false
access_key_id = SECRET
secret_access_key = SECRET
endpoint = httpsCOLON//imas.ams3.digitaloceanspaces.com
acl = private
I've tried all manner of combinations of the endpoint that I can think of, with and without the subdomain.
httpsCOLON//ams3.digitaloceanspaces.com/imas
httpsCOLON//imas.ams3.digitaloceanspaces.com/backstop
A log from the command with the -vv
flag
This is the initial pushing of the files:
[2023-05-12 +15:36:56 +UTC] Sending files back to storage: imas:backstop/2023-05-12-15-36-49_www.mha.systems/backstop
2023/05/12 15:36:56 DEBUG : rclone: Version "v1.45" starting with parameters ["rclone" "copy" "/src" "imas:backstop/2023-05-12-15-36-49_www.mha.systems/backstop" "-vv"]
2023/05/12 15:36:56 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/05/12 15:36:56 DEBUG : pacer: Reducing sleep to 0s
2023/05/12 15:36:56 INFO : S3 bucket backstop path 2023-05-12-15-36-49_www.mha.systems/backstop/: Waiting for checks to finish
2023/05/12 15:36:56 INFO : S3 bucket backstop path 2023-05-12-15-36-49_www.mha.systems/backstop/: Waiting for transfers to finish
2023/05/12 15:36:56 INFO : backstop.json: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/cookies.json: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/imageStub.jpg: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/puppet/clickAndHoverHelper.js: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/puppet/loadCookies.js: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/puppet/ignoreCSP.js: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/puppet/interceptImages.js: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/puppet/onBefore.js: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/puppet/overrideCSS.js: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/puppet/onReady.js: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/playwright/clickAndHoverHelper.js: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/playwright/loadCookies.js: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/playwright/interceptImages.js: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/playwright/onBefore.js: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/playwright/onReady.js: Copied (new)
2023/05/12 15:36:56 INFO : backstop_data/engine_scripts/playwright/overrideCSS.js: Copied (new)
2023/05/12 15:36:56 INFO :
Transferred: 13.818k / 13.818 kBytes, 100%, 42.105 kBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 16 / 16, 100%
Elapsed time: 300ms
2023/05/12 15:36:56 DEBUG : 16 go routines active
2023/05/12 15:36:56 DEBUG : rclone: Version "v1.45" finishing with parameters ["rclone" "copy" "/src" "imas:backstop/2023-05-12-15-36-49_www.mha.systems/backstop" "-vv"]
But when the next container tries to pull them down again:
[2023-05-12 +15:51:17 +UTC] Pulling files from storage: imas:backstop/2023-05-12-15-36-49_www.mha.systems/backstop
2023/05/12 15:51:17 DEBUG : rclone: Version "v1.45" starting with parameters ["rclone" "copy" "imas:backstop/2023-05-12-15-36-49_www.mha.systems/backstop" "/src" "-vv"]
2023/05/12 15:51:17 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/05/12 15:51:17 DEBUG : pacer: Reducing sleep to 0s
2023/05/12 15:51:17 INFO : Local file system at /src: Waiting for checks to finish
2023/05/12 15:51:17 INFO : Local file system at /src: Waiting for transfers to finish
2023/05/12 15:51:17 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 100ms
2023/05/12 15:51:17 DEBUG : 5 go routines active
2023/05/12 15:51:17 DEBUG : rclone: Version "v1.45" finishing with parameters ["rclone" "copy" "imas:backstop/2023-05-12-15-36-49_www.mha.systems/backstop" "/src" "-vv"]
If I then hop into the container to run commands things get a bit more confusing:
ls displays nothing:
root@b673a8d6c98a:/src# rclone ls -vv imas:
2023/05/12 15:55:24 DEBUG : rclone: Version "v1.45" starting with parameters ["rclone" "ls" "-vv" "imas:"]
2023/05/12 15:55:24 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/05/12 15:55:24 DEBUG : pacer: Reducing sleep to 0s
2023/05/12 15:55:24 DEBUG : 4 go routines active
2023/05/12 15:55:24 DEBUG : rclone: Version "v1.45" finishing with parameters ["rclone" "ls" "-vv" "imas:"]
root@b673a8d6c98a:/src#
But if I add --dump bodies
it looks like rclone is actually getting the results:
root@b673a8d6c98a:/src# rclone ls -vv imas:
2023/05/12 15:55:24 DEBUG : rclone: Version "v1.45" starting with parameters ["rclone" "ls" "-vv" "imas:"]
2023/05/12 15:55:24 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/05/12 15:55:24 DEBUG : pacer: Reducing sleep to 0s
2023/05/12 15:55:24 DEBUG : 4 go routines active
2023/05/12 15:55:24 DEBUG : rclone: Version "v1.45" finishing with parameters ["rclone" "ls" "-vv" "imas:"]
root@b673a8d6c98a:/src# rclone ls -vv --dump bodies imas:
2023/05/12 15:55:43 DEBUG : rclone: Version "v1.45" starting with parameters ["rclone" "ls" "-vv" "--dump" "bodies" "imas:"]
2023/05/12 15:55:43 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/05/12 15:55:43 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/05/12 15:55:43 DEBUG : HTTP REQUEST (req 0xc0002f7700)
2023/05/12 15:55:43 DEBUG : GET / HTTP/1.1
Host: imas.ams3.digitaloceanspaces.com
User-Agent: rclone/v1.45
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230512T155543Z
Accept-Encoding: gzip
2023/05/12 15:55:43 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/05/12 15:55:44 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/05/12 15:55:44 DEBUG : HTTP RESPONSE (req 0xc0002f7700)
2023/05/12 15:55:44 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 12 May 2023 15:55:44 GMT
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
X-Amz-Request-Id: tx000000000000023223ae7-00645e6180-3ea93469-ams3c
22f5
<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>imas</Name><Prefix></Prefix><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>backstop</Key><LastModified>2023-03-17T18:32:10.441Z</LastModified><ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag><Size>0</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop-init-action.log</Key><LastModified>2023-05-12T15:36:57.364Z</LastModified><ETag>"fa957247dedfe801a54817059d95a6ff"</ETag><Size>5091</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop-reference-action.log</Key><LastModified>2023-05-12T15:51:19.373Z</LastModified><ETag>"8aabd40575745188679769452c3fe71f"</ETag><Size>4642</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/</Key><LastModified>2023-05-12T15:36:50.252Z</LastModified><ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag><Size>0</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop.json</Key><LastModified>2023-05-12T15:37:07.466Z</LastModified><ETag>"03c34bbcba5c4c19e03c22e73b848655"</ETag><Size>7834</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/cookies.json</Key><LastModified>2023-05-12T15:36:56.535Z</LastModified><ETag>"2abd7ff522da804b77889b9ca521389c"</ETag><Size>273</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/imageStub.jpg</Key><LastModified>2023-05-12T15:36:56.552Z</LastModified><ETag>"17fccaafaf2d54068adbc6cc8b51cbe4"</ETag><Size>2900</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/playwright/clickAndHoverHelper.js</Key><LastModified>2023-05-12T15:36:56.631Z</LastModified><ETag>"6820b012d1de4bfe04188b35368b8bbb"</ETag><Size>1501</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/playwright/interceptImages.js</Key><LastModified>2023-05-12T15:36:56.647Z</LastModified><ETag>"6b71e60a66355f2c62d2047e265456a4"</ETag><Size>766</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/playwright/loadCookies.js</Key><LastModified>2023-05-12T15:36:56.665Z</LastModified><ETag>"3885016017c9b325f36a8a3f3757f5aa"</ETag><Size>429</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/playwright/onBefore.js</Key><LastModified>2023-05-12T15:36:56.675Z</LastModified><ETag>"221f9137fb097bf0f4bf3347c50583d1"</ETag><Size>147</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/playwright/onReady.js</Key><LastModified>2023-05-12T15:36:56.680Z</LastModified><ETag>"82987ca9b6e6972c5c719bd827524b69"</ETag><Size>230</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/playwright/overrideCSS.js</Key><LastModified>2023-05-12T15:36:56.697Z</LastModified><ETag>"5c81b1d93f139061dff61a8c401d3964"</ETag><Size>582</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/puppet/clickAndHoverHelper.js</Key><LastModified>2023-05-12T15:36:56.560Z</LastModified><ETag>"6b669c56ade946f4a86aca672a42e116"</ETag><Size>1380</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/puppet/ignoreCSP.js</Key><LastModified>2023-05-12T15:36:56.571Z</LastModified><ETag>"53ada96e0b40abb75d2d18d0e6a959a7"</ETag><Size>1933</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/puppet/interceptImages.js</Key><LastModified>2023-05-12T15:36:56.574Z</LastModified><ETag>"6d36f5f87cb4024eda7812e9018d35d3"</ETag><Size>959</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/puppet/loadCookies.js</Key><LastModified>2023-05-12T15:36:56.584Z</LastModified><ETag>"a8b1939b2416e7aafdf29fd8c6238bf8"</ETag><Size>846</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/puppet/onBefore.js</Key><LastModified>2023-05-12T15:36:56.604Z</LastModified><ETag>"735e924eb6834516da37107f073955e0"</ETag><Size>102</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/puppet/onReady.js</Key><LastModified>2023-05-12T15:36:56.621Z</LastModified><ETag>"d6b86cac53ab73f2c29c70036475265c"</ETag><Size>195</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/backstop_data/engine_scripts/puppet/overrideCSS.js</Key><LastModified>2023-05-12T15:36:56.630Z</LastModified><ETag>"637080d0d25653c66489aaa3c25a02e9"</ETag><Size>573</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>backstop/2023-05-12-15-36-49_www.mha.systems/backstop/created.txt</Key><LastModified>2023-05-12T15:36:50.297Z</LastModified><ETag>"19ca14e7ea6328a42e0eb13d585e4c22"</ETag><Size>2</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>jobs/wp-nwlondonremovals_98d7b11e-28ef-48e1-9d64-a4cdfe550811/update-word-press-action.log</Key><LastModified>2023-04-03T11:33:04.917Z</LastModified><ETag>"0cad542131115d9977cc7d5f9f119a4d"</ETag><Size>120</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>test.txt</Key><LastModified>2023-05-12T15:07:17.959Z</LastModified><ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag><Size>0</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Contents><Key>test.txt/test.txt</Key><LastModified>2023-05-12T15:07:17.984Z</LastModified><ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag><Size>0</Size><StorageClass>STANDARD</StorageClass><Owner><ID>1812396</ID><DisplayName>1812396</DisplayName></Owner><Type>Normal</Type></Contents><Marker></Marker></ListBucketResult>
0
2023/05/12 15:55:44 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/05/12 15:55:44 DEBUG : pacer: Reducing sleep to 0s
2023/05/12 15:55:44 DEBUG : 4 go routines active
2023/05/12 15:55:44 DEBUG : rclone: Version "v1.45" finishing with parameters ["rclone" "ls" "-vv" "--dump" "bodies" "imas:"]