Rcd /operations/stat is not as expected with Azure ADLS directory

Help and Support
1

Hello Experts,

I am using rclone in rcd mode (http://myip:5572).

I have a need to identify whether the requested source is file or directory, so using below end point to get that info -
http://myip:5572/operations/stat

Now it works fine with AWS S3, GCS but facing one issue with ADLS (Azure data store).

Response is as expected when the source is a file but when source is directory its response is unexpected.
Please take a look at below 3 scenarios request & responses -
Scenario one is working as expected but not 2 & 3.

http://:5572/operations/stat

Scenario 1)
Request -

{
	"fs": "appsettingchanges_6019_ADLS_82c880d2_bb71_45d1_96be_08db1bc37687:",
	"remote": "/o9datalake/Folder1/In_SalesDomain.csv"
}

Response - 200
Response body -

{
	"item": {
		"Path": "/o9datalake/Folder1/In_SalesDomain.csv",
		"Name": "In_SalesDomain.csv",
		"Size": 11413,
		"MimeType": "text/csv; charset=utf-8",
		"ModTime": "2022-10-05T03:59:16.000000000Z",
		"IsDir": false,
		"Tier": "Hot"
	}
}

Scenario 2)

Request -

{
	"fs": "appsettingchanges_6019_ADLS_82c880d2_bb71_45d1_96be_08db1bc37687:",
	"remote": "/o9datalake/Folder1"
}

Response - 500
Response body -

{
	"error": "bucket or container name is needed in remote",
	"input": {
		"fs": "appsettingchanges_6019_ADLS_82c880d2_bb71_45d1_96be_08db1bc37687:",
		"remote": "/o9datalake/Folder1"
	},
	"path": "operations/stat",
	"status": 500
}

Scenario 3)
Request -

{
	"fs": "appsettingchanges_6019_ADLS_82c880d2_bb71_45d1_96be_08db1bc37687:",
	"remote": "o9datalake/Folder1/"
}

Response - 200
Response body -

{
	"item": null
}

Remote :

image
image1026×228 46.9 KB

Rclone version - rclone v1.57.0

Could you please suggest me on this ?

Thanks,
Mahendra

2

might update to latest stable v1.61.1 and test again.

3

@asdffdsa Thanks for the quick reply.
I tried with "rclone v1.61.1" version also but no luck, its same response.

4

Assuming the bucket is called o9datalake then you want to remove the leading / here and just use "remote": "o9datalake/Folder1"

5

Thanks for the quick response @ncw .

Yes I tried removing leading '/' ("remote": "o9datalake/Folder1/") , I got response code 200 but response body was like below (same mentioned in scenario 3)-
{
** "item": null**
}

One more thing to note is at the same time if I try remote as single file path , it works as expected:
"remote": "/o9datalake/Folder1/In_SalesDomain.csv"
This works well

6

It looks like the trailing / is causing the problem here

$ rclone rc --loopback operations/stat fs=TestAzureBlob: remote=rclone/rclone1test
{
	"item": {
		"IsDir": true,
		"MimeType": "inode/directory",
		"ModTime": "2023-03-06T10:35:17.659526872Z",
		"Name": "rclone1test",
		"Path": "rclone/rclone1test",
		"Size": -1
	}
}

vs

$ rclone rc --loopback operations/stat fs=TestAzureBlob: remote=rclone/rclone1test/
{
	"item": null
}

I'm not quite sure why that should be. Can you confirm that works for you? If it does I'll investigate further.

1 Like
7

Hi @ncw .

Yes you are right, I was about to update the same.
After removing trailing /, it worked as expected.

The reason why I thought its an issue is, leading trailing slash works fine for with '/sync/copy' end point (for data copy) but with '/operations/stat' behaviour is different.

To summaries, for '/operations/stat' below are the cases :

Working paths:
/o9datalake/Folder1/In_SalesDomain.csv
/o9datalake/Folder1/In_SalesDomain.csv
o9datalake/Folder1

Failing paths:
/o9datalake/Folder1/
/o9datalake/Folder1
o9datalake/Folder1/

Thanks

8

I think the reason this is like this is that rclone knows instantly a path ending in a trailing / is a directory so it doesn't bother querying the remote and so doesn't fetch any information. Note that every directory exists on bucket based backends as they don't really have the concept of directory.

Without the trailing / rclone does query the remote (as it could refer to a file) and you do get a proper indication as to whether the directory exists or not.

I'm not 100% happy with this behaviour though - can you open a new issue on Github about this with some info and a link to this forum post - thanks!

9

Done.

Thanks

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.