Rclone bisync Box Failed to create file system: is a directory not a file

NomadIT · September 23, 2024, 12:36pm

What is the problem you are having with rclone?

I have this command running via task scheduler on Windows Server 2019 every 5 minutes to transfer to/from a UNC path to Box (unsupported, I understand, but it's usually working). Occasionally, maybe once every couple of hours, I will get the following in the logs and the bisync will fail.

CRITICAL: Failed to create file system for "box:Path1\Folder": is a directory not a file

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.0

os/version: Microsoft Windows Server 2019 Standard 1809 (64 bit)
os/kernel: 10.0.17763.6293 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.23.1
go/linking: static
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Box

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone bisync "box:Path1\Folder" "\\Path2\Folder" --create-empty-src-dirs --recover --resilient --log-file "C:\logfile.txt" --log-level debug

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[box]
type = box
box_sub_type = enterprise
token = XXX

A log from the command that you were trying to run with the `-vv` flag

2024/09/23 07:45:03 DEBUG : rclone: Version "v1.68.0" starting with parameters ["C:\\rclone\\rclone.exe" "bisync" "box:Path1\\Folder" "\\\\Path2\\Folder" "--create-empty-src-dirs" "--recover" "--resilient" "--log-file" "C:\\Logfile.txt" "--log-level" "debug"]
2024/09/23 07:45:03 DEBUG : Creating backend with remote "box:Path1\\Folder"
2024/09/23 07:45:03 DEBUG : Using config file from "C:\\Users\\Username\\AppData\\Roaming\\rclone\\rclone.conf"
2024/09/23 07:45:03 DEBUG : box: Loaded invalid token from config file - ignoring
2024/09/23 07:45:04 DEBUG : box: got fatal oauth error: oauth2: "invalid_grant" "Refresh token has expired"
2024/09/23 07:45:04 DEBUG : box root 'Path1/Folder': Token expired but no uploads in progress - doing nothing
2024/09/23 07:45:04 DEBUG : Config file has changed externally - reloading
2024/09/23 07:45:04 DEBUG : box: Loaded fresh token from config file
2024/09/23 07:45:04 DEBUG : box: Loaded new refresh token from config file
2024/09/23 07:45:04 CRITICAL: Failed to create file system for "box:Path1\\Folder": is a directory not a file

asdffdsa · September 23, 2024, 12:53pm

welcome to the forum,

what is unsupported?

i would try this syntax
"box:Path1/Folder"

NomadIT · September 23, 2024, 12:55pm

Oh, at least according to the Bisync documentation, Box doesn't seem to be a "supported" platform for the command. I don't know how recently that list had been updated, though, and it does seems to be working most of the time. Just wanted to acknowledge the documentation, though

asdffdsa · September 23, 2024, 1:08pm

where is it documented that box is not supported for bisync command?

that log you posted shows that the token expired.
just a guess, that might cause the error.

fwiw, never a good idea to create files in the root of the c: drive

NomadIT · September 23, 2024, 1:13pm

I'm not creating files in the root of the C: drive. I just sanitized where the log file goes.

It shows the token expired, but then three lines down it says it got a new token from the config file. That seems like a normal part of the process?

asdffdsa · September 23, 2024, 3:50pm

Automatic backup when "failed to copy: is a directory not a file"

NomadIT · September 23, 2024, 4:10pm

I'm sorry, but I don't see how this applies in my case. The situation in the linked thread concerns rclone sync, not bisync. It also pertains to a situation where a file ".git" becomes a folder between synchronizations, which isn't happening here. In my case, box:Path1/Folder and \Path2\Folder are just totally normal folders. Not .git folders, not hidden folders, etc.

asdffdsa · September 23, 2024, 4:11pm

ok. just wanted share that link.

now we know it does not apply to your case.

asdffdsa · September 23, 2024, 7:49pm

afiak, the error is not a bisync error, not a sync error.
it is a generic file system error that can occur with any number of commands and backends.

hopefully @nielash would know?

NomadIT · September 23, 2024, 8:13pm

I tried this this morning and didn't get any errors for a while. Then, just a little bit ago, I got another one of these errors, so unfortunately, this didn't seem to solve the issue.

This might be true, but it's also not very actionable. Is there anything I can do to get more detailed logs about what the issue is? Is -vv more verbose than --log-level debug? It seems to be an issue with rclone reading the cloud storage side, which is why I bring up the unsupported nature of Box as a cloud endpoint. Can I provide more information to assist with troubleshooting if there's something Box is doing differently from other cloud platforms when enumerating files and folder lists?

Alternatively, despite putting in --retries 3, it won't retry again. I'm not sure if there's something I'm misunderstanding about how that flag works, but if there was a way to get it to try and connect again, that would kind of brute force through the problem since it doesn't error out on subsequent runs.

asdffdsa · September 23, 2024, 8:23pm

no, there are equivalent and there is not a -vvv
might try a --dump flag

well, that exact error is mentioned in the source code for box...

github.com

rclone/rclone/blob/f8d782c02de5e42c354f281b4f32c4128fa5db1e/backend/box/box.go#L803


      
          		return shouldRetry(ctx, resp, err)
          	})
          	if err != nil {
          		if apiErr, ok := err.(*api.Error); ok && apiErr.Code == "item_name_in_use" {
          			var conflict api.PreUploadCheckConflict
          			err = json.Unmarshal(apiErr.ContextInfo, &conflict)
          			if err != nil {
          				return nil, fmt.Errorf("pre-upload check: JSON decode failed: %w", err)
          			}
          			if conflict.Conflicts.Type != api.ItemTypeFile {
          				return nil, fs.ErrorIsDir
          			}
          			return &conflict.Conflicts, nil
          		}
          		return nil, fmt.Errorf("pre-upload check: %w", err)
          	}
          	return nil, nil
          }
          
          // Put the object
          //

NomadIT · September 23, 2024, 8:24pm

This is where I'm out of my depth. Can you explain what this bit of code means? I'm guessing that Box is telling Rclone the wrong information about the directory?

asdffdsa · September 23, 2024, 8:36pm

that is the default value.
i guess that is bisync is playing it safe, the error is critical, so no retries are attempted.

cannot tell you for sure and i could wrong about it being a box issue.

"preUploadCheck checks to see if a file can be uploaded"
i would check if the name appears in the source and/or dest, is it a file, directory, both or what?

NomadIT · September 23, 2024, 8:42pm

Okay. To be very very precise here, I'm running rclone bisync "box:ABC/XYZ" "\\10.1.1.1\XYZ", where ABC is the (pretend) name of the root folder on the Box side, 10.1.1.1 is the IP address of the "local" machine, and XYZ is the name of both the child directory on the Box side and the share on the "local" side. Neither of these have any files or directories in them that are named XYZ. Is there an issue with having both directories on both sides named the same?

asdffdsa · September 23, 2024, 8:49pm

sorry, not precise at all.
might be clear to you, but imho, super confusing.

before you posted box:Path1\Folder
now, you post box:ABC/XYZ

the slashes are not consistent.
the redacted names are not consistent.

please, can you just post the exact command and a full debug log, with zero redactions, or just as little as possible.
that will also help other forum members

NomadIT · September 23, 2024, 9:11pm

I changed them around after you suggested doing so.

rclone bisync "box:NAS/Uploads" "\\10.1.1.10\Uploads" --create-empty-src-dirs --recover --resilient --log-file "C:\logfile.txt" --log-level debug

What I posted above is the full debug log of the erroring run.

nielash · September 24, 2024, 12:43am

Bisync maintainer here. Box is still failing some bisync integration tests (that most other backends pass), which is why it is not officially supported yet. However, the error on those tests (Item with the same name already exists) does not look related to the error here, so I would be surprised if that's the problem. It should otherwise be working.

That is for errors which occur during the command. Here, we are not even able to get that far because the box fs fails to be created at the outset.

The debug log posted above still has the backslashes in it... it would be really helpful to see a log of what happened after you tried fixing the slashes, to rule out that the error is somehow related to the slashes (for example, maybe an issue related to shell quoting/escaping).

There shouldn't be.

NomadIT · September 24, 2024, 1:58am

Here you go:

2024/09/23 17:00:11 DEBUG : rclone: Version "v1.68.0" starting with parameters ["C:\\rclone\\rclone.exe" "bisync" "box:NAS/Upload" "\\\\10.1.1.10\\upload" "--create-empty-src-dirs" "--recover" "--resilient" "--log-file" "C:\\Logfile.txt" "--log-level" "debug" "--retries" "3"]
2024/09/23 17:00:11 DEBUG : Creating backend with remote "box:NAS/Upload"
2024/09/23 17:00:11 DEBUG : Using config file from "C:\\Users\\username\\AppData\\Roaming\\rclone\\rclone.conf"
2024/09/23 17:00:11 DEBUG : box root 'NAS/Upload': Token expired but no uploads in progress - doing nothing
2024/09/23 17:00:11 DEBUG : box: Loaded invalid token from config file - ignoring
2024/09/23 17:00:13 DEBUG : box: got fatal oauth error: oauth2: "invalid_grant" "Refresh token has expired"
2024/09/23 17:00:13 DEBUG : Config file has changed externally - reloading
2024/09/23 17:00:13 DEBUG : box: Loaded fresh token from config file
2024/09/23 17:00:13 DEBUG : box: Loaded new refresh token from config file
2024/09/23 17:00:14 CRITICAL: Failed to create file system for "box:NAS/Upload": is a directory not a file

I am hoping to use bisync in an official business context eventually. We currently have enterprise Box Support hours that I would be willing to put towards this sort of thing if it'll help to get Box moved to being an officially supported platform. Let me know if there is anything I can help to facilitate this.

nielash · September 24, 2024, 3:35am

Thank you. That is helpful. Can you try a couple more things to narrow this down:

Try with --fs-cache-expire-duration 0 (to rule out cache issues)
If that doesn't change anything, can you try changing Path2 to a normal local path (ideally not one starting with \\ and an IP address) and see if that works as expected?
(One thing I'm wondering is if the \\ is somehow making it get parsed as a file instead of a directory, and that makes it expect Path1 to be a file too.)

If none of those make any difference, I'll try to pull out my Windows laptop tomorrow and see if I can reproduce it.

Thanks! I suspect the problem is on rclone's end rather than box's end, and I haven't looked too closely at it yet, but just from the looks of that error, I'm guessing it may be that we are trying to copy or move a file over itself and not using the proper API request options for that. You might consider asking them if they happen to know if there's a way we're supposed to be specifying in the call that we want the file to be overwritten instead of erroring.

I think this may be the same issue:

github.com/rclone/rclone

Server side copy - Box - item_name_in_use 409 error with existing files

opened 09:22PM - 04 Sep 19 UTC

a6

bug Remote: Box

Linked from [forum posting](https://forum.rclone.org/t/server-side-copy-box-item…-name-in-use-409-error-with-existing-files/11365). If I attempt a server side copy from one box path to another, any file existing in both paths that has been updated in the src path will not be updated in the dst path. Instead, an error is thrown: DEBUG : : Sizes differ (src 32 vs dst 25) ERROR : : Failed to copy: Error "item_name_in_use" (409): Item with the same name already exists The command is something like: rclone copy boxRemote:path1 boxRemote:path2 where path2 is intended to have a copy of all files that exist in path1. The first time this copy is called, everything works because there are no existing files of path1 in path2. However, if later, a file is updated in path1 and then the above copy command is attempted, this `item_name_in_use` error is thrown. This is with rclone 1.48 on Ubuntu x64 16.04. I suspect this is not desired behavior, because the same types of copy commands work just fine if src is local instead of a box remote.

And here is a log of the failing bisync tests.

Otherwise, if you are interested in taking out a support contract, I'm sure @ncw would appreciate that and it would help the bug fix be prioritized on the rclone end

NomadIT · September 24, 2024, 1:50pm

I'll give this a try and let you know.

Unfortunately I can't. The server that rclone is running from is acting as a go-between for Box and an IBM Power8 system that is providing files through SMB share. And because I have this running from a Windows server's task scheduler headless, I can't map the drive. I have to specify UNC path. I guess worst comes to worst, I can fire up a Linux system to run all of this on, but I am infinitely more comfortable in a Windows environment and my co-workers don't know Linux at all, so I'd be the only one able to support this system.