Error on MacOS with launchd and default user directories

Help and Support
1

What is the problem you are having with rclone?

When launchd launches a shell script, which in turn executes an rclone sync command, for some source directories an exit code of 1 is given and an error (e.g.: failed to open directory "": open /Users/arnaud/Desktop: operation not permitted) is logged.

The script behaves as expected when manually evoked from a local Terminal. The script behaves as expected when providing rsync with a newly created directory instead.

The directories that result in the error include:

  • /Users/arnaud/Desktop
  • /Users/arnaud/Documents
  • /Users/arnaud/Applications

Considering the limited scope of the failure, I suspect this might be a MacOS account/folder protection problem. But have been unable to resolve the problem.

What is your rclone version (output from rclone version)

rclone v1.53.3

  • os/arch: darwin/amd64
  • go version: go1.15.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)

MacOS Big Sur 11.1, 64-bit, Intel

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The shell script (sync-to-remote.sh):

#!/bin/sh

rclone sync "/Users/arnaud/Desktop" gdrive:test --log-file=/Users/arnaud/test/logfile.txt --log-level DEBUG

The plist file (com.av.sync-to-remote.plist):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>com.av.sync-to-remote</string>

  <key>ProgramArguments</key>
  <array>
    <string>/Users/Shared/av-tool-sync-to-remote/sync-to-remote.sh</string>
  </array>

  <key>RootDirectory</key>
  <string>/Users/Shared/av-tool-sync-to-remote</string>

<!-- Tried with and without UserName and GroupName -->
  <key>UserName></key> 
  <string>arnaud</string> 

  <key>GroupName</key> 
  <string>wheel</string>
  
  <key>EnvironmentVariables</key>
  <dict>
     <key>PATH</key>
     <string>/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin</string>
  </dict>

<!-- Every minute for test purposes -->
  <key>StartInterval</key>
  <integer>60</integer>

  <key>StandardOutPath</key>
  <string>/tmp/sync-to-remote.stdout</string>

  <key>StandardErrorPath</key>
  <string>/tmp/sync-to-remote.stderr</string>

  <key>Nice</key>
  <integer>19</integer>

<!-- Tried different types -->
  <key>ProcessType</key>
  <string>Background</string>
</dict>
</plist>

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = [redacted]
client_secret = [redacted]
scope = drive
token = [redacted]

A log from the command with the -vv flag

2021/01/02 10:34:42 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "sync" "Users/arnaud/Desktop/" "gdrive:test" "--log-file=/Users/arnaud/test/logfile.txt" "--log-level" "DEBUG"]
2021/01/02 10:34:42 DEBUG : Creating backend with remote "Users/arnaud/Desktop/"
2021/01/02 10:34:42 DEBUG : Using config file from "/Users/arnaud/.config/rclone/rclone.conf"
2021/01/02 10:34:42 DEBUG : fs cache: renaming cache item "Users/arnaud/Desktop/" to be canonical "/Users/arnaud/Desktop"
2021/01/02 10:34:42 DEBUG : Creating backend with remote "gdrive:test"
2021/01/02 10:34:43 ERROR : : failed to open directory "": open /Users/arnaud/Desktop: operation not permitted
2021/01/02 10:34:43 DEBUG : Google drive root 'test': Waiting for checks to finish
2021/01/02 10:34:43 DEBUG : Google drive root 'test': Waiting for transfers to finish
2021/01/02 10:34:43 ERROR : Google drive root 'test': not deleting files as there were IO errors
2021/01/02 10:34:43 ERROR : Google drive root 'test': not deleting directories as there were IO errors
2021/01/02 10:34:43 INFO  : There was nothing to transfer
2021/01/02 10:34:43 ERROR : Attempt 1/3 failed with 2 errors and: not deleting files as there were IO errors
2021/01/02 10:34:43 ERROR : : failed to open directory "": open /Users/arnaud/Desktop: operation not permitted
2021/01/02 10:34:44 DEBUG : Google drive root 'test': Waiting for checks to finish
2021/01/02 10:34:44 DEBUG : Google drive root 'test': Waiting for transfers to finish
2021/01/02 10:34:44 ERROR : Google drive root 'test': not deleting files as there were IO errors
2021/01/02 10:34:44 ERROR : Google drive root 'test': not deleting directories as there were IO errors
2021/01/02 10:34:44 INFO  : There was nothing to transfer
2021/01/02 10:34:44 ERROR : Attempt 2/3 failed with 2 errors and: not deleting files as there were IO errors
2021/01/02 10:34:44 ERROR : : failed to open directory "": open /Users/arnaud/Desktop: operation not permitted
2021/01/02 10:34:44 DEBUG : Google drive root 'test': Waiting for checks to finish
2021/01/02 10:34:44 DEBUG : Google drive root 'test': Waiting for transfers to finish
2021/01/02 10:34:44 ERROR : Google drive root 'test': not deleting files as there were IO errors
2021/01/02 10:34:44 ERROR : Google drive root 'test': not deleting directories as there were IO errors
2021/01/02 10:34:44 INFO  : There was nothing to transfer
2021/01/02 10:34:44 ERROR : Attempt 3/3 failed with 2 errors and: not deleting files as there were IO errors
2021/01/02 10:34:44 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 2 (retrying may help)
Elapsed time:         2.5s
3

Is there anything I can add to make looking into this easier?

5

You are missing a / before Users in the actual command you are executing.

6

Well spotted.

However, the fourth line in the log shows that the app translates paths into canonical form, pre-empting inconsistent paths like that one.

Now, I did update the script with your suggestion. No reason for me to hold onto bad habits.

7

hi,
i noticed that, but lacking unix command skills, i did not post about it.
and i noticed that rclone seems to deal with that, adding the leading slash.
fs cache: renaming cache item "Users/arnaud/Desktop/" to be canonical "/Users/arnaud/Desktop"

8

You can see without the slash it does not work, but works with:

felix@gemini:~$ rclone lsd home/felix -vv
2021/01/10 12:38:32 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "lsd" "home/felix" "-vv"]
2021/01/10 12:38:32 DEBUG : Creating backend with remote "home/felix"
2021/01/10 12:38:32 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/01/10 12:38:32 DEBUG : fs cache: renaming cache item "home/felix" to be canonical "/home/felix/home/felix"
2021/01/10 12:38:32 ERROR : : error listing: directory not found
2021/01/10 12:38:32 DEBUG : 2 go routines active
2021/01/10 12:38:32 Failed to lsd with 2 errors: last error was: directory not found

and

felix@gemini:~$ rclone lsd /home/felix -vv
2021/01/10 12:38:36 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "lsd" "/home/felix" "-vv"]
2021/01/10 12:38:36 DEBUG : Creating backend with remote "/home/felix"
2021/01/10 12:38:36 DEBUG : Using config file from "/opt/rclone/rclone.conf"
          -1 2020-12-14 08:03:51        -1 .aspnet
          -1 2020-12-21 07:50:44        -1 .cache
          -1 2020-12-14 11:50:23        -1 .config
          -1 2020-12-21 07:48:37        -1 .local
          -1 2020-12-14 08:53:34        -1 .npm
          -1 2020-12-18 12:15:17        -1 .ssh
          -1 2020-12-14 08:02:36        -1 .step
          -1 2020-10-10 12:07:50        -1 GetScripts
          -1 2020-12-02 15:00:42        -1 bin
          -1 2021-01-01 13:11:03        -1 logs
          -1 2020-05-05 13:11:01        -1 output
          -1 2020-06-26 20:37:51        -1 pkg
          -1 2020-05-20 10:26:16        -1 pprof
          -1 2020-12-15 14:09:46        -1 scripts
          -1 2020-11-19 16:20:16        -1 src
          -1 2021-01-02 16:10:29        -1 test
2021/01/10 12:38:36 DEBUG : 2 go routines active
9

How are you launching it?

Where is the plist file located?

10

@Animosity022

Thank you for taking an interest.

1. Path name

The log you provided seems to show that the renaming is failing here. For the canonical name I would expect '/home/felix', instead of the shown '/home/felix/home/felix'. You might have found a bug in the path renaming code.

Either way, probably nothing to do with the problem presented here. I've updated the script to use the correct canonical name - debug output no longer shows a path translation -. It still throws exit code 1 when invoked from Dgraph, while the same command works just fine when started by me from the command line.

2. Launching and plist location

The rclone command is called from a shell script, which is called by launchd. Both the script and launchd plist config can be found in the original post description.

The plist config is placed in /Library/LaunchAgents.

The script runs as expected, with the exception that rclone does not want to sync from default user folders such as Desktop and Documents. Instead rclone exits with exit code 1.

11

Instead of a sync, can you change it and run a rclone ls on that bad path and debug the results into a log file.

12

And no big, it’s relative to the path you are running it from. So yours is running from / and mine from my working directory.

So it’s using your current working directory of the command which is exactly what it should do if no slash is provided. You got lucky :wink:

13

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.