PHP with Exec - Strange Behaviour When Requesting Container Information

Help and Support
1

What is the problem you are having with rclone?

I am unable to retrieve container information using 'rclone size BackupUtility' however 'rclone config dump' works OK as does 'rclone version'

What is your rclone version (output from rclone version)

rclone v1.53.1

  • os/arch: linux/amd64
  • go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux CentOS 8 64bit running PHP 7.3

Which cloud storage system are you using? (eg Google Drive)

S3 / DigitalOcean

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/local/directadmin/plugins/backup-utility/scripts/rclone/rclone size BackupUtility

The rclone config contents with secrets removed.

[BackupUtility]
type = s3
env_auth = false
endpoint = objects-us-east-1.dream.io
upload_concurrency = 4
force_path_style = true
encoding = Slash,InvalidUtf8,Dot
memory_pool_use_mmap = true
provider = DigitalOcean
region = nl-ams
chunk_size = 5M
no_check_bucket = true
secret_access_key = XXXxXXxXXXXXxXXXXxxXXXXXXxxxxXXX
upload_cutoff = 200M
max_upload_parts = 10000
use_accelerate_endpoint = true
list_chunk = 1000
leave_parts_on_error = true
access_key_id = XXXXXXXXXXXXXX
location_constraint = us-standard
acl = default
bucket_acl = private
copy_cutoff = 4.656G
disable_checksum = true
v2_auth = true
memory_pool_flush_time = 1m0s

A log from the command with the -vv flag

No error is returned, just an empty string.

Strangely, I am able to see the config and my remote in listremotes. Here is the PHP and the Output:

<?php
echo "<pre>";
$return = NULL;$retVal = NULL;
$command = $installDir . 'rclone version';
$return = exec($command, $retVal);
var_dump($command);
var_dump($return);	
var_dump($retVal);

$return = NULL;$retVal = NULL;
$command = $installDir . 'rclone listremotes';
$return = exec($command, $retVal);
var_dump($command);
var_dump($return);	
var_dump($retVal);		

$return = NULL;$retVal = NULL;
$command = $installDir . 'rclone size BackupUtility -vv';
$return = exec($command, $retVal);
var_dump($command);
var_dump($return);	
var_dump($retVal);		
?>

And the output:

string(75) "/usr/local/directadmin/plugins/backup-utility/scripts/rclone/rclone version"
string(20) "- go version: go1.15"
array(3) {
  [0]=>
  string(14) "rclone v1.53.1"
  [1]=>
  string(22) "- os/arch: linux/amd64"
  [2]=>
  string(20) "- go version: go1.15"
}
string(79) "/usr/local/directadmin/plugins/backup-utility/scripts/rclone/rclone listremotes"
string(14) "BackupUtility:"
array(1) {
  [0]=>
  string(14) "BackupUtility:"
}
string(90) "/usr/local/directadmin/plugins/backup-utility/scripts/rclone/rclone size BackupUtility -vv"
string(0) ""
array(0) {
}

Just to say - the commands run fine from the CLI.

2

There should be a colon after the remote in the size command: rclone size BackupUtility: -vv

3

Thanks for your reply. Yes I tried that (see below) - I have tried with forward slash too. No joy.

$return = NULL;$retVal = NULL;
$command = $installDir . 'rclone size BackupUtility: -vv';
$return = exec($command, $retVal);
var_dump($command);
var_dump($return);	
var_dump($retVal);

Produces:

string(91) "/usr/local/directadmin/plugins/backup-utility/scripts/rclone/rclone size BackupUtility: -vv"
string(0) ""
array(0) {
}
4

you should use rclone log file with debug output.

5

I did the following:

.../rclone size BackupUtility: -vv --log-file=rclone_log.txt

But no file was created, does rclone log to any place by default and is there a way to configure the instance instead of passing it to the CLI?

Kind regards

6

not sure what you mean?

as for the log file, use a full path and then you will know where it should be.

7

Thank you - here it is:

[admin@staging1 ~]$ cat rclone_log.txt 
2020/11/04 11:55:34 DEBUG : rclone: Version "v1.53.1" starting with parameters ["/usr/local/directadmin/plugins/backup-utility/scripts/rclone/rclone" "size" "BackupUtility:" "-vv" "--log-file=/home/admin/rclone_log.txt"]
2020/11/04 11:55:34 DEBUG : Using config file from "/home/admin/.config/rclone/rclone.conf"
2020/11/04 11:55:34 DEBUG : Creating backend with remote "BackupUtility:"
2020/11/04 11:55:34 DEBUG : Using v2 auth
2020/11/04 11:55:35 DEBUG : 4 go routines active
2020/11/04 11:55:35 Failed to size: InvalidAccessKeyId: 
	status code: 403, request id: tx00000000000000028869a-005fa2dd07-3f89b5c5-us-east-1-iad1, host id:

But from the CLI it runs fine. See below.

[admin@staging1 ~]$ /usr/local/directadmin/plugins/backup-utility/scripts/rclone/rclone size BackupUtility --log-file=/home/admin/rclone_log.txt
Total objects: 1
Total size: 13 Bytes (13 Bytes)
[admin@staging1 ~]$ cat rclone_log.txt 
[admin@staging1 ~]$

This is what I see when running without the colon:

2020/11/04 12:00:20 DEBUG : rclone: Version "v1.53.1" starting with parameters ["/usr/local/directadmin/plugins/backup-utility/scripts/rclone/rclone" "size" "BackupUtility" "-vv" "--log-file=/home/admin/rclone_log.txt"]
2020/11/04 12:00:20 DEBUG : Creating backend with remote "BackupUtility"
2020/11/04 12:00:20 DEBUG : Using config file from "/home/admin/.config/rclone/rclone.conf"
2020/11/04 12:00:20 DEBUG : fs cache: renaming cache item "BackupUtility" to be canonical "/usr/local/directadmin/BackupUtility"
2020/11/04 12:00:20 ERROR : : error listing: directory not found
2020/11/04 12:00:20 DEBUG : 2 go routines active
2020/11/04 12:00:20 Failed to size with 2 errors: last error was: directory not found

If I login as admin (the user that PHP runs as) and try to do the following it is seen that I do not have permissions on that directory. Where it is getting that directory from, specifically, I have no idea.

[admin@staging1 ~]$ touch /usr/local/directadmin/BackupUtility

touch: cannot touch '/usr/local/directadmin/BackupUtility': Permission denied
8

Paste the output and logs of CLI command with colon

9

Here it is:

[admin@staging1 ~]$ /usr/local/directadmin/plugins/backup-utility/scripts/rclone/rclone size BackupUtility: --log-file=/home/admin/rclone_log.txt
[admin@staging1 ~]$ cat rclone_log.txt 
2020/11/04 12:45:20 Failed to size: InvalidAccessKeyId: 
	status code: 403, request id: tx0000000000000057d9e3b-005fa2e8b0-3991a6f5-us-east-1-iad1, host id: 
[admin@staging1 ~]$

Without colon:

[admin@staging1 ~]$ /usr/local/directadmin/plugins/backup-utility/scripts/rclone/rclone size BackupUtility --log-file=/home/admin/rclone_log.txt
Total objects: 1
Total size: 13 Bytes (13 Bytes)
[admin@staging1 ~]$
10

Without colon is a useless result since it is just giving you the size of a local directory named BackupUtility instead of the actual remote so you can skip that from any further consideration.

From the logs, it looks like you have some access key issue for the remote which needs to be fixed.

11

Ah - Yes, understood. I gave my config earlier and am able to use functionality but as you say, from a local folder. Is there something I may be missing (or be incorrect) from my config? I had tried everything before discovering the no colon approach.

Kind regards

12

a remote must have a colon in it.

the error in the log file is InvalidAccessKeyId

so make sure that value is correct in the config file
access_key_id = XXXXXXXXXXXXXX

13

So that explains that. Not sure why the local nature of my setup wasn't picked up on, but it's all working now; including from PHP.

Many thanks

14

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.