Rclone copy command work on terminal but not in script

gaurav_basra · January 11, 2024, 12:22pm

I'm uploading the recorded files to aws s3 when user end the call.

I'm using this command to copy my file to s3 bucket

rclone copy /usr/local/recordings/ AWS:newjitsi-call-recording -v --log-file=/var/log/jitsi/amazons33_upload.log

This command is working on the terminal of ubuntu 22.04 server but it is not working in .sh file.

Please help me !!!!!!!!!!!!!!!!!!!!!!!!!!

kapitainsky · January 11, 2024, 12:27pm

As you are new to this forum then maybe you are not aware that without providing details you were asked for when opening this thread nobody is going to keep guessing what you are doing.

Help people to help you.

gaurav_basra · January 11, 2024, 12:55pm

Hey @kapitainsky , Yes I'm newbie here sorry about that

What is the problem you are having with rclone?

The problem is that the command I'm using to copy my file and paste to s3 worked as I expect on the terminal of ubuntu (22.04) but it was not working on bash scripts file.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.1

os/version: ubuntu 22.04 (64 bit)
os/kernel: 5.15.0-89-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.21.5
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Amazon S3

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy /usr/local/recordings/ AWS:newjitsi-call-recording -v --log-file=/var/log/jitsi/amazons33_upload.log

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
region = us-east-1
acl = bucket-owner-full-control

A log from the command that you were trying to run with the `-vv` flag

There is no log if I the command ran from the bash scripts

Create TopicClose

kapitainsky · January 11, 2024, 12:58pm

Could you please replace it with (change -v to -vv):

-vv --log-file=/var/log/jitsi/amazons33_upload.log

Then try to run your .sh script and post content of amazons33_upload.log here. It should tell us what is going on.

gaurav_basra · January 11, 2024, 12:59pm

sure @kapitainsky , Just give me one minute. I will be right back

gaurav_basra · January 11, 2024, 1:04pm

Hey @kapitainsky, I just do the same changes but nothing happen and log file is also empty

rclone move /usr/local/recordings/ AWS:newjitsi-call-recording --v --log-file=/var/log/jitsi/amazons33_upload.log

kapitainsky · January 11, 2024, 1:05pm

Then it means that nothing runs:) what is your .sh script content? How do you run it? From terminal? From cron?

gaurav_basra · January 11, 2024, 1:08pm

@kapitainsky , I can give you the surety that the file is running as I'm also deleting the file when it is uploaded

Here is content

#!/bin/bash
RECORDINGS_DIR=$1

aws s3 cp --recursive ${RECORDINGS_DIR} AWS://

if [ -d "$RECORDINGS_DIR" ]; then
    rclone copy /usr/local/recordings/ AWS:newjitsi-call-recording --v --log-file=/var/log/jitsi/amazons33_upload.log
    rm -r ${RECORDINGS_DIR}
fi

exit 0

kapitainsky · January 11, 2024, 1:11pm

Change --v to -vv please and temporarily remove --log-file=/var/log/jitsi/amazons33_upload.log

Run your script and post what is displayed in terminal here.

If still nothing then change your script a bit and try again:

#!/bin/bash
RECORDINGS_DIR=$1

echo "RECORDINGS_DIR: $RECORDINGS_DIR"

aws s3 cp --recursive ${RECORDINGS_DIR} AWS://

if [ -d "$RECORDINGS_DIR" ]; then
echo "Running rclone now"
    rclone copy /usr/local/recordings/ AWS:newjitsi-call-recording --v --log-file=/var/log/jitsi/amazons33_upload.log
    rm -r ${RECORDINGS_DIR}
fi

exit 0

gaurav_basra · January 11, 2024, 1:18pm

I just ran this command to my terminal

rclone copy /usr/local/recordings/ AWS:newjitsi-call-recording -vv

output

rclone copy /usr/local/recordings/ AWS:newjitsi-call-recording -vv
2024/01/11 13:16:00 DEBUG : rclone: Version "v1.65.1" starting with parameters ["rclone" "copy" "/usr/local/recordings/" "AWS:newjitsi-call-recording" "-vv"]
2024/01/11 13:16:00 DEBUG : Creating backend with remote "/usr/local/recordings/"
2024/01/11 13:16:00 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/01/11 13:16:00 DEBUG : fs cache: renaming cache item "/usr/local/recordings/" to be canonical "/usr/local/recordings"
2024/01/11 13:16:00 DEBUG : Creating backend with remote "AWS:newjitsi-call-recording"
2024/01/11 13:16:00 DEBUG : S3 bucket newjitsi-call-recording: Waiting for checks to finish
2024/01/11 13:16:00 DEBUG : S3 bucket newjitsi-call-recording: Waiting for transfers to finish
2024/01/11 13:16:00 INFO  : There was nothing to transfer
2024/01/11 13:16:00 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.0s

2024/01/11 13:16:00 DEBUG : 6 go routines activ

kapitainsky · January 11, 2024, 1:20pm

All looks good. There is nothing to do for rclone...

gaurav_basra · January 11, 2024, 1:24pm

No @kapitainsky , I ran this command on the terminal and all the files are deleted already. As I told you on the topic that the command is running as I expected on the terminal but it is not working from the bash script.

gaurav_basra · January 11, 2024, 1:25pm

#!/bin/bash
RECORDINGS_DIR=$1

echo "RECORDINGS_DIR: $RECORDINGS_DIR"

aws s3 cp --recursive ${RECORDINGS_DIR} AWS://

if [ -d "$RECORDINGS_DIR" ]; then
echo "Running rclone now"
    rclone copy /usr/local/recordings/ AWS:newjitsi-call-recording --v --log-file=/var/log/jitsi/amazons33_upload.log
    rm -r ${RECORDINGS_DIR}
fi

exit 0

@kapitainsky , I just replace my script with your code but still no luck

gaurav_basra · January 11, 2024, 1:30pm

@kapitainsky check now the log file If there is a file and I run this command on the terminal

2024/01/11 13:28:20 DEBUG : rclone: Version "v1.65.1" starting with parameters ["rclone" "copy" "/usr/local/recordings/" "AWS:newjitsi-call-recording" "-vv" "--log-file=/var/log/jitsi/amazons33_upload.log"]
2024/01/11 13:28:20 DEBUG : Creating backend with remote "/usr/local/recordings/"
2024/01/11 13:28:20 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/01/11 13:28:20 DEBUG : fs cache: renaming cache item "/usr/local/recordings/" to be canonical "/usr/local/recordings"
2024/01/11 13:28:20 DEBUG : Creating backend with remote "AWS:newjitsi-call-recording"
2024/01/11 13:28:20 DEBUG : flqyfzfkcodiqvyz/metadata.json: Need to transfer - File not found at Destination
2024/01/11 13:28:20 DEBUG : flqyfzfkcodiqvyz/shinetest001_2024-01-11-13-27-25.mp4: Need to transfer - File not found at Destination
2024/01/11 13:28:20 DEBUG : S3 bucket newjitsi-call-recording: Waiting for checks to finish
2024/01/11 13:28:20 DEBUG : S3 bucket newjitsi-call-recording: Waiting for transfers to finish
2024/01/11 13:28:20 DEBUG : flqyfzfkcodiqvyz/metadata.json: md5 = ec813756d35a634d15c76d60735a7dcf OK
2024/01/11 13:28:20 INFO  : flqyfzfkcodiqvyz/metadata.json: Copied (new)
2024/01/11 13:28:20 DEBUG : flqyfzfkcodiqvyz/shinetest001_2024-01-11-13-27-25.mp4: md5 = 040cd23887f505f6486cc41f3f64c931 OK
2024/01/11 13:28:20 INFO  : flqyfzfkcodiqvyz/shinetest001_2024-01-11-13-27-25.mp4: Copied (new)
2024/01/11 13:28:20 INFO  :
Transferred:      106.958 KiB / 106.958 KiB, 100%, 0 B/s, ETA -
Transferred:            2 / 2, 100%
Elapsed time:         0.2s

2024/01/11 13:28:20 DEBUG : 11 go routines active

kapitainsky · January 11, 2024, 1:41pm

Can you in terminal run modified script:

$ mysript.sh

Here will be some output

and post what it shows. Not from log file.

gaurav_basra · January 12, 2024, 4:35am

@kapitainsky,

/etc/jitsi/jibri/rclone.amazonS3.upload.sh

OUTPUT ->

RECORDINGS_DIR: 
/etc/jitsi/jibri/rclone.amazonS3.upload.sh: line 6: aws: command not found

gaurav_basra · January 12, 2024, 5:49am

@kapitainsky ,
After fixing this aws: command not found, I found one more issue that rclone don't have permission to write in log file.

After giving the permission to rclone here is the log file

2024/01/12 05:43:18 DEBUG : rclone: Version "v1.65.1" starting with parameters ["rclone" "copy" "/usr/local/recordings/" "AWS:newjitsi-call-recording" "-vv" "--log-file=/var/log/jitsi/amazons33_upload.log"]
2024/01/12 05:43:18 DEBUG : Creating backend with remote "/usr/local/recordings/"
2024/01/12 05:43:18 DEBUG : Using config file from "/home/jibri/.config/rclone/rclone.conf"
2024/01/12 05:43:18 DEBUG : fs cache: renaming cache item "/usr/local/recordings/" to be canonical "/usr/local/recordings"
2024/01/12 05:43:18 DEBUG : Creating backend with remote "AWS:newjitsi-call-recording"
2024/01/12 05:43:18 Failed to create file system for "AWS:newjitsi-call-recording": didn't find section in config file

kapitainsky · January 12, 2024, 6:58am

You do not have rclone.conf file with remote configuration in /home/jibri/.config/rclone.

Your previous command was run as root:

with config file in /root/.config/rclone directory

depending on user you use rclone it requires config file in different location.

Other option is to use --config flag, e.g.:

--config = /path/to/rclone.conf

gaurav_basra · January 12, 2024, 7:05am

Okay, thanks for sending me the problem but @kapitainsky, why it is working from the terminal?

And where to use this --config flag, Please write down the solution properly.

Thanks you so much buddy!

kapitainsky · January 12, 2024, 7:19am

Where you run from terminal I can see that you run it as root:

and it all works so there is config there.

When you run your script:

I can see you run it as jibri and there is no config there.

Make sure you have your config file in right locations.

Or use --config flag

rclone copy .... --config=/path/to/rclone.conf