What is the problem you are having with rclone?
I want to perform rclone copy as a job on remote HPC so that I do not have to be logged in during the execution. The problem is that it does not perform any copy.
Run the command 'rclone version' and share the full output of the command.
rclone v1.60.0
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Here is the job script that I submit
#!/bin/bash -x
# Job name:
#SBATCH --job-name=rclone
# Max running time (DD-HH:MM:SS)
#SBATCH --time=00-00:02:00
# Project:
#SBATCH --account=[some number]
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=1G
#SBATCH --ntasks=1
# Save output as 'job-name-job-id.txt'
#SBATCH --output=/cluster/home/[my_username]/%x-%j.txt
## Set up job environment:
set -o errexit # Exit the script on any error
set -o nounset # Treat any unset variables as an error
source="/cluster/work/users/[my_username]/[some_name]"
dest=“[some path]"
/cluster/home/[my_username]/bin/rclone -vv --config /cluster/home/[my_username]/.config/rclone/rclone.conf copy $source simula:$dest >> /cluster/home/[my_username]/rclone.log 2>&1
The rclone config contents with secrets removed.
type = drive
client_id = 512279872525-tocr2phr1o2214cfn6khcq4i7n794ur0.apps.googleusercontent.com
scope = drive
token = {"access_token":"ya29.a0AeTM1idetvUvB5wFP3O1wrAMosQzKSCSGZqniPy0cwZLe1DZA6ju2pulfM__e4X7gUCHLgp0PHGnpdKGPj5rWtxXEUmtuqddpPA4nRGZC5wojcWzGye53lGcgYe9lu6Q0UoIJwkdpEuWa1LnumIet-NA6ms0goIaCgYKAWISARESFQHWtWOm75HQzPHGyd88J381TxhWAg0166","token_type":"Bearer","refresh_token":"1//0cfOIlp0pDhpDCgYIARAAGAwSNwF-L9IrXfQ8xaBYFxKbzjbDVH0ZkWwESIF0jC1IKzZq9l1OkNECq2Ea7xluScZa_3_Rd5c-NKc","expiry":"2022-12-12T23:54:27.189470176+01:00"}
team_drive =
A log from the command with the -vv flag
2022/12/13 16:59:16 DEBUG : rclone: Version "v1.60.0" starting with parameters ["/cluster/home/[my_username]/bin/rclone" "-vv" "--config" "/cluster/home//[my_username]/.config/rclone/rclone.conf" "copy" "/cluster/work/users/[my_username]/case9_225k_1203" "simula:turtleFSI/case9_225k_1203"]
2022/12/13 16:59:16 DEBUG : Creating backend with remote "/cluster/work/users//[my_username]/case9_225k_1203"
2022/12/13 16:59:16 DEBUG : Using config file from "/cluster/home//[my_username]/.config/rclone/rclone.conf"
2022/12/13 16:59:16 DEBUG : Creating backend with remote "simula:turtleFSI/case9_225k_1203"
2022/12/13 16:59:16 DEBUG : simula: Loaded invalid token from config file - ignoring
2022/12/13 17:00:16 DEBUG : simula: Token refresh failed try 1/5: Post "https://oauth2.googleapis.com/token": dial tcp 142.250.74.170:443: i/o timeout
2022/12/13 17:00:17 DEBUG : simula: Loaded invalid token from config file - ignoring
2022/12/13 17:01:17 DEBUG : simula: Token refresh failed try 2/5: Post "https://oauth2.googleapis.com/token": dial tcp 216.58.207.202:443: i/o timeout
2022/12/13 17:01:18 DEBUG : simula: Loaded invalid token from config file - ignoring
I tried to refresh the token from rclone config reconnect simula: but somehow I could not log in to my google account with the error message saying something went wrong try again . Using rclone from command line works absolutely fine.