Sync within .sh file

Help and Support
1

I am working with a group of students for a final project, we have to web scrape data on individual AWS EC2 instances and use cloud storage to hold everyone's combined collected data. Within my personal EC2 instance, it opens every night at 11:50 PM and closes at 11:59 PM. Within that time I have a cron job that runs a docker.sh file. The docker.sh file opens my docker container with a mounted local folder, runs the collection code, closes the docker container, and finally prunes the docker containers.

I now want to add a line to the .sh file that syncs my local EC2 folder to our team's GDrive. The command I am using is "rclone sync /home/ec2-user/environment/SteamDataCollection remote:/SteamDataCollection"

This command works great on it's own when I directly type it into my command line, but when I add it following a && in my docker.sh file I get the errors:
2020/11/28 05:25:11 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
2020/11/28 05:25:11 Failed to create file system for "remote:/SteamDataCollection": didn't find section in config file

I would look to my team for help usually, but I have been designated to set up our team's cloud storage to our GDrive. My team is unaware that I have even chose rclone to connect our EC2 instances to GDrive as they are focusing on other parts of the project.

The goal of the EC2 instance, and really the commands within the .sh file is to make data collection and pushing to cloud storage fully automated. But if the sync command isn't working in the .sh file then we will have to manually type it everytime.

Does anyone have any experience working with rclone sync and .sh files, or have any advice as to how I could automate rclone within a .sh file to push to GDrive?

2

Hi, welcome to the forum!

When running rclone on its own, which configuration file does it use then? If you run command rclone config file it will show you the path. The error messages when running in docker.sh says /root/.config/rclone/rclone.conf, so my first guess would be this is caused by different users (root/non-root).

3

Running the path command gives /home/ec2-user/.config/rclone/rclone.conf so it is defiantly a user issue, how can I ensure my .sh command uses the non-root config file?

4

The best solution I think would be to configure the cron job to run as the specific user, instead of root. E.g. see this:

Alternatively, you could set the --config option in the cron job command line, to point it to a specific rclone config file.

5

Thank you, I will look into those options

6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.