I am working with a group of students for a final project, we have to web scrape data on individual AWS EC2 instances and use cloud storage to hold everyone's combined collected data. Within my personal EC2 instance, it opens every night at 11:50 PM and closes at 11:59 PM. Within that time I have a cron job that runs a docker.sh file. The docker.sh file opens my docker container with a mounted local folder, runs the collection code, closes the docker container, and finally prunes the docker containers.
I now want to add a line to the .sh file that syncs my local EC2 folder to our team's GDrive. The command I am using is "rclone sync /home/ec2-user/environment/SteamDataCollection remote:/SteamDataCollection"
This command works great on it's own when I directly type it into my command line, but when I add it following a && in my docker.sh file I get the errors:
2020/11/28 05:25:11 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
2020/11/28 05:25:11 Failed to create file system for "remote:/SteamDataCollection": didn't find section in config file
I would look to my team for help usually, but I have been designated to set up our team's cloud storage to our GDrive. My team is unaware that I have even chose rclone to connect our EC2 instances to GDrive as they are focusing on other parts of the project.
The goal of the EC2 instance, and really the commands within the .sh file is to make data collection and pushing to cloud storage fully automated. But if the sync command isn't working in the .sh file then we will have to manually type it everytime.
Does anyone have any experience working with rclone sync and .sh files, or have any advice as to how I could automate rclone within a .sh file to push to GDrive?