ACD BAN/Alternatives discussion (error HTTP code 429)

Amazon sucks on more levels than just this. Final straw for me.

1 Like

Yeah, I have to admit, I was pretty disappointed to find that

  • their app sucks bigly
  • the Synology Cloud Station ACD sync sucks bigly
  • Prime Photos doesn’t have search or image recognition like GPhotos does (it’s available in Family Vault apparently, but that’s still US-only)
  • Transfer rates can be really slow
  • Document support is nowhere near as good as GDrive

Can you elaborate on how to execute the recursive sync? The odrive linux CLI documentation is really poor and I’m at a loss here how to make odrive download my ACD content to the server. Is there a possibility to choose which directories to download?

Exactly the same issue here, here’s my current attempt

python “$HOME/.odrive-agent/bin/odrive.py” sync ~/odrive-agent-mount/cloud provider/backup/fileso.cloudf

How do I make this recursive? I’ve tried the normal arguments…

Edit: I see you need to script it, can anyone help with this?

Guys, what the situation the Dropbox Business? It is usable with plex?

I did not have much time, therefore it is ugly, but just works.

recursive sync directories:

find ~/odrive-agent-mount/ -type f -name "*.cloudf" -exec python "$HOME/.odrive-agent/bin/odrive.py" sync "{}" \;

Since this just syncs only one layer, you need to call this multiple times, until you have reached the max-depth of your directory structure.

Since I didn’t know and didn’t want to know about my depth or structure, I just went with the brute force approach.

this script: https://pastebin.com/KDB1XVYR

#!/bin/bash
find ~/odrive-agent-mount/ -type f -name “.cloudf" -exec python “$HOME/.odrive-agent/bin/odrive.py” sync “{}” ;
find ~/odrive-agent-mount/ -type f -name "
.cloudf” -exec python “$HOME/.odrive-agent/bin/odrive.py” sync “{}” ;
find ~/odrive-agent-mount/ -type f -name “.cloudf" -exec python “$HOME/.odrive-agent/bin/odrive.py” sync “{}” ;
find ~/odrive-agent-mount/ -type f -name "
.cloudf” -exec python “$HOME/.odrive-agent/bin/odrive.py” sync “{}” ;
find ~/odrive-agent-mount/ -type f -name “.cloudf" -exec python “$HOME/.odrive-agent/bin/odrive.py” sync “{}” ;
find ~/odrive-agent-mount/ -type f -name "
.cloudf” -exec python “$HOME/.odrive-agent/bin/odrive.py” sync “{}” ;
find ~/odrive-agent-mount/ -type f -name “.cloudf" -exec python “$HOME/.odrive-agent/bin/odrive.py” sync “{}” ;
find ~/odrive-agent-mount/ -type f -name "
.cloudf” -exec python “$HOME/.odrive-agent/bin/odrive.py” sync “{}” ;
find ~/odrive-agent-mount/ -type f -name “*.cloudf” -exec python “$HOME/.odrive-agent/bin/odrive.py” sync “{}” ;

I had to call it twice. Once everything is synced find should not detect any “cloudf” files anymore and no output should be generated by the script anymore.

Now you have all directories with tons of “*.cloud” files, which represent your files.

I just launched screen and ran this call three times in parallel.

find ~/odrive-agent-mount/ -type f -name “*.cloud” -exec python “$HOME/.odrive-agent/bin/odrive.py” sync {} ;

Edit: It seems performance is a little bit flacky per file, since ACD stores them on different servers. Currently I am using 4 threads on eu-west-1c to get a more stable download stream.

At first it may complain about some cloud files missing, but once the three threads get out of sync enough, this should not be an issue anymore.

Once the three threads are finished, I am going to call it once again to get anything which may failed.

To verify everything is downloaded check for cloud and cloudf files when finished.

find ~/odrive-agent-mount/ -type f -name *.cloud
find ~/odrive-agent-mount/ -type f -name *.cloudf

Both should not report anything when you have finished syncing.

3 Likes

This looks great, though for some reason it’s not doing anything, testing the find works. Yet with the exec part it just hangs without doing anything. Any ideas?

give it some time. if you have many directories. it took me about 45-60min to just sync the structure. it seems odrive is slow that way.

You should see output of paths periodically during the directory sync.

once you have the structure, download itself is handled via ACD servers, with rclone like performance.

I mean, I can run a single sync line and it will work. I try to run your script (just the one line) and it hangs. I thought it would work at a lower and lower depth each time you ran it?

Awesome! And I just got done uploading my network drive as my first online backup. Guess it’s time to look at something else now.

There’s a missing " that’s why it wouldn’t work :smiley:

Thanks so much for all this Phillip, really appreciated!

Here’s the fix:

find ~/odrive-agent-mount/ -type f -name “*.cloudf” -exec python “$HOME/.odrive-agent/bin/odrive.py” sync “{}” ;

To watch status in realtime:

while true; do “$HOME/.odrive-agent/bin/odrive” status; sleep 1; done

Not not it tells you much of anything

Odrive errors on some files due to file name length. I was getting the same issue with the ACD windows client.

what is the best copy/move setting for gdrive? I’m getting 11-12MB/sec only, from DE server location to whereever Google hosts my files at the end…

checkers, retries, transfers, etc

what are you guys using?

I’m not getting the performance you got :frowning:

Same machine specs, zone: europe-west1-d

I wonder if increasing the number of threads will increase the speed? What do you think?

Using www.cloudhq.net/ to move ACD to GDrive. And guess what…free 15 day trial!!!

Its also SUPER easy to get it going

edit: crap ---- paid cloud accounts is limited to 4 GB during trial
it will cost $20 to sync it all.

With the recent problems for both rclone and acd_cli being blocked by Amazon, I decided to diversify my cloud storage options. Google Cloud Drive with an unlimited plan was as my first choice.

But I was facing a couple of challenges:

  • About 12 TB of data to transfer from Amazon to Google cloud storage
  • Home bandwidth limitations – upstream of about 9 MB/s and downstream of about 4 MB/s, and me and my family also need this bandwidth for something else than cloud syncs :slight_smile:
  • Possible integration of other cloud services for additional backup and security

So on top of Google Cloud Drive I also investigated how to best sync first my current data from Amazon to Google, and then keep it in sync in the future.

I first considered a virtual private server, this could potentially give a decent performance, but the rclone / acd_cli problems would just be the same, as these servers usually run some Linux implementation and rclone would be the only option for syncing.

When looking for other options, I ran across cloud sync services by various providers – these services do the sync on their servers, you use a simple web interface to configure and monitor sync operations.

So I decided to go for a free trial of cloudHQ (https:// www.cloudhq.net/) and was very positively surprised.

After setting up and activating an account (no payment information required for the 15-day trial) a very simple user interface guides you through the initial setup.

Most major cloud providers are supported – Amazon, Google, Dropbox, Microsoft One Drive and Office 365 and many others, but also E-Mail syncs between various providers and application level syncs for example between Salesforce and Google Contact/Calendars. Beyond my use case above, there are many possibilities I hadn’t even considered before.

During the setup you simply select two providers, decide if it is a one-way or two-way sync and grant cloudHQ access to your cloud data, and then the initial transfer starts – cloudHQ will notify you by mail when it’s done. There is an excellent dashboard to monitor and control all activities.

After the initial transfer is done, cloudHQ will continue to monitor all activities and will keep everything in sync continuously without any need for further manual syncs. And all this works magically in the background without consuming any of my local bandwidth or computers.

Pricing is a bit high – it’s $9.90 per month for a premium plan which you would need if you host large data volumes. However for low-end users, there is a free plan and as mentioned a free 15 day trial for all plans and a 30-day money back offer.

3 Likes

I was under the impression that $9.90 was only if one paid for the entire year and if one wanted to pay monthly, it is $19.80 per month? Maybe I missed a different available option?

Correct - that’s the monthly price when paying for a year. However, I do not only want a one-time sync but after the initial transfer to google I would reverse the sync direction so that my “main” cloud provider for rclone is Google drive and all news stuff and changes are copied to Amazon as an additional backup.And being able to integrate other provider like my One Drive / Office 365 / Dropbox accounts makes it worth the money for me.

And there is a 30-days “no questions asked” money-back policy in case it does not work as well as expected.

OK, thank you for the follow up.

FYI
Transfered 40TB from ACD to GDrive via Google Cloud a week ago.
Cost < USD$3 (VM fee not included)