OK, that may make it difficult to optimize.
The bottleneck is most likely due to high load on the shared host, since I can sync 70K files from GDrive to an ultra small SFTP server in less than a minute, when nothing to transfer.
You may be able to work around the slow target using top-up sync, but that requires a fairly good technical understanding.