I thought of this post when I created my issue. Could the md5s be making them copy over and over like in my situation?
https://github.com/ncw/rclone/issues/1430