Sftp sync breaks after a few files

What is the problem you are having with rclone?

I'm trying to sync a directory via sftp to a remote server (wpengine), after the first few files I receive error messages that the connection was closed. wpengine support does not see anything unusual on their server. I ran tcpdump during the sync and see that for each file a new ssh connection is created.

-) Is that a know issue?
-) Is it possible to use the same connection for the whole sync?

What is your rclone version (output from rclone version)

rclone v1.47.0

  • os/arch: linux/amd64
  • go version: go1.11.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux bamboo-0 4.19.72 #1 SMP Sat Sep 14 10:06:40 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Which cloud storage system are you using? (eg Google Drive)

SFTP / WPEngine (wordpress)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync . remote:wp-content/themes/theme --sftp-set-modtime=false --exclude ".git/" --no-update-modtime -v --transfers=1

The rclone config contents with secrets removed.

[contantemx]
type = sftp
port = 2222
host = remote
user = user
pass = *** ENCRYPTED ***
set_modtime = false

A log from the command with the -vv flag

2020/07/03 15:24:37 DEBUG : inc/acf/acf-json/group_5de82930edc3e.json: Modification times differ by -58m39.110080075s: 2020-07-03 15:24:35.110080075 +0000 UTC, 2020-07-03 14:25:56 +0000 UTC
2020/07/03 15:24:37 DEBUG : inc/acf/acf-json/group_5e53e01760fa4.json: Modification times differ by -12m9.158079444s: 2020-07-03 15:24:35.158079444 +0000 UTC, 2020-07-03 15:12:26 +0000 UTC
2020/07/03 15:24:37 DEBUG : inc/acf/acf-json/group_5e39d4cebc560.json: Modification times differ by -12m10.146079602s: 2020-07-03 15:24:35.146079602 +0000 UTC, 2020-07-03 15:12:25 +0000 UTC
2020/07/03 15:24:37 DEBUG : inc/acf/acf-json/group_5e69274103991.json: Modification times differ by -14m19.164079365s: 2020-07-03 15:24:35.164079365 +0000 UTC, 2020-07-03 15:10:16 +0000 UTC
2020/07/03 15:24:37 DEBUG : inc/acf/acf-json/group_5dceeacc3aee8.json: Modification times differ by -1h0m24.070080601s: 2020-07-03 15:24:35.070080601 +0000 UTC, 2020-07-03 14:24:11 +0000 UTC
2020/07/03 15:24:37 DEBUG : inc/acf/acf-json/group_5e42fb2d724cc.json: Modification times differ by -14m20.152079523s: 2020-07-03 15:24:35.152079523 +0000 UTC, 2020-07-03 15:10:15 +0000 UTC
2020/07/03 15:24:37 INFO  : phpcs.xml.dist: Copied (replaced existing)
2020/07/03 15:24:37 DEBUG : templates/footer/footer.php: Modification times differ by -1h3m53.289077723s: 2020-07-03 15:24:35.289077723 +0000 UTC, 2020-07-03 14:20:42 +0000 UTC
2020/07/03 15:24:38 DEBUG : templates/blocks/slide-content/slide-content.css: Modification times differ by -1h0m30.271077959s: 2020-07-03 15:24:35.271077959 +0000 UTC, 2020-07-03 14:24:05 +0000 UTC
2020/07/03 15:24:38 DEBUG : templates/blocks/slide-content/slide-content.js: Modification times differ by -1h0m30.275077907s: 2020-07-03 15:24:35.275077907 +0000 UTC, 2020-07-03 14:24:05 +0000 UTC
2020/07/03 15:24:38 DEBUG : templates/blocks/slide-content/slide-content.php: Modification times differ by -1h0m30.283077802s: 2020-07-03 15:24:35.283077802 +0000 UTC, 2020-07-03 14:24:05 +0000 UTC
2020/07/03 15:24:38 INFO  : sftp://remotedev-bamboo-deploy@remotedev.sftp.wpengine.com:2222/wp-content/themes/theme: Waiting for checks to finish
2020/07/03 15:24:38 INFO  : sftp://remotedev-bamboo-deploy@remotedev.sftp.wpengine.com:2222/wp-content/themes/theme: Waiting for transfers to finish
2020/07/03 15:24:38 INFO  : prepros-6.config: Copied (replaced existing)
2020/07/03 15:24:38 INFO  : screenshot.png: Copied (replaced existing)
2020/07/03 15:24:38 INFO  : search.php: Copied (replaced existing)
2020/07/03 15:24:38 INFO  : sidebar.php: Copied (replaced existing)
2020/07/03 15:24:38 INFO  : single.php: Copied (replaced existing)
2020/07/03 15:24:39 INFO  : style-editor.css: Copied (replaced existing)
2020/07/03 15:24:39 INFO  : style.css: Copied (replaced existing)
2020/07/03 15:24:40 INFO  : archive.php: Copied (replaced existing)
2020/07/03 15:24:40 INFO  : functions.php: Copied (replaced existing)
2020/07/03 15:24:40 INFO  : comments.php: Copied (replaced existing)
2020/07/03 15:24:41 INFO  : index.php: Copied (replaced existing)
2020/07/03 15:24:41 INFO  : footer.php: Copied (replaced existing)
2020/07/03 15:24:42 INFO  : header.php: Copied (replaced existing)
2020/07/03 15:24:42 INFO  : page.php: Copied (replaced existing)
2020/07/03 15:24:42 INFO  : templates/loop/content-career.php: Copied (replaced existing)
2020/07/03 15:24:43 INFO  : templates/loop/content-none.php: Copied (replaced existing)
2020/07/03 15:24:43 INFO  : templates/loop/content-page.php: Copied (replaced existing)
2020/07/03 15:24:44 INFO  : templates/loop/content-search.php: Copied (replaced existing)
2020/07/03 15:24:44 INFO  : templates/loop/content.php: Copied (replaced existing)
2020/07/03 15:24:44 INFO  : inc/acf_blocks.php: Copied (replaced existing)
2020/07/03 15:24:45 INFO  : inc/customizer.php: Copied (replaced existing)
2020/07/03 15:24:45 INFO  : inc/template-tags.php: Copied (replaced existing)
2020/07/03 15:24:46 INFO  : languages/lendoncompl.pot: Copied (replaced existing)
2020/07/03 15:24:46 INFO  : languages/readme.txt: Copied (replaced existing)
2020/07/03 15:24:46 INFO  : assets/js/customizer.js: Copied (replaced existing)
2020/07/03 15:24:47 INFO  : assets/js/skip-link-focus-fix.js: Copied (replaced existing)
2020/07/03 15:24:47 INFO  : assets/js/scripts.js: Copied (replaced existing)
2020/07/03 15:24:47 ERROR : assets/sass/_global.scss: Failed to copy: Update SetModTime failed: SetModTime stat failed: stat failed: stat: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56624->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:47 ERROR : assets/sass/_import.scss: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56626->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:48 ERROR : assets/sass/style-editor.scss: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: EOF
2020/07/03 15:24:48 ERROR : assets/sass/_variables.scss: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56634->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:48 ERROR : assets/sass/style.scss: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56640->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:48 ERROR : assets/sass/utilities/_mixins.scss: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56644->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:48 ERROR : assets/sass/libraries/_normalize.scss: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56650->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:48 ERROR : assets/sass/libraries/_wpcore.scss: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: EOF
2020/07/03 15:24:48 ERROR : assets/sass/layout/_footer.scss: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56662->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:48 ERROR : assets/sass/layout/_header.scss: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56664->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:48 ERROR : assets/sass/fonts/Gotham-Bold.woff: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56668->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:48 ERROR : assets/sass/fonts/Gotham-Bold.woff2: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56672->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:49 ERROR : assets/sass/fonts/Gotham-Book.woff: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56674->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:49 ERROR : assets/sass/fonts/Gotham-Book.woff2: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56678->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:49 ERROR : assets/sass/fonts/Gotham-Light.woff: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56682->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:49 ERROR : assets/sass/fonts/Gotham-Light.woff2: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56688->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:49 ERROR : assets/sass/fonts/Gotham-Medium.woff: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56694->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:49 ERROR : assets/sass/fonts/Gotham-Medium.woff2: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: EOF
2020/07/03 15:24:49 ERROR : assets/sass/elements/_buttons.scss: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56702->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:49 ERROR : assets/sass/elements/_contact-form-seven.scss: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: EOF
2020/07/03 15:24:49 ERROR : assets/sass/elements/_theme-specific.scss: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56714->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:49 ERROR : assets/sass/elements/_trustarc.scss: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56716->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:50 ERROR : inc/acf/acf-json/group_5db0a1ea9fe6a.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56720->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:50 ERROR : inc/acf/acf-json/group_5dcec4ec90d59.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56724->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:50 ERROR : inc/acf/acf-json/group_5dd45b03ccc14.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56734->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:50 ERROR : inc/acf/acf-json/group_5dd47fcc02536.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56736->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:50 ERROR : inc/acf/acf-json/group_5ddb2383d5195.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56744->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:50 ERROR : inc/acf/acf-json/group_5de4470607738.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56750->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:50 ERROR : inc/acf/acf-json/group_5dd481008dc7b.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56760->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:50 ERROR : inc/acf/acf-json/group_5e2b30db25ae5.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56764->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:50 ERROR : inc/acf/acf-json/group_5e33540db158c.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56766->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:51 ERROR : inc/acf/acf-json/group_5e35ed6f3fb37.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56770->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:51 ERROR : inc/acf/acf-json/group_5de82930edc3e.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56776->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:51 ERROR : inc/acf/acf-json/group_5e53e01760fa4.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56780->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:51 ERROR : inc/acf/acf-json/group_5e39d4cebc560.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56786->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:51 ERROR : inc/acf/acf-json/group_5e69274103991.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56794->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:51 ERROR : inc/acf/acf-json/group_5dceeacc3aee8.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: EOF
2020/07/03 15:24:51 ERROR : inc/acf/acf-json/group_5e42fb2d724cc.json: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: EOF
2020/07/03 15:24:51 ERROR : templates/footer/footer.php: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56802->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:51 ERROR : templates/blocks/slide-content/slide-content.css: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56804->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:51 ERROR : templates/blocks/slide-content/slide-content.js: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56806->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:52 ERROR : templates/blocks/slide-content/slide-content.php: Failed to copy: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56808->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:52 ERROR : sftp://remotedev-bamboo-deploy@remotedev.sftp.wpengine.com:2222/wp-content/themes/theme: not deleting files as there were IO errors
2020/07/03 15:24:52 ERROR : sftp://remotedev-bamboo-deploy@remotedev.sftp.wpengine.com:2222/wp-content/themes/theme: not deleting directories as there were IO errors
2020/07/03 15:24:52 ERROR : Attempt 1/3 failed with 43 errors and: Update: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56808->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:52 DEBUG : .git: Excluded
2020/07/03 15:24:52 ERROR : : error reading destination directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56810->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:52 INFO  : sftp://remotedev-bamboo-deploy@remotedev.sftp.wpengine.com:2222/wp-content/themes/theme: Waiting for checks to finish
2020/07/03 15:24:52 INFO  : sftp://remotedev-bamboo-deploy@remotedev.sftp.wpengine.com:2222/wp-content/themes/theme: Waiting for transfers to finish
2020/07/03 15:24:52 ERROR : sftp://remotedev-bamboo-deploy@remotedev.sftp.wpengine.com:2222/wp-content/themes/theme: not deleting files as there were IO errors
2020/07/03 15:24:52 ERROR : sftp://remotedev-bamboo-deploy@remotedev.sftp.wpengine.com:2222/wp-content/themes/theme: not deleting directories as there were IO errors
2020/07/03 15:24:52 ERROR : Attempt 2/3 failed with 2 errors and: not deleting files as there were IO errors
2020/07/03 15:24:52 DEBUG : .git: Excluded
2020/07/03 15:24:52 ERROR : : error reading destination directory: List failed: dirExists: couldn't connect SSH: ssh: handshake failed: read tcp yyy.yyy.yyy.yyy:56816->xxx.xxx.xxx.xxx:2222: read: connection reset by peer
2020/07/03 15:24:52 INFO  : sftp://remotedev-bamboo-deploy@remotedev.sftp.wpengine.com:2222/wp-content/themes/theme: Waiting for checks to finish
2020/07/03 15:24:52 INFO  : sftp://remotedev-bamboo-deploy@remotedev.sftp.wpengine.com:2222/wp-content/themes/theme: Waiting for transfers to finish
2020/07/03 15:24:52 ERROR : sftp://remotedev-bamboo-deploy@remotedev.sftp.wpengine.com:2222/wp-content/themes/theme: not deleting files as there were IO errors
2020/07/03 15:24:52 ERROR : sftp://remotedev-bamboo-deploy@remotedev.sftp.wpengine.com:2222/wp-content/themes/theme: not deleting directories as there were IO errors
2020/07/03 15:24:52 ERROR : Attempt 3/3 failed with 2 errors and: not deleting files as there were IO errors
2020/07/03 15:24:52 Failed to sync: not deleting files as there were IO errors

I have updated to the newest release 1.52.2 and the issue is gone ...

That is the best kind of bug - one which is already fixed :slight_smile:

You can reduce the number of connections to the minimum with --transfers 1 --checkers 1

Rclone should then use 2 connections. If you need it to use 1 connection then use the --check-first flag - rclone will do its listing before doing the transferring.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.