Copy from Sharepoint of huge amount of files

Help and Support
1

What is the problem you are having with rclone?
Hello,

I'm facing a issue when I trying to copy to local disk a huge amount of files (1500000) from Sharepoint
I've splitted the dowload to differnt pc with different pubblic IPs but almost all of them are facing the issue e wont be able to complete the download of all the files.
This branch has 209 GB but only 8

I'm getting a lot of:
error reading source directory: couldn't list files: 429 TOO MANY REQUESTS: 429 Too Many Requests

Run the command 'rclone version' and share the full output of the command.
rclone v1.58.0

Which cloud storage system are you using? (eg Google Drive)
Sharepoint

The command you were trying to run (eg rclone copy /tmp remote:tmp)
Paste command here
rclone copy --progress --ignore-size --ignore-checksum --update _:"/Documenti condivisi/6. OPERATIONS/2. EPC Projects/2019" "D:\2. EPC Projects\2019" --log-file=C:\temp\SharePoint_log6.txt --log-level ERROR
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 26967 / 26967, 100%
Elapsed time: 6m23.7s

The rclone config contents with secrets removed.
Paste config here

[_]
type = webdav
url = https://.sharepoint.com/sites/_
vendor = sharepoint
user =
pass =*****

A log from the command with the -vv flag

I've removed a lot of rows: all of them have "Unchanged skipping" or "Sizes identical"
total rows number in log is 27901

2022/04/29 07:42:52 DEBUG : rclone: Version "v1.58.0" starting with parameters ["rclone" "copy" "-vv" "_****:/Documenti condivisi/6. OPERATIONS/2. *** Projects/2019" "D:\2. *** Projects\2019" "--log-file=C:\temp\SharePoint_logD.txt"]
2022/04/29 07:42:52 DEBUG : Creating backend with remote "*:/Documenti condivisi/6. OPERATIONS/2. *** Projects/2019"
2022/04/29 07:42:52 DEBUG : Using config file from "C:\Users\Sergio\AppData\Roaming\rclone\rclone.conf"
2022/04/29 07:42:52 DEBUG : found headers:
2022/04/29 07:42:53 DEBUG : fs cache: rming cache item "*****:/Documenti condivisi/6. OPERATIONS/2. *** Projects/2019" to be canonical "*****_:Documenti condivisi/6. OPERATIONS/2. *** Projects/2019"
2022/04/29 07:42:53 DEBUG : Creating backend with remote "D:\2. *** Projects\2019"
2022/04/29 07:42:53 DEBUG : fs cache: rming cache item "D:\2. *** Projects\2019" to be canonical "//?/D:/2. *** Projects/2019"
2022/04/29 07:42:53 DEBUG : MANUALE_UTENTE_RID.PDF: Sizes identical
2022/04/29 07:42:53 DEBUG : MANUALE_UTENTE_RID.PDF: Unchanged skipping
2022/04/29 07:42:53 DEBUG : Utilizzo della vettura privata.pdf: Sizes identical
2022/04/29 07:42:53 DEBUG : Utilizzo della vettura privata.pdf: Unchanged skipping
2022/04/29 07:42:53 DEBUG : ManualeUtenteAreaClienti GSE.pdf: Sizes identical
2022/04/29 07:42:53 DEBUG : ManualeUtenteAreaClienti GSE.pdf: Unchanged skipping
2022/04/29 07:42:54 DEBUG : 0. Template/Contacts- Porto Torres 2.xlsx: Sizes identical
2022/04/29 07:42:54 DEBUG : 0. Template/Contacts-*** Porto Torres 2.xlsx: Unchanged skipping
2022/04/29 07:42:54 DEBUG : 1. Coordinamento *** Italia/****** *** ITALIA-rev 191006.xlsx: Sizes identical
2022/04/29 07:42:54 DEBUG : 1. Coordinamento *** Italia/****** *** ITALIA-rev 191006.xlsx: Unchanged skipping
2022/04/29 07:42:54 DEBUG : 1. Coordinamento *** Italia/****** *** ITALIA-rev 192105.xlsx: Sizes identical
2022/04/29 07:42:54 DEBUG : 1. Coordinamento *** Italia/****** *** ITALIA-rev 192105.xlsx: Unchanged skipping
2022/04/29 07:42:54 DEBUG : xxxxxxxxx - Messico/Ejemplo_Curve-S.xlsx: Sizes identical
2022/04/29 07:42:54 DEBUG : xxxxxxxxx - Messico/Ejemplo_Curve-S.xlsx: Unchanged skipping
2022/04/29 07:42:54 DEBUG : xxxxxxxxx - Messico/WBS da elaborare.xlsx: Sizes identical
2022/04/29 07:42:54 DEBUG : xxxxxxxxx - Messico/WBS da elaborare.xlsx: Unchanged skipping
2022/04/29 07:42:54 DEBUG : F19CT002***- *** Volpiano/02. Documents/*** Volpiano-Contacts.xlsx: Sizes identical
2022/04/29 07:42:54 DEBUG : F19CT002***- *** Volpiano/02. Documents/*** Volpiano-Contacts.xlsx: Unchanged skipping
2022/04/29 07:42:54 DEBUG : F19CT002***- *** Volpiano/02. Documents/*** Volpiano-Spese sostenute.xlsx: Sizes identical
2022/04/29 07:42:54 DEBUG : F19CT002***- *** Volpiano/02. Documents/*** Volpiano-Spese sostenute.xlsx: Unchanged skipping
2022/04/29 07:42:54 DEBUG : F19CT002***- *** Volpiano/02. Documents/hardcopy.log: Sizes identical
2022/04/29 07:42:54 DEBUG : F19CT002***- *** Volpiano/02. Documents/hardcopy.log: Unchanged skipping
2022/04/29 07:42:54 DEBUG : F19CT002***- *** Volpiano/02. Documents/Testata Archivio Cartaceo-Project Name-DA FARE.doc: Sizes identical
2022/04/29 07:42:54 DEBUG : F19CT002***- *** Volpiano/02. Documents/Testata Archivio Cartaceo-Project Name-DA FARE.doc: Unchanged skipping
2022/04/29 07:42:54 DEBUG : F18CT007 - Aldi Landriano (2018)/02. Documents/20180830_LNDC_VVF_PraticaProtocollata.zip: Sizes identical
2022/04/29 07:42:54 DEBUG : F18CT007 - Aldi Landriano (2018)/02. Documents/20180830_LNDC_VVF_PraticaProtocollata.zip: Unchanged skipping
2022/04/29 07:42:54 DEBUG : F18CT007 - Aldi Landriano (2018)/02. Documents/Contacts-LANDRIANO.xlsx: Sizes identical
2022/04/29 07:42:54 DEBUG : F18CT007 - Aldi Landriano (2018)/02. Documents/Contacts-LANDRIANO.xlsx: Unchanged skipping
2022/04/29 07:42:54 DEBUG : F18CT007 - Aldi Landriano (2018)/02. Documents/Dogane e RDE.7z: Sizes identical
2022/04/29 07:42:54 DEBUG : F18CT007 - Aldi Landriano (2018)/02. Documents/Dogane e RDE.7z: Unchanged skipping
2022/04/29 07:42:54 DEBUG : F18CT007 - Aldi Landriano (2018)/02. Documents/Elenco convenzioni Vailog.JPG: Sizes identical
2022/04/29 07:42:54 DEBUG : F18CT007 - Aldi Landriano (2018)/02. Documents/Elenco convenzioni Vailog.JPG: Unchanged skipping
2022/04/29 07:42:54 DEBUG : F18CT007 - Aldi Landriano (2018)/02. Documents/Instanza RID Vailog Landriano_inviata.JPG: Sizes identical
identical
023700DGRV00280 - Relazione Tecnica Generale ‎(1)/desktop.ini: Unchanged skipping
2022/04/29 07:43:40 DEBUG : pacer: Reducing sleep to 12.65625ms
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00303 - 023700DBQB00303 - Piano della Qualità di Progetto ‎(1)/023700DBQB00303_EXDE00_16 revMCA.docx: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00303 - 023700DBQB00303 - Piano della Qualità di Progetto ‎(1)/023700DBQB00303_EXDE00_16 revMCA.docx: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00303 - 023700DBQB00303 - Piano della Qualità di Progetto ‎(1)/023700DBQB00303_EXDE00_16.zip: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00303 - 023700DBQB00303 - Piano della Qualità di Progetto ‎(1)/023700DBQB00303_EXDE00_16.zip: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00303 - 023700DBQB00303 - Piano della Qualità di Progetto ‎(1)/CS_023700DBQB00303_EXDE00_16.docx: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00303 - 023700DBQB00303 - Piano della Qualità di Progetto ‎(1)/CS_023700DBQB00303_EXDE00_16.docx: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00303 - 023700DBQB00303 - Piano della Qualità di Progetto ‎(1)/desktop.ini: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00303 - 023700DBQB00303 - Piano della Qualità di Progetto ‎(1)/desktop.ini: Unchanged skipping
2022/04/29 07:43:40 DEBUG : pacer: Reducing sleep to 10ms
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00319 - 023700DECA00319 - Impianto messa a terra/023700DECA00319_EXDE00_04.zip: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00319 - 023700DECA00319 - Impianto messa a terra/023700DECA00319_EXDE00_04.zip: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00319 - 023700DECA00319 - Impianto messa a terra/CS_023700DECA00319_EXDE00_04-.docx: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00319 - 023700DECA00319 - Impianto messa a terra/CS_023700DECA00319_EXDE00_04-.docx: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00319 - 023700DECA00319 - Impianto messa a terra/CS_023700DECA00319_EXDE00_04.docx: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00319 - 023700DECA00319 - Impianto messa a terra/CS_023700DECA00319_EXDE00_04.docx: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00319 - 023700DECA00319 - Impianto messa a terra/desktop.ini: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00319 - 023700DECA00319 - Impianto messa a terra/desktop.ini: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00320 - 023700DEDG00320 - Plan. area sottocampi e posiz. SB ‎(1)/023700DEDG00320_EXDE00_03.zip: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00320 - 023700DEDG00320 - Plan. area sottocampi e posiz. SB ‎(1)/023700DEDG00320_EXDE00_03.zip: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00320 - 023700DEDG00320 - Plan. area sottocampi e posiz. SB ‎(1)/023700DEDG00320_EXDE00_03_rev.ELE.pdf: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00320 - 023700DEDG00320 - Plan. area sottocampi e posiz. SB ‎(1)/023700DEDG00320_EXDE00_03_rev.ELE.pdf: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00320 - 023700DEDG00320 - Plan. area sottocampi e posiz. SB ‎(1)/023700DEDG00320_EXDE01_03.zip: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00320 - 023700DEDG00320 - Plan. area sottocampi e posiz. SB ‎(1)/023700DEDG00320_EXDE01_03.zip: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00320 - 023700DEDG00320 - Plan. area sottocampi e posiz. SB ‎(1)/023700DEDG00320_EXDE01_03_comm.pdf: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00320 - 023700DEDG00320 - Plan. area sottocampi e posiz. SB ‎(1)/023700DEDG00320_EXDE01_03_comm.pdf: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00320 - 023700DEDG00320 - Plan. area sottocampi e posiz. SB ‎(1)/CS_023700DEDG00320_EXDE00_03.docx: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00320 - 023700DEDG00320 - Plan. area sottocampi e posiz. SB ‎(1)/CS_023700DEDG00320_EXDE00_03.docx: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00320 - 023700DEDG00320 - Plan. area sottocampi e posiz. SB ‎(1)/CS_023700DEDG00320_EXDE01_03.docx: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00320 - 023700DEDG00320 - Plan. area sottocampi e posiz. SB ‎(1)/CS_023700DEDG00320_EXDE01_03.docx: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00320 - 023700DEDG00320 - Plan. area sottocampi e posiz. SB ‎(1)/desktop.ini: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00320 - 023700DEDG00320 - Plan. area sottocampi e posiz. SB ‎(1)/desktop.ini: Unchanged skipping
2022/04/29 07:43:40 DEBUG : pacer: low level retry 1/10 (error 429 TOO MANY REQUESTS: 429 Too Many Requests)
2022/04/29 07:43:40 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022/04/29 07:43:40 DEBUG : pacer: Reducing sleep to 15ms
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00278 - 023700DGPF00278 - Piano di Approv/023700DGPF00278_EXDE00_04.zip: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00451 - 023700DACA00451 - Rel. calcolo fond. MTR/CS_023700DACA00451_EXDE00_1532.docx: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00451 - 023700DACA00451 - Rel. calcolo fond. MTR/desktop.ini: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00451 - 023700DACA00451 - Rel. calcolo fond. MTR/desktop.ini: Unchanged skipping
2022/04/29 07:43:40 DEBUG : pacer: low level retry 2/10 (error 429 TOO MANY REQUESTS: 429 Too Many Requests)
2022/04/29 07:43:40 DEBUG : pacer: Rate limited, increasing sleep to 22.5ms
2022/04/29 07:43:40 DEBUG : pacer: Reducing sleep to 16.875ms
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/023700DACA00452_EXDE00_35.zip: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/023700DACA00452_EXDE00_35.zip: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/023700DACA00452_EXDE01_46.zip: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/023700DACA00452_EXDE01_46.zip: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/023700DACA00452_EXDE01_46_rev CIVI.pdf: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/023700DACA00452_EXDE01_46_rev CIVI.pdf: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/023700DACA00452_EXDE02_45.zip: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/023700DACA00452_EXDE02_45.zip: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/Allegato 3 Relazione Tecnica Generale - (250X660)_rev CIVI.pdf: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/Allegato 3 Relazione Tecnica Generale - (250X660)_rev CIVI.pdf: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/Allegato 4 Relazione di calcolo Cabina - (250x660)rev CIVI.pdf: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/Allegato 4 Relazione di calcolo Cabina - (250x660)rev CIVI.pdf: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/CS_023700DACA00452_EXDE01_46.docx: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/CS_023700DACA00452_EXDE01_46.docx: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/CS_023700DACA00452_EXDE02_45.docx: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/CS_023700DACA00452_EXDE02_45.docx: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/desktop.ini: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00452 - 023700DACA00452 - Rel. calcolo fondaz.POD1ePOD2/desktop.ini: Unchanged skipping
2022/04/29 07:43:40 DEBUG : pacer: Reducing sleep to 12.65625ms
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00450 - 023700DAST00450 - Spec. esecuzione prove pull-out/023700DAST00450_EXDE00_17.zip: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00450 - 023700DAST00450 - Spec. esecuzione prove pull-out/023700DAST00450_EXDE00_17.zip: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00450 - 023700DAST00450 - Spec. esecuzione prove pull-out/023700DAST00450_EXDE00_17.zip: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00450 - 023700DAST00450 - Spec. esecuzione prove pull-out/023700DAST00450_EXDE00_17.zip: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00450 - 023700DAST00450 - Spec. esecuzione prove pull-out/023700DAST00450_EXDE02_18.zip: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00450 - 023700DAST00450 - Spec. esecuzione prove pull-out/023700DAST00450_EXDE02_18.zip: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00450 - 023700DAST00450 - Spec. esecuzione prove pull-out/CS_023700DAST00450_EXDE02_18.docx: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00450 - 023700DAST00450 - Spec. esecuzione prove pull-out/CS_023700DAST00450_EXDE02_18.docx: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00450 - 023700DAST00450 - Spec. esecuzione prove pull-out/desktop.ini: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00450 - 023700DAST00450 - Spec. esecuzione prove pull-out/desktop.ini: Unchanged skipping
2022/04/29 07:43:40 DEBUG : pacer: Reducing sleep to 10ms
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00420 - 023700DADG00420 - Viabilità, accessi e rencizione/023700DADG00420_EXDE00_02.zip: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00420 - 023700DADG00420 - Viabilità, accessi e rencizione/023700DADG00420_EXDE00_02.zip: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00420 - 023700DADG00420 - Viabilità, accessi e rencizione/023700DADG00420_EXDE00_02_rev CIVI.pdf: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00420 - 023700DADG00420 - Viabilità, accessi e rencizione/023700DADG00420_EXDE00_02_rev CIVI.pdf: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00420 - 023700DADG00420 - Viabilità, accessi e rencizione/023700DADG00420_EXDE01_02.zip: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00420 - 023700DADG00420 - Viabilità, accessi e rencizione/023700DADG00420_EXDE01_02.zip: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00420 - 023700DADG00420 - Viabilità, accessi e rencizione/023700DADG00420_EXDE01_02_rev .pdf: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00420 - 023700DADG00420 - Viabilità, accessi e rencizione/023700DADG00420_EXDE01_02_rev .pdf: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00420 - 023700DADG00420 - Viabilità, accessi e rencizione/CS_023700DADG00420_EXDE00_02.docx: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00420 - 023700DADG00420 - Viabilità, accessi e rencizione/CS_023700DADG00420_EXDE00_02.docx: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00420 - 023700DADG00420 - Viabilità, accessi e rencizione/CS_023700DADG00420_EXDE01_02.docx: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00420 - 023700DADG00420 - Viabilità, accessi e rencizione/CS_023700DADG00420_EXDE01_02.docx: Unchanged skipping
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00420 - 023700DADG00420 - Viabilità, accessi e rencizione/desktop.ini: Sizes identical
2022/04/29 07:43:40 DEBUG : F19CT002***- *** Volpiano/01. Design/02. Executive Design/00_Consegne EDAM (backup Drive)/EDAM/Working area/00420 - 023700DADG00420 - Viabilità, accessi e rencizione/desktop.ini: Unchanged skipping
2022/04/29 07:43:40 DEBUG : Local file system at //?/D:/2. *** Projects/2019: Waiting for checks to finish
2022/04/29 07:43:40 DEBUG : Local file system at //?/D:/2. *** Projects/2019: Waiting for transfers to finish
2022/04/29 07:43:40 INFO :
Transferred: 3.006 MiB / 3.006 MiB, 100%, 279.853 KiB/s, ETA 0s
Checks: 13538 / 13538, 100%
Transferred: 1 / 1, 100%
Elapsed time: 48.5s

2022/04/29 07:43:40 DEBUG : 9 go routines active

2

This usually indicates you are hitting the server too hard.

Probably the most effective thing to do will be to use --tpslimit to limit the number of transactions you are doing per second to something the server does not rate limit. Try --tpslimit 10 and reduce the 10 until the server no longer gives 429 errors.

3

Ty Nick

I'll try and let you know.

4

I decreased the value till 1, but nothing is changed.

rclone copy --progress --ignore-size --ignore-checksum --update --tpslimit 1 *****_****:"/Documenti condivisi/6. OPERATIONS/2. EPC Projects/2019" "D:\2. EPC Projects\2019" --log-file=C:\temp\SharePoint_logD.txt --log-level ERROR
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 40593 / 40593, 100%
Elapsed time: 3h8m53.7s

The copied files still 88,5 GB # 13534

What else can I do?

5

What should have happened? That appeared to exit with no errors?

6

These are all the errors for the run with tpslimt 1 (less then before ....)

2022/04/29 18:59:40 ERROR : Attempt 1/3 failed with 1 errors and: couldn't list files: 429 TOO MANY REQUESTS: 429 Too Many Requests
2022/04/29 19:28:07 ERROR : 0. Template/0. xxxxxxx - PROJECT NAME/02. Documents/13. Manuals: error reading source directory: couldn't list files: 429 TOO MANY REQUESTS: 429 Too Many Requests
2022/04/29 19:34:55 ERROR : F19CT001 - ARUP Chiesi/02. Documents/04. Offerts-PR-RDA/04.03 RDA/2019.02.08-Inverter/0.OLD: error reading source directory: couldn't list files: 429 TOO MANY REQUESTS: 429 Too Many Requests
2022/04/29 19:58:04 ERROR : F19CT001*** - ENI Porto Torres 2/02. Documents/04. Offerts-PER-RDA/04.02 PER/PER_Civil+EM/2. Integrazioni: error reading source directory: couldn't list files: 429 TOO MANY REQUESTS: 429 Too Many Requests
2022/04/29 19:58:10 ERROR : F19CT001*** - *** Porto Torres 2/01. Design/Topografia Impianto/Planimetria suddivisione aree: error reading source directory: couldn't list files: 429 TOO MANY REQUESTS: 429 Too Many Requests
2022/04/29 20:12:04 ERROR : Attempt 2/3 failed with 5 errors and: march failed with 4 error(s): first error: couldn't list files: 429 TOO MANY REQUESTS: 429 Too Many Requests
2022/04/29 21:09:04 ERROR : Attempt 3/3 succeeded

The command used:
rclone copy --progress --ignore-size --ignore-checksum --update --tpslimit 1 *****_****:"/Documenti condivisi/6. OPERATIONS/2. EPC Projects/2019" "D:\2. EPC Projects\2019" --log-file=C:\temp\SharePoint_logD.txt --log-level ERROR
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 40593 / 40593, 100%
Elapsed time: 3h8m53.7s

7

Looks like the 3rd run succeeded.

You can make the limit less than 1 eg --tpslimit 0.5

8

Now it's running with 0.5

But the dir in local disk still 88,5 GB
The dir in Sharepont 209
Nothing have been copied on the last run.
I can't understend why....

9

Post a full log with -vv and share it.

All the answers are in the debug log.

10

From you previous log, it looks like it never finished listing the directory, so it never found new files to copy

However the final run looked like it succeeded

So it should have worked then...

11

I can confirm all files are there.

Sharepoint 13538
local disk 13538

Not sure why Sharepoint web interface show more then 200 GB and localy I have 88 GB
I checked some big files and are consistent (not corrupted)

Seem ok, ty!

12

hi,

perhaps that includes files in the trashban/recyclebin.

13

Yes and/or files versioning ..

14

Checkout rclone cleanup and the no_versions flag in the onedrive backend.

15

Hi,

I got this new error:

2022/05/03 07:16:40 ERROR : contatori-importati: error reading source directory: couldn't list files: 413 Request Entity Too Large
2022/05/03 07:16:40 ERROR : contatori: error reading source directory: couldn't list files: 413 Request Entity Too Large
2022/05/03 07:17:14 ERROR : Attempt 1/3 failed with 3 errors and: march failed with 2 error(s): first error: couldn't list files: 413 Request Entity Too Large
2022/05/03 07:17:14 ERROR : contatori-importati: error reading source directory: couldn't list files: 413 Request Entity Too Large
2022/05/03 07:17:14 ERROR : contatori: error reading source directory: couldn't list files: 413 Request Entity Too Large
2022/05/03 07:17:14 ERROR : Attempt 2/3 failed with 3 errors and: march failed with 2 error(s): first error: couldn't list files: 413 Request Entity Too Large
2022/05/03 07:17:15 ERROR : contatori-importati: error reading source directory: couldn't list files: 413 Request Entity Too Large
2022/05/03 07:17:15 ERROR : contatori: error reading source directory: couldn't list files: 413 Request Entity Too Large

Have anyone any suggestion?

16

Sounds like the response to the directory listing is too large which is weird.

Do you have a very large directory?

Try reducing this to 500 say, --onedrive-list-chunk 500 and see if that helps. It should reduce the maximum size of the directory listing chunk.

  --onedrive-list-chunk int   Size of listing chunk (default 1000)
17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.