Is there a flag for IBM COS S3 to move from bucket to another without DOWN and UP again? Similar to GDrive --drive-server-side-across-configs.
Thanks
rclone supports server-side-moves between buckets on S3. no flags needed.
when you posted there was a template of questions for you to answer???
No matter, moving between two buckets from same account or different accounts.
I couldn't find any related topic to IBM COS server-side move.
This is the output (It stucks like this):
rclone move COSUP_1:cml1 COSUP_1:cml1-1 --transfers 5 --s3-chunk-size 1024M --no-traverse --ignore-existing -P
Transferred: 0 B / 154.435 TiB, 0%, 0 B/s, ETA -
Checks: 0 / 5, 0%
Transferred: 0 / 1560, 0%
Elapsed time: 1m24.5s
Checking:
Transferring:
- 00151qlol0254hld10cbsn…7h54ab8nnf4as7hn8gsmfg: 0% /10.375Gi, 0/s, -
- 00ivtik940g8gsjt74boai…t6kq5tbg6ctu6medl9smus: 0% /10.366Gi, 0/s, -
- 013kp5qvpiugcc47s9tugn…e3f4frndqim443ash4s9hk: 0% /10.383Gi, 0/s, -
- 016emgvjaeg2hk5etrb0eu…l2t6v8jqscnfb7qcj96pbk: 0% /10.387Gi, 0/s, -
- 02al7mu8i0i0l63mo69gab…13kvhltlk98m1madfkprqg: 0% /10.386Gi, 0/s, -
it takes time to transfer the files....
just rclone move -vv a single small file, then you will understand how it all works.
Thank you for replying.
Actually, I am comparing with G Drive move. G Drive is superfast in moving hundreds of large files. After some research, I realized that S3 based providers do not support server-side move but copy. That must be the case.
This is the case of two buckets from different accounts, It downloads and upload to the second bucket. The thing I am looking for is to move hundreds of files in seconds:
rclone move COSUP_1:cml1 COSUP_2:cml2 --transfers 5 --s3-chunk-size 1024M --no-traverse --ignore-existing -P -vv
2024/01/03 21:31:04 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "move" "COSUP_1:cml1" "COSUP_2:cml2" "--transfers" "5" "--s3-chunk-size" "1024M" "--no-traverse" "--ignore-existing" "-P" "-vv"]
2024/01/03 21:31:04 DEBUG : Creating backend with remote "COSUP_1:cml1"
2024/01/03 21:31:04 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/01/03 21:31:04 DEBUG : COSUP_1: detected overridden config - adding "{kysdd}" suffix to name
2024/01/03 21:31:04 DEBUG : fs cache: renaming cache item "COSUP_1:cml1" to be canonical "COSUP_1{kysdd}:cml1"
2024/01/03 21:31:04 DEBUG : Creating backend with remote "COSUP_2:cml2"
2024/01/03 21:31:04 DEBUG : COSUP_2: detected overridden config - adding "{kysdd}" suffix to name
2024/01/03 21:31:04 DEBUG : fs cache: renaming cache item "COSUP_2:cml2" to be canonical "COSUP_2{kysdd}:cml2"
2024-01-03 21:31:30 DEBUG : S3 bucket cml2: Waiting for checks to finish
2024-01-03 21:31:30 DEBUG : S3 bucket cml2: Waiting for transfers to finish
Transferred: 466.980 MiB / 15.534 TiB, 0%, 49.223 MiB/s, ETA 5w3d2h26m8s
Checks: 0 / 5, 0%
Transferred: 0 / 1561, 0%
Elapsed time: 36.0s
Checking:
Transferring:
- 00ivtik940g8gsjt74boai…t6kq5tbg6ctu6medl9smus: 0% /10.366Gi, 12.107Mi/s, 32m44s
- 013kp5qvpiugcc47s9tugn…e3f4frndqim443ash4s9hk: 0% /10.383Gi, 18.374Mi/s, 34m1s
- 00151qlol0254hld10cbsn…7h54ab8nnf4as7hn8gsmfg: 0% /10.375Gi, 10.750Mi/s, 37m47s
- 016emgvjaeg2hk5etrb0eu…l2t6v8jqscnfb7qcj96pbk: 0% /10.387Gi, 6.499Mi/s, 26m5s
- 02al7mu8i0i0l63mo69gab…13kvhltlk98m1madfkprqg: 0% /10.386Gi, 5.123Mi/s, 37m36s
rclone simulates move as server-side-copy then delete.
that should not be the case with S3.
so on the machine that rclone is running on, rclone uses GiB+ of memory and GiB+ of network bandwidth?
if IBM S3 does not support server-side-copy, we can update the rclone docs...
just rclone move -vv on a single small file, then we would know for sure.
can you please post the full debug output.
also, can you post the output of rclone backend features COSUP_1:
rclone backend features COSUP_1 :
rclone backend features COSUP_1:
{
"Name": "COSUP_1",
"Root": "",
"String": "S3 root",
"Precision": 1,
"Hashes": [
"md5"
],
"Features": {
"About": false,
"BucketBased": true,
"BucketBasedRootOK": true,
"CanHaveEmptyDirectories": false,
"CaseInsensitive": false,
"ChangeNotify": false,
"CleanUp": true,
"Command": true,
"Copy": true,
"DirCacheFlush": false,
"DirMove": false,
"Disconnect": false,
"DuplicateFiles": false,
"FilterAware": false,
"GetTier": true,
"IsLocal": false,
"ListR": true,
"MergeDirs": false,
"Move": false,
"OpenWriterAt": false,
"PublicLink": true,
"Purge": true,
"PutStream": true,
"PutUnchecked": false,
"ReadMetadata": true,
"ReadMimeType": true,
"ServerSideAcrossConfigs": false,
"SetTier": true,
"SetWrapper": false,
"Shutdown": false,
"SlowHash": false,
"SlowModTime": true,
"UnWrap": false,
"UserInfo": false,
"UserMetadata": true,
"WrapFs": false,
"WriteMetadata": true,
"WriteMimeType": true
},
"MetadataInfo": {
"System": {
"btime": {
"Help": "Time of file birth (creation) read from Last-Modified header",
"Type": "RFC 3339",
"Example": "2006-01-02T15:04:05.999999999Z07:00",
"ReadOnly": true
},
"cache-control": {
"Help": "Cache-Control header",
"Type": "string",
"Example": "no-cache",
"ReadOnly": false
},
"content-disposition": {
"Help": "Content-Disposition header",
"Type": "string",
"Example": "inline",
"ReadOnly": false
},
"content-encoding": {
"Help": "Content-Encoding header",
"Type": "string",
"Example": "gzip",
"ReadOnly": false
},
"content-language": {
"Help": "Content-Language header",
"Type": "string",
"Example": "en-US",
"ReadOnly": false
},
"content-type": {
"Help": "Content-Type header",
"Type": "string",
"Example": "text/plain",
"ReadOnly": false
},
"mtime": {
"Help": "Time of last modification, read from rclone metadata",
"Type": "RFC 3339",
"Example": "2006-01-02T15:04:05.999999999Z07:00",
"ReadOnly": false
},
"tier": {
"Help": "Tier of the object",
"Type": "string",
"Example": "GLACIER",
"ReadOnly": true
}
},
"Help": "User metadata is stored as x-amz-meta- keys. S3 metadata keys are case insensitive and are always returned in lower case."
}
}
Moving a single file speed is 15Mi/s:
rclone move COSUP_1:cml1/u620uk17dakla8a3nngbq686m4e2rrn835hep3k9cfml2ikfbpvll8n0r08dui6f9rmcg46t19pe42k4vhd980bdi81sj0hpqslfv2cprloqfc8bl6if9r2ip7h0ss91t6q0tkls5pqd2faotka31lkktg COSUP_2:cml2 --transfers 5 --s3-chunk-size 1024M --no-traverse --ignore-existing -P -vv
2024/01/03 22:34:15 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "move" "COSUP_1:cml1/u620uk17dakla8a3nngbq686m4e2rrn835hep3k9cfml2ikfbpvll8n0r08dui6f9rmcg46t19pe42k4vhd980bdi81sj0hpqslfv2cprloqfc8bl6if9r2ip7h0ss91t6q0tkls5pqd2faotka31lkktg" "COSUP_2:cml2" "--transfers" "5" "--s3-chunk-size" "1024M" "--no-traverse" "--ignore-existing" "-P" "-vv"]
2024/01/03 22:34:15 DEBUG : Creating backend with remote "COSUP_1:cml1/u620uk17dakla8a3nngbq686m4e2rrn835hep3k9cfml2ikfbpvll8n0r08dui6f9rmcg46t19pe42k4vhd980bdi81sj0hpqslfv2cprloqfc8bl6if9r2ip7h0ss91t6q0tkls5pqd2faotka31lkktg"
2024/01/03 22:34:15 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/01/03 22:34:15 DEBUG : COSUP_1: detected overridden config - adding "{kysdd}" suffix to name
2024/01/03 22:34:16 DEBUG : fs cache: adding new entry for parent of "COSUP_1:cml1/u620uk17dakla8a3nngbq686m4e2rrn835hep3k9cfml2ikfbpvll8n0r08dui6f9rmcg46t19pe42k4vhd980bdi81sj0hpqslfv2cprloqfc8bl6if9r2ip7h0ss91t6q0tkls5pqd2faotka31lkktg", "COSUP_1{kysdd}:cml1"
2024/01/03 22:34:16 DEBUG : Creating backend with remote "COSUP_2:cml2"
2024/01/03 22:34:16 DEBUG : COSUP_2: detected overridden config - adding "{kysdd}" suffix to name
2024/01/03 22:34:16 DEBUG : fs cache: renaming cache item "COSUP_2:cml2" to be canonical "COSUP_2{kysdd}:cml2"
2024-01-03 22:34:16 DEBUG : u620uk17dakla8a3nngbq686m4e2rrn835hep3k9cfml2ikfbpvll8n0r08dui6f9rmcg46t19pe42k4vhd980bdi81sj0hpqslfv2cprloqfc8bl6if9r2ip7h0ss91t6q0tkls5pqd2faotka31lkktg: Need to transfer - File not found at Destination
Transferred: 500.996 MiB / 10.378 GiB, 0%, 14.549 MiB/s, ETA 18m20s
Checks: 0 / 1, 0%
Transferred: 0 / 1, 0%
Elapsed time: 36.2s
Checking:
Transferring:
* u620uk17dakla8a3nngbq6…tkls5pqd2faotka31lkktg: 0% /10.378Gi, 14.678Mi/s, 10m17s
As I understand, it is not possible to move large number of huge files as I do in GDrive. Right?
sorry getting confused, did you change the source and dest remotes in that last command?
before, the source and dest were the same remote
now, the source and dest are not the same remote
-
server-side will not work when the source and dest remotes are not the same.
-
server-side should work when the source and dest remotes are the same.
maybe with IBM S3 will not work.
if you would post the full output of something like this, on a very small sized file.
rclone move COSUP_1:cml1/small.txt COSUP_1:cml1-1 -vv
really, that is the most important bit of information.
server-side or not, rclone can move the files.
but as you can see, it can be a long slow process.
It was from bucket 1 to bucket 2 on the same remote. The others have different remotes.
It moves small files:
rclone move COSUP_1:cml1/image1.png COSUP_2:cml2 --transfers 5 --s3-chunk-size 1024M --no-traverse --ignore-existing -P -vv
2024/01/03 22:57:21 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "move" "COSUP_1:cml1/image1.png" "COSUP_2:cml2" "--transfers" "5" "--s3-chunk-size" "1024M" "--no-traverse" "--ignore-existing" "-P" "-vv"]
2024/01/03 22:57:21 DEBUG : Creating backend with remote "COSUP_1:cml1/image1.png"
2024/01/03 22:57:21 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/01/03 22:57:21 DEBUG : COSUP_1: detected overridden config - adding "{kysdd}" suffix to name
2024/01/03 22:57:22 DEBUG : fs cache: adding new entry for parent of "COSUP_1:cml1/image1.png", "COSUP_1{kysdd}:cml1"
2024/01/03 22:57:22 DEBUG : Creating backend with remote "COSUP_2:cml2"
2024/01/03 22:57:22 DEBUG : COSUP_2: detected overridden config - adding "{kysdd}" suffix to name
2024/01/03 22:57:22 DEBUG : fs cache: renaming cache item "COSUP_2:cml2" to be canonical "COSUP_2{kysdd}:cml2"
2024-01-03 22:57:23 DEBUG : image1.png: Need to transfer - File not found at Destination
2024-01-03 22:57:24 DEBUG : image1.png: md5 = 2f414221e9377915de7888b503b0b892 OK
2024-01-03 22:57:24 INFO : image1.png: Copied (new)
2024-01-03 22:57:24 INFO : image1.png: Deleted
Transferred: 501.406 KiB / 501.406 KiB, 100%, 501.358 KiB/s, ETA 0s
Checks: 2 / 2, 100%
Deleted: 1 (files), 0 (dirs)
Renamed: 1
Transferred: 1 / 1, 100%
Elapsed time: 2.9s
2024/01/03 22:57:24 INFO :
Transferred: 501.406 KiB / 501.406 KiB, 100%, 501.358 KiB/s, ETA 0s
Checks: 2 / 2, 100%
Deleted: 1 (files), 0 (dirs)
Renamed: 1
Transferred: 1 / 1, 100%
Elapsed time: 2.9s
2024/01/03 22:57:24 DEBUG : 7 go routines active
As I said before, I am looking for instantly moving large numbers of huge files as I do in GDrive. In the IBM case It downloads then uploads to the destination unfortunately
in my testing, instant is not possible with S3. except maybe for wasabi.
if the source and dest are the same remote, then server-side-copy should work.
that has been the case with all the S3 providers i have tested and used with rclone.
so i would be surprised that is not the case with IBM S3
with amazon and google, can rent a super cheap cloud vm in the same region as the bucket.
i would imagine that ibm offers the same.
and in that case, not charged for egress, not using local machine, local internet, etc...
with that large amount of data to move, i would contact IBM tech support.
there might be a way.
Thank you for your advice. This looks the only option for huge data migration within IBM cloud.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.