Google cloud storage 使用gsutil rsync的校验和

我正在使用gsutil rsync从谷歌云存储下载大量公共数据文件。偶尔会有一些文件下载失败。为了确保我拥有所有请求的文件，我在启用校验和的情况下再次运行gsutil。在第二次运行期间，gsutil报告它正在为下载的文件计算校验和。我在下面附上了一些示例输出。在本例中，它在第一次rsync期间下载了29个文件，但仅报告它在第二次rsync期间正在计算16个文件的校验和

gsutil是否没有计算校验和并对某些文件执行rsync，或者只是没有报告正在执行校验和

肯

---

gsutil rsync-c

仅在源和目标大小匹配时计算校验和。这节省了时间，因为不需要计算校验和来确定需要下载的大小匹配文件。

请将日志缩短到相关行。这一屏幕的输出在我看来是非常重复的，我认为这是另一种方式。它仅在源和目标大小与mtime匹配时计算校验和。如果它们不匹配，它将始终重新复制文件。如果它们确实匹配，它将进行校验和以确认它们是相同的。（来自gsutil帮助：-c导致rsync命令在源和目标的大小以及mtime（如果可用）匹配的情况下计算和比较文件的校验和（而不是比较mtime）。抱歉，谢谢您的更正。我已经更新了我的答案。

mix> gsutil -m rsync -R -P gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX /csrpc1/NEXRAD/level2/2017/201702/20170201/KHGX
Building synchronization state...
Starting synchronization
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131210000_20170131215959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131190000_20170131195959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131230000_20170131235959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131200000_20170131205959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131220000_20170131225959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201040000_20170201045959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201090000_20170201095959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201020000_20170201025959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201010000_20170201015959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201030000_20170201035959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201180000_20170201185959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201130000_20170201135959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201060000_20170201065959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201100000_20170201105959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201110000_20170201115959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201120000_20170201125959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201170000_20170201175959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201150000_20170201155959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201160000_20170201165959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201050000_20170201055959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201000000_20170201005959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201080000_20170201085959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201070000_20170201075959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201140000_20170201145959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201200000_20170201205959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201190000_20170201195959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201210000_20170201215959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201220000_20170201225959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201230000_20170201235959.tar...
- [29/29 files][387.3 MiB/387.3 MiB] 100% Done                                  
Operation completed over 29 objects/387.3 MiB.                                   
mix> gsutil -m rsync -R -P -c gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX /csrpc1/NEXRAD/level2/2017/201702/20170201/KHGX
Building synchronization state...
Starting synchronization
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131190000_20170131195959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131200000_20170131205959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131210000_20170131215959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131220000_20170131225959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201000000_20170201005959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201010000_20170201015959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201020000_20170201025959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201030000_20170201035959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201040000_20170201045959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201050000_20170201055959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201160000_20170201165959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201170000_20170201175959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201180000_20170201185959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201190000_20170201195959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201200000_20170201205959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201210000_20170201215959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201210000_20170201215959.tar...
mix>