Warning: file_get_contents(/data/phpspider/zhask/data//catemap/4/unix/3.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Unix 在linux中使用hadoop fsck命令时是否可以跳过文件检查?_Unix_Hadoop_Hdfs - Fatal编程技术网

Unix 在linux中使用hadoop fsck命令时是否可以跳过文件检查?

Unix 在linux中使用hadoop fsck命令时是否可以跳过文件检查?,unix,hadoop,hdfs,Unix,Hadoop,Hdfs,我想使用hadoop fsck命令跳过指定路径上的文件检查。我们能做到吗? 我正在使用以下命令: hadoop fsck>/output.txt 我也检查了hdfs指南,但是没有什么可以从上面的命令中排除路径 请提供帮助。从hadoop 2.9.0开始,没有办法在hadoop fsck命令中指定排除路径 但是您可以使用WebHDFS REST API获得与fsck相同的文件系统健康信息。使用此API,我们可以使用LISTSTATUS API获取目录中所有文件的信息,或者使用GETFILESTAT

我想使用hadoop fsck命令跳过指定路径上的文件检查。我们能做到吗? 我正在使用以下命令:

hadoop fsck>/output.txt

我也检查了hdfs指南,但是没有什么可以从上面的命令中排除路径


请提供帮助。

从hadoop 2.9.0开始,没有办法在hadoop fsck命令中指定排除路径

但是您可以使用WebHDFS REST API获得与fsck相同的文件系统健康信息。使用此API,我们可以使用LISTSTATUS API获取目录中所有文件的信息,或者使用GETFILESTATUS API获取单个文件的信息

对于目录:

curl -i  "http://<HOST>:<PORT>/webhdfs/v1/<DIRECTORY_PATH>?op=LISTSTATUS"
curl-i“http://:/webhdfs/v1/?op=LISTSTATUS”
档案:

curl -i  "http://<HOST>:<PORT>/webhdfs/v1/<FILE_PATH>?op=GETFILESTATUS"
curl-i“http://:/webhdfs/v1/?op=GETFILESTATUS”
它们将返回带有FileStatuses JSON对象的响应

请在下面找到NN为目录返回的响应示例:

curl -i "http://<NN_HOST>:<HTTP_PORT>/webhdfs/v1/<DIRECTORY_PATH>?op=LISTSTATUS"
HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26.hwx)

{"FileStatuses":{"FileStatus":[
{"accessTime":1489059994224,"blockSize":134217728,"childrenNum":0,"fileId":209158298,"group":"hdfs","length":0,"modificationTime":1489059994227,"owner":"XXX","pathSuffix":"_SUCCESS","permission":"644","replication":3,"storagePolicy":0,"type":"FILE"},
{"accessTime":1489059969939,"blockSize":134217728,"childrenNum":0,"fileId":209158053,"group":"hdfs","length":0,"modificationTime":1489059986846,"owner":"XXX","pathSuffix":"part-m-00000","permission":"644","replication":3,"storagePolicy":0,"type":"FILE"},
{"accessTime":1489059982614,"blockSize":134217728,"childrenNum":0,"fileId":209158225,"group":"hdfs","length":0,"modificationTime":1489059993497,"owner":"XXX","pathSuffix":"part-m-00001","permission":"644","replication":3,"storagePolicy":0,"type":"FILE"},
{"accessTime":1489059977524,"blockSize":134217728,"childrenNum":0,"fileId":209158188,"group":"hdfs","length":0,"modificationTime":1489059983034,"owner":"XXX","pathSuffix":"part-m-00002","permission":"644","replication":3,"storagePolicy":0,"type":"FILE"}]}}
curl-i“http://:/webhdfs/v1/?op=LISTSTATUS”
HTTP/1.1200ok
缓存控制:没有缓存
内容类型:application/json
传输编码:分块
服务器:Jetty(6.1.26.hwx)
{“文件状态”:{“文件状态”:[
{“accessTime”:1489059994224,“blockSize”:134217728,“childrenNum”:0,“fileId”:209158298,“group”:“hdfs”,“length”:0,“modificationTime”:1489059994227,“owner”:“XXX”,“pathSuffix”:“_SUCCESS”,“permission”:“644”,“replication”:3,“storagePolicy”:0,“type”:“FILE”},
{“accessTime”:1489059969939,“blockSize”:134217728,“childrenNum”:0,“fileId”:209158053,“group”:“hdfs”,“length”:0,“modificationTime”:1489059986846,“owner”:“XXX”,“pathSuffix”:“part-m-00000”,“permission”:“644”,“replication”:3,“storagePolicy”:0,“type”:“FILE”},
{“accessTime”:1489059982614,“blockSize”:134217728,“childrenNum”:0,“fileId”:209158225,“group”:“hdfs”,“length”:0,“modificationTime”:1489059993497,“owner”:“XXX”,“pathSuffix”:“part-m-00001”,“permission”:“644”,“replication”:3,“storagePolicy”:0,“type”:“FILE”},
{“accessTime”:1489059977524,“blockSize”:134217728,“childrenNum”:0,“fileId”:209158188,“group”:“hdfs”,“length”:0,“modificationTime”:1489059983034,“owner”:“XXX”,“pathSuffix”:“part-m-00002”,“permission”:“644”,“replication”:3,“storagePolicy”:0,“type”:“FILE”}}}