Web crawler 如何使用nutch 1.2对特定URL进行爬网
我正在使用nutch-1.2,但无法将配置文件限制为仅对给定URL进行爬网Web crawler 如何使用nutch 1.2对特定URL进行爬网,web-crawler,nutch,Web Crawler,Nutch,我正在使用nutch-1.2,但无法将配置文件限制为仅对给定URL进行爬网 我的crawl-urlfilter.txt文件是 # Each non-comment, non-blank line contains a regular expression # prefixed by '+' or '-'. The first matching pattern in the file # determines whether a URL is included or ign
我的crawl-urlfilter.txt文件是
# Each non-comment, non-blank line contains a regular expression
# prefixed by '+' or '-'. The first matching pattern in the file
# determines whether a URL is included or ignored. If no pattern
# matches, the URL is ignored.
# skip file:, ftp:, & mailto: urls
-^(file|ftp|mailto):
# skip image and other suffixes we can't yet parse
-\.(gif|GIF|js|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$
# skip URLs containing certain characters as probable queries, etc.
-[?*!@=]
# skip URLs with slash-delimited segment that repeats 3+ times, to break loops
-.*(/[^/]+)/[^/]+\1/[^/]+\1/
# accept hosts in MY.DOMAIN.NAME
-.*
+^http://([a-z0-9]*\.)abc.com/[0-9].*[a-z]*.html$
-^http://([a-z0-9]*\.)*abc.com/[-a-z/]*/$
-^http://([a-z0-9]*\.)*abc.com/[-a-z]*.php$
谢谢我想我在什么地方读到过,这个配置是自上而下评估的 自从
-.
匹配所有内容-nutch将忽略进一步的条目。我的crawl-urlfilter.txt在底部看起来与之类似(按顺序排列),并且运行得非常好:
# accept hosts in MY.DOMAIN.NAME
+^http://localhost:8080/
# skip everything else
-.
感谢您的快速回复,我已经尝试了您的建议,但仍然不起作用。我必须更改任何其他配置文件。