Python 在不爬网的情况下抓取网站；搜索引擎；页面的一节

最近，我一直在尝试开发一个网络爬虫，使用Scrapy从特定领域提取所有PDF文档，尽管我能以某种方式实现这一目标，但由于该网站的“搜索引擎”部分，它仍然陷入某种“无限循环”

我有什么办法可以避免这种行为吗

以下是到目前为止已经编写的爬虫程序的代码（如果是意大利面条，很抱歉，仍在努力改进..）：

我试图通过添加一个上限（n_迭代）来限制递归，即爬虫应该挖掘的深度。它的工作，但仍然，我想过滤的链接，它爬到一个适当的方式

非常感谢你帮助我，祝你一周愉快

class AirbusSpider(CrawlSpider):
    name = "test"
    tag = "aircraft"
    n_iterations = 3
    allowed_domains = ['airbus.com']

    start_urls = [
        'https://www.airbus.com'
    ]

def start_requests(self):
    for url in self.start_urls:
        yield scrapy.Request(url=url, callback=self.parse)

def parse(self, response):
    yield scrapy.Request(url=response.url+'/'+self.tag+'.html', callback=self.parse_item, meta={'iteration': self.n_iterations})


def parse_item(self, response):
    current_iteration = response.meta['iteration']
    if(current_iteration>0):

        temp_links = []
        valuable_links = []
        current_iteration-=1

        #all "somewhat-valuable" links to crawl into
        links = response.selector.xpath('//a[contains(@href,"aircraft")]/@href').extract() #all hrefs of current page

        for link in links:
            valuable_links.append(link)

        temp_links = set(valuable_links) 
        valuable_links = list(temp_links)

        for link in valuable_links:
            link = str(link.encode('utf-8'))
            if '.pdf' not in link:
                if(link[0] == '/' or link[0] == '#'):
                    yield scrapy.Request(url=self.start_urls[0]+link, callback=self.parse_item, meta={'iteration': current_iteration})
                else:
                    yield scrapy.Request(url=link, callback=self.parse_item, meta={'iteration': current_iteration})
            else:
                if(link[0] == '/'  or link[0] == '#'):
                    #increase the priority of these requests
                    yield scrapy.Request(url=self.start_urls[0]+link, callback=self.download_file, priority = 1)
                else:
                    yield scrapy.Request(url=link, callback=self.download_file, priority = 1)

def download_file(self, response):
    item = PdfLink()
    item['link'] = str(response.url)

    return item