Javascript 如何避免使用ChereIO进行网页抓取时出现错误403_Javascript_Node.js_Web Scraping_Request_Cheerio

Javascript 如何避免使用ChereIO进行网页抓取时出现错误403

javascript node.js web-scraping

Javascript 如何避免使用ChereIO进行网页抓取时出现错误403,javascript,node.js,web-scraping,request,cheerio,Javascript,Node.js,Web Scraping,Request,Cheerio,我正在浏览一个网站，我有一系列链接： 'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Abercorn', 'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Longueuil', 'http://www.adventi

我正在浏览一个网站，我有一系列链接：

 'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Abercorn',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Longueuil',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Sainte-Anne-De-Bellevue',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Shawinigan',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Chateauguay',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Mont-Laurier',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Saint-Georges',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Sherbrooke',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Chicoutimi',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Montreal',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Saint-Henri-De-Levis',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Stukely-Sud',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Drummondville',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Montreal-Est',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Saint-Hubert',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Trois-Rivieres',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Gatineau',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Montreal-Nord',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Saint-Jerome',
  "http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Val-D'or",
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Granby',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Montreal-Ouest',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Saint-Lambert',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Verdun',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Lachine',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Quebec',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Saint-Laurent',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Warwick',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Lasalle',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Rigaud',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Saint-Leonard',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Westmount',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Laval',
  'http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Roxboro'

但当我请求时，其中一些链接返回错误403-禁止

Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Verdun
Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Granby
Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Saint-Lambert
Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Val-D'or
Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Lasalle
Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Laval
Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Warwick
Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Quebec
Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Westmount
Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Roxboro
Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Saint-Laurent
Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Rigaud
Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Saint-Leonard
Error null Forbidden http://www.adventistdirectory.org/SearchResults.aspx?CtryCode=CA&StateProv=QC&City=Lachine

当我使用链接较少的列表时，它工作得非常好

这是我的密码：

 const request = require('request');
  const cheerio = require('cheerio');
      
  function readChurches(cities){
        const churches = []
        for (let index = 0; index < cities[0].length; index++){
            const city = cities[0][index];
            churches.push(new Promise((resolve, reject) => {
                const church = [] 
                let options = {
                    url: city,
                    headers: {
                         'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'
                    }
                };
                
                request(options, (error, response, html) => {
                    if(!error && response.statusCode == 200) {
                        const $ = cheerio.load(html);
                        const $$ = cheerio.load($('table').find('tbody').eq(1).find('tr').eq(1).find('td').eq(1).html())
        
                        $$('a').each((i, el) => {
                            const item = $(el).attr('href')
                            if(item != undefined){
                                if(item.includes('ViewEntity')) {
                                    church.push(`http://www.adventistdirectory.org${item}`);
                                }    
                            }
                        });
                        resolve(church);
                    } else {
                        console.log('Error',error,response.statusMessage,city)
                        reject(error)
                    }
                });
            }))
        }
    
        return Promise.all(churches);
    }

我正在运行它：

async function run(){
    
        let provinces = []
        provinces.push(`http://www.adventistdirectory.org/BrowseStateProv.aspx?CtryCode=CA&StateProv=QC`)
        
        let cities = await readCities(provinces);
    
        const churches = await readChurches(cities);
        const stuff = await doStuff(churches)
        
        console.log('Churches: ', stuff);
    
        console.log('End')
        
    } catch (error) {
        console.log('Error', error)
    }
}

我在我的控制台上看到：

Promise { <pending> }
Promise {
  [
    'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=19653'
  ]
}
Promise { <pending> }
Promise {
  [
    'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=19637'
  ]
}
Promise {
  [
    'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=54633'
  ]
}
Promise {
  [
    'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=31155'
  ]
}
Promise {
  [
    'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15271'
  ]
}
Promise { <pending> }
Promise {
  [
    'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=30783'
  ]
}
Promise { <pending> }
Promise { <pending> }
Promise {
  [
    'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15265'
  ]
}
Promise {
  [
    'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15255'
  ]
}
Promise {
  [
    'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15251'
  ]
}
Promise { <pending> }
Promise {
  [
    'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15247'
  ]
}
Promise {
  [
    'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=19645'
  ]
}
Promise {
  [
    'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=32838'
  ]
}
Promise {
  [
    'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=29973'
  ]
}
Promise { <pending> }

Promise{}
允诺{
[
'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=19653'
]
}
承诺{}
允诺{
[
'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=19637'
]
}
允诺{
[
'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=54633'
]
}
允诺{
[
'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=31155'
]
}
允诺{
[
'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15271'
]
}
承诺{}
允诺{
[
'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=30783'
]
}
承诺{}
承诺{}
允诺{
[
'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15265'
]
}
允诺{
[
'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15255'
]
}
允诺{
[
'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15251'
]
}
承诺{}
允诺{
[
'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=15247'
]
}
允诺{
[
'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=19645'
]
}
允诺{
[
'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=32838'
]
}
允诺{
[
'http://www.adventistdirectory.org/ViewEntity.aspx?EntityID=29973'
]
}
承诺{}

它无法访问

console.log（'Churches:'，stuff）
发回的错误是您向其发出请求的服务器的选择，因此没有通用的方法来“避免”它。您可能发出的请求太快，并且由于使用了太多的带宽而被阻塞
使用Promise.all是指同时发出所有请求。您需要进行排序循环，以使请求一次一个地进行
因此，类似这样的方法可能会奏效：
const wait = async (time) =>
  new Promise((res, rej) => setTimeout(() => res(), time));

async function doStuff() {
  const results = [];
  for(let i in churches) {
    await wait(1000);
    const result = await churches[i];
    console.log(result);
    results.push(result);
  }
}

发回的错误是您选择向其发出请求的服务器，因此没有通用的方法来“避免”它。您可能发出的请求太快，并且由于使用了太多的带宽而被阻塞
使用Promise.all是指同时发出所有请求。您需要进行排序循环，以使请求一次一个地进行
因此，类似这样的方法可能会奏效：
const wait = async (time) =>
  new Promise((res, rej) => setTimeout(() => res(), time));

async function doStuff() {
  const results = [];
  for(let i in churches) {
    await wait(1000);
    const result = await churches[i];
    console.log(result);
    results.push(result);
  }
}

您可能会受到速率限制（尽管这不是正确的状态代码）-尝试在请求之间睡觉？我已经这样做了，但没有work@BrunoLopesBacelar你试过我的建议一次提出一个要求吗？这可能还是太快了，你可能不得不进一步限制它。当你自己只做一次请求时，请求是否有效？@chrispytoes我在描述中添加了更多信息。用你的想法是行不通的。我仍然没有得到回应。@BrunoLopesBacelar你记录的是实际的承诺，而不是结果。检查我的更新答案。你可能会受到速率限制（虽然这不是正确的状态代码）-尝试在请求之间睡觉？我已经做了，但没有work@BrunoLopesBacelar你试过我的建议一次提出一个要求吗？这可能还是太快了，你可能不得不进一步限制它。当你自己只做一次请求时，请求是否有效？@chrispytoes我在描述中添加了更多信息。用你的想法是行不通的。我仍然没有得到回应。@BrunoLopesBacelar你记录的是实际的承诺，而不是结果。检查我的更新答案。谢谢，我只需要在我的readChurches函数中添加wait wait（1000）就可以了。再次感谢。谢谢，我只需要在我的readChurches函数中添加wait wait（1000）就行了。再次感谢。
const wait = async (time) =>
  new Promise((res, rej) => setTimeout(() => res(), time));

async function doStuff() {
  const results = [];
  for(let i in churches) {
    await wait(1000);
    const result = await churches[i];
    console.log(result);
    results.push(result);
  }
}