Javascript 木偶绘制者：多用户登录和抓取

我正在使用Apify和PuppeterCrawler为多个用户刮取页面。我必须让每个用户登录到系统并刮取5个页面，然后注销并继续使用下一个用户

最好的方法是什么？为每个用户调用爬虫程序，或者只调用一次爬虫程序，让它处理登录/注销

---

我从Apify云中扩展了这个示例，并在Apify云中运行它。现在我正在修改request.userData对象，方法是在其上添加一个标签“login”，这样就可以初步处理登录案例。登录后，相关的5个页面将被删除。

我认为这两个选项都非常有效。拥有多个爬虫当然更简单，尽管在一个爬虫中完成所有工作可能更高效（因为您可以同时处理所有用户）。我主张从第一个选项开始，直到你更好地感觉到如何正确处理第二个选项

我提供的这个版本是最简单的，因为它假设您访问的页面会自动重定向到登录页面并从中重定向。如果不是这样，您只需要使用标签即可

    // Let's assume you have some object with your users.
    // This may in fact be loaded from input or somewhere else but for simplicity, let's define it right away
    const users = {
        stevejobs: {
            credentials: {
                username: 'stevejobs@gmail.com',
                password: '123',
            },
            cookies: null, // Cookies can be also loaded so you can use persistent login
        },
        billgates: {
            credentials: {
                username: 'billgates@gmail.com',
                password: '123',
            },
            cookies: null,
        },
        // etc...
    };

    const myUrls = ['https://resource1.com', 'https://resource2.com']; // replace with real URLs

    // initialize request queue
    const requestQueue = await Apify.openRequestQueue();

    // Now we will loop over the users and for each of them define a crawler and run it
    for (const user of Object.keys(users)) {

        // enqueue some pages
        for (const url of myUrls)
            await requestQueue.addRequest({
                url,
                uniqueKey: `${url}_${user}` // Otherwise the queue would dedup them
            });
        }

        const crawler = new Apify.PuppeteerCrawler({
            requestQueue,
            gotoFunction: async ({ page, request }) => {
                // if you have cookies, you simply add them to the page
                const { cookies } = users[user];
                if (cookies) {
                    await page.setCookie(...cookies);
                }
                return page.goto(request.url);
            },
            handlePageFunction: async ({ page, request }) => {
                // Check if you are logged in by some selector, if not log in
                const loggedIn = $('am-i-logged'); // Change to real selector
                if (!loggedIn) {
                    // log in with credentials
                    const { username, password } = users[user].credentials;
                    // do your login
                    // ...
                    // wait for redirect
                    // then we save cookies
                    const cookies = await page.cookies();
                    users[user].cookies = cookies;
                }
                // Usually the log in page will redirect directly to the resource so we can scrape data right away
                const data = scrapeData(); // replace with real function
                await Apify.pushData(data);
            }
        })
        await crawler.run();
    }