Javascript 在不使用模块的情况下从另一个网站抓取数据
我正在尝试使用node.js和 使用此后端代码Javascript 在不使用模块的情况下从另一个网站抓取数据,javascript,node.js,web-scraping,Javascript,Node.js,Web Scraping,我正在尝试使用node.js和 使用此后端代码 import { fetch } from 'wix-fetch'; export function fetchData() { let url = 'https://www.brainyquote.com/topics/hacker'; let option = { "method": "GET" } return fetch(url, option) .then(result => { return result.te
import { fetch } from 'wix-fetch';
export function fetchData() {
let url = 'https://www.brainyquote.com/topics/hacker';
let option = {
"method": "GET"
}
return fetch(url, option)
.then(result => {
return result.text();
}).catch(reason => {
return reason;
})
}
fetchData().then(function (result) {
console.log(result);
})
但是我想要的是只获取引号，然后将它们添加到我的数据库集合中，而不使用像这样的模块 首先，你在后端做的事情可以在前端完成。但实际上，我们不需要在前端发生任何事情。前端应该只接收来自后端的报价。刮取和保存到数据库只能在后端进行 Cron job启动scraper->scraper完成他的工作并将被盗的内容保存到DB->server使用端点服务内容 你真的需要像
cheeriophantom.js- 使用任何可以轻松从整个页面提取html元素的工具创建一个scraper。此脚本应连接到数据库并将项目保存在那里
- 例如，使用
每N小时/分钟运行一次刮板节点cron - 在服务器上创建提供这些引用的端点
您的前端不应该参与任何类型的刮削或发射刮刀。它应该只是显示数据。 刮削是一件坏事，因为你基本上窃取属于其他作者的内容，所以也许你应该考虑找到提供相似内容的API。 但如果你真的想刮-这里有一个小教程为你刮 首先，你在后端做的事情可以在前端完成。但实际上，我们不需要在前端发生任何事情。前端应该只接收来自后端的报价。刮取和保存到数据库只能在后端进行 Cron job启动scraper->scraper完成他的工作并将被盗的内容保存到DB->server使用端点服务内容 你真的需要像
cheeriophantom.js- 使用任何可以轻松从整个页面提取html元素的工具创建一个scraper。此脚本应连接到数据库并将项目保存在那里
- 例如，使用
每N小时/分钟运行一次刮板节点cron - 在服务器上创建提供这些引用的端点
您的前端不应该参与任何类型的刮削或发射刮刀。它应该只显示数据。如果没有cheerio，您将不得不自己解析HTML内容，这将非常痛苦。您必须分析HTML响应，将响应加载到字符串中，然后使用regex或其他方法解析出所需的部分 下面是一些如何使用regex执行此操作的示例：
如果没有cheerio，您将不得不自己解析HTML内容，这将是一件痛苦的事情。您必须分析HTML响应，将响应加载到字符串中，然后使用regex或其他方法解析出所需的部分 下面是一些如何使用regex执行此操作的示例： 使用wix的解决方案： 将返回：
[
'Very smart people are often tricked by hackers, by phishing. I don't exclude myself from that. It's about being smarter than a hacker. Not about being smart.',
'Harper Reed',
'I'm a hacker, but I'm the good kind of hackers. And I've never been a criminal.',
'Mikko Hypponen',
'At the end of the day, my goal was to be the best hacker.',
'Kevin Mitnick',
'Software Engineering might be science; but that's not what I do. I'm a hacker, not an engineer.',
'Jamie Zawinski',
'If you give a hacker a new toy, the first thing he'll do is take it apart to figure out how it works.',
'Jamie Zawinski',
'Social engineering has become about 75% of an average hacker's toolkit, and for the most successful hackers, it reaches 90% or more.',
'John McAfee',
'I'm a really good hacker, but I'm not a sensible person.',
'Richard D. James',
'A hacker is someone who uses a combination of high-tech cybertools and social engineering to gain illicit access to someone else's data.',
'John McAfee',
'The hacker mindset doesn't actually see what happens on the other side, to the victim.',
'Kevin Mitnick',
'I look like a geeky hacker, but I don't know anything about computers.',
'Justin Long',
'The hacker community may be small, but it possesses the skills that are driving the global economies of the future.',
'Heather Brooke',
'I'm a bit of a hacker fanatic and know a fair bit about that industry and cyber crime and cyber warfare.',
'Seth Gordon',
'It's true, I had hacked into a lot of companies, and took copies of the source code to analyze it for security bugs. If I could locate security bugs, I could become better at hacking into their systems. It was all towards becoming a better hacker.',
'Kevin Mitnick',
'It's not enough to have a hacker culture anymore. You have to have a design culture, too.',
'Robert Scoble',
'If you go to a coffee shop or at the airport, and you're using open wireless, I would use a VPN service that you could subscribe for 10 bucks a month. Everything is encrypted in an encryption tunnel, so a hacker cannot tamper with your connection.',
'Kevin Mitnick',
'I wasn't a hacker for the money, and it wasn't to cause damage.',
'Kevin Mitnick',
'I'm not an economist; I'm a hacker who has spent his career exploring and repairing large networks.',
'Dan Kaminsky',
'In the '80s, society created a caricature of what a hacker or a programmer looked like: a guy wearing a hoodie, drinking energy drinks, sitting in a basement somewhere coding. Today, programmers look like the men we see in the show 'Silicon Valley' on HBO. If you look at the message girls are getting, it's saying, 'This is not for you.'',
'Reshma Saujani',
'I don't condone anyone causing damage in my name, or doing anything malicious in support of my plight. There are more productive ways to help me. As a hacker myself, I never intentionally damaged anything.',
'Kevin Mitnick',
'I think Linux is a great thing, in the big picture. It's a great hacker's tool, and it has a lot of potential to become something more.',
'Jamie Zawinski',
'Bitcoin is here to stay. There would be a hacker uproar to anyone who attempted to take credit for the patent of cryptocurrency. And I wouldn't want to be on the receiving end of hacker fury.',
'Adam Draper',
'It was on a bulletin board that I first learned about hacker culture, the 'Let's just break through this wall and see what's on the other side' mentality.',
'Harper Reed',
'Everything about Mark Zuckerberg is pure hacker. Hackers don't take realities of the world for granted; they seek to break and rebuild what they don't like. They seek to outsmart the world.',
'Sarah Lacy',
'If you're a juvenile delinquent today, you're a hacker. You live in your parent's house; they haven't seen you for two months. They put food outside your door, and you're shutting down a government of a foreign country from your computer.',
'John Waters',
'The key to social engineering is influencing a person to do something that allows the hacker to gain access to information or your network.',
'Kevin Mitnick',
'A smartphone links patients' bodies and doctors' computers, which in turn are connected to the Internet, which in turn is connected to any smartphone anywhere. The new devices could put the management of an individual's internal organs in the hands of every hacker, online scammer, and digital vandal on Earth.',
'Charles C. Mann'
]
const数据=[“非常聪明的人经常被黑客、网络钓鱼所欺骗。我不排斥自己。这是关于比黑客更聪明。不是关于聪明。”，“哈珀·里德”，“我是黑客，但我是黑客中的好人。我从来都不是罪犯。”，“米科·海波宁”，最后，我的目标是成为最好的黑客呃，“‘凯文·米尼克’，‘软件工程可能是科学；但我不是这样做的。我是黑客，不是工程师’，‘杰米·扎温斯基’，‘如果你给一个黑客一个新玩具，他要做的第一件事就是把它拆开，弄清楚它是如何工作的’，‘杰米·扎温斯基’，‘社会工程已经成为一个普通黑客75%的工具箱，以及对于最成功的黑客来说，这一比例达到90%或更高。、“约翰·麦卡菲”、“我是一个非常优秀的黑客，但我不是一个明智的人。”、“理查德·詹姆斯”、“黑客是一个利用高科技网络工具和社会工程相结合非法访问他人数据的人。”、“约翰·麦卡菲”、“黑客思维方式不是一个明智的人。”“我看起来像一个极客黑客，但我对计算机一无所知”，“贾斯汀·朗”，“黑客社区可能很小，但它拥有推动未来全球经济发展的技能”，“希瑟·布鲁克”，“我是一个黑客狂热分子，知道一个f关于那个行业、网络犯罪和网络战争的广播节目。“Seth Gordon”、“It'；是真的，我入侵了很多公司，并复制了源代码来分析它的安全漏洞。如果我能找到安全漏洞，我就可以更好地入侵他们的系统。这都是为了成为一个更好的黑客。”“Kevin Mitnick”有黑客文化是不够的。你也必须有设计文化。，“罗伯特·斯科布尔”，“如果你去咖啡馆或机场，你使用开放无线，我会使用VPN服务，你可以每月订阅10美元。所有东西都在加密隧道中加密，所以黑客不能篡改你的电脑。”“你的关系”，“凯文·米尼克”，“我不是为了钱而做的黑客，也不是为了引起da的注意。”
const https = require('https');
https.get('https://www.brainyquote.com/topics/hacker', (res) => {
console.log('statusCode:', res.statusCode);
console.log('headers:', res.headers);
const data = [];
res.on('data', (d) => {
data.push(d);
});
res.on('end', ()=>{
const result = data
.join("")
.match(/(?<=title="(view quote|view author)">)(.*?)(?=<\/a>)/g)
console.log(result);
});
}).on('error', (e) => {
console.error(e);
});
[
'Very smart people are often tricked by hackers, by phishing. I don't exclude myself from that. It's about being smarter than a hacker. Not about being smart.',
'Harper Reed',
'I'm a hacker, but I'm the good kind of hackers. And I've never been a criminal.',
'Mikko Hypponen',
'At the end of the day, my goal was to be the best hacker.',
'Kevin Mitnick',
'Software Engineering might be science; but that's not what I do. I'm a hacker, not an engineer.',
'Jamie Zawinski',
'If you give a hacker a new toy, the first thing he'll do is take it apart to figure out how it works.',
'Jamie Zawinski',
'Social engineering has become about 75% of an average hacker's toolkit, and for the most successful hackers, it reaches 90% or more.',
'John McAfee',
'I'm a really good hacker, but I'm not a sensible person.',
'Richard D. James',
'A hacker is someone who uses a combination of high-tech cybertools and social engineering to gain illicit access to someone else's data.',
'John McAfee',
'The hacker mindset doesn't actually see what happens on the other side, to the victim.',
'Kevin Mitnick',
'I look like a geeky hacker, but I don't know anything about computers.',
'Justin Long',
'The hacker community may be small, but it possesses the skills that are driving the global economies of the future.',
'Heather Brooke',
'I'm a bit of a hacker fanatic and know a fair bit about that industry and cyber crime and cyber warfare.',
'Seth Gordon',
'It's true, I had hacked into a lot of companies, and took copies of the source code to analyze it for security bugs. If I could locate security bugs, I could become better at hacking into their systems. It was all towards becoming a better hacker.',
'Kevin Mitnick',
'It's not enough to have a hacker culture anymore. You have to have a design culture, too.',
'Robert Scoble',
'If you go to a coffee shop or at the airport, and you're using open wireless, I would use a VPN service that you could subscribe for 10 bucks a month. Everything is encrypted in an encryption tunnel, so a hacker cannot tamper with your connection.',
'Kevin Mitnick',
'I wasn't a hacker for the money, and it wasn't to cause damage.',
'Kevin Mitnick',
'I'm not an economist; I'm a hacker who has spent his career exploring and repairing large networks.',
'Dan Kaminsky',
'In the '80s, society created a caricature of what a hacker or a programmer looked like: a guy wearing a hoodie, drinking energy drinks, sitting in a basement somewhere coding. Today, programmers look like the men we see in the show 'Silicon Valley' on HBO. If you look at the message girls are getting, it's saying, 'This is not for you.'',
'Reshma Saujani',
'I don't condone anyone causing damage in my name, or doing anything malicious in support of my plight. There are more productive ways to help me. As a hacker myself, I never intentionally damaged anything.',
'Kevin Mitnick',
'I think Linux is a great thing, in the big picture. It's a great hacker's tool, and it has a lot of potential to become something more.',
'Jamie Zawinski',
'Bitcoin is here to stay. There would be a hacker uproar to anyone who attempted to take credit for the patent of cryptocurrency. And I wouldn't want to be on the receiving end of hacker fury.',
'Adam Draper',
'It was on a bulletin board that I first learned about hacker culture, the 'Let's just break through this wall and see what's on the other side' mentality.',
'Harper Reed',
'Everything about Mark Zuckerberg is pure hacker. Hackers don't take realities of the world for granted; they seek to break and rebuild what they don't like. They seek to outsmart the world.',
'Sarah Lacy',
'If you're a juvenile delinquent today, you're a hacker. You live in your parent's house; they haven't seen you for two months. They put food outside your door, and you're shutting down a government of a foreign country from your computer.',
'John Waters',
'The key to social engineering is influencing a person to do something that allows the hacker to gain access to information or your network.',
'Kevin Mitnick',
'A smartphone links patients' bodies and doctors' computers, which in turn are connected to the Internet, which in turn is connected to any smartphone anywhere. The new devices could put the management of an individual's internal organs in the hands of every hacker, online scammer, and digital vandal on Earth.',
'Charles C. Mann'
]