Java Can';t使用URLConnection连接到特定网站,读取超时
我正在使用以下代码:Java Can';t使用URLConnection连接到特定网站,读取超时,java,web-scraping,Java,Web Scraping,我正在使用以下代码: public static void main(String[] args) throws IOException { String EngLink; URL EngUrl; URLConnection EngCon; String cookiesHeader; InputStream EngIs; BufferedReader EngBr; String line; String EngPageHtml=""
public static void main(String[] args) throws IOException {
String EngLink;
URL EngUrl;
URLConnection EngCon;
String cookiesHeader;
InputStream EngIs;
BufferedReader EngBr;
String line;
String EngPageHtml="";
EngLink="https://www.zomato.com/";
EngUrl = new URL(EngLink);
EngCon = (HttpURLConnection) EngUrl.openConnection();
EngCon.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 (.NET CLR 3.5.30729)");
EngIs = EngCon.getInputStream();
EngBr = new BufferedReader(new InputStreamReader(EngIs,"UTF-8"));
while ((line = EngBr.readLine()) != null) {
EngPageHtml = EngPageHtml + "\n" + line;
}
System.out.println(EngPageHtml);
}
我想做的是获取网站的原始html。
但是,当我运行代码时,会出现以下错误:
Exception in thread "main" java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.security.ssl.InputRecord.readFully(Unknown Source)
at sun.security.ssl.InputRecord.read(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at sun.security.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
at project1.Aaa.main(Aaa.java:33)
我正在使用这段代码成功地获取多个其他站点的HTML,但这段代码不起作用
有什么问题?我该如何解决
编辑:在firefox中加载站点,从中获取cookie并将其传入:
EngCon.setRequestProperty("Cookie",cookie);
使页面加载,但这不好,因为它不能反复使用。通过添加另一个请求属性解决了这一问题:
EngCon.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
不需要其他任何东西您尝试过这个url吗?您需要使用“HttpsURLConnection”而不是“HttpURLConnection”,并且需要导入“javax.net.ssl.HttpsURLConnection”。@ChamaraMaduranga
HttpsURLConnection
是HttpURLConnection
的子类,因此如果您不需要访问任何ssl功能,使用HttpURLConnection
非常好。查看我在mozzilla中得到的cookie,特别是“ak_bmsc”和“bm_sv”的两个值。其他为PHPSESSID=null;fbcity=93;zl=en;fbtrack=null;dpr=2;G_ENABLED_IDPS=谷歌;csrf=null;使用这种方法可以进行页面加载,但是我显然更喜欢不需要我从mozilla复制cookie的方法,因为我不知道这种方法可以工作多久读取超时
错误通常都是由代理或防火墙引起的。如果存在防火墙(例如,来自公司网络),则必须设置网络属性以设置代理和可选身份验证:https.proxyHost
,https.proxyPort
,等等。