Fatal编程技术网
  • java/
  • Can'；t使用Java中的WebClient连接到特定的https网站

Can'；t使用Java中的WebClient连接到特定的https网站

java web-scraping

Can'；t使用Java中的WebClient连接到特定的https网站,java,web-scraping,webclient,htmlunit,Java,Web Scraping,Webclient,Htmlunit,小结：我创建了一个简单的代码块，我在过去成功地使用过多次，但它不适用于此网站（“https://mid-usa.com/login）因为某种原因。我可以通过443从我的计算机远程登录到mid-usa.com，我可以在浏览器中打开此页面 WebClient client = new WebClient(BrowserVersion.BEST_SUPPORTED); client.getOptions().setSSLClientProtocols(new String[] { &q

小结：我创建了一个简单的代码块，我在过去成功地使用过多次，但它不适用于此网站（“https://mid-usa.com/login）因为某种原因。我可以通过443从我的计算机远程登录到mid-usa.com，我可以在浏览器中打开此页面

    WebClient client = new WebClient(BrowserVersion.BEST_SUPPORTED);
    client.getOptions().setSSLClientProtocols(new String[] { "TLSv1.2","TLSv1.1"});
    client.getOptions().setCssEnabled(false);
    client.getOptions().setJavaScriptEnabled(false);
    client.getOptions().setUseInsecureSSL(true);
    client.getOptions().setTimeout(10000);
    HtmlPage page = (HtmlPage) client.getPage("https://mid-usa.com/login");
我之所以将setSSLClientProtocols设置为TLSv1.2，是因为在查看运行扫描仪的网站协议时，我认为这会有所帮助（通过阅读其他帖子），但事实并非如此。我也只留下了TLSv1.2，但仍然得到了以下打印结果：

Jul 03, 2020 3:23:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://mid-usa.com:443: Connection reset
Jul 03, 2020 3:23:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://mid-usa.com:443
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://mid-usa.com:443: Connection reset
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://mid-usa.com:443
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://mid-usa.com:443: Connection reset
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://mid-usa.com:443
Connection resetjava.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(Unknown Source)
    at java.net.SocketInputStream.read(Unknown Source)
    at sun.security.ssl.InputRecord.readFully(Unknown Source)
    at sun.security.ssl.InputRecord.read(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
    at com.gargoylesoftware.htmlunit.httpclient.HtmlUnitSSLConnectionSocketFactory.connectSocket(HtmlUnitSSLConnectionSocketFactory.java:184)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
    at com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:193)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1537)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1456)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:447)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:368)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:520)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:501)
    at ScraperBase.autoLogin(ScraperBase.java:25)
    at MidwayDataCollection$4.actionPerformed(MidwayDataCollection.java:222)
    at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
    at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
    at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
    at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
    at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
    at java.awt.Component.processMouseEvent(Unknown Source)
    at javax.swing.JComponent.processMouseEvent(Unknown Source)
    at java.awt.Component.processEvent(Unknown Source)
    at java.awt.Container.processEvent(Unknown Source)
    at java.awt.Component.dispatchEventImpl(Unknown Source)
    at java.awt.Container.dispatchEventImpl(Unknown Source)
    at java.awt.Component.dispatchEvent(Unknown Source)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
    at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
    at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
    at java.awt.Container.dispatchEventImpl(Unknown Source)
    at java.awt.Window.dispatchEventImpl(Unknown Source)
    at java.awt.Component.dispatchEvent(Unknown Source)
    at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
    at java.awt.EventQueue.access$500(Unknown Source)
    at java.awt.EventQueue$3.run(Unknown Source)
    at java.awt.EventQueue$3.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.awt.EventQueue$4.run(Unknown Source)
    at java.awt.EventQueue$4.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.awt.EventQueue.dispatchEvent(Unknown Source)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
    at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
    at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
    at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
    at java.awt.EventDispatchThread.run(Unknown Source)
有什么想法吗？

我认为问题在于服务器想要使用的密码套件在以后的JDK中被禁用了

当我通过
curl-v连接时https://mid-usa.com >/dev/null
我得到：

*使用TLSv1.2/DES-CBC3-SHA的SSL连接

Ignoring disabled cipher suite: SSL_RSA_WITH_3DES_EDE_CBC_SHA for TLSv1.2
No available cipher suite for TLSv1.2
main, handling exception: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
这映射到JSSE（）中的SSL_RSA_和_3DES_EDE_CBC_SHA

我将您的示例改写为仅使用URLConnection：

        URL url = new URL("https://mid-usa.com");
        URLConnection urlConnection = url.openConnection();
        InputStream is = urlConnection.getInputStream();
并使用
-Djavax.net.debug=all-Dhttps.cipherSuites=“SSL\u RSA\u with\u 3DES\u EDE\u CBC\u SHA”运行它

所以我认为服务器想要使用JDK不允许的密码

这个Jira问题似乎类似——服务器在发送握手后会断开连接


所以现在的问题是：如何启用密码套件，我还没有弄明白怎么做
 看看这是否有帮助，谢谢你的链接。在搜索我的问题时，我注意到了这一点，并进行了扫描，发现它支持TLSv1.2，这就是我在代码中设置它的原因。我还尝试使用较低级别的库（如本文中所述）手动创建套接字。我仍然得到一个连接重置。我不知道为什么，我也没有主意了。不过浏览器工作正常。在重定向的情况下，我确实添加了重定向“client.getOptions（）.setRedirectEnabled（true）；”，但我仍然遇到了连接重置的相同问题。我只是不知道为什么它只针对这个特定的网站。我对它进行了扫描并启用了协议。我增加了超时时间。我可以将URL更改为任何其他内容，而且没有问题，但我可以使用实际浏览器轻松访问此页面。关于我在哪里可以准确地发现这个网站有什么特别之处，我需要克服吗？是的，8u171和j10 up在
jdk.tls.disabledAlgorithms
安全属性中默认禁用3DES密码套件；看见这可以通过（1）编辑j8/lib/security/java.security或j9+/conf/security/java.security（2）在别处创建编辑的java.security，并用sysprop java.security.properties指向它，请参见和（3）在代码早期调用security.setProperty（在加载TLS类之前）。SSLabs为此给服务器一个C级！你们真了不起。非常感谢你。我从你的链接中学到了很多，并且能够解决这个问题。默认情况下，java.security文件启用了编辑它的功能（security.overridePropertiesFile=true）。我能够在我的代码中设置以下内容，而无需编辑环境中的任何内容：
Security.setProperty（“jdk.tls.disabledAlgorithms”，“RC4，MD5withRSA”）；Security.setProperty（“jdk.tls.legacyAlgorithms”、“tls_RSA_WITH_3DES_EDE_CBC_SHA”）；System.setProperty（“https.protocols”、“TLSv1、TLSv1.1”）