AWS NLB&x2B；NGINX入口+；WebSocket==502网关错误或握手错误

入口控制器配置（以下文件无更改）

服务

kind: Service
apiVersion: v1
metadata:
  name: websockets-service
  namespace: development
  annotations:
    argocd.argoproj.io/sync-wave: "10"
spec:
  selector:
    app: project-websocket
  ports:
  #- name: http
  #  protocol: TCP
  #  port: 80
  #  targetPort: 9000
  - name: https
    protocol: TCP
    port: 443
    targetPort: 443
  type: ClusterIP

入口

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: websockets-ingress
  namespace: development
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"

    # nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/websocket-services: websockets-service
    nginx.org/websocket-services: websockets-service

    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header HOST $host;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_pass_request_headers on;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "Upgrade";
    argocd.argoproj.io/sync-wave: "10"
spec:
  tls:
    - hosts:
      - backend-dev.project.com
      secretName: tls-secret
  rules:
  - host: backend-dev.project.com
    http:
      paths:
      - path: /ws/
        backend:
          serviceName: websockets-service
          servicePort: 443

正在尝试各种CLI websocket客户端：

➜ ws --insecure wss://backend-dev.project.com/ws/table/123
websocket: bad handshake

➜ websocat -k  wss://backend-dev.project.com/ws/table/123
websocat: WebSocketError: Received unexpected status code (502 Bad Gateway)
websocat: error running

如果我从服务中选择代理端口，则一切正常：

k port-forward svc/websockets-service 8443:443
Forwarding from 127.0.0.1:8443 -> 443
Forwarding from [::1]:8443 -> 443
Handling connection for 8443

➜ websocat -k wss://localhost:8443/ws/table/123
{"connect": {"channel_name": "specific.239f96e4a93a470688a12fc0bc8d0374!9d92256f6f65446386fb17b59d1aac57", "table_token": "123"}}

and in logs I see

127.0.0.1:50070 - - [21/Nov/2020:05:11:50] "WSCONNECTING /ws/table/123" - -
127.0.0.1:50070 - - [21/Nov/2020:05:11:50] "WSCONNECT /ws/table/123" - -

SSL信息

➜ curl -vvI https://backend-dev.project.com
*   Trying 44.235.41.143:443...
* Connected to backend-dev.project.com (44.235.41.143) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.project.com
*  start date: Oct 28 00:00:00 2020 GMT
*  expire date: Oct 28 23:59:59 2021 GMT
*  subjectAltName: host "backend-dev.project.com" matched cert's "*.project.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5579ce4a9af0)
> HEAD / HTTP/2
> Host: backend-dev.project.com
> user-agent: curl/7.69.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
HTTP/2 200 
< date: Sat, 21 Nov 2020 05:13:22 GMT
date: Sat, 21 Nov 2020 05:13:22 GMT
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< content-length: 143552
content-length: 143552
< vary: Accept-Encoding
vary: Accept-Encoding
< server-timing: TimerPanel_utime;dur=31.268000000011398;desc="User CPU time", TimerPanel_stime;dur=0.0;desc="System CPU time", TimerPanel_total;dur=31.268000000011398;desc="Total CPU time", TimerPanel_total_time;dur=31.340599060058594;desc="Elapsed time", SQLPanel_sql_time;dur=0;desc="SQL 0 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
server-timing: TimerPanel_utime;dur=31.268000000011398;desc="User CPU time", TimerPanel_stime;dur=0.0;desc="System CPU time", TimerPanel_total;dur=31.268000000011398;desc="Total CPU time", TimerPanel_total_time;dur=31.340599060058594;desc="Elapsed time", SQLPanel_sql_time;dur=0;desc="SQL 0 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-frame-options: DENY
x-frame-options: DENY
< vary: Cookie, Origin
vary: Cookie, Origin
< set-cookie: csrftoken=zuBDnumLdRgcY5cICoIKnpAn98sdiGKdpiFKiiG7KvLr7bCPeOdMjzNK0mgTRUQZ; expires=Sat, 20 Nov 2021 05:13:22 GMT; Max-Age=31449600; Path=/; SameSite=Lax
set-cookie: csrftoken=zuBDnumLdRgcY5cICoIKnpAn98sdiGKdpiFKiiG7KvLr7bCPeOdMjzNK0mgTRUQZ; expires=Sat, 20 Nov 2021 05:13:22 GMT; Max-Age=31449600; Path=/; SameSite=Lax
< strict-transport-security: max-age=15724800; includeSubDomains
strict-transport-security: max-age=15724800; includeSubDomains

< 
* Connection #0 to host backend-dev.project.com left intact

➜ curl-vvIhttps://backend-dev.project.com
*正在尝试44.235.41.143:443。。。
*已连接到backend-dev.project.com（44.235.41.143）端口443（#0）
*阿尔卑斯山，提供h2
*ALPN，提供http/1.1
*已成功设置证书验证位置：
*CAfile:/etc/pki/tls/certs/ca-bundle.crt
卡帕斯：没有
*TLSv1.3（输出），TLS握手，客户端问候（1）：
*TLSv1.3（IN）、TLS握手、服务器hello（2）：
*TLSv1.3（IN）、TLS握手、加密扩展（8）：
*TLSv1.3（IN），TLS握手，证书（11）：
*TLSv1.3（IN）、TLS握手、证书验证（15）：
*TLSv1.3（IN），TLS握手，完成（20）：
*TLSv1.3（OUT），TLS更改密码，更改密码规范（1）：
*TLSv1.3（输出），TLS握手，完成（20）：
*使用TLSv1.3/TLS_AES_256_GCM_SHA384的SSL连接
*ALPN，服务器接受使用h2
*服务器证书：
*主题：CN=*.project.com
*开始日期：10月28日00:00:00格林威治标准时间2020
*到期日期：10月28日23:59:59格林威治标准时间2021
*subjectAltName:主机“backend-dev.project.com”匹配证书“*.project.com”
*发行人：C=GB；ST=大曼彻斯特；L=萨尔福德；O=Sectigo有限公司；CN=Sectigo RSA域验证安全服务器CA
*SSL证书验证正常。
*使用HTTP2，服务器支持多用途
*连接状态已更改（HTTP/2已确认）
*升级后正在将流缓冲区中的HTTP/2数据复制到连接缓冲区：len=0
*使用流ID:1（易处理0x5579ce4a9af0）
>HEAD/HTTP/2
>主机：backend-dev.project.com
>用户代理：curl/7.69.1
>接受：*/*
> 
*TLSv1.3（IN）、TLS握手、新闻会话单（4）：
*TLSv1.3（IN）、TLS握手、新闻会话单（4）：
*旧SSL会话ID已过时，正在删除
*连接状态已更改（最大并发流==128）！

WebSocket和SSE（服务器发送的事件）是AWS的难题。 对于SSE，我发现只有ALB工作正常

好消息：AWS有ALB入口控制器
坏消息：就是这样。不与nginx入口集成

您可以混合使用alb入口控制器和nginx入口控制器：
它归结为安装

alb ingress controller

和

ingress nginx

（裸机图表），然后创建一个入口资源，该资源指向

alb ingress controller

到

ingress nginx

的节点端口服务

比如：

kind: Ingress
metadata:
  name: "alb-ingress-to-nginx-ingress"
  labels:
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: "internet-facing"
    alb.ingress.kubernetes.io/target-type: "ip"
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-southeast-2:yyy:certificate/xxxx
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
spec:
  rules:
  - host: 'toto.edu'
    http:
      paths:
      - backend:
          serviceName: nginx-controller
          servicePort: http

资源：

---

我遇到了同样的问题。你找到解决办法了吗？