Amazon ec2 ELB生成504网关超时，带有2个EC2实例-数据包未到达服务器

我有一个面向互联网的ELB，连接到2个服务于一个网页的EC2实例。已使用AWS Opsworks部署基础结构

配置如下：

面向互联网的ELB：

• 存在于两个可用区域中
• 已启用跨区域负载平衡
• 空闲超时：60秒
• 粘性：LBCookieStickinessPolicy，expirationPeriod='0'

为网页提供服务的EC2实例（demo1.virtualtoptraining.com）

• 两个服务器上的配置完全相同
• 作为模块运行Apache2和PHP5

DNS配置：

demo1.virtualtoptraining.com-->加载平衡器的CNAME到DNS名称

当我使用JMETER对demo1.virtualtoptraining.com运行测试时，我看到大约20%的请求出现错误。报告的错误是504网关超时。这些错误并不取决于加载条件，因为在web浏览器上使用单个用户导航时可能会出现此错误。错误是100%可复制的

尝试了以下操作：

• 我一直在玩apache定时器，因为我认为这可能与web服务器没有及时响应负载平衡器有关，但我没有得到任何改进

• 我一直在使用同一可用区域上的服务器。错误仍然存在

经过一些调查，我得出结论，我获得504 GATEWAY_超时的请求到达负载平衡器，但它们从未转发到web服务器，因为我在任何web服务器访问日志中都看不到它们

我在下面展示了一个包含所有日志的问题示例。我在浏览器上导航到网页时得到了以下示例：

示例序列

从网页注销（转发至首页）

登上头版

获取登录页面

登录后凭证

在凭据验证后遵循重定向

获取主页。错误-->在任何服务器访问日志上都看不到此获取操作

我现在列出了访问日志和ELB日志，我用***（message#）突出显示了这些消息

服务器1访问日志：

ubuntu@php-app1:/var/log/apache2$ tail -f moodle281-access.log
172.31.40.186 - - [09/Jun/2015:22:44:52 +0000] "POST /login/index.php HTTP/1.1" 303 1018 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:53 +0000] "GET / HTTP/1.1" 200 39933 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:54 +0000] "GET /course/view.php?id=2 HTTP/1.1" 200 55453 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:55 +0000] "GET /course/view.php?id=2 HTTP/1.1" 200 55453 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:55 +0000] "GET /mod/forum/discuss.php?d=1 HTTP/1.1" 200 47600 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:56 +0000] "POST /mod/forum/post.php HTTP/1.1" 404 26456 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:56 +0000] "GET /user/index.php?id=2 HTTP/1.1" 200 82175 "-" "Apache-HttpClient/4.2.6 (java 1.5)"

****(1) 172.31.19.167 - - [09/Jun/2015:22:48:52 +0000] "GET /login/logout.php?sesskey=fXjerOYZEo HTTP/1.1" 303 862 "http://demo1.virtualtoptraining.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"
****(4) 172.31.19.167 - - [09/Jun/2015:22:48:55 +0000] "GET /login/index.php HTTP/1.1" 200 6088 "http://demo1.virtualtoptraining.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"
****(5) 172.31.19.167 - - [09/Jun/2015:22:49:00 +0000] "GET /login/index.php?testsession=2 HTTP/1.1" 303 800 "http://demo1.virtualtoptraining.com/login/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"

....

****  (2) 172.31.19.167 - - [09/Jun/2015:22:48:52 +0000] "GET / HTTP/1.1" 200 7078 "http://demo1.virtualtoptraining.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"
****  (3) 172.31.19.167 - - [09/Jun/2015:22:49:00 +0000] "POST /login/index.php HTTP/1.1" 303 1010 "http://demo1.virtualtoptraining.com/login/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"

....

015-06-09T22:48:52.128826Z itoptest2 151.182.141.178:52554 172.31.25.91:80 0.000092 0.204629 0.000058 303 303 0 434 "GET http://demo1.virtualtoptraining.com:80/login/logout.php?sesskey=fXjerOYZEo HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:48:52.382853Z itoptest2 151.182.141.178:52554 172.31.34.25:80 0.000082 0.456022 0.00007 200 200 0 6500 "GET http://demo1.virtualtoptraining.com:80/ HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:48:55.853509Z itoptest2 151.182.141.178:52554 172.31.25.91:80 0.000105 0.292976 0.000046 200 200 0 5617 "GET http://demo1.virtualtoptraining.com:80/login/index.php HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:49:00.382245Z itoptest2 151.182.141.178:52554 172.31.34.25:80 0.000105 0.545512 0.000068 303 303 36 463 "POST http://demo1.virtualtoptraining.com:80/login/index.php HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:49:00.980174Z itoptest2 151.182.141.178:52554 172.31.25.91:80 0.00007 0.219837 0.000044 303 303 0 434 "GET http://demo1.virtualtoptraining.com:80/login/index.php?testsession=2 HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:49:01.248300Z itoptest2 151.182.141.178:52554 - -1 -1 -1 504 0 0 0 "GET http://demo1.virtualtoptraining.com:80/ HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -

服务器2访问日志：

ubuntu@php-app1:/var/log/apache2$ tail -f moodle281-access.log
172.31.40.186 - - [09/Jun/2015:22:44:52 +0000] "POST /login/index.php HTTP/1.1" 303 1018 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:53 +0000] "GET / HTTP/1.1" 200 39933 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:54 +0000] "GET /course/view.php?id=2 HTTP/1.1" 200 55453 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:55 +0000] "GET /course/view.php?id=2 HTTP/1.1" 200 55453 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:55 +0000] "GET /mod/forum/discuss.php?d=1 HTTP/1.1" 200 47600 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:56 +0000] "POST /mod/forum/post.php HTTP/1.1" 404 26456 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:56 +0000] "GET /user/index.php?id=2 HTTP/1.1" 200 82175 "-" "Apache-HttpClient/4.2.6 (java 1.5)"

****(1) 172.31.19.167 - - [09/Jun/2015:22:48:52 +0000] "GET /login/logout.php?sesskey=fXjerOYZEo HTTP/1.1" 303 862 "http://demo1.virtualtoptraining.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"
****(4) 172.31.19.167 - - [09/Jun/2015:22:48:55 +0000] "GET /login/index.php HTTP/1.1" 200 6088 "http://demo1.virtualtoptraining.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"
****(5) 172.31.19.167 - - [09/Jun/2015:22:49:00 +0000] "GET /login/index.php?testsession=2 HTTP/1.1" 303 800 "http://demo1.virtualtoptraining.com/login/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"

....

****  (2) 172.31.19.167 - - [09/Jun/2015:22:48:52 +0000] "GET / HTTP/1.1" 200 7078 "http://demo1.virtualtoptraining.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"
****  (3) 172.31.19.167 - - [09/Jun/2015:22:49:00 +0000] "POST /login/index.php HTTP/1.1" 303 1010 "http://demo1.virtualtoptraining.com/login/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"

....

015-06-09T22:48:52.128826Z itoptest2 151.182.141.178:52554 172.31.25.91:80 0.000092 0.204629 0.000058 303 303 0 434 "GET http://demo1.virtualtoptraining.com:80/login/logout.php?sesskey=fXjerOYZEo HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:48:52.382853Z itoptest2 151.182.141.178:52554 172.31.34.25:80 0.000082 0.456022 0.00007 200 200 0 6500 "GET http://demo1.virtualtoptraining.com:80/ HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:48:55.853509Z itoptest2 151.182.141.178:52554 172.31.25.91:80 0.000105 0.292976 0.000046 200 200 0 5617 "GET http://demo1.virtualtoptraining.com:80/login/index.php HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:49:00.382245Z itoptest2 151.182.141.178:52554 172.31.34.25:80 0.000105 0.545512 0.000068 303 303 36 463 "POST http://demo1.virtualtoptraining.com:80/login/index.php HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:49:00.980174Z itoptest2 151.182.141.178:52554 172.31.25.91:80 0.00007 0.219837 0.000044 303 303 0 434 "GET http://demo1.virtualtoptraining.com:80/login/index.php?testsession=2 HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:49:01.248300Z itoptest2 151.182.141.178:52554 - -1 -1 -1 504 0 0 0 "GET http://demo1.virtualtoptraining.com:80/ HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -

负载平衡器访问日志：

ubuntu@php-app1:/var/log/apache2$ tail -f moodle281-access.log
172.31.40.186 - - [09/Jun/2015:22:44:52 +0000] "POST /login/index.php HTTP/1.1" 303 1018 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:53 +0000] "GET / HTTP/1.1" 200 39933 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:54 +0000] "GET /course/view.php?id=2 HTTP/1.1" 200 55453 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:55 +0000] "GET /course/view.php?id=2 HTTP/1.1" 200 55453 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:55 +0000] "GET /mod/forum/discuss.php?d=1 HTTP/1.1" 200 47600 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:56 +0000] "POST /mod/forum/post.php HTTP/1.1" 404 26456 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
172.31.40.186 - - [09/Jun/2015:22:44:56 +0000] "GET /user/index.php?id=2 HTTP/1.1" 200 82175 "-" "Apache-HttpClient/4.2.6 (java 1.5)"

****(1) 172.31.19.167 - - [09/Jun/2015:22:48:52 +0000] "GET /login/logout.php?sesskey=fXjerOYZEo HTTP/1.1" 303 862 "http://demo1.virtualtoptraining.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"
****(4) 172.31.19.167 - - [09/Jun/2015:22:48:55 +0000] "GET /login/index.php HTTP/1.1" 200 6088 "http://demo1.virtualtoptraining.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"
****(5) 172.31.19.167 - - [09/Jun/2015:22:49:00 +0000] "GET /login/index.php?testsession=2 HTTP/1.1" 303 800 "http://demo1.virtualtoptraining.com/login/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"

....

****  (2) 172.31.19.167 - - [09/Jun/2015:22:48:52 +0000] "GET / HTTP/1.1" 200 7078 "http://demo1.virtualtoptraining.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"
****  (3) 172.31.19.167 - - [09/Jun/2015:22:49:00 +0000] "POST /login/index.php HTTP/1.1" 303 1010 "http://demo1.virtualtoptraining.com/login/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"

....

015-06-09T22:48:52.128826Z itoptest2 151.182.141.178:52554 172.31.25.91:80 0.000092 0.204629 0.000058 303 303 0 434 "GET http://demo1.virtualtoptraining.com:80/login/logout.php?sesskey=fXjerOYZEo HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:48:52.382853Z itoptest2 151.182.141.178:52554 172.31.34.25:80 0.000082 0.456022 0.00007 200 200 0 6500 "GET http://demo1.virtualtoptraining.com:80/ HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:48:55.853509Z itoptest2 151.182.141.178:52554 172.31.25.91:80 0.000105 0.292976 0.000046 200 200 0 5617 "GET http://demo1.virtualtoptraining.com:80/login/index.php HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:49:00.382245Z itoptest2 151.182.141.178:52554 172.31.34.25:80 0.000105 0.545512 0.000068 303 303 36 463 "POST http://demo1.virtualtoptraining.com:80/login/index.php HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:49:00.980174Z itoptest2 151.182.141.178:52554 172.31.25.91:80 0.00007 0.219837 0.000044 303 303 0 434 "GET http://demo1.virtualtoptraining.com:80/login/index.php?testsession=2 HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
2015-06-09T22:49:01.248300Z itoptest2 151.182.141.178:52554 - -1 -1 -1 504 0 0 0 "GET http://demo1.virtualtoptraining.com:80/ HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -

正如您所见，在任何web服务器访问日志中都没有看到生成504网关超时的请求

---

您知道问题可能是什么吗？

在负载测试场景中，您是否可能没有为ELB提供足够的加速期？确保监视CloudWatch上的ELB激增队列。丢弃的喘振请求返回5xx错误

另外，在针对AmazonEC2ELB进行测试时，配置JMeter的stalecheck也很重要。这是因为Amazon将用一个更大的ELB热交换您的ELB，而不是主动扩展您正在使用的ELB。这意味着底层寻址发生了变化，因此JMeter必须检查陈旧的连接。可能是JMeter正在向旧ELB发送请求，而旧ELB又有网关超时，因为它无法再发送到您的应用服务器

查看AWS的深入ELB测试指南：

此外，这也是JMeter的stalecheck属性所在的位置

../bin/hc.参数：

# Default value since JMeter 2.11, 
# also uncomment hc.parameters.file=hc.parameters to enable this check:
#http.connection.stalecheck$Boolean=false

---

您还需要编辑jmeter.properties文件，以便它实际使用hc.parameters设置，如上面的注释所述。

您是否添加并启用了？如果没有，我建议尝试一下，它应该可以解决您的问题。

我们对该主题做了进一步的调查：

• 请求到达服务器（通过运行tcpdump找到），但Apache没有响应

• Apache错误日志显示，对于“未响应的请求”，Apache由于核心转储而崩溃。在日志中发现以下错误：
  AH00051:子pid 11419出口信号总线错误（7）

• 堆芯倾倒分析显示如下：
  警告：无法加载4个库的共享库符号，例如/usr/lib/php5/20121212/sasl.so。 使用“info sharedlibrary”命令查看完整的列表。 您需要“设置solib搜索路径”还是“设置系统根”？ [已启用使用libthread_db的线程调试] 使用主机libthread_db library“/lib/x86_64-linux-gnu/libthread_db.so.1”。 核心由“/usr/sbin/apache2-k start”生成。 程序以信号SIGBUS终止，总线错误。 0在Zend/Zend_语言_扫描器上进行lex_扫描（zendlval=）。c:1091 1091 Zend/Zend_language_scanner.c：没有这样的文件或目录。 （gdb）

---

在分析所有这些信息后，我们得出结论认为问题与PHP错误有关，因为存在从应用程序的两个不同实例对文件的并发访问。因此，此访问未正确协调，导致崩溃。

无论负载如何，问题都会重现。单个用户访问应用程序时可能会出现这种情况。一开始我认为这可能与缺乏预警有关，但后来我能够以很小的负载进行复制。关于陈旧的连接，我会看一看，让你知道发生了什么。我启用了DNS缓存管理器，但问题仍然存在