Amazon web services 允许VPC中的Lambda访问同一VPC中的Elasticsearch域
我正在学习如何使用Amazon服务,尤其是我目前想要创建一个简单的云形成脚本设置:一个VPC,其中有一个用JS编写的lambda,可以访问同一VPC中的Elasticsearch服务 不知怎么的,我无法让它工作。从lambda到Elasticsearch域的所有请求始终超时。然而,在同一VPC中运行Amazon Linux 2的EC2实例中,从相同的JS代码或curl(即使没有任何额外授权,也只是卷曲ES域端点)发出的相同请求可以正常工作,并且我可以从EC2实例(被SSHed到其中)与Elasticsearch通信 同时,lambda能够访问VPC中的Aurora群集,因此lambda不能访问VPC资源不是一般问题 请告诉我,我在设置的云形成描述中做错了什么?下面是我的云形成模板的相关摘录和JS代码的示例,该代码能够从EC2实例访问ES服务,但不能对lambda执行相同的操作:Amazon web services 允许VPC中的Lambda访问同一VPC中的Elasticsearch域,amazon-web-services,elasticsearch,amazon-ec2,amazon-cloudformation,amazon-vpc,Amazon Web Services,elasticsearch,Amazon Ec2,Amazon Cloudformation,Amazon Vpc,我正在学习如何使用Amazon服务,尤其是我目前想要创建一个简单的云形成脚本设置:一个VPC,其中有一个用JS编写的lambda,可以访问同一VPC中的Elasticsearch服务 不知怎么的,我无法让它工作。从lambda到Elasticsearch域的所有请求始终超时。然而,在同一VPC中运行Amazon Linux 2的EC2实例中,从相同的JS代码或curl(即使没有任何额外授权,也只是卷曲ES域端点)发出的相同请求可以正常工作,并且我可以从EC2实例(被SSHed到其中)与Elast
AWSTemplateFormatVersion: 2010-09-09
Description: The AWS CloudFormation tutorial
Resources:
SomeDeploymentBucket:
Type: 'AWS::S3::Bucket'
AppLogGroup:
Type: 'AWS::Logs::LogGroup'
Properties:
LogGroupName: /aws/lambda/some-lambda
# ========= The Lambda Execution Role =========
IamRoleLambdaExecution:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Policies:
- PolicyName: !Join
- '-'
- - dev
- some-app
- lambda
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 's3:*'
- 'rds-db:connect'
- 'rds:*'
- 'es:*'
Resource: '*'
Path: /
RoleName: !Join
- '-'
- - some-app
- dev
- eu-west-1
- lambdaRole
ManagedPolicyArns:
- !Join
- ''
- - 'arn:'
- !Ref 'AWS::Partition'
- ':iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'
# ========= The Lambda =========
AppLambdaFunction:
Type: 'AWS::Lambda::Function'
Properties:
Code:
S3Bucket: !Ref SomeDeploymentBucket
S3Key: >-
tutorial/some-app/dev/1545610972669-2018-12-24T00:22:52.669Z/some-app.zip
FunctionName: some-lambda
Handler: app.server
MemorySize: 1024
Role: !GetAtt
- IamRoleLambdaExecution
- Arn
Runtime: nodejs8.10
Timeout: 6
VpcConfig:
SecurityGroupIds:
- !Ref xxxVPCSecurityGroup
SubnetIds:
- !Ref xxxLambdaSubnet
DependsOn:
- AppLogGroup
- IamRoleLambdaExecution
# ========= VPC =========
xxxVPC:
Type: 'AWS::EC2::VPC'
Properties:
CidrBlock: 172.31.0.0/16
InstanceTenancy: default
EnableDnsSupport: 'true'
EnableDnsHostnames: 'true'
xxxVPCSecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupName: VPC SG
GroupDescription: VPC Security Group
VpcId: !Ref xxxVPC
xxxLambdaSubnet:
Type: 'AWS::EC2::Subnet'
Properties:
VpcId: !Ref xxxVPC
CidrBlock: 172.31.32.0/20
# ========= Elasticsearch =========
xxxESSecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupName: ES SG
GroupDescription: ES Security group
VpcId: !Ref xxxVPC
SecurityGroupIngress:
- IpProtocol: -1
FromPort: 0
ToPort: 65535
SourceSecurityGroupId: !Ref xxxVPCSecurityGroup
xxxElasticSearch:
Type: 'AWS::Elasticsearch::Domain'
Properties:
AccessPolicies:
Version: 2012-10-17
Statement:
- Action:
- 'es:*'
- 'ec2:*'
- 's3:*'
Principal:
AWS:
- '*'
Resource: '*'
Effect: Allow
DomainName: es-xxx-domain
AdvancedOptions:
rest.action.multi.allow_explicit_index: 'true'
ElasticsearchVersion: 6.3
ElasticsearchClusterConfig:
InstanceCount: 2
InstanceType: m3.medium.elasticsearch
DedicatedMasterEnabled: 'false'
VPCOptions:
SecurityGroupIds:
- !Ref xxxESSecurityGroup
SubnetIds:
- !Ref xxxLambdaSubnet
JS代码(没有使用creds签名的版本,但签名时也不起作用):
在同一VPC中的EC2实例中执行此操作可以从ES域获得响应,而不会出现任何问题:
我非常感谢您的帮助,因为我已经陷入了困境。注意到您的设置中存在两个问题
AWSTemplateFormatVersion: 2010-09-09
Description: The AWS CloudFormation tutorial
Resources:
SomeDeploymentBucket:
Type: 'AWS::S3::Bucket'
AppLogGroup:
Type: 'AWS::Logs::LogGroup'
Properties:
LogGroupName: /aws/lambda/some-lambda
# ========= The Lambda Execution Role =========
IamRoleLambdaExecution:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Policies:
- PolicyName: !Join
- '-'
- - dev
- some-app
- lambda
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 's3:*'
- 'rds-db:connect'
- 'rds:*'
- 'es:*'
Resource: '*'
Path: /
RoleName: !Join
- '-'
- - some-app
- dev
- eu-west-1
- lambdaRole
ManagedPolicyArns:
- !Join
- ''
- - 'arn:'
- !Ref 'AWS::Partition'
- ':iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'
# ========= The Lambda =========
AppLambdaFunction:
Type: AWS::Lambda::Function
DependsOn:
- AppLogGroup
- IamRoleLambdaExecution
Properties:
FunctionName: some-lambda
Handler: index.lambda_handler
Runtime: python2.7
Timeout: 60
MemorySize: 1024
Role: !GetAtt
- IamRoleLambdaExecution
- Arn
VpcConfig:
SecurityGroupIds:
- !Ref xxxVPCSecurityGroup
SubnetIds:
- !Ref xxxLambdaSubnet1
- !Ref xxxLambdaSubnet2
Code:
ZipFile: !Sub |
from __future__ import print_function
import boto3
iam = boto3.client('iam')
def lambda_handler(event, context):
print('called lambda_handler')
# ========= VPC =========
xxxVPC:
Type: 'AWS::EC2::VPC'
Properties:
CidrBlock: 172.31.0.0/16
InstanceTenancy: default
EnableDnsSupport: 'true'
EnableDnsHostnames: 'true'
xxxVPCSecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupName: VPC SG
GroupDescription: VPC Security Group
VpcId: !Ref xxxVPC
xxxLambdaSubnet1:
Type: 'AWS::EC2::Subnet'
Properties:
VpcId: !Ref xxxVPC
CidrBlock: 172.31.32.0/20
xxxLambdaSubnet2:
Type: 'AWS::EC2::Subnet'
Properties:
VpcId: !Ref xxxVPC
CidrBlock: 172.31.16.0/20
# ========= Elasticsearch =========
xxxESSecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupName: ES SG
GroupDescription: ES Security group
VpcId: !Ref xxxVPC
SecurityGroupIngress:
- IpProtocol: -1
FromPort: 0
ToPort: 65535
SourceSecurityGroupId: !Ref xxxVPCSecurityGroup
xxxElasticSearch:
Type: 'AWS::Elasticsearch::Domain'
Properties:
AccessPolicies:
Version: 2012-10-17
Statement:
- Action:
- 'es:*'
- 'ec2:*'
- 's3:*'
Principal:
AWS:
- '*'
Resource: '*'
Effect: Allow
DomainName: es-xxx-domain
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 20
VolumeType: "gp2"
AdvancedOptions:
rest.action.multi.allow_explicit_index: 'true'
ElasticsearchClusterConfig:
InstanceCount: 1
InstanceType: m4.large.elasticsearch
DedicatedMasterEnabled: 'false'
VPCOptions:
SecurityGroupIds:
- !Ref xxxESSecurityGroup
SubnetIds:
- !Ref xxxLambdaSubnet1
更新的Lambda函数处理程序代码
import urllib2
def lambda_handler(event, context):
print('called lambda_handler')
data = ''
url = 'https://vpc-es-xxx-domain-fixthis.es.amazonaws.com'
req = urllib2.Request(url, data, {'Content-Type': 'application/json'})
f = urllib2.urlopen(req)
for x in f:
print(x)
f.close()
Lambda函数和ES域是否在VPC内的同一子网中?如果它们在不同的子网上,请确保相应的路由表彼此之间有路由。@ben5556是的,lambda和ES在同一子网中,即示例中的
xxxlambda子网
。我改进了代码,使其更加简洁。您是在运行一个自我管理的elasticsearch群集还是一个AWS elasticsearch服务群集?@RenéGonzálezVenegas这是一个与同一VPC关联的AWS elasticsearch服务群集。您可以在我的问题中提供的第一个代码示例中看到它的配置(向下滚动示例到最底部,在那里您可以看到配置的elasticsearch部分)
AWSTemplateFormatVersion: 2010-09-09
Description: The AWS CloudFormation tutorial
Resources:
SomeDeploymentBucket:
Type: 'AWS::S3::Bucket'
AppLogGroup:
Type: 'AWS::Logs::LogGroup'
Properties:
LogGroupName: /aws/lambda/some-lambda
# ========= The Lambda Execution Role =========
IamRoleLambdaExecution:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Policies:
- PolicyName: !Join
- '-'
- - dev
- some-app
- lambda
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 's3:*'
- 'rds-db:connect'
- 'rds:*'
- 'es:*'
Resource: '*'
Path: /
RoleName: !Join
- '-'
- - some-app
- dev
- eu-west-1
- lambdaRole
ManagedPolicyArns:
- !Join
- ''
- - 'arn:'
- !Ref 'AWS::Partition'
- ':iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'
# ========= The Lambda =========
AppLambdaFunction:
Type: AWS::Lambda::Function
DependsOn:
- AppLogGroup
- IamRoleLambdaExecution
Properties:
FunctionName: some-lambda
Handler: index.lambda_handler
Runtime: python2.7
Timeout: 60
MemorySize: 1024
Role: !GetAtt
- IamRoleLambdaExecution
- Arn
VpcConfig:
SecurityGroupIds:
- !Ref xxxVPCSecurityGroup
SubnetIds:
- !Ref xxxLambdaSubnet1
- !Ref xxxLambdaSubnet2
Code:
ZipFile: !Sub |
from __future__ import print_function
import boto3
iam = boto3.client('iam')
def lambda_handler(event, context):
print('called lambda_handler')
# ========= VPC =========
xxxVPC:
Type: 'AWS::EC2::VPC'
Properties:
CidrBlock: 172.31.0.0/16
InstanceTenancy: default
EnableDnsSupport: 'true'
EnableDnsHostnames: 'true'
xxxVPCSecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupName: VPC SG
GroupDescription: VPC Security Group
VpcId: !Ref xxxVPC
xxxLambdaSubnet1:
Type: 'AWS::EC2::Subnet'
Properties:
VpcId: !Ref xxxVPC
CidrBlock: 172.31.32.0/20
xxxLambdaSubnet2:
Type: 'AWS::EC2::Subnet'
Properties:
VpcId: !Ref xxxVPC
CidrBlock: 172.31.16.0/20
# ========= Elasticsearch =========
xxxESSecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupName: ES SG
GroupDescription: ES Security group
VpcId: !Ref xxxVPC
SecurityGroupIngress:
- IpProtocol: -1
FromPort: 0
ToPort: 65535
SourceSecurityGroupId: !Ref xxxVPCSecurityGroup
xxxElasticSearch:
Type: 'AWS::Elasticsearch::Domain'
Properties:
AccessPolicies:
Version: 2012-10-17
Statement:
- Action:
- 'es:*'
- 'ec2:*'
- 's3:*'
Principal:
AWS:
- '*'
Resource: '*'
Effect: Allow
DomainName: es-xxx-domain
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 20
VolumeType: "gp2"
AdvancedOptions:
rest.action.multi.allow_explicit_index: 'true'
ElasticsearchClusterConfig:
InstanceCount: 1
InstanceType: m4.large.elasticsearch
DedicatedMasterEnabled: 'false'
VPCOptions:
SecurityGroupIds:
- !Ref xxxESSecurityGroup
SubnetIds:
- !Ref xxxLambdaSubnet1
import urllib2
def lambda_handler(event, context):
print('called lambda_handler')
data = ''
url = 'https://vpc-es-xxx-domain-fixthis.es.amazonaws.com'
req = urllib2.Request(url, data, {'Content-Type': 'application/json'})
f = urllib2.urlopen(req)
for x in f:
print(x)
f.close()