Java ApacheFlink（v1.6.0）验证Elasticsearch接收器（v6.4）_Java_<img Src="//i.stack.imgur.com/RUiNP.png" Height="16" Width="18" Alt="" Class="sponsor Tag Img">elasticsearch_Apache Flink_Flink Streaming

Java ApacheFlink（v1.6.0）验证Elasticsearch接收器（v6.4）

java apache-flink

Java ApacheFlink（v1.6.0）验证Elasticsearch接收器（v6.4）,java,elasticsearch,apache-flink,flink-streaming,Java,elasticsearch,Apache Flink,Flink Streaming,我正在使用ApacheFlink v1.6.0，我正在尝试编写ElasticSearchV6.4.0，它托管在中。我在对弹性云集群进行身份验证时遇到问题我已经能够让Flink写入本地Elasticsearch v6.4.0节点，该节点没有使用以下代码进行加密： /* Elasticsearch Configuration */ List<HttpHost> httpHosts = new ArrayList<>(); httpHosts.add(new Http

我正在使用ApacheFlink v1.6.0，我正在尝试编写ElasticSearchV6.4.0，它托管在中。我在对弹性云集群进行身份验证时遇到问题

我已经能够让Flink写入本地Elasticsearch v6.4.0节点，该节点没有使用以下代码进行加密：

/*
    Elasticsearch Configuration
*/
List<HttpHost> httpHosts = new ArrayList<>();
httpHosts.add(new HttpHost("127.0.0.1", 9200, "http"));

// use a ElasticsearchSink.Builder to create an ElasticsearchSink
ElasticsearchSink.Builder<ObjectNode> esSinkBuilder = new ElasticsearchSink.Builder<>(
        httpHosts,
        new ElasticsearchSinkFunction<ObjectNode>() {
            private IndexRequest createIndexRequest(ObjectNode payload) {

                // remove the value node so the fields are at the base of the json payload
                JsonNode jsonOutput = payload.get("value");

                return Requests.indexRequest()
                        .index("raw-payload")
                        .type("payload")
                        .source(jsonOutput.toString(), XContentType.JSON);
            }

            @Override
            public void process(ObjectNode payload, RuntimeContext ctx, RequestIndexer indexer) {
                indexer.add(createIndexRequest(payload));
            }
        }
);

// set number of events to be seen before writing to Elasticsearch
esSinkBuilder.setBulkFlushMaxActions(1);

// finally, build and add the sink to the job's pipeline
stream.addSink(esSinkBuilder.build());

我在执行作业时遇到以下错误：

14:49:54,700 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Stopped Akka RPC service.
Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: org.elasticsearch.ElasticsearchStatusException: method [HEAD], host [https://XXXXXXXXXXXXXX.europe-west1.gcp.cloud.es.io:9243], URI [/], status line [HTTP/1.1 401 Unauthorized]
    at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:623)
    at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:123)
    at com.downuk.AverageStockSalePrice.main(AverageStockSalePrice.java:146)
Caused by: org.elasticsearch.ElasticsearchStatusException: method [HEAD], host [https://XXXXXXXXXXXXXX.europe-west1.gcp.cloud.es.io:9243], URI [/], status line [HTTP/1.1 401 Unauthorized]
    at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:625)

有人能帮我指出哪里出了问题吗

override def configureRestClientBuilder(restClientBuilder: RestClientBuilder): Unit = {
        // TODO Additional rest client args go here - authentication headers for secure connections etc...
      }
    })

我希望这能帮助你

我希望这能对您有所帮助。

在阅读了Flink示例和Elasticsearch文档后，我能够解决这个问题

结果是我试图在上面设置错误的配置：

restClientBuilder.setDefaultHeaders(...);

不是实际需要的设置：

restClientBuilder.setHttpClientConfigCallback(...);

一旦您使用了正确的自定义配置，其余的就相当简单了。因此，第一部分缺失的是：

// provide a RestClientFactory for custom configuration on the internally created REST client
esSinkBuilder.setRestClientFactory(
    restClientBuilder -> {
        restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
            @Override
            public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {

                // elasticsearch username and password
                CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
                credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("$USERNAME", "$PASSWORD"));

                return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
            }
        });
    }
);

最后是Elasticsearch Sink的完整片段：

/*
    Elasticsearch Configuration
*/
List<HttpHost> httpHosts = new ArrayList<>();
httpHosts.add(new HttpHost("127.0.0.1", 9200, "http"));

// use a ElasticsearchSink.Builder to create an ElasticsearchSink
ElasticsearchSink.Builder<ObjectNode> esSinkBuilder = new ElasticsearchSink.Builder<>(
        httpHosts,
        new ElasticsearchSinkFunction<ObjectNode>() {
            private IndexRequest createIndexRequest(ObjectNode payload) {

                // remove the value node so the fields are at the base of the json payload
                JsonNode jsonOutput = payload.get("value");

                return Requests.indexRequest()
                        .index("raw-payload")
                        .type("payload")
                        .source(jsonOutput.toString(), XContentType.JSON);
            }

            @Override
            public void process(ObjectNode payload, RuntimeContext ctx, RequestIndexer indexer) {
                indexer.add(createIndexRequest(payload));
            }
        }
);

// set number of events to be seen before writing to Elasticsearch
esSinkBuilder.setBulkFlushMaxActions(1);

// provide a RestClientFactory for custom configuration on the internally created REST client
esSinkBuilder.setRestClientFactory(
    restClientBuilder -> {
        restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
            @Override
            public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {

                // elasticsearch username and password
                CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
                credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("$USERNAME", "$PASSWORD"));

                return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
            }
        });
    }
);

// finally, build and add the sink to the job's pipeline
stream.addSink(esSinkBuilder.build());

/*
弹性搜索配置
*/
List httpHosts=new ArrayList（）；
添加（新的HttpHost（“127.0.0.1”，9200，“http”）；
//使用ElasticsearchSink.Builder创建ElasticsearchSink
ElasticsearchSink.Builder esSinkBuilder=新建ElasticsearchSink.Builder(
httpHosts，
新的ElasticsearchSinkFunction（）{
私有IndexRequest createIndexRequest（ObjectNode负载）{
//删除value节点，使字段位于json负载的底部
JsonNode jsonOutput=payload.get（“值”）；
返回请求。indexRequest（）
.索引（“原始有效载荷”）
.类型（“有效载荷”）
.source（jsonOutput.toString（），XContentType.JSON）；
}
@凌驾
公共无效进程（ObjectNode负载、RuntimeContext ctx、RequestIndexer索引器）{
add（createIndexRequest（负载））；
}
}
);
//设置写入Elasticsearch之前要查看的事件数
esSinkBuilder.setBulkFlushMaxActions（1）；
//为内部创建的REST客户端上的自定义配置提供RestClientFactory
esSinkBuilder.setRestClientFactory(
restClientBuilder->{
restClientBuilder.setHttpClientConfigCallback（新的restClientBuilder.HttpClientConfigCallback（）{
@凌驾
公共HttpAsyncClientBuilder自定义HttpClient（HttpAsyncClientBuilder httpClientBuilder）{
//elasticsearch用户名和密码
CredentialsProvider CredentialsProvider=新的BasicCredentialsProvider（）；
setCredentials（AuthScope.ANY，新用户名密码凭据（“$USERNAME”，“$PASSWORD”））；
返回httpClientBuilder.setDefaultCredentialsProvider（credentialsProvider）；
}
});
}
);
//最后，构建接收器并将其添加到作业的管道中
stream.addSink（esSinkBuilder.build（））；

我希望这能帮助其他被困在同一个地方的人

看了Flink示例和Elasticsearch文档后，我就能够解决这个问题了

结果是我试图在上面设置错误的配置：

restClientBuilder.setDefaultHeaders(...);

不是实际需要的设置：

restClientBuilder.setHttpClientConfigCallback(...);

一旦您使用了正确的自定义配置，其余的就相当简单了。因此，第一部分缺失的是：

// provide a RestClientFactory for custom configuration on the internally created REST client
esSinkBuilder.setRestClientFactory(
    restClientBuilder -> {
        restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
            @Override
            public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {

                // elasticsearch username and password
                CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
                credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("$USERNAME", "$PASSWORD"));

                return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
            }
        });
    }
);

最后是Elasticsearch Sink的完整片段：

/*
    Elasticsearch Configuration
*/
List<HttpHost> httpHosts = new ArrayList<>();
httpHosts.add(new HttpHost("127.0.0.1", 9200, "http"));

// use a ElasticsearchSink.Builder to create an ElasticsearchSink
ElasticsearchSink.Builder<ObjectNode> esSinkBuilder = new ElasticsearchSink.Builder<>(
        httpHosts,
        new ElasticsearchSinkFunction<ObjectNode>() {
            private IndexRequest createIndexRequest(ObjectNode payload) {

                // remove the value node so the fields are at the base of the json payload
                JsonNode jsonOutput = payload.get("value");

                return Requests.indexRequest()
                        .index("raw-payload")
                        .type("payload")
                        .source(jsonOutput.toString(), XContentType.JSON);
            }

            @Override
            public void process(ObjectNode payload, RuntimeContext ctx, RequestIndexer indexer) {
                indexer.add(createIndexRequest(payload));
            }
        }
);

// set number of events to be seen before writing to Elasticsearch
esSinkBuilder.setBulkFlushMaxActions(1);

// provide a RestClientFactory for custom configuration on the internally created REST client
esSinkBuilder.setRestClientFactory(
    restClientBuilder -> {
        restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
            @Override
            public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {

                // elasticsearch username and password
                CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
                credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("$USERNAME", "$PASSWORD"));

                return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
            }
        });
    }
);

// finally, build and add the sink to the job's pipeline
stream.addSink(esSinkBuilder.build());

/*
弹性搜索配置
*/
List httpHosts=new ArrayList（）；
添加（新的HttpHost（“127.0.0.1”，9200，“http”）；
//使用ElasticsearchSink.Builder创建ElasticsearchSink
ElasticsearchSink.Builder esSinkBuilder=新建ElasticsearchSink.Builder(
httpHosts，
新的ElasticsearchSinkFunction（）{
私有IndexRequest createIndexRequest（ObjectNode负载）{
//删除value节点，使字段位于json负载的底部
JsonNode jsonOutput=payload.get（“值”）；
返回请求。indexRequest（）
.索引（“原始有效载荷”）
.类型（“有效载荷”）
.source（jsonOutput.toString（），XContentType.JSON）；
}
@凌驾
公共无效进程（ObjectNode负载、RuntimeContext ctx、RequestIndexer索引器）{
add（createIndexRequest（负载））；
}
}
);
//设置写入Elasticsearch之前要查看的事件数
esSinkBuilder.setBulkFlushMaxActions（1）；
//为内部创建的REST客户端上的自定义配置提供RestClientFactory
esSinkBuilder.setRestClientFactory(
restClientBuilder->{
restClientBuilder.setHttpClientConfigCallback（新的restClientBuilder.HttpClientConfigCallback（）{
@凌驾
公共HttpAsyncClientBuilder自定义HttpClient（HttpAsyncClientBuilder httpClientBuilder）{
//elasticsearch用户名和密码
CredentialsProvider CredentialsProvider=新的BasicCredentialsProvider（）；
setCredentials（AuthScope.ANY，新用户名密码凭据（“$USERNAME”，“$PASSWORD”））；
返回httpClientBuilder.setDefaultCredentialsProvider（credentialsProvider）；
}
});
}
);
//最后，构建接收器并将其添加到作业的管道中
stream.addSink（esSinkBuilder.build（））；

我希望这能帮助其他被困在同一个地方的人