Google cloud platform 如何在使用Google Cloud Dataflow清除云Memorystore中的缓存后插入数据？

如果数据流要处理的输入文件有数据，我正在执行一项清除memorystore缓存的任务。这意味着，如果输入文件没有记录，则不会刷新memorystore，但输入文件甚至有一条记录，则应刷新memorystore，然后处理输入文件

我的数据流应用程序是一个多管道应用程序，它读取、处理数据，然后将数据存储在memorystore中。管道正在成功执行。但是，内存存储的刷新正在工作，但在刷新之后，插入没有发生

我编写了一个函数，在检查输入文件是否有记录后刷新memorystore

FlushingMemorystore.java

package com.click.example.functions;

import afu.org.checkerframework.checker.nullness.qual.Nullable;
import com.google.auto.value.AutoValue;
import org.apache.beam.sdk.io.redis.RedisConnectionConfiguration;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.vendor.grpc.v1p26p0.com.google.common.base.Preconditions;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.Pipeline;

public class FlushingMemorystore {


    private static final Logger LOGGER = LoggerFactory.getLogger(FlushingMemorystore.class);

    public static FlushingMemorystore.Read read() {
        return (new AutoValue_FlushingMemorystore_Read.Builder())
                .setConnectionConfiguration(RedisConnectionConfiguration.create()).build();
    }

    @AutoValue
    public abstract static class Read extends PTransform<PCollection<Long>, PDone> {

        public Read() {
        }

        @Nullable
        abstract RedisConnectionConfiguration connectionConfiguration();

        @Nullable
        abstract Long expireTime();
        abstract FlushingMemorystore.Read.Builder toBuilder();

        public FlushingMemorystore.Read withEndpoint(String host, int port) {
            Preconditions.checkArgument(host != null, "host cannot be null");
            Preconditions.checkArgument(port > 0, "port cannot be negative or 0");
            return this.toBuilder().setConnectionConfiguration(this.connectionConfiguration().withHost(host).withPort(port)).build();
        }

        public FlushingMemorystore.Read withAuth(String auth) {
            Preconditions.checkArgument(auth != null, "auth cannot be null");
            return this.toBuilder().setConnectionConfiguration(this.connectionConfiguration().withAuth(auth)).build();
        }

        public FlushingMemorystore.Read withTimeout(int timeout) {
            Preconditions.checkArgument(timeout >= 0, "timeout cannot be negative");
            return this.toBuilder().setConnectionConfiguration(this.connectionConfiguration().withTimeout(timeout)).build();
        }

        public FlushingMemorystore.Read withConnectionConfiguration(RedisConnectionConfiguration connectionConfiguration) {
            Preconditions.checkArgument(connectionConfiguration != null, "connection cannot be null");
            return this.toBuilder().setConnectionConfiguration(connectionConfiguration).build();
        }

        public FlushingMemorystore.Read withExpireTime(Long expireTimeMillis) {
            Preconditions.checkArgument(expireTimeMillis != null, "expireTimeMillis cannot be null");
            Preconditions.checkArgument(expireTimeMillis > 0L, "expireTimeMillis cannot be negative or 0");
            return this.toBuilder().setExpireTime(expireTimeMillis).build();
        }

        public PDone expand(PCollection<Long> input) {
            Preconditions.checkArgument(this.connectionConfiguration() != null, "withConnectionConfiguration() is required");
            input.apply(ParDo.of(new FlushingMemorystore.Read.ReadFn(this)));
            return PDone.in(input.getPipeline());
        }

        private static class ReadFn extends DoFn<Long, String> {
            private static final int DEFAULT_BATCH_SIZE = 1000;
            private final FlushingMemorystore.Read spec;
            private transient Jedis jedis;
            private transient Pipeline pipeline;
            private int batchCount;

            public ReadFn(FlushingMemorystore.Read spec) {
                this.spec = spec;
            }

            @Setup
            public void setup() {
                this.jedis = this.spec.connectionConfiguration().connect();
            }

            @StartBundle
            public void startBundle() {
                this.pipeline = this.jedis.pipelined();
                this.pipeline.multi();
                this.batchCount = 0;
            }

            @ProcessElement
            public void processElement(DoFn<Long, String>.ProcessContext c) {
                Long count = c.element();
                batchCount++;

                if(count==null && count < 0) {
                    LOGGER.info("No Records are there in the input file");
                } else {
                    if (pipeline.isInMulti()) {
                        pipeline.exec();
                        pipeline.sync();
                        jedis.flushDB();
                    }
                    LOGGER.info("*****The memorystore is flushed*****");
                }
            }

            @FinishBundle
            public void finishBundle() {
                if (this.pipeline.isInMulti()) {
                    this.pipeline.exec();
                    this.pipeline.sync();
                }
                this.batchCount=0;
            }

            @Teardown
            public void teardown() {
                this.jedis.close();
            }

        }

        @AutoValue.Builder
        abstract static class Builder {

            Builder() {
            }

            abstract FlushingMemorystore.Read.Builder setExpireTime(Long expireTimeMillis);

            abstract FlushingMemorystore.Read build();

            abstract FlushingMemorystore.Read.Builder setConnectionConfiguration(RedisConnectionConfiguration connectionConfiguration);

        }

    }

}

数据流执行得很好，它也会刷新memorystore，但在此之后插入就不起作用了。你能指出我错在哪里吗？ 任何解决问题的办法都是值得赞赏的。提前谢谢

编辑：

 StorageToRedisOptions options = PipelineOptionsFactory.fromArgs(args)
                .withValidation()
                .as(StorageToRedisOptions.class);

        Pipeline p = Pipeline.create(options);

        PCollection<String> lines = p.apply(
                "ReadLines", TextIO.read().from(options.getInputFile()));

        /**
         * Flushing the Memorystore if there are records in the input file
         */
        lines.apply("Checking Data in input file", Count.globally())
                .apply("Flushing the data store", FlushingMemorystore.read()
                        .withConnectionConfiguration(RedisConnectionConfiguration
                        .create(options.getRedisHost(), options.getRedisPort())));

 dataset.apply(SOME_DATASET_TRANSFORMATION, RedisIO.write()
                .withMethod(RedisIO.Write.Method.SADD)
                .withConnectionConfiguration(RedisConnectionConfiguration
                        .create(options.getRedisHost(), options.getRedisPort())));

提供评论中要求的其他信息

使用的运行时是Java11，它使用ApacheBeamSDK for 2.24.0

如果输入文件有记录，它将用一些逻辑处理数据。例如，如果输入文件包含以下数据：

abcabc|Bruce|Wayne|2000
abbabb|Tony|Stark|3423

数据流将统计记录的数量，在本例中为2，并根据逻辑处理id、名字等，然后存储在memorystore中。此输入文件每天都会出现，因此，如果输入文件有记录，则应清除（或刷新）memorystore

---

虽然管道没有中断，但我认为我遗漏了一些东西。

我怀疑这里的问题是，您需要确保在RedisIO.write步骤发生之前运行（并完成）“刷新”步骤。Beam具有可用于此操作的变换

为了实现这一点，我们可以使用flushing pttransform的输出作为我们已经刷新了数据库的信号，并且我们只在完成刷新后才向数据库写入数据。刷新DoFn的

过程调用如下所示：

@ProcessElement
public void processElement（DoFn.ProcessContext c）{
长计数=c.元素（）；
if（count==null&&count<0）{
info（“输入文件中没有记录”）；
}否则{
if（pipeline.isInMulti（））{
pipeline.exec（）；
sync（）；
绝地武士；
}
LOGGER.info（“******内存存储已刷新******”）；
}
c、 输出（“就绪”）；
}

一旦有信号指示数据库已刷新，我们可以使用它等待，然后再将新数据写入其中：

Pipeline p=Pipeline.create（选项）；
p收集线=p应用(
“ReadLines”，TextIO.read（）.from（options.getInputFile（））；
/**
*如果输入文件中有记录，则刷新Memorystore
*/
PCollection flushedSignal=线路
.apply（“检查输入文件中的数据”，Count.globally（））
.apply（“刷新数据存储”，FlushingMemorystore.read（））
.带连接配置（重新连接配置
.create（options.getRedisHost（），options.getRedisPort（））；
//然后，我们使用刷新信号开始写入Redis：
数据集
.apply（等待打开（flushedSignal））
.apply（一些数据集转换，RedisIO.write（）
.withMethod（RedisIO.Write.Method.SADD）
.带连接配置（重新连接配置
.create（options.getRedisHost（），options.getRedisPort（））；
在我申请Wait.on转换后，问题得到了解决，Pablo的回答已经解释了这一点。但是，我不得不将FlushingMemorystore.java稍微重写为flushSignal标志的PCollection

下面是函数：

package com.click.example.functions;

import afu.org.checkerframework.checker.nullness.qual.Nullable;
import com.google.auto.value.AutoValue;
import org.apache.beam.sdk.io.redis.RedisConnectionConfiguration;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.vendor.grpc.v1p26p0.com.google.common.base.Preconditions;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.Pipeline;

public class FlushingMemorystore extends DoFn<Long, String> {

    private static final Logger LOGGER = LoggerFactory.getLogger(FlushingMemorystore.class);

    public static FlushingMemorystore.Read read() {
        return (new AutoValue_FlushingMemorystore_Read.Builder())
                .setConnectionConfiguration(RedisConnectionConfiguration.create()).build();
    }

    @AutoValue
    public abstract static class Read extends PTransform<PCollection<Long>, PCollection<String>> {

        public Read() {
        }

        @Nullable
        abstract RedisConnectionConfiguration connectionConfiguration();

        @Nullable
        abstract Long expireTime();
        abstract FlushingMemorystore.Read.Builder toBuilder();

        public FlushingMemorystore.Read withEndpoint(String host, int port) {
            Preconditions.checkArgument(host != null, "host cannot be null");
            Preconditions.checkArgument(port > 0, "port cannot be negative or 0");
            return this.toBuilder().setConnectionConfiguration(this.connectionConfiguration().withHost(host).withPort(port)).build();
        }

        public FlushingMemorystore.Read withAuth(String auth) {
            Preconditions.checkArgument(auth != null, "auth cannot be null");
            return this.toBuilder().setConnectionConfiguration(this.connectionConfiguration().withAuth(auth)).build();
        }

        public FlushingMemorystore.Read withTimeout(int timeout) {
            Preconditions.checkArgument(timeout >= 0, "timeout cannot be negative");
            return this.toBuilder().setConnectionConfiguration(this.connectionConfiguration().withTimeout(timeout)).build();
        }

        public FlushingMemorystore.Read withConnectionConfiguration(RedisConnectionConfiguration connectionConfiguration) {
            Preconditions.checkArgument(connectionConfiguration != null, "connection cannot be null");
            return this.toBuilder().setConnectionConfiguration(connectionConfiguration).build();
        }

        public FlushingMemorystore.Read withExpireTime(Long expireTimeMillis) {
            Preconditions.checkArgument(expireTimeMillis != null, "expireTimeMillis cannot be null");
            Preconditions.checkArgument(expireTimeMillis > 0L, "expireTimeMillis cannot be negative or 0");
            return this.toBuilder().setExpireTime(expireTimeMillis).build();
        }

        public PCollection<String> expand(PCollection<Long> input) {
            Preconditions.checkArgument(this.connectionConfiguration() != null, "withConnectionConfiguration() is required");
           return input.apply(ParDo.of(new FlushingMemorystore.Read.ReadFn(this)));
        }

        @Setup
        public Jedis setup() {
            return this.connectionConfiguration().connect();
        }

        private static class ReadFn extends DoFn<Long, String> {
            private static final int DEFAULT_BATCH_SIZE = 1000;
            private final FlushingMemorystore.Read spec;
            private transient Jedis jedis;
            private transient Pipeline pipeline;
            private int batchCount;

            public ReadFn(FlushingMemorystore.Read spec) {
                this.spec = spec;
            }

            @Setup
            public void setup() {
                this.jedis = this.spec.connectionConfiguration().connect();
            }

            @StartBundle
            public void startBundle() {
                this.pipeline = this.jedis.pipelined();
                this.pipeline.multi();
                this.batchCount = 0;
            }

            @ProcessElement
            public void processElement(@Element Long count, OutputReceiver<String> out) {
                batchCount++;

                if(count!=null && count > 0) {
                    if (pipeline.isInMulti()) {
                        pipeline.exec();
                        pipeline.sync();
                        jedis.flushDB();
                        LOGGER.info("*****The memorystore is flushed*****");
                    }
                    out.output("SUCCESS");
                } else {
                    LOGGER.info("No Records are there in the input file");
                    out.output("FAILURE");
                }

            }

            @FinishBundle
            public void finishBundle() {
                if (this.pipeline.isInMulti()) {
                    this.pipeline.exec();
                    this.pipeline.sync();
                }
                this.batchCount=0;
            }

            @Teardown
            public void teardown() {
                this.jedis.close();
            }

        }

        @AutoValue.Builder
        abstract static class Builder {

            Builder() {
            }

            abstract FlushingMemorystore.Read.Builder setExpireTime(Long expireTimeMillis);

            abstract FlushingMemorystore.Read build();

          abstract FlushingMemorystore.Read.Builder setConnectionConfiguration(RedisConnectionConfiguration connectionConfiguration);

        }

    }

}

package com.click.example.functions；
导入afu.org.checkerframework.checker.nullness.qual.Nullable；
导入com.google.auto.value.AutoValue；
导入org.apache.beam.sdk.io.redis.RedisConnectionConfiguration；
导入org.apache.beam.sdk.transforms.DoFn；
导入org.apache.beam.sdk.transforms.ptTransform；
导入org.apache.beam.sdk.transforms.ParDo；
导入org.apache.beam.sdk.values.PCollection；
导入org.apache.beam.vendor.grpc.v1p26p0.com.google.common.base.premissions；
导入org.slf4j.Logger；
导入org.slf4j.LoggerFactory；
导入redis.clients.jedis.jedis；
导入redis.clients.jedis.Pipeline；
公共类FlushingMemorystore扩展了DoFn{
私有静态最终记录器Logger=LoggerFactory.getLogger（FlushingMemorystore.class）；
公共静态FlushingMemorystore.Read读取（）{
返回（新的自动值\u FlushingMemorystore\u Read.Builder（））
.setConnectionConfiguration（重新连接配置.create（））.build（）；
}
@自动值
公共抽象静态类读取扩展了pttransform{
公众阅读（）{
}
@可空
抽象重新连接配置connectionConfiguration（）；
@可空
抽象长呼气时间（）；
抽象FlushingMemorystore.Read.Builder toBuilder（）；
公共FlushingMemorystore.ReadWithEndpoint（字符串主机，int端口）{
前提条件.checkArgument（主机！=null，“主机不能为null”）；
先决条件.checkArgument（端口>0，“端口不能为负或0”）；
返回此.toBuilder（）.setConnectionConfiguration（this.connectionConfiguration（）.withHost（host）.withPort（port））.build（）；
}
public FlushingMemorystore.ReadWithAuth（字符串身份验证）{
预条件.checkArgument（auth！=null，“auth不能为null”）；
返回this.toBuilder（）.setConnectionConfiguration（this.connectionConfiguration（）.withAuth（auth））.build（）；
}
公共FlushingMemorystore.ReadWithTimeout（int timeout）{
前提条件