如何使用Java Spark拼花写入停止JVM核心转储

如何使用Java Spark拼花写入停止JVM核心转储,java,apache-spark,Java,Apache Spark,Spark遇到问题,尝试“手动”创建数据集,然后将其写入parquet。我已经简化了这个例子,这样你就可以看到发生了什么 我有两个类(它们来自图书馆,所以不容易更改) 如果我从行程中除掉一些成员,它就会开始工作。如果我去掉isPrivate()方法,它就会开始工作 有人知道这里发生了什么吗?我最后不得不更改rawcinerary类定义,将isPrivate()方法更改为与bean不同的方法。完成后,一切都正常了。可能是因为Encoders.bean无法为isPrivate创建编码器吗?我最终不得

Spark遇到问题,尝试“手动”创建数据集,然后将其写入parquet。我已经简化了这个例子,这样你就可以看到发生了什么

我有两个类(它们来自图书馆,所以不容易更改)

如果我从行程中除掉一些成员,它就会开始工作。如果我去掉isPrivate()方法,它就会开始工作


有人知道这里发生了什么吗?

我最后不得不更改rawcinerary类定义,将isPrivate()方法更改为与bean不同的方法。完成后,一切都正常了。可能是因为Encoders.bean无法为isPrivate创建编码器吗?我最终不得不更改RawInventurey类定义,以将isPrivate()方法更改为与bean不同的方法。完成后,一切都正常了。可能是因为Encoders.bean无法为isPrivate创建编码器?
public class RawSearch implements Serializable {

    protected long timestamp;

    @Nullable
    protected List<RawItinerary> itineraries;

}

public class RawItinerary implements Serializable {

    protected long timestamp;

    @Nullable
    protected String originCityCode;

    @Nullable
    protected String destinationCityCode;

    protected int departDate;
    protected int returnDate;
    protected String currencyCode;

    protected double totalAmount;
    protected double totalTaxAmount;
    protected double totalSurchargeAmount;

    @Nullable
    protected String validatingCarrier;

    protected boolean multiCarrier;

    @Nullable
    protected String cabin;

    public boolean isPrivate() {
        return false;
    }

}
       List<RawSearch> searches = new ArrayList<>();

        RawSearch rawSearch = new RawSearch();

        rawSearch.setTimestamp(System.currentTimeMillis());
        rawSearch.setItineraries(new ArrayList<>());
        searches.add(rawSearch);

        RawItinerary itinerary = new RawItinerary();

        itinerary.setCabin("Coach");
        itinerary.setCurrencyCode("USD");
        itinerary.setDepartDate(20200215);
        itinerary.setDestinationCityCode("EDI");
        itinerary.setMultiCarrier(false);
        itinerary.setOriginCityCode("LON");
        itinerary.setReturnDate(0);
        itinerary.setTimestamp(System.currentTimeMillis());
        itinerary.setValidatingCarrier("BA");

        rawSearch.getItineraries().add(itinerary);

        SparkSession sparkSession = SparkSession.builder().appName("Bob").master("local[1]").getOrCreate();

        Dataset<RawSearch> searchDataset = sparkSession.createDataset(searches, Encoders.bean(RawSearch.class));

        searchDataset.write()
                .parquet("/Users/graemewallace/Downloads/bob-" + System.currentTimeMillis());

        sparkSession.close();
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0xa) at pc=0x0000000109adccda, pid=56878, tid=0x0000000000008903
#
# JRE version: Java(TM) SE Runtime Environment (8.0_241-b07) (build 1.8.0_241-b07)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.241-b07 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.dylib+0xdccda]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#