如何为创建类标记以使用java构建广播变量spark?
我需要在JAVA函数中创建一个borodcast,一个字符串列表的广播如何为创建类标记以使用java构建广播变量spark?,java,apache-spark,Java,Apache Spark,我需要在JAVA函数中创建一个borodcast,一个字符串列表的广播 Broadcast<List<String>>broadcastSp = sc.broadcast(Tvalue,classTag<T>evidence); broadcasts sp=sc.broadcast(Tvalue,classtag证据); T值是“my_list”(list),但问题是如何创建第二部分classTagevidence,不需要这样做。在Java中工作时,您不应
Broadcast<List<String>>broadcastSp = sc.broadcast(Tvalue,classTag<T>evidence);
broadcasts sp=sc.broadcast(Tvalue,classtag证据);
T值是“my_list”(
list)
,但问题是如何创建第二部分classTagevidence
,不需要这样做。在Java中工作时,您不应该使用org.apache.spark.SparkContext
,它的设计考虑了Scala。而是使用org.apache.spark.api.java.JavaSparkContext
引用官方文件:
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.SparkConf;
SparkConf conf = new SparkConf().setAppName("broadcast").setMaster("local[*]");
JavaSparkContext sc = new JavaSparkContext(conf);
它的广播方法不需要ClassTags
():
broadcastVar=sc.Broadcast(新的int[]{1,2,3});
这是我的功能:
public static JavaSparkContext sc;
public static Broadcast < List < String >> broadcastP;
public static void main(String args[]) throws Exception {
sc = new JavaSparkContext("local", "test");
.
.
.
private static List < String > fFtItemSets(JavaRDD < String > base_initiale) throws Exception {
List < String > kMinusOneSets;
List < String > k_sets;
int i = 1;
k_sets = remplir_ksets_intiale(base_initiale);
System.out.println(k_sets);
List < String > k_sets1 = k_sets;
int NB = k_sets.size();
while (NB > 1) {
kMinusOneSets = k_sets;
k_sets = Jointure(kMinusOneSets);
k_sets = Elagage(k_sets, k_sets1, i);
System.out.println(k_sets);
Broadcast < List < String >> broadcastSp = sc.broadcast(k - sets, List < String > );
//main.sc.broadcast(k_sets);
System.out.println(broadcastSp.value());
k_sets = tr_freq(k_sets, broadcastnSp, base_initiale, i);
k_sets1 = k_sets;
NB++;
i++;
}
return k_sets;
}
publicstaticjavasparkcontext-sc;
公共静态广播>broadcastP;
公共静态void main(字符串args[])引发异常{
sc=新的JavaSparkContext(“本地”、“测试”);
.
.
.
私有静态列表fFtItemSets(JavaRDDbase_initiale)引发异常{
列出kMinusOneSets;
列出k_集;
int i=1;
k_集=remplir_ksets_initial(基首字母);
系统输出打印LN(k_集);
列表k_集1=k_集;
int NB=k_set.size();
而(NB>1){
kMinusOneSets=k_集;
k_集=接合(Kminusonessets);
k_集=Elagage(k_集,k_集1,i);
系统输出打印LN(k_集);
Broadcast>broadcasts sp=sc.Broadcast(k组,List);
//主播、主播(k_台);
System.out.println(broadcastSp.value());
k_集=tr_频率(k_集,广播NSP,基首字母,i);
k_集1=k_集;
NB++;
i++;
}
返回k_集;
}
my_list是一个字符串列表问题在以下部分:Broadcast broadcasts sp=sc.Broadcast(k-set,classTag);如果您想在问题中添加新信息,请使用链接。答案应仅用于实际答案。谢谢@user10465661,但我必须创建类标记,我与您分享我的功能
public static JavaSparkContext sc;
public static Broadcast < List < String >> broadcastP;
public static void main(String args[]) throws Exception {
sc = new JavaSparkContext("local", "test");
.
.
.
private static List < String > fFtItemSets(JavaRDD < String > base_initiale) throws Exception {
List < String > kMinusOneSets;
List < String > k_sets;
int i = 1;
k_sets = remplir_ksets_intiale(base_initiale);
System.out.println(k_sets);
List < String > k_sets1 = k_sets;
int NB = k_sets.size();
while (NB > 1) {
kMinusOneSets = k_sets;
k_sets = Jointure(kMinusOneSets);
k_sets = Elagage(k_sets, k_sets1, i);
System.out.println(k_sets);
Broadcast < List < String >> broadcastSp = sc.broadcast(k - sets, List < String > );
//main.sc.broadcast(k_sets);
System.out.println(broadcastSp.value());
k_sets = tr_freq(k_sets, broadcastnSp, base_initiale, i);
k_sets1 = k_sets;
NB++;
i++;
}
return k_sets;
}