Python Spark mllib线性回归给出了非常糟糕的结果
当我尝试使用Spark mllib的LinearRegressionWithGD使用Python进行线性回归时，我得到了非常糟糕的结果 我研究了类似的问题，比如：Python Spark mllib线性回归给出了非常糟糕的结果,python,apache-spark,pyspark,linear-regression,apache-spark-mllib,Python,Apache Spark,Pyspark,Linear Regression,Apache Spark Mllib,当我尝试使用Spark mllib的LinearRegressionWithGD使用Python进行线性回归时，我得到了非常糟糕的结果 我研究了类似的问题，比如： 我很清楚，关键是正确调整参数 我也知道随机梯度下降不一定能找到最优解（就像交替最小二乘法一样），因为它有可能陷入局部极小值。但至少我希望能找到一个好的模型 这是我的设置，我选择使用《统计教育杂志》和相应的。我从这篇论文（以及复制JMP中的结果）中了解到，如果我只使用数值场，我应该得到类似于以下等式的结果（R^2约为44%，R
from collections import Iterable
from pyspark import SparkContext
from pyspark.mllib.regression import LabeledPoint
from pyspark.mllib.regression import LinearRegressionWithSGD
from pyspark.mllib.evaluation import RegressionMetrics
def f(n):
return float(n)
if __name__ == "__main__":
sc = SparkContext(appName="LinearRegressionExample")
# CSV file format:
# 0 1 2 3 4 5 6 7 8 9 10 11
# Price, Mileage, Make, Model, Trim, Type, Cylinder, Liter, Doors, Cruise, Sound, Leather
raw_data = sc.textFile('file:///home/ccastroh/training/pyspark/kuiper.csv')
# Grabbing numerical values only (for now)
data = raw_data \
.map(lambda x : x.split(',')) \
.map(lambda x : [f(x[0]), f(x[1]), f(x[6]), f(x[8]), f(x[9]), f(x[10]), f(x[11])])
points = data.map(lambda x : LabeledPoint(x[0], x[1:])).cache()
print "Num, Iterations, Step, MiniBatch, RegParam, RegType, Intercept?, Validation?, " + \
"RMSE, R2, EXPLAINED VARIANCE, INTERCEPT, WEIGHTS..."
i = 0
for ite in [10, 100, 1000]:
for stp in [1, 1e-01, 1e-02, 1e-03, 1e-04, 1e-05, 1e-06, 1e-07, 1e-08, 1e-09, 1e-10]:
for mini in [0.2, 0.4, 0.6, 0.8, 1.0]:
for regP in [0.0, 0.1, 0.01, 0.001]:
for regT in [None, 'l1', 'l2']:
for intr in [True]:
for vald in [False, True]:
i += 1
message = str(i) + \
"," + str(ite) + \
"," + str(stp) + \
"," + str(mini) + \
"," + str(regP) + \
"," + str(regT) + \
"," + str(intr) + \
"," + str(vald)
model = LinearRegressionWithSGD.train(points, iterations=ite, step=stp, \
miniBatchFraction=mini, regParam=regP, regType=regT, intercept=intr, \
validateData=vald)
predictions_observations = points \
.map(lambda p : (float(model.predict(p.features)), p.label)).cache()
metrics = RegressionMetrics(predictions_observations)
message += "," + str(metrics.rootMeanSquaredError) \
+ "," + str(metrics.r2) \
+ "," + str(metrics.explainedVariance)
message += "," + str(model.intercept)
for weight in model.weights:
message += "," + str(weight)
print message
sc.stop()
- 在很多次跑步中，我都得到了NaN的截距和重量
- 我得到的最高R^2是-0.89。我甚至不知道你会得到一个负的R^2。结果表明，负值表示选择的模型
- 我得到的最低RMSE是13600，比预期的7400差得多