Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/sockets/2.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Python Spark Streaming应用程序无法在端口上接收字符串_Python_Sockets_Apache Spark_Pyspark_Spark Streaming - Fatal编程技术网

Python Spark Streaming应用程序无法在端口上接收字符串

Python Spark Streaming应用程序无法在端口上接收字符串,python,sockets,apache-spark,pyspark,spark-streaming,Python,Sockets,Apache Spark,Pyspark,Spark Streaming,我想写一个Spark Streamin应用程序,它获取一个带有随机整数的流并对其进行计数。这是我写的Spark应用程序: from pyspark import SparkContext from pyspark.streaming import StreamingContext sc = SparkContext("local[2]", "IntegerCount") # 2 threads, app name ssc = StreamingContext(sc, 1) # sc, time

我想写一个Spark Streamin应用程序,它获取一个带有随机整数的流并对其进行计数。这是我写的Spark应用程序:

from pyspark import SparkContext
from pyspark.streaming import StreamingContext

sc = SparkContext("local[2]", "IntegerCount") # 2 threads, app name
ssc = StreamingContext(sc, 1) # sc, time interval for batch update.

nums = ssc.socketTextStream("localhost", 8000) # stream data from TCP; source, port

# create key,value pairs
tests = nums.map(lambda num: (int(num), 1))

# Count each integer in each batch
intCounts = tests.reduceByKey(lambda x, y: x + y)

# Print
intCounts.pprint()

ssc.start()             # Start the computation
ssc.awaitTermination()  # Wait for the computation to terminate
我将随机数发送到该服务器的端口8000.py:

import socket
from random import randint

host = 'localhost'
port = 8000
address = (host, port)

server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(address)
server_socket.listen(5)


print "Listening for client . . ."
conn, address = server_socket.accept()
print "Connected to client at ", address
#pick a large output buffer size because i dont necessarily know how big the incoming packet is
while True:
    output = str(randint(0, 10))
    conn.send(output)
当我运行Server.py和Spark应用程序时,连接成功建立。但是,我看到一个空输出,即:

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
-------------------------------------------
Time: 2017-07-16 22:36:11
-------------------------------------------

-------------------------------------------
Time: 2017-07-16 22:36:12
-------------------------------------------

我不知道问题出在哪里,请帮助我了解发生了什么?

解决了,我发送了带“\n”的字符串,结果成功了

import socket
from random import randint

host = 'localhost'
port = 8000
address = (host, port)

server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(address)
server_socket.listen(5)


print "Listening for client . . ."
conn, address = server_socket.accept()
print "Connected to client at ", address
#pick a large output buffer size because i dont necessarily know how big the incoming packet is
while True:
    output = str(randint(0, 10)) + "\n"  ### THAT IS THE FIX.
    conn.send(output)

没人知道这个问题???