获取错误名称'spark'未定义

Question

这是我使用的代码：

df = None

from pyspark.sql.functions import lit

for category in file_list_filtered:

    data_files = os.listdir('HMP_Dataset/'+category)

    for data_file in data_files:

        print(data_file)

        temp_df = spark.read.option('header', 'false').option('delimiter', ' ').csv('HMP_Dataset/'+category+'/'+data_file, schema = schema)

        temp_df = temp_df.withColumn('class', lit(category))

        temp_df = temp_df.withColumn('source', lit(data_file))

        if df is None:

            df = temp_df

        else:

            df = df.union(temp_df)

我得到了这个错误：

NameError                                 Traceback (most recent call last)

<ipython-input-4-4296b4e97942> in <module>

      9     for data_file in data_files:

     10         print(data_file)

---> 11         temp_df = spark.read.option('header', 'false').option('delimiter', ' ').csv('HMP_Dataset/'+category+'/'+data_file, schema = schema)

     12         temp_df = temp_df.withColumn('class', lit(category))

     13         temp_df = temp_df.withColumn('source', lit(data_file))

NameError: name 'spark' is not defined

我该如何解决？

Answer 1

初始化 Spark Session，然后spark在您的循环中使用。df = Nonefrom pyspark.sql.functions import litfrom pyspark.sql import SparkSessionspark = SparkSession.builder.appName('app_name').getOrCreate()for category in file_list_filtered:...

Answer 2

尝试定义sparkvarfrom pyspark.context import SparkContextfrom pyspark.sql.session import SparkSessionsc = SparkContext('local')spark = SparkSession(sc)