Pyspark：将元组类型 RDD 转换为 DataFrame

Question

我有一个复杂tuple类型的 RDD，比如

(20190701, [11,21,31], [('A',10), ('B', 20)])

schema 可以自己定义。

那么如何把它变成一个DataFrame，像这样：

date | 0 | 1 | 2 | A | B 
20190701 | 11 | 21 | 31 | 10 | 20

Answer 1

单程：from pyspark.sql import Rowrdd = sc.parallelize([(20190701, [11,21,31], [('A',10), ('B', 20)])])# customize a Row class based on schema    MRow = Row("date", "0", "1", "2", "A", "B")rdd.map(lambda x: MRow(x[0], *x[1], *map(lambda e:e[1],x[2]))).toDF().show()+--------+---+---+---+---+---+|    date|  0|  1|  2|  A|  B|+--------+---+---+---+---+---+|20190701| 11| 21| 31| 10| 20|+--------+---+---+---+---+---+或者另一种方式：rdd.map(lambda x: Row(date=x[0], **dict((str(i), e) for i,e in list(enumerate(x[1])) + x[2]))).toDF().show()+---+---+---+---+---+--------+|  0|  1|  2|  A|  B|    date|+---+---+---+---+---+--------+| 11| 21| 31| 10| 20|20190701|+---+---+---+---+---+--------+

Answer 2

rdd = sc.parallelize((20190701, [11,21,31], [('A',10), ('B', 20)]))elements = rdd.take(3)a = [elements[0]] + (elements[1]) + [elements[2][0][1], elements[2][1][1]]import pandas as pdsdf = spark.createDataFrame(pd.DataFrame([20190701, 11, 21, 31, 10, 20]).T)