Spark Core-RDD三部曲(上)

 * Internally, each RDD is characterized by five main properties:
 *
 *  - A list of partitions
 *  - A function for computing each split
 *  - A list of dependencies on other RDDs
 *  - Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned)
 *  - Optionally, a list of preferred locations to compute each split on (e.g. block locations for an HDFS file)
复制

// 创建配置对象
val sparkConf = new SparkConf().setMaster("local[*]").setAppName("WordCount");
val sc = new SparkContext(sparkConf)
  
/** Word Count **/
val fileRDD: RDD[String] = sc.textFile("data/input/word.txt") // (1)读取,一行一行的读取文件
val flatRDD: RDD[String] = fileRDD.flatMap( _.split(" ") )    // (2)切分,一行字符切成单个单词
val mapRDD: RDD[(String, Int)] = flatRDD.map((_,1))           // (3)转换,数据结构 word => (word, 1)
val reduceRDD: RDD[(String, Int)] = mapRDD.reduceByKey(_+_)   // (4)聚合,相同相加
复制