第72课：Spark UDF和UDAF解密学习笔记
第72课：Spark UDF和UDAF解密学习笔记
本期内容：
1 Spark UDF实战
2 Spark UDAF实战
UDAF=USER DEFINE AGGREGATE FUNCTION
下面直接实战编写UDF和UDAF：
package SparkSQLByScala
import org.apache.spark.sql.expressions.{MutableAggregationBuffer, UserDefinedAggregateFunction}
import org.apache.spark.sql.types._
import org.apache.spark.sql.{Row, SQLContext}
import org.apache.spark.{SparkContext, SparkConf}
/**
* 使用Scala开发集群运行的Spark WordCount程序
* @author DT大数据梦工厂
* 新浪微博：http://weibo.com/ilovepains/
* Created by hp on 2016/3/31.
* 通过案例实战Spark SQL下的UDF和UDAF的具体使用：
* UDF: User Defined Function，用户自定义的函数，函数的输入是一条具体的数据记录，实现上讲就是普通的Scala函数；
* UDAF：User Defined Aggregation Function，用户自定义的聚合函数，函数本身作用于数据集合，能够在聚合操作的基础上进行自定义操作；
*
* 实质上讲，例如说UDF会被Spark SQL中的Catalyst封装成为Expression，最终会通过eval方法来计算输入的数据Row（此处的Row和DataFrame
* 中的Row没有任何关系）
*
*/
object SparkSQLUDFUDAF {
def main (args: Array[String]) {
/**
* 第1步：创建Spark的配置对象SparkConf，设置Spark程序的运行时的配置信息，
* 例如说通过setMaster来设置程序要链接的Spark集群的Master的URL,如果设置
* 为local，则代表Spark程序在本地运行，特别适合于机器配置条件非常差（例如
* 只有1G的内存）的初学者 *
*/
val conf =new SparkConf() //创建SparkConf对象
conf.setAppName("SparkSQLUDFUDAF")//设置应用程序的名称，在程序运行的监控界面可以看到名称
// conf.setMaster("spark://Master:7077") //此时，程序在Spark集群
conf.setMaster("local[4]")
/**
* 第2步：创建SparkContext对象
* SparkContext是Spark程序所有功能的唯一入口，无论是采用Scala、Java、Python、R等都必须有一个SparkContext
* SparkContext核心作用：初始化Spark应用程序运行所需要的核心组件，包括DAGScheduler、TaskScheduler、SchedulerBackend
* 同时还会负责Spark程序往Master注册程序等
* SparkContext是整个Spark应用程序中最为至关重要的一个对象
*/
val sc =new SparkContext(conf)//创建SparkContext对象，通过传入SparkConf实例来定制Spark运行的具体参数和配置信息
val sqlContext =new SQLContext(sc)//构建SQL上下文
//模拟实际使用的数据
val bigData = Array("Spark","Spark", "Hadoop", "Spark", "Hadoop", "Spark", "Spark", "Hadoop", "Spark", "Hadoop")
/**
* 基于提供的数据创建DataFrame
*/
val bigDataRDD = sc.parallelize(bigData)
val bigDataRDDRow =bigDataRDD.map(item => Row(item))
val structType = StructType(Array(StructField("word", StringType,true)))
val bigDataDF =sqlContext.createDataFrame(bigDataRDDRow,structType)
bigDataDF.registerTempTable("bigDataTable")//注册成为临时表
/**
* 通过SQLContext注册UDF，在Scala 2.10.x版本UDF函数最多可以接受22个输入参数
*/
sqlContext.udf.register("computeLength", (input: String) => input.length)
//直接在SQL语句中使用UDF，就像使用SQL自动的内部函数一样
sqlContext.sql("select word, computeLength(word) as length from bigDataTable").show
sqlContext.udf.register("wordCount",new MyUDAF)
sqlContext.sql("select word,wordCount(word) as count,computeLength(word) as length" +
" from bigDataTable group by word").show()
while(true)()
}
}
/**
* 按照模板实现UDAF
*/
class MyUDAF extends UserDefinedAggregateFunction {
/**
* 该方法指定具体输入数据的类型
* @return
*/
override def inputSchema: StructType = StructType(Array(StructField("input", StringType,true)))
/**
* 在进行聚合操作的时候所要处理的数据的结果的类型
* @return
*/
override def bufferSchema: StructType = StructType(Array(StructField("count", IntegerType,true)))
/**
* 指定UDAF函数计算后返回的结果类型
* @return
*/
override def dataType: DataType = IntegerType
override def deterministic: Boolean =true
/**
* 在Aggregate之前每组数据的初始化结果
* @param buffer
*/
override def initialize(buffer: MutableAggregationBuffer): Unit = {buffer(0) =0}
/**
* 在进行聚合的时候，每当有新的值进来，对分组后的聚合如何进行计算
* 本地的聚合操作，相当于Hadoop MapReduce模型中的Combiner
* @param buffer
* @param input
*/
override def update(buffer: MutableAggregationBuffer, input: Row): Unit = {
buffer(0) = buffer.getAs[Int](0) +1
}
/**
* 最后在分布式节点进行Local Reduce完成后需要进行全局级别的Merge操作
* @param buffer1
* @param buffer2
*/
override def merge(buffer1: MutableAggregationBuffer, buffer2: Row): Unit = {
buffer1(0) = buffer1.getAs[Int](0) + buffer2.getAs[Int](0)
}
/**
* 返回UDAF最后的计算结果
* @param buffer
* @return
*/
override def evaluate(buffer: Row): Any = buffer.getAs[Int](0)
}
在Eclipse中运行如下：
16/04/13 23:54:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
16/04/13 23:54:38 INFO Executor: Finished task 61.0 in stage 5.0 (TID 70). 1609 bytes result sent to driver
16/04/13 23:54:38 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 12 ms
16/04/13 23:54:38 INFO Executor: Finished task 59.0 in stage 5.0 (TID 68). 1609 bytes result sent to driver
16/04/13 23:54:38 INFO Executor: Finished task 47.0 in stage 5.0 (TID 56). 1609 bytes result sent to driver
16/04/13 23:54:38 INFO TaskSetManager: Starting task 62.0 in stage 5.0 (TID 71, localhost, partition 63,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:38 INFO TaskSetManager: Starting task 63.0 in stage 5.0 (TID 72, localhost, partition 64,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:38 INFO Executor: Running task 62.0 in stage 5.0 (TID 71)
16/04/13 23:54:38 INFO TaskSetManager: Starting task 64.0 in stage 5.0 (TID 73, localhost, partition 65,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:38 INFO TaskSetManager: Finished task 61.0 in stage 5.0 (TID 70) in 184 ms on localhost (59/199)
16/04/13 23:54:38 INFO TaskSetManager: Finished task 59.0 in stage 5.0 (TID 68) in 250 ms on localhost (60/199)
16/04/13 23:54:38 INFO Executor: Running task 63.0 in stage 5.0 (TID 72)
16/04/13 23:54:39 INFO Executor: Running task 64.0 in stage 5.0 (TID 73)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 6 ms
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 46 ms
16/04/13 23:54:39 INFO Executor: Finished task 63.0 in stage 5.0 (TID 72). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 45 ms
16/04/13 23:54:39 INFO TaskSetManager: Starting task 65.0 in stage 5.0 (TID 74, localhost, partition 66,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO Executor: Finished task 64.0 in stage 5.0 (TID 73). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO TaskSetManager: Finished task 63.0 in stage 5.0 (TID 72) in 107 ms on localhost (61/199)
16/04/13 23:54:39 INFO Executor: Running task 65.0 in stage 5.0 (TID 74)
16/04/13 23:54:39 INFO TaskSetManager: Starting task 66.0 in stage 5.0 (TID 75, localhost, partition 67,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 64.0 in stage 5.0 (TID 73) in 121 ms on localhost (62/199)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:39 INFO Executor: Running task 66.0 in stage 5.0 (TID 75)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:39 INFO TaskSetManager: Starting task 67.0 in stage 5.0 (TID 76, localhost, partition 68,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO Executor: Finished task 66.0 in stage 5.0 (TID 75). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO Executor: Running task 67.0 in stage 5.0 (TID 76)
16/04/13 23:54:39 INFO TaskSetManager: Starting task 68.0 in stage 5.0 (TID 77, localhost, partition 69,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 66.0 in stage 5.0 (TID 75) in 43 ms on localhost (63/199)
16/04/13 23:54:39 INFO Executor: Running task 68.0 in stage 5.0 (TID 77)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:39 INFO Executor: Finished task 67.0 in stage 5.0 (TID 76). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO TaskSetManager: Starting task 69.0 in stage 5.0 (TID 78, localhost, partition 70,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO Executor: Running task 69.0 in stage 5.0 (TID 78)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 60.0 in stage 5.0 (TID 69) in 426 ms on localhost (64/199)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 47.0 in stage 5.0 (TID 56) in 881 ms on localhost (65/199)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 67.0 in stage 5.0 (TID 76) in 67 ms on localhost (66/199)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:39 INFO Executor: Finished task 69.0 in stage 5.0 (TID 78). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO TaskSetManager: Starting task 70.0 in stage 5.0 (TID 79, localhost, partition 71,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 69.0 in stage 5.0 (TID 78) in 36 ms on localhost (67/199)
16/04/13 23:54:39 INFO Executor: Running task 70.0 in stage 5.0 (TID 79)
16/04/13 23:54:39 INFO Executor: Finished task 62.0 in stage 5.0 (TID 71). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO TaskSetManager: Starting task 71.0 in stage 5.0 (TID 80, localhost, partition 72,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 62.0 in stage 5.0 (TID 71) in 501 ms on localhost (68/199)
16/04/13 23:54:39 INFO Executor: Running task 71.0 in stage 5.0 (TID 80)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:39 INFO Executor: Finished task 65.0 in stage 5.0 (TID 74). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO TaskSetManager: Starting task 72.0 in stage 5.0 (TID 81, localhost, partition 73,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO Executor: Finished task 71.0 in stage 5.0 (TID 80). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO Executor: Running task 72.0 in stage 5.0 (TID 81)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 65.0 in stage 5.0 (TID 74) in 315 ms on localhost (69/199)
16/04/13 23:54:39 INFO TaskSetManager: Starting task 73.0 in stage 5.0 (TID 82, localhost, partition 74,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 71.0 in stage 5.0 (TID 80) in 93 ms on localhost (70/199)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
16/04/13 23:54:39 INFO Executor: Running task 73.0 in stage 5.0 (TID 82)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:39 INFO Executor: Finished task 73.0 in stage 5.0 (TID 82). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO TaskSetManager: Starting task 74.0 in stage 5.0 (TID 83, localhost, partition 75,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO Executor: Finished task 72.0 in stage 5.0 (TID 81). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO TaskSetManager: Starting task 75.0 in stage 5.0 (TID 84, localhost, partition 76,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 73.0 in stage 5.0 (TID 82) in 69 ms on localhost (71/199)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 72.0 in stage 5.0 (TID 81) in 80 ms on localhost (72/199)
16/04/13 23:54:39 INFO Executor: Running task 74.0 in stage 5.0 (TID 83)
16/04/13 23:54:39 INFO Executor: Running task 75.0 in stage 5.0 (TID 84)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 26 ms
16/04/13 23:54:39 INFO Executor: Finished task 75.0 in stage 5.0 (TID 84). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 28 ms
16/04/13 23:54:39 INFO TaskSetManager: Starting task 76.0 in stage 5.0 (TID 85, localhost, partition 77,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO Executor: Finished task 68.0 in stage 5.0 (TID 77). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO Executor: Running task 76.0 in stage 5.0 (TID 85)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 75.0 in stage 5.0 (TID 84) in 78 ms on localhost (73/199)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:39 INFO Executor: Finished task 74.0 in stage 5.0 (TID 83). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO Executor: Finished task 70.0 in stage 5.0 (TID 79). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO TaskSetManager: Starting task 77.0 in stage 5.0 (TID 86, localhost, partition 78,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 68.0 in stage 5.0 (TID 77) in 430 ms on localhost (74/199)
16/04/13 23:54:39 INFO Executor: Running task 77.0 in stage 5.0 (TID 86)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 5 ms
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:39 INFO Executor: Finished task 76.0 in stage 5.0 (TID 85). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO Executor: Finished task 77.0 in stage 5.0 (TID 86). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO TaskSetManager: Starting task 78.0 in stage 5.0 (TID 87, localhost, partition 79,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO Executor: Running task 78.0 in stage 5.0 (TID 87)
16/04/13 23:54:39 INFO TaskSetManager: Starting task 79.0 in stage 5.0 (TID 88, localhost, partition 80,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO TaskSetManager: Starting task 80.0 in stage 5.0 (TID 89, localhost, partition 81,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:39 INFO Executor: Finished task 78.0 in stage 5.0 (TID 87). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO TaskSetManager: Starting task 81.0 in stage 5.0 (TID 90, localhost, partition 82,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO Executor: Running task 81.0 in stage 5.0 (TID 90)
16/04/13 23:54:39 INFO TaskSetManager: Starting task 82.0 in stage 5.0 (TID 91, localhost, partition 83,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 70.0 in stage 5.0 (TID 79) in 413 ms on localhost (75/199)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 74.0 in stage 5.0 (TID 83) in 165 ms on localhost (76/199)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 76.0 in stage 5.0 (TID 85) in 102 ms on localhost (77/199)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 77.0 in stage 5.0 (TID 86) in 75 ms on localhost (78/199)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 78.0 in stage 5.0 (TID 87) in 76 ms on localhost (79/199)
16/04/13 23:54:39 INFO Executor: Running task 79.0 in stage 5.0 (TID 88)
16/04/13 23:54:39 INFO Executor: Running task 80.0 in stage 5.0 (TID 89)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:39 INFO Executor: Finished task 79.0 in stage 5.0 (TID 88). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:39 INFO Executor: Finished task 80.0 in stage 5.0 (TID 89). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:39 INFO Executor: Running task 82.0 in stage 5.0 (TID 91)
16/04/13 23:54:39 INFO Executor: Finished task 81.0 in stage 5.0 (TID 90). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO TaskSetManager: Starting task 83.0 in stage 5.0 (TID 92, localhost, partition 84,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO TaskSetManager: Starting task 84.0 in stage 5.0 (TID 93, localhost, partition 85,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
16/04/13 23:54:39 INFO Executor: Running task 83.0 in stage 5.0 (TID 92)
16/04/13 23:54:39 INFO Executor: Running task 84.0 in stage 5.0 (TID 93)
16/04/13 23:54:39 INFO Executor: Finished task 82.0 in stage 5.0 (TID 91). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO TaskSetManager: Starting task 85.0 in stage 5.0 (TID 94, localhost, partition 86,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 11 ms
16/04/13 23:54:39 INFO TaskSetManager: Starting task 86.0 in stage 5.0 (TID 95, localhost, partition 87,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 29 ms
16/04/13 23:54:39 INFO Executor: Finished task 84.0 in stage 5.0 (TID 93). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO Executor: Finished task 83.0 in stage 5.0 (TID 92). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO Executor: Running task 85.0 in stage 5.0 (TID 94)
16/04/13 23:54:39 INFO Executor: Running task 86.0 in stage 5.0 (TID 95)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 81.0 in stage 5.0 (TID 90) in 161 ms on localhost (80/199)
16/04/13 23:54:39 INFO TaskSetManager: Starting task 87.0 in stage 5.0 (TID 96, localhost, partition 88,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO TaskSetManager: Starting task 88.0 in stage 5.0 (TID 97, localhost, partition 89,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:39 INFO Executor: Finished task 85.0 in stage 5.0 (TID 94). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO Executor: Running task 87.0 in stage 5.0 (TID 96)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 84.0 in stage 5.0 (TID 93) in 68 ms on localhost (81/199)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 83.0 in stage 5.0 (TID 92) in 76 ms on localhost (82/199)
16/04/13 23:54:39 INFO Executor: Running task 88.0 in stage 5.0 (TID 97)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:39 INFO TaskSetManager: Finished task 79.0 in stage 5.0 (TID 88) in 212 ms on localhost (83/199)
16/04/13 23:54:39 INFO TaskSetManager: Starting task 89.0 in stage 5.0 (TID 98, localhost, partition 90,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 82.0 in stage 5.0 (TID 91) in 198 ms on localhost (84/199)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 85.0 in stage 5.0 (TID 94) in 97 ms on localhost (85/199)
16/04/13 23:54:39 INFO Executor: Running task 89.0 in stage 5.0 (TID 98)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:39 INFO Executor: Finished task 86.0 in stage 5.0 (TID 95). 1973 bytes result sent to driver
16/04/13 23:54:39 INFO Executor: Finished task 89.0 in stage 5.0 (TID 98). 1609 bytes result sent to driver
16/04/13 23:54:39 INFO TaskSetManager: Finished task 80.0 in stage 5.0 (TID 89) in 269 ms on localhost (86/199)
16/04/13 23:54:39 INFO TaskSetManager: Starting task 90.0 in stage 5.0 (TID 99, localhost, partition 91,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO TaskSetManager: Starting task 91.0 in stage 5.0 (TID 100, localhost, partition 92,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 86.0 in stage 5.0 (TID 95) in 121 ms on localhost (87/199)
16/04/13 23:54:39 INFO TaskSetManager: Finished task 89.0 in stage 5.0 (TID 98) in 65 ms on localhost (88/199)
16/04/13 23:54:39 INFO Executor: Running task 90.0 in stage 5.0 (TID 99)
16/04/13 23:54:39 INFO Executor: Running task 91.0 in stage 5.0 (TID 100)
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:39 INFO Executor: Finished task 90.0 in stage 5.0 (TID 99). 1609 bytes result sent to driver
16/04/13 23:54:40 INFO TaskSetManager: Starting task 92.0 in stage 5.0 (TID 101, localhost, partition 93,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:40 INFO TaskSetManager: Finished task 90.0 in stage 5.0 (TID 99) in 155 ms on localhost (89/199)
16/04/13 23:54:40 INFO Executor: Running task 92.0 in stage 5.0 (TID 101)
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:40 INFO Executor: Finished task 92.0 in stage 5.0 (TID 101). 1609 bytes result sent to driver
16/04/13 23:54:40 INFO TaskSetManager: Starting task 93.0 in stage 5.0 (TID 102, localhost, partition 94,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:40 INFO Executor: Running task 93.0 in stage 5.0 (TID 102)
16/04/13 23:54:40 INFO TaskSetManager: Finished task 92.0 in stage 5.0 (TID 101) in 223 ms on localhost (90/199)
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:40 INFO Executor: Finished task 93.0 in stage 5.0 (TID 102). 1609 bytes result sent to driver
16/04/13 23:54:40 INFO TaskSetManager: Starting task 94.0 in stage 5.0 (TID 103, localhost, partition 95,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:40 INFO TaskSetManager: Finished task 93.0 in stage 5.0 (TID 102) in 60 ms on localhost (91/199)
16/04/13 23:54:40 INFO Executor: Running task 94.0 in stage 5.0 (TID 103)
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:40 INFO Executor: Finished task 94.0 in stage 5.0 (TID 103). 1609 bytes result sent to driver
16/04/13 23:54:40 INFO TaskSetManager: Starting task 95.0 in stage 5.0 (TID 104, localhost, partition 96,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:40 INFO TaskSetManager: Finished task 94.0 in stage 5.0 (TID 103) in 57 ms on localhost (92/199)
16/04/13 23:54:40 INFO Executor: Running task 95.0 in stage 5.0 (TID 104)
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 30 ms
16/04/13 23:54:40 INFO Executor: Finished task 87.0 in stage 5.0 (TID 96). 1609 bytes result sent to driver
16/04/13 23:54:40 INFO TaskSetManager: Starting task 96.0 in stage 5.0 (TID 105, localhost, partition 97,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:40 INFO TaskSetManager: Finished task 87.0 in stage 5.0 (TID 96) in 692 ms on localhost (93/199)
16/04/13 23:54:40 INFO Executor: Running task 96.0 in stage 5.0 (TID 105)
16/04/13 23:54:40 INFO Executor: Finished task 88.0 in stage 5.0 (TID 97). 1609 bytes result sent to driver
16/04/13 23:54:40 INFO TaskSetManager: Starting task 97.0 in stage 5.0 (TID 106, localhost, partition 98,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:40 INFO TaskSetManager: Finished task 88.0 in stage 5.0 (TID 97) in 746 ms on localhost (94/199)
16/04/13 23:54:40 INFO Executor: Running task 97.0 in stage 5.0 (TID 106)
16/04/13 23:54:40 INFO Executor: Finished task 96.0 in stage 5.0 (TID 105). 1609 bytes result sent to driver
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:40 INFO Executor: Finished task 91.0 in stage 5.0 (TID 100). 1609 bytes result sent to driver
16/04/13 23:54:40 INFO TaskSetManager: Starting task 98.0 in stage 5.0 (TID 107, localhost, partition 99,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:40 INFO TaskSetManager: Starting task 99.0 in stage 5.0 (TID 108, localhost, partition 100,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:40 INFO Executor: Finished task 95.0 in stage 5.0 (TID 104). 1609 bytes result sent to driver
16/04/13 23:54:40 INFO Executor: Finished task 97.0 in stage 5.0 (TID 106). 1609 bytes result sent to driver
16/04/13 23:54:40 INFO TaskSetManager: Starting task 100.0 in stage 5.0 (TID 109, localhost, partition 101,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:40 INFO TaskSetManager: Finished task 96.0 in stage 5.0 (TID 105) in 120 ms on localhost (95/199)
16/04/13 23:54:40 INFO Executor: Running task 98.0 in stage 5.0 (TID 107)
16/04/13 23:54:40 INFO Executor: Running task 99.0 in stage 5.0 (TID 108)
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:40 INFO Executor: Finished task 98.0 in stage 5.0 (TID 107). 1609 bytes result sent to driver
16/04/13 23:54:40 INFO Executor: Running task 100.0 in stage 5.0 (TID 109)
16/04/13 23:54:40 INFO Executor: Finished task 99.0 in stage 5.0 (TID 108). 1609 bytes result sent to driver
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:40 INFO TaskSetManager: Finished task 91.0 in stage 5.0 (TID 100) in 719 ms on localhost (96/199)
16/04/13 23:54:40 INFO TaskSetManager: Starting task 101.0 in stage 5.0 (TID 110, localhost, partition 102,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:40 INFO TaskSetManager: Starting task 102.0 in stage 5.0 (TID 111, localhost, partition 103,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:40 INFO Executor: Finished task 100.0 in stage 5.0 (TID 109). 1609 bytes result sent to driver
16/04/13 23:54:40 INFO TaskSetManager: Starting task 103.0 in stage 5.0 (TID 112, localhost, partition 104,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:40 INFO TaskSetManager: Starting task 104.0 in stage 5.0 (TID 113, localhost, partition 105,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:40 INFO TaskSetManager: Finished task 95.0 in stage 5.0 (TID 104) in 345 ms on localhost (97/199)
16/04/13 23:54:40 INFO TaskSetManager: Finished task 97.0 in stage 5.0 (TID 106) in 133 ms on localhost (98/199)
16/04/13 23:54:40 INFO TaskSetManager: Finished task 99.0 in stage 5.0 (TID 108) in 109 ms on localhost (99/199)
16/04/13 23:54:40 INFO TaskSetManager: Finished task 100.0 in stage 5.0 (TID 109) in 92 ms on localhost (100/199)
16/04/13 23:54:40 INFO Executor: Running task 101.0 in stage 5.0 (TID 110)
16/04/13 23:54:40 INFO Executor: Running task 102.0 in stage 5.0 (TID 111)
16/04/13 23:54:40 INFO Executor: Running task 103.0 in stage 5.0 (TID 112)
16/04/13 23:54:40 INFO Executor: Running task 104.0 in stage 5.0 (TID 113)
16/04/13 23:54:40 INFO TaskSetManager: Finished task 98.0 in stage 5.0 (TID 107) in 221 ms on localhost (101/199)
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 147 ms
16/04/13 23:54:40 INFO Executor: Finished task 104.0 in stage 5.0 (TID 113). 1609 bytes result sent to driver
16/04/13 23:54:40 INFO TaskSetManager: Starting task 105.0 in stage 5.0 (TID 114, localhost, partition 106,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:40 INFO Executor: Running task 105.0 in stage 5.0 (TID 114)
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 3 ms
16/04/13 23:54:40 INFO Executor: Finished task 105.0 in stage 5.0 (TID 114). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO TaskSetManager: Starting task 106.0 in stage 5.0 (TID 115, localhost, partition 107,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO Executor: Finished task 102.0 in stage 5.0 (TID 111). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO TaskSetManager: Finished task 104.0 in stage 5.0 (TID 113) in 388 ms on localhost (102/199)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 105.0 in stage 5.0 (TID 114) in 64 ms on localhost (103/199)
16/04/13 23:54:41 INFO TaskSetManager: Starting task 107.0 in stage 5.0 (TID 116, localhost, partition 108,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO Executor: Running task 106.0 in stage 5.0 (TID 115)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 102.0 in stage 5.0 (TID 111) in 417 ms on localhost (104/199)
16/04/13 23:54:41 INFO Executor: Running task 107.0 in stage 5.0 (TID 116)
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:41 INFO Executor: Finished task 107.0 in stage 5.0 (TID 116). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:41 INFO Executor: Finished task 106.0 in stage 5.0 (TID 115). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO TaskSetManager: Starting task 108.0 in stage 5.0 (TID 117, localhost, partition 109,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO TaskSetManager: Starting task 109.0 in stage 5.0 (TID 118, localhost, partition 110,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 107.0 in stage 5.0 (TID 116) in 133 ms on localhost (105/199)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 106.0 in stage 5.0 (TID 115) in 144 ms on localhost (106/199)
16/04/13 23:54:41 INFO Executor: Running task 108.0 in stage 5.0 (TID 117)
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:41 INFO Executor: Running task 109.0 in stage 5.0 (TID 118)
16/04/13 23:54:41 INFO Executor: Finished task 108.0 in stage 5.0 (TID 117). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO TaskSetManager: Starting task 110.0 in stage 5.0 (TID 119, localhost, partition 111,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:41 INFO Executor: Finished task 109.0 in stage 5.0 (TID 118). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO TaskSetManager: Starting task 111.0 in stage 5.0 (TID 120, localhost, partition 112,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 108.0 in stage 5.0 (TID 117) in 100 ms on localhost (107/199)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 109.0 in stage 5.0 (TID 118) in 119 ms on localhost (108/199)
16/04/13 23:54:41 INFO Executor: Running task 110.0 in stage 5.0 (TID 119)
16/04/13 23:54:41 INFO Executor: Running task 111.0 in stage 5.0 (TID 120)
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 17 ms
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:41 INFO Executor: Finished task 110.0 in stage 5.0 (TID 119). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO Executor: Finished task 111.0 in stage 5.0 (TID 120). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO TaskSetManager: Starting task 112.0 in stage 5.0 (TID 121, localhost, partition 113,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO Executor: Running task 112.0 in stage 5.0 (TID 121)
16/04/13 23:54:41 INFO TaskSetManager: Starting task 113.0 in stage 5.0 (TID 122, localhost, partition 114,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 110.0 in stage 5.0 (TID 119) in 124 ms on localhost (109/199)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 111.0 in stage 5.0 (TID 120) in 104 ms on localhost (110/199)
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:41 INFO Executor: Running task 113.0 in stage 5.0 (TID 122)
16/04/13 23:54:41 INFO Executor: Finished task 112.0 in stage 5.0 (TID 121). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 3 ms
16/04/13 23:54:41 INFO Executor: Finished task 113.0 in stage 5.0 (TID 122). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO TaskSetManager: Starting task 114.0 in stage 5.0 (TID 123, localhost, partition 115,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO TaskSetManager: Starting task 115.0 in stage 5.0 (TID 124, localhost, partition 116,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 112.0 in stage 5.0 (TID 121) in 74 ms on localhost (111/199)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 113.0 in stage 5.0 (TID 122) in 66 ms on localhost (112/199)
16/04/13 23:54:41 INFO Executor: Running task 114.0 in stage 5.0 (TID 123)
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:41 INFO Executor: Running task 115.0 in stage 5.0 (TID 124)
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:41 INFO Executor: Finished task 115.0 in stage 5.0 (TID 124). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO TaskSetManager: Starting task 116.0 in stage 5.0 (TID 125, localhost, partition 117,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 115.0 in stage 5.0 (TID 124) in 71 ms on localhost (113/199)
16/04/13 23:54:41 INFO Executor: Running task 116.0 in stage 5.0 (TID 125)
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:41 INFO Executor: Finished task 116.0 in stage 5.0 (TID 125). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO TaskSetManager: Starting task 117.0 in stage 5.0 (TID 126, localhost, partition 118,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 116.0 in stage 5.0 (TID 125) in 68 ms on localhost (114/199)
16/04/13 23:54:41 INFO Executor: Running task 117.0 in stage 5.0 (TID 126)
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:41 INFO Executor: Finished task 103.0 in stage 5.0 (TID 112). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:41 INFO Executor: Finished task 101.0 in stage 5.0 (TID 110). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO Executor: Finished task 117.0 in stage 5.0 (TID 126). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO Executor: Finished task 114.0 in stage 5.0 (TID 123). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO TaskSetManager: Starting task 118.0 in stage 5.0 (TID 127, localhost, partition 119,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO TaskSetManager: Starting task 119.0 in stage 5.0 (TID 128, localhost, partition 120,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO TaskSetManager: Starting task 120.0 in stage 5.0 (TID 129, localhost, partition 121,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO TaskSetManager: Starting task 121.0 in stage 5.0 (TID 130, localhost, partition 122,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 103.0 in stage 5.0 (TID 112) in 1263 ms on localhost (115/199)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 101.0 in stage 5.0 (TID 110) in 1270 ms on localhost (116/199)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 117.0 in stage 5.0 (TID 126) in 402 ms on localhost (117/199)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 114.0 in stage 5.0 (TID 123) in 521 ms on localhost (118/199)
16/04/13 23:54:41 INFO Executor: Running task 118.0 in stage 5.0 (TID 127)
16/04/13 23:54:41 INFO Executor: Running task 119.0 in stage 5.0 (TID 128)
16/04/13 23:54:41 INFO Executor: Running task 120.0 in stage 5.0 (TID 129)
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:41 INFO Executor: Finished task 118.0 in stage 5.0 (TID 127). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO Executor: Finished task 120.0 in stage 5.0 (TID 129). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO Executor: Running task 121.0 in stage 5.0 (TID 130)
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:41 INFO Executor: Finished task 121.0 in stage 5.0 (TID 130). 1609 bytes result sent to driver
16/04/13 23:54:41 INFO TaskSetManager: Starting task 122.0 in stage 5.0 (TID 131, localhost, partition 123,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 118.0 in stage 5.0 (TID 127) in 80 ms on localhost (119/199)
16/04/13 23:54:41 INFO TaskSetManager: Finished task 120.0 in stage 5.0 (TID 129) in 78 ms on localhost (120/199)
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 8 ms
16/04/13 23:54:41 INFO Executor: Running task 122.0 in stage 5.0 (TID 131)
16/04/13 23:54:42 INFO Executor: Finished task 119.0 in stage 5.0 (TID 128). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Starting task 123.0 in stage 5.0 (TID 132, localhost, partition 124,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 124.0 in stage 5.0 (TID 133, localhost, partition 125,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 125.0 in stage 5.0 (TID 134, localhost, partition 126,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO Executor: Running task 124.0 in stage 5.0 (TID 133)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO TaskSetManager: Finished task 119.0 in stage 5.0 (TID 128) in 226 ms on localhost (121/199)
16/04/13 23:54:42 INFO Executor: Finished task 124.0 in stage 5.0 (TID 133). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Running task 125.0 in stage 5.0 (TID 134)
16/04/13 23:54:42 INFO Executor: Running task 123.0 in stage 5.0 (TID 132)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 126.0 in stage 5.0 (TID 135, localhost, partition 127,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO TaskSetManager: Finished task 124.0 in stage 5.0 (TID 133) in 128 ms on localhost (122/199)
16/04/13 23:54:42 INFO Executor: Finished task 123.0 in stage 5.0 (TID 132). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Running task 126.0 in stage 5.0 (TID 135)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO Executor: Finished task 125.0 in stage 5.0 (TID 134). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Finished task 126.0 in stage 5.0 (TID 135). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Starting task 127.0 in stage 5.0 (TID 136, localhost, partition 128,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 128.0 in stage 5.0 (TID 137, localhost, partition 129,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 129.0 in stage 5.0 (TID 138, localhost, partition 130,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 123.0 in stage 5.0 (TID 132) in 159 ms on localhost (123/199)
16/04/13 23:54:42 INFO Executor: Running task 127.0 in stage 5.0 (TID 136)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 125.0 in stage 5.0 (TID 134) in 156 ms on localhost (124/199)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 126.0 in stage 5.0 (TID 135) in 47 ms on localhost (125/199)
16/04/13 23:54:42 INFO Executor: Running task 128.0 in stage 5.0 (TID 137)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO Executor: Running task 129.0 in stage 5.0 (TID 138)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO Executor: Finished task 128.0 in stage 5.0 (TID 137). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO TaskSetManager: Starting task 130.0 in stage 5.0 (TID 139, localhost, partition 131,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 128.0 in stage 5.0 (TID 137) in 35 ms on localhost (126/199)
16/04/13 23:54:42 INFO Executor: Finished task 129.0 in stage 5.0 (TID 138). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Running task 130.0 in stage 5.0 (TID 139)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 131.0 in stage 5.0 (TID 140, localhost, partition 132,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 129.0 in stage 5.0 (TID 138) in 44 ms on localhost (127/199)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO Executor: Running task 131.0 in stage 5.0 (TID 140)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 257 ms
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO Executor: Finished task 131.0 in stage 5.0 (TID 140). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Starting task 132.0 in stage 5.0 (TID 141, localhost, partition 133,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO Executor: Running task 132.0 in stage 5.0 (TID 141)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 131.0 in stage 5.0 (TID 140) in 80 ms on localhost (128/199)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO Executor: Finished task 132.0 in stage 5.0 (TID 141). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Finished task 122.0 in stage 5.0 (TID 131). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Starting task 133.0 in stage 5.0 (TID 142, localhost, partition 134,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 132.0 in stage 5.0 (TID 141) in 22 ms on localhost (129/199)
16/04/13 23:54:42 INFO Executor: Running task 133.0 in stage 5.0 (TID 142)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 134.0 in stage 5.0 (TID 143, localhost, partition 135,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO TaskSetManager: Finished task 122.0 in stage 5.0 (TID 131) in 358 ms on localhost (130/199)
16/04/13 23:54:42 INFO Executor: Running task 134.0 in stage 5.0 (TID 143)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO Executor: Finished task 134.0 in stage 5.0 (TID 143). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Starting task 135.0 in stage 5.0 (TID 144, localhost, partition 136,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO Executor: Running task 135.0 in stage 5.0 (TID 144)
16/04/13 23:54:42 INFO Executor: Finished task 133.0 in stage 5.0 (TID 142). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Starting task 136.0 in stage 5.0 (TID 145, localhost, partition 137,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 5 ms
16/04/13 23:54:42 INFO Executor: Finished task 135.0 in stage 5.0 (TID 144). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Running task 136.0 in stage 5.0 (TID 145)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 133.0 in stage 5.0 (TID 142) in 60 ms on localhost (131/199)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 137.0 in stage 5.0 (TID 146, localhost, partition 138,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO TaskSetManager: Finished task 135.0 in stage 5.0 (TID 144) in 54 ms on localhost (132/199)
16/04/13 23:54:42 INFO Executor: Finished task 136.0 in stage 5.0 (TID 145). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Running task 137.0 in stage 5.0 (TID 146)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 138.0 in stage 5.0 (TID 147, localhost, partition 139,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 136.0 in stage 5.0 (TID 145) in 55 ms on localhost (133/199)
16/04/13 23:54:42 INFO Executor: Running task 138.0 in stage 5.0 (TID 147)
16/04/13 23:54:42 INFO Executor: Finished task 127.0 in stage 5.0 (TID 136). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 40 ms
16/04/13 23:54:42 INFO TaskSetManager: Finished task 134.0 in stage 5.0 (TID 143) in 129 ms on localhost (134/199)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 139.0 in stage 5.0 (TID 148, localhost, partition 140,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 127.0 in stage 5.0 (TID 136) in 355 ms on localhost (135/199)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO Executor: Finished task 130.0 in stage 5.0 (TID 139). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Running task 139.0 in stage 5.0 (TID 148)
16/04/13 23:54:42 INFO Executor: Finished task 138.0 in stage 5.0 (TID 147). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Starting task 140.0 in stage 5.0 (TID 149, localhost, partition 141,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO Executor: Running task 140.0 in stage 5.0 (TID 149)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 141.0 in stage 5.0 (TID 150, localhost, partition 142,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 138.0 in stage 5.0 (TID 147) in 123 ms on localhost (136/199)
16/04/13 23:54:42 INFO Executor: Running task 141.0 in stage 5.0 (TID 150)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO Executor: Finished task 140.0 in stage 5.0 (TID 149). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO TaskSetManager: Starting task 142.0 in stage 5.0 (TID 151, localhost, partition 143,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO Executor: Running task 142.0 in stage 5.0 (TID 151)
16/04/13 23:54:42 INFO Executor: Finished task 139.0 in stage 5.0 (TID 148). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO TaskSetManager: Starting task 143.0 in stage 5.0 (TID 152, localhost, partition 144,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 140.0 in stage 5.0 (TID 149) in 33 ms on localhost (137/199)
16/04/13 23:54:42 INFO Executor: Finished task 141.0 in stage 5.0 (TID 150). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Finished task 139.0 in stage 5.0 (TID 148) in 41 ms on localhost (138/199)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 144.0 in stage 5.0 (TID 153, localhost, partition 145,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO Executor: Running task 143.0 in stage 5.0 (TID 152)
16/04/13 23:54:42 INFO Executor: Finished task 142.0 in stage 5.0 (TID 151). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Finished task 141.0 in stage 5.0 (TID 150) in 36 ms on localhost (139/199)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO Executor: Finished task 143.0 in stage 5.0 (TID 152). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Running task 144.0 in stage 5.0 (TID 153)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 145.0 in stage 5.0 (TID 154, localhost, partition 146,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 80 ms
16/04/13 23:54:42 INFO TaskSetManager: Starting task 146.0 in stage 5.0 (TID 155, localhost, partition 147,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 121.0 in stage 5.0 (TID 130) in 773 ms on localhost (140/199)
16/04/13 23:54:42 INFO Executor: Finished task 137.0 in stage 5.0 (TID 146). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Running task 145.0 in stage 5.0 (TID 154)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 143.0 in stage 5.0 (TID 152) in 129 ms on localhost (141/199)
16/04/13 23:54:42 INFO Executor: Running task 146.0 in stage 5.0 (TID 155)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO TaskSetManager: Finished task 142.0 in stage 5.0 (TID 151) in 147 ms on localhost (142/199)
16/04/13 23:54:42 INFO Executor: Finished task 145.0 in stage 5.0 (TID 154). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Starting task 147.0 in stage 5.0 (TID 156, localhost, partition 148,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 148.0 in stage 5.0 (TID 157, localhost, partition 149,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 130.0 in stage 5.0 (TID 139) in 506 ms on localhost (143/199)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 137.0 in stage 5.0 (TID 146) in 340 ms on localhost (144/199)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 145.0 in stage 5.0 (TID 154) in 138 ms on localhost (145/199)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO Executor: Running task 147.0 in stage 5.0 (TID 156)
16/04/13 23:54:42 INFO Executor: Finished task 146.0 in stage 5.0 (TID 155). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Running task 148.0 in stage 5.0 (TID 157)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO Executor: Finished task 147.0 in stage 5.0 (TID 156). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Finished task 148.0 in stage 5.0 (TID 157). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Starting task 149.0 in stage 5.0 (TID 158, localhost, partition 150,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 150.0 in stage 5.0 (TID 159, localhost, partition 151,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 151.0 in stage 5.0 (TID 160, localhost, partition 152,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 146.0 in stage 5.0 (TID 155) in 95 ms on localhost (146/199)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 148.0 in stage 5.0 (TID 157) in 73 ms on localhost (147/199)
16/04/13 23:54:42 INFO Executor: Running task 149.0 in stage 5.0 (TID 158)
16/04/13 23:54:42 INFO Executor: Running task 150.0 in stage 5.0 (TID 159)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO Executor: Finished task 150.0 in stage 5.0 (TID 159). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Running task 151.0 in stage 5.0 (TID 160)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO Executor: Finished task 151.0 in stage 5.0 (TID 160). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO Executor: Finished task 149.0 in stage 5.0 (TID 158). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Starting task 152.0 in stage 5.0 (TID 161, localhost, partition 153,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 281 ms
16/04/13 23:54:42 INFO TaskSetManager: Finished task 147.0 in stage 5.0 (TID 156) in 165 ms on localhost (148/199)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 151.0 in stage 5.0 (TID 160) in 98 ms on localhost (149/199)
16/04/13 23:54:42 INFO Executor: Finished task 144.0 in stage 5.0 (TID 153). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Starting task 153.0 in stage 5.0 (TID 162, localhost, partition 154,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO Executor: Running task 152.0 in stage 5.0 (TID 161)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 154.0 in stage 5.0 (TID 163, localhost, partition 155,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 155.0 in stage 5.0 (TID 164, localhost, partition 156,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 150.0 in stage 5.0 (TID 159) in 115 ms on localhost (150/199)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 149.0 in stage 5.0 (TID 158) in 118 ms on localhost (151/199)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO TaskSetManager: Finished task 144.0 in stage 5.0 (TID 153) in 323 ms on localhost (152/199)
16/04/13 23:54:42 INFO Executor: Running task 153.0 in stage 5.0 (TID 162)
16/04/13 23:54:42 INFO Executor: Finished task 152.0 in stage 5.0 (TID 161). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Running task 154.0 in stage 5.0 (TID 163)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO Executor: Finished task 154.0 in stage 5.0 (TID 163). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO Executor: Running task 155.0 in stage 5.0 (TID 164)
16/04/13 23:54:42 INFO Executor: Finished task 153.0 in stage 5.0 (TID 162). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Starting task 156.0 in stage 5.0 (TID 165, localhost, partition 157,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 157.0 in stage 5.0 (TID 166, localhost, partition 158,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 158.0 in stage 5.0 (TID 167, localhost, partition 159,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO Executor: Finished task 155.0 in stage 5.0 (TID 164). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Finished task 152.0 in stage 5.0 (TID 161) in 137 ms on localhost (153/199)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 154.0 in stage 5.0 (TID 163) in 58 ms on localhost (154/199)
16/04/13 23:54:42 INFO TaskSetManager: Finished task 153.0 in stage 5.0 (TID 162) in 60 ms on localhost (155/199)
16/04/13 23:54:42 INFO Executor: Running task 156.0 in stage 5.0 (TID 165)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 159.0 in stage 5.0 (TID 168, localhost, partition 160,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO Executor: Running task 157.0 in stage 5.0 (TID 166)
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO Executor: Running task 159.0 in stage 5.0 (TID 168)
16/04/13 23:54:42 INFO Executor: Finished task 156.0 in stage 5.0 (TID 165). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 32 ms
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:42 INFO Executor: Finished task 157.0 in stage 5.0 (TID 166). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Finished task 155.0 in stage 5.0 (TID 164) in 71 ms on localhost (156/199)
16/04/13 23:54:42 INFO Executor: Running task 158.0 in stage 5.0 (TID 167)
16/04/13 23:54:42 INFO Executor: Finished task 159.0 in stage 5.0 (TID 168). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:42 INFO TaskSetManager: Starting task 160.0 in stage 5.0 (TID 169, localhost, partition 161,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 161.0 in stage 5.0 (TID 170, localhost, partition 162,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO Executor: Finished task 158.0 in stage 5.0 (TID 167). 1609 bytes result sent to driver
16/04/13 23:54:42 INFO TaskSetManager: Starting task 162.0 in stage 5.0 (TID 171, localhost, partition 163,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO TaskSetManager: Starting task 163.0 in stage 5.0 (TID 172, localhost, partition 164,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:42 INFO Executor: Running task 160.0 in stage 5.0 (TID 169)
16/04/13 23:54:42 INFO Executor: Running task 161.0 in stage 5.0 (TID 170)
16/04/13 23:54:43 INFO TaskSetManager: Finished task 157.0 in stage 5.0 (TID 166) in 95 ms on localhost (157/199)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
16/04/13 23:54:43 INFO Executor: Running task 162.0 in stage 5.0 (TID 171)
16/04/13 23:54:43 INFO Executor: Running task 163.0 in stage 5.0 (TID 172)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:43 INFO Executor: Finished task 163.0 in stage 5.0 (TID 172). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Finished task 159.0 in stage 5.0 (TID 168) in 83 ms on localhost (158/199)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 12 ms
16/04/13 23:54:43 INFO TaskSetManager: Finished task 156.0 in stage 5.0 (TID 165) in 119 ms on localhost (159/199)
16/04/13 23:54:43 INFO TaskSetManager: Finished task 158.0 in stage 5.0 (TID 167) in 118 ms on localhost (160/199)
16/04/13 23:54:43 INFO Executor: Finished task 162.0 in stage 5.0 (TID 171). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Starting task 164.0 in stage 5.0 (TID 173, localhost, partition 165,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO Executor: Running task 164.0 in stage 5.0 (TID 173)
16/04/13 23:54:43 INFO TaskSetManager: Starting task 165.0 in stage 5.0 (TID 174, localhost, partition 166,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:43 INFO TaskSetManager: Finished task 162.0 in stage 5.0 (TID 171) in 55 ms on localhost (161/199)
16/04/13 23:54:43 INFO Executor: Running task 165.0 in stage 5.0 (TID 174)
16/04/13 23:54:43 INFO TaskSetManager: Finished task 163.0 in stage 5.0 (TID 172) in 129 ms on localhost (162/199)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:43 INFO Executor: Finished task 165.0 in stage 5.0 (TID 174). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Starting task 166.0 in stage 5.0 (TID 175, localhost, partition 167,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO TaskSetManager: Finished task 165.0 in stage 5.0 (TID 174) in 98 ms on localhost (163/199)
16/04/13 23:54:43 INFO Executor: Running task 166.0 in stage 5.0 (TID 175)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:43 INFO Executor: Finished task 166.0 in stage 5.0 (TID 175). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Starting task 167.0 in stage 5.0 (TID 176, localhost, partition 168,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO TaskSetManager: Finished task 166.0 in stage 5.0 (TID 175) in 18 ms on localhost (164/199)
16/04/13 23:54:43 INFO Executor: Running task 167.0 in stage 5.0 (TID 176)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:43 INFO Executor: Finished task 160.0 in stage 5.0 (TID 169). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Starting task 168.0 in stage 5.0 (TID 177, localhost, partition 169,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO TaskSetManager: Finished task 160.0 in stage 5.0 (TID 169) in 230 ms on localhost (165/199)
16/04/13 23:54:43 INFO Executor: Running task 168.0 in stage 5.0 (TID 177)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 3 ms
16/04/13 23:54:43 INFO Executor: Finished task 168.0 in stage 5.0 (TID 177). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Starting task 169.0 in stage 5.0 (TID 178, localhost, partition 170,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO TaskSetManager: Finished task 168.0 in stage 5.0 (TID 177) in 79 ms on localhost (166/199)
16/04/13 23:54:43 INFO Executor: Running task 169.0 in stage 5.0 (TID 178)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:43 INFO Executor: Finished task 169.0 in stage 5.0 (TID 178). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Starting task 170.0 in stage 5.0 (TID 179, localhost, partition 171,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO TaskSetManager: Finished task 169.0 in stage 5.0 (TID 178) in 40 ms on localhost (167/199)
16/04/13 23:54:43 INFO Executor: Running task 170.0 in stage 5.0 (TID 179)
16/04/13 23:54:43 INFO Executor: Finished task 161.0 in stage 5.0 (TID 170). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Starting task 171.0 in stage 5.0 (TID 180, localhost, partition 172,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO TaskSetManager: Finished task 161.0 in stage 5.0 (TID 170) in 402 ms on localhost (168/199)
16/04/13 23:54:43 INFO Executor: Running task 171.0 in stage 5.0 (TID 180)
16/04/13 23:54:43 INFO Executor: Finished task 164.0 in stage 5.0 (TID 173). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Starting task 172.0 in stage 5.0 (TID 181, localhost, partition 173,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO TaskSetManager: Finished task 164.0 in stage 5.0 (TID 173) in 399 ms on localhost (169/199)
16/04/13 23:54:43 INFO Executor: Running task 172.0 in stage 5.0 (TID 181)
16/04/13 23:54:43 INFO Executor: Finished task 167.0 in stage 5.0 (TID 176). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Starting task 173.0 in stage 5.0 (TID 182, localhost, partition 174,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO TaskSetManager: Finished task 167.0 in stage 5.0 (TID 176) in 456 ms on localhost (170/199)
16/04/13 23:54:43 INFO Executor: Running task 173.0 in stage 5.0 (TID 182)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:43 INFO Executor: Finished task 173.0 in stage 5.0 (TID 182). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO TaskSetManager: Starting task 174.0 in stage 5.0 (TID 183, localhost, partition 175,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:43 INFO Executor: Finished task 172.0 in stage 5.0 (TID 181). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Finished task 173.0 in stage 5.0 (TID 182) in 63 ms on localhost (171/199)
16/04/13 23:54:43 INFO Executor: Running task 174.0 in stage 5.0 (TID 183)
16/04/13 23:54:43 INFO TaskSetManager: Starting task 175.0 in stage 5.0 (TID 184, localhost, partition 176,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:43 INFO Executor: Finished task 174.0 in stage 5.0 (TID 183). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Finished task 172.0 in stage 5.0 (TID 181) in 275 ms on localhost (172/199)
16/04/13 23:54:43 INFO Executor: Running task 175.0 in stage 5.0 (TID 184)
16/04/13 23:54:43 INFO TaskSetManager: Starting task 176.0 in stage 5.0 (TID 185, localhost, partition 177,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO Executor: Running task 176.0 in stage 5.0 (TID 185)
16/04/13 23:54:43 INFO TaskSetManager: Finished task 174.0 in stage 5.0 (TID 183) in 51 ms on localhost (173/199)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:43 INFO Executor: Finished task 175.0 in stage 5.0 (TID 184). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:43 INFO TaskSetManager: Starting task 177.0 in stage 5.0 (TID 186, localhost, partition 178,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO TaskSetManager: Finished task 175.0 in stage 5.0 (TID 184) in 47 ms on localhost (174/199)
16/04/13 23:54:43 INFO Executor: Running task 177.0 in stage 5.0 (TID 186)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:43 INFO Executor: Finished task 177.0 in stage 5.0 (TID 186). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Starting task 178.0 in stage 5.0 (TID 187, localhost, partition 179,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO TaskSetManager: Finished task 177.0 in stage 5.0 (TID 186) in 89 ms on localhost (175/199)
16/04/13 23:54:43 INFO Executor: Running task 178.0 in stage 5.0 (TID 187)
16/04/13 23:54:43 INFO Executor: Finished task 170.0 in stage 5.0 (TID 179). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Starting task 179.0 in stage 5.0 (TID 188, localhost, partition 180,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO Executor: Finished task 176.0 in stage 5.0 (TID 185). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:43 INFO Executor: Finished task 178.0 in stage 5.0 (TID 187). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO Executor: Finished task 171.0 in stage 5.0 (TID 180). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO Executor: Running task 179.0 in stage 5.0 (TID 188)
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 2 ms
16/04/13 23:54:43 INFO Executor: Finished task 179.0 in stage 5.0 (TID 188). 1609 bytes result sent to driver
16/04/13 23:54:43 INFO TaskSetManager: Finished task 170.0 in stage 5.0 (TID 179) in 646 ms on localhost (176/199)
16/04/13 23:54:43 INFO TaskSetManager: Starting task 180.0 in stage 5.0 (TID 189, localhost, partition 181,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO TaskSetManager: Starting task 181.0 in stage 5.0 (TID 190, localhost, partition 182,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO TaskSetManager: Starting task 182.0 in stage 5.0 (TID 191, localhost, partition 183,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:43 INFO TaskSetManager: Starting task 183.0 in stage 5.0 (TID 192, localhost, partition 184,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 176.0 in stage 5.0 (TID 185) in 312 ms on localhost (177/199)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 178.0 in stage 5.0 (TID 187) in 204 ms on localhost (178/199)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 171.0 in stage 5.0 (TID 180) in 640 ms on localhost (179/199)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 179.0 in stage 5.0 (TID 188) in 175 ms on localhost (180/199)
16/04/13 23:54:44 INFO Executor: Running task 180.0 in stage 5.0 (TID 189)
16/04/13 23:54:44 INFO Executor: Running task 181.0 in stage 5.0 (TID 190)
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:44 INFO Executor: Finished task 180.0 in stage 5.0 (TID 189). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:44 INFO Executor: Finished task 181.0 in stage 5.0 (TID 190). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO Executor: Running task 182.0 in stage 5.0 (TID 191)
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:44 INFO Executor: Finished task 182.0 in stage 5.0 (TID 191). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO Executor: Running task 183.0 in stage 5.0 (TID 192)
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:44 INFO Executor: Finished task 183.0 in stage 5.0 (TID 192). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO TaskSetManager: Starting task 184.0 in stage 5.0 (TID 193, localhost, partition 185,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO TaskSetManager: Starting task 185.0 in stage 5.0 (TID 194, localhost, partition 186,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO TaskSetManager: Starting task 186.0 in stage 5.0 (TID 195, localhost, partition 187,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO TaskSetManager: Starting task 187.0 in stage 5.0 (TID 196, localhost, partition 188,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO Executor: Running task 186.0 in stage 5.0 (TID 195)
16/04/13 23:54:44 INFO Executor: Running task 185.0 in stage 5.0 (TID 194)
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:44 INFO Executor: Finished task 186.0 in stage 5.0 (TID 195). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO TaskSetManager: Starting task 188.0 in stage 5.0 (TID 197, localhost, partition 189,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 183.0 in stage 5.0 (TID 192) in 218 ms on localhost (181/199)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 186.0 in stage 5.0 (TID 195) in 114 ms on localhost (182/199)
16/04/13 23:54:44 INFO Executor: Running task 187.0 in stage 5.0 (TID 196)
16/04/13 23:54:44 INFO Executor: Running task 184.0 in stage 5.0 (TID 193)
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 3 ms
16/04/13 23:54:44 INFO Executor: Finished task 187.0 in stage 5.0 (TID 196). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO Executor: Finished task 184.0 in stage 5.0 (TID 193). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO TaskSetManager: Starting task 189.0 in stage 5.0 (TID 198, localhost, partition 190,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO TaskSetManager: Starting task 190.0 in stage 5.0 (TID 199, localhost, partition 191,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO Executor: Running task 188.0 in stage 5.0 (TID 197)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 182.0 in stage 5.0 (TID 191) in 267 ms on localhost (183/199)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 187.0 in stage 5.0 (TID 196) in 146 ms on localhost (184/199)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 184.0 in stage 5.0 (TID 193) in 155 ms on localhost (185/199)
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 58 ms
16/04/13 23:54:44 INFO Executor: Running task 189.0 in stage 5.0 (TID 198)
16/04/13 23:54:44 INFO Executor: Running task 190.0 in stage 5.0 (TID 199)
16/04/13 23:54:44 INFO Executor: Finished task 185.0 in stage 5.0 (TID 194). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO TaskSetManager: Finished task 181.0 in stage 5.0 (TID 190) in 306 ms on localhost (186/199)
16/04/13 23:54:44 INFO Executor: Finished task 188.0 in stage 5.0 (TID 197). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO TaskSetManager: Starting task 191.0 in stage 5.0 (TID 200, localhost, partition 192,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO TaskSetManager: Starting task 192.0 in stage 5.0 (TID 201, localhost, partition 193,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:44 INFO Executor: Finished task 190.0 in stage 5.0 (TID 199). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO TaskSetManager: Starting task 193.0 in stage 5.0 (TID 202, localhost, partition 194,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 185.0 in stage 5.0 (TID 194) in 230 ms on localhost (187/199)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 188.0 in stage 5.0 (TID 197) in 120 ms on localhost (188/199)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 190.0 in stage 5.0 (TID 199) in 94 ms on localhost (189/199)
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 132 ms
16/04/13 23:54:44 INFO Executor: Running task 193.0 in stage 5.0 (TID 202)
16/04/13 23:54:44 INFO Executor: Finished task 189.0 in stage 5.0 (TID 198). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO Executor: Running task 192.0 in stage 5.0 (TID 201)
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:44 INFO TaskSetManager: Starting task 194.0 in stage 5.0 (TID 203, localhost, partition 195,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO Executor: Running task 191.0 in stage 5.0 (TID 200)
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/04/13 23:54:44 INFO Executor: Finished task 192.0 in stage 5.0 (TID 201). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO Executor: Finished task 193.0 in stage 5.0 (TID 202). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO TaskSetManager: Starting task 195.0 in stage 5.0 (TID 204, localhost, partition 196,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO Executor: Finished task 191.0 in stage 5.0 (TID 200). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO TaskSetManager: Starting task 196.0 in stage 5.0 (TID 205, localhost, partition 197,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO TaskSetManager: Starting task 197.0 in stage 5.0 (TID 206, localhost, partition 198,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO Executor: Running task 194.0 in stage 5.0 (TID 203)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 189.0 in stage 5.0 (TID 198) in 276 ms on localhost (190/199)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 192.0 in stage 5.0 (TID 201) in 231 ms on localhost (191/199)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 193.0 in stage 5.0 (TID 202) in 197 ms on localhost (192/199)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 191.0 in stage 5.0 (TID 200) in 235 ms on localhost (193/199)
16/04/13 23:54:44 INFO Executor: Running task 195.0 in stage 5.0 (TID 204)
16/04/13 23:54:44 INFO Executor: Running task 197.0 in stage 5.0 (TID 206)
16/04/13 23:54:44 INFO Executor: Running task 196.0 in stage 5.0 (TID 205)
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:44 INFO Executor: Finished task 197.0 in stage 5.0 (TID 206). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:44 INFO Executor: Finished task 196.0 in stage 5.0 (TID 205). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO TaskSetManager: Starting task 198.0 in stage 5.0 (TID 207, localhost, partition 199,NODE_LOCAL, 1999 bytes)
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 5 ms
16/04/13 23:54:44 INFO Executor: Finished task 195.0 in stage 5.0 (TID 204). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO Executor: Running task 198.0 in stage 5.0 (TID 207)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 197.0 in stage 5.0 (TID 206) in 279 ms on localhost (194/199)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 196.0 in stage 5.0 (TID 205) in 288 ms on localhost (195/199)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 180.0 in stage 5.0 (TID 189) in 824 ms on localhost (196/199)
16/04/13 23:54:44 INFO TaskSetManager: Finished task 195.0 in stage 5.0 (TID 204) in 292 ms on localhost (197/199)
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:44 INFO Executor: Finished task 198.0 in stage 5.0 (TID 207). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/04/13 23:54:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/04/13 23:54:44 INFO TaskSetManager: Finished task 198.0 in stage 5.0 (TID 207) in 60 ms on localhost (198/199)
16/04/13 23:54:44 INFO Executor: Finished task 194.0 in stage 5.0 (TID 203). 1609 bytes result sent to driver
16/04/13 23:54:44 INFO TaskSetManager: Finished task 194.0 in stage 5.0 (TID 203) in 343 ms on localhost (199/199)
16/04/13 23:54:44 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool
16/04/13 23:54:44 INFO DAGScheduler: ResultStage 5 (show at SparkSQLUDFUDAF.scala:68) finished in 9.998 s
16/04/13 23:54:44 INFO DAGScheduler: Job 3 finished: show at SparkSQLUDFUDAF.scala:68, took 10.309674 s
+------+-----+------+
| word|count|length|
+------+-----+------+
| Spark| 6| 5|
|Hadoop| 4| 6|
+------+-----+------+
16/04/13 23:54:44 INFO BlockManagerInfo: Removed broadcast_3_piece0 on localhost:60618 in memory (size: 6.5 KB, free: 1773.7 MB)
下面看一下UDFRegistration的源码：
/**
* Functions for registering user-defined functions. Use[[SQLContext.udf]] to access this.
*
* @since 1.3.0
*/
class UDFRegistrationprivate[sql] (sqlContext: SQLContext)extends Logging {
private val functionRegistry= sqlContext.functionRegistry
protected[sql]def registerPython(name:String, udf: UserDefinedPythonFunction): Unit = {
log.debug(
s"""
| Registering new PythonUDF:
| name: $name
| command: ${udf.command.toSeq}
| envVars: ${udf.envVars}
| pythonIncludes: ${udf.pythonIncludes}
| pythonExec: ${udf.pythonExec}
| dataType: ${udf.dataType}
""".stripMargin)
functionRegistry.registerFunction(name, udf.builder)
}
/**
* Register a user-defined aggregate function (UDAF).
*
* @param name the name of the UDAF.
* @param udaf the UDAF needs to be registered.
* @return the registered UDAF.
*/
defregister(
name: String,
udaf: UserDefinedAggregateFunction): UserDefinedAggregateFunction = {
def builder(children:Seq[Expression]) = ScalaUDAF(children, udaf)
functionRegistry.registerFunction(name, builder)
udaf
}
// scalastyle:off
/* register 0-22 were generated by this script
(0 to 22).map { x =>
val types = (1 to x).foldRight("RT")((i, s) => {s"A$i, $s"})
val typeTags = (1 to x).map(i => s"A${i}: TypeTag").foldLeft("RT: TypeTag")(_ + ", " + _)
val inputTypes = (1 to x).foldRight("Nil")((i, s) => {s"ScalaReflection.schemaFor[A$i].dataType :: $s"})
println(s"""
/**
* Register a Scala closure of ${x} arguments as user-defined function (UDF).
* @tparam RT return type of UDF.
* @since 1.3.0
*/
def register[$typeTags](name: String, func: Function$x[$types]): UserDefinedFunction = {
val dataType = ScalaReflection.schemaFor[RT].dataType
val inputTypes = Try($inputTypes).getOrElse(Nil)
def builder(e: Seq[Expression]) = ScalaUDF(func, dataType, e, inputTypes)
functionRegistry.registerFunction(name, builder)
UserDefinedFunction(func, dataType, inputTypes)
}""")
}
(1 to 22).foreach { i =>
val extTypeArgs = (1 to i).map(_ => "_").mkString(", ")
val anyTypeArgs = (1 to i).map(_ => "Any").mkString(", ")
val anyCast = s".asInstanceOf[UDF$i[$anyTypeArgs, Any]]"
val anyParams = (1 to i).map(_ => "_: Any").mkString(", ")
println(s"""
|/**
| * Register a user-defined function with ${i} arguments.
| * @since 1.3.0
| */
|def register(name: String, f: UDF$i[$extTypeArgs, _], returnType: DataType) = {
| functionRegistry.registerFunction(
| name,
| (e: Seq[Expression]) => ScalaUDF(f$anyCast.call($anyParams), returnType, e))
|}""".stripMargin)
}
*/
/**
* Register a Scala closure of 0 arguments as user-defined function (UDF).
* @tparam RT return type of UDF.
* @since 1.3.0
*/
def register[RT: TypeTag](name:String, func: Function0[RT]): UserDefinedFunction = {
val dataType = ScalaReflection.schemaFor[RT].dataType
val inputTypes = Try(Nil).getOrElse(Nil)
def builder(e: Seq[Expression]) = ScalaUDF(func, dataType, e, inputTypes)
functionRegistry.registerFunction(name, builder)
UserDefinedFunction(func, dataType, inputTypes)
}
FunctionRegistry的源码如下：
object FunctionRegistry {
type FunctionBuilder= Seq[Expression] => Expression
val expressions:Map[String, (ExpressionInfo,FunctionBuilder)] = Map(
// misc non-aggregate functions
expression[Abs]("abs"),
expression[CreateArray]("array"),
expression[Coalesce]("coalesce"),
expression[Explode]("explode"),
expression[Greatest]("greatest"),
expression[If]("if"),
expression[IsNaN]("isnan"),
expression[IsNull]("isnull"),
expression[IsNotNull]("isnotnull"),
expression[Least]("least"),
expression[Coalesce]("nvl"),
expression[Rand]("rand"),
expression[Randn]("randn"),
expression[CreateStruct]("struct"),
expression[CreateNamedStruct]("named_struct"),
expression[Sqrt]("sqrt"),
expression[NaNvl]("nanvl"),
// math functions
expression[Acos]("acos"),
expression[Asin]("asin"),
expression[Atan]("atan"),
expression[Atan2]("atan2"),
expression[Bin]("bin"),
expression[Cbrt]("cbrt"),
expression[Ceil]("ceil"),
expression[Ceil]("ceiling"),
expression[Cos]("cos"),
expression[Cosh]("cosh"),
expression[Conv]("conv"),
expression[EulerNumber]("e"),
expression[Exp]("exp"),
expression[Expm1]("expm1"),
expression[Floor]("floor"),
expression[Factorial]("factorial"),
expression[Hypot]("hypot"),
expression[Hex]("hex"),
expression[Logarithm]("log"),
expression[Log]("ln"),
expression[Log10]("log10"),
expression[Log1p]("log1p"),
expression[Log2]("log2"),
expression[UnaryMinus]("negative"),
expression[Pi]("pi"),
expression[Pow]("pow"),
expression[Pow]("power"),
expression[Pmod]("pmod"),
expression[UnaryPositive]("positive"),
expression[Rint]("rint"),
expression[Round]("round"),
expression[ShiftLeft]("shiftleft"),
expression[ShiftRight]("shiftright"),
expression[ShiftRightUnsigned]("shiftrightunsigned"),
expression[Signum]("sign"),
expression[Signum]("signum"),
expression[Sin]("sin"),
expression[Sinh]("sinh"),
expression[Tan]("tan"),
expression[Tanh]("tanh"),
expression[ToDegrees]("degrees"),
expression[ToRadians]("radians"),
// aggregate functions
expression[HyperLogLogPlusPlus]("approx_count_distinct"),
expression[Average]("avg"),
expression[Corr]("corr"),
expression[Count]("count"),
expression[First]("first"),
expression[First]("first_value"),
expression[Last]("last"),
expression[Last]("last_value"),
expression[Max]("max"),
expression[Average]("mean"),
expression[Min]("min"),
expression[StddevSamp]("stddev"),
expression[StddevPop]("stddev_pop"),
expression[StddevSamp]("stddev_samp"),
expression[Sum]("sum"),
expression[VarianceSamp]("variance"),
expression[VariancePop]("var_pop"),
expression[VarianceSamp]("var_samp"),
expression[Skewness]("skewness"),
expression[Kurtosis]("kurtosis"),
// string functions
expression[Ascii]("ascii"),
expression[Base64]("base64"),
expression[Concat]("concat"),
expression[ConcatWs]("concat_ws"),
expression[Encode]("encode"),
expression[Decode]("decode"),
expression[FindInSet]("find_in_set"),
expression[FormatNumber]("format_number"),
expression[GetJsonObject]("get_json_object"),
expression[InitCap]("initcap"),
expression[JsonTuple]("json_tuple"),
expression[Lower]("lcase"),
expression[Lower]("lower"),
expression[Length]("length"),
expression[Levenshtein]("levenshtein"),
expression[RegExpExtract]("regexp_extract"),
expression[RegExpReplace]("regexp_replace"),
expression[StringInstr]("instr"),
expression[StringLocate]("locate"),
expression[StringLPad]("lpad"),
expression[StringTrimLeft]("ltrim"),
expression[FormatString]("format_string"),
expression[FormatString]("printf"),
expression[StringRPad]("rpad"),
expression[StringRepeat]("repeat"),
expression[StringReverse]("reverse"),
expression[StringTrimRight]("rtrim"),
expression[SoundEx]("soundex"),
expression[StringSpace]("space"),
expression[StringSplit]("split"),
expression[Substring]("substr"),
expression[Substring]("substring"),
expression[SubstringIndex]("substring_index"),
expression[StringTranslate]("translate"),
expression[StringTrim]("trim"),
expression[UnBase64]("unbase64"),
expression[Upper]("ucase"),
expression[Unhex]("unhex"),
expression[Upper]("upper"),
// datetime functions
expression[AddMonths]("add_months"),
expression[CurrentDate]("current_date"),
expression[CurrentTimestamp]("current_timestamp"),
expression[CurrentTimestamp]("now"),
expression[DateDiff]("datediff"),
expression[DateAdd]("date_add"),
expression[DateFormatClass]("date_format"),
expression[DateSub]("date_sub"),
expression[DayOfMonth]("day"),
expression[DayOfYear]("dayofyear"),
expression[DayOfMonth]("dayofmonth"),
expression[FromUnixTime]("from_unixtime"),
expression[FromUTCTimestamp]("from_utc_timestamp"),
expression[Hour]("hour"),
expression[LastDay]("last_day"),
expression[Minute]("minute"),
expression[Month]("month"),
expression[MonthsBetween]("months_between"),
expression[NextDay]("next_day"),
expression[Quarter]("quarter"),
expression[Second]("second"),
expression[ToDate]("to_date"),
expression[ToUnixTimestamp]("to_unix_timestamp"),
expression[ToUTCTimestamp]("to_utc_timestamp"),
expression[TruncDate]("trunc"),
expression[UnixTimestamp]("unix_timestamp"),
expression[WeekOfYear]("weekofyear"),
expression[Year]("year"),
// collection functions
expression[Size]("size"),
expression[SortArray]("sort_array"),
expression[ArrayContains]("array_contains"),
// misc functions
expression[Crc32]("crc32"),
expression[Md5]("md5"),
expression[Sha1]("sha"),
expression[Sha1]("sha1"),
expression[Sha2]("sha2"),
expression[SparkPartitionID]("spark_partition_id"),
expression[InputFileName]("input_file_name"),
expression[MonotonicallyIncreasingID]("monotonically_increasing_id")
)
val builtin: SimpleFunctionRegistry = {
val fr = newSimpleFunctionRegistry
expressions.foreach { case (name, (info,builder)) => fr.registerFunction(name, info, builder) }
fr
}
/** See usage above. */
defexpression[T <: Expression](name:String)
(implicit tag: ClassTag[T]): (String, (ExpressionInfo,FunctionBuilder)) = {
// See if we can find a constructor that accepts Seq[Expression]
valvarargCtor = Try(tag.runtimeClass.getDeclaredConstructor(classOf[Seq[_]])).toOption
val builder = (expressions:Seq[Expression]) => {
if (varargCtor.isDefined) {
// If there is an apply method that accepts Seq[Expression], use that one.
Try(varargCtor.get.newInstance(expressions).asInstanceOf[Expression])match {
case Success(e) => e
case Failure(e) =>throw new AnalysisException(e.getMessage)
}
} else {
// Otherwise, find an ctor method that matches the number of arguments, and use that.
valparams = Seq.fill(expressions.size)(classOf[Expression])
val f = Try(tag.runtimeClass.getDeclaredConstructor(params : _*))match {
case Success(e) =>
e
case Failure(e) =>
throw new AnalysisException(s"Invalid number of arguments for function$name")
}
Try(f.newInstance(expressions : _*).asInstanceOf[Expression]) match {
case Success(e) => e
case Failure(e) =>throw new AnalysisException(e.getMessage)
}
}
}
val clazz = tag.runtimeClass
val df = clazz.getAnnotation(classOf[ExpressionDescription])
if (df != null) {
(name,
(new ExpressionInfo(clazz.getCanonicalName, name, df.usage(), df.extended()),
builder))
} else {
(name, (new ExpressionInfo(clazz.getCanonicalName, name), builder))
}
}
}
可以看出SparkSQL的内置函数也是和UDF一样注册的。
以上内容是王家林老师DT大数据梦工厂《 IMF传奇行动》第72课的学习笔记。
王家林老师是Spark、Flink、Docker、Android技术中国区布道师。Spark亚太研究院院长和首席专家，DT大数据梦工厂创始人，Android软硬整合源码级专家，英语发音魔术师，健身狂热爱好者。
微信公众账号：DT_Spark
电话：18610086859
QQ:1740415547
微信号：18610086859
新浪微博：ilovepains
注意！
本站转载的文章为个人学习借鉴使用，本站对版权不负任何法律责任。如果侵犯了您的隐私权益，请联系我们删除。