CrossValidator

CrossValidator implements K-fold cross validation.

The k-fold is to split the input RDD to the array of (rdd1, rdd2) of size k. rdd1 take (k-1)/k of RDD, and rdd2 take 1/k of RDD. The former will be used as the train set and the latter is the validation set.

The fit method runs the models with different parameter setup, evaluate the model and choose the best one to be returned. In details, each model will run k time with per parameter set, and in each round it updates the metrics. After k runs, the evaluator will chose the best mode from the accumulated metrics.