Spark 2.0.0 support

[a-roberts]

A few API changes and lots of project file changes

The v2p0 folder contains the code that we'll run with Spark 2 so contains the API changes, it's hard to tell what's new as they're entirely new files but to summarise I have

• used foreachrdd not foreach for streaming
• used awaitTerminationOrTimeout
• reduced the defaults in config/config.py to a tiny scale factor (I think people will want to build it and see if it works in a tiny environment before scaling up, previous defaults included 20 G driver memory) and uses your $SPARK_HOME

[a-roberts]

There's a problem here, if we do sbt package from the mllib directory then run the 25 mllib tests, there are four failures as certain test properties have been removed, thanks to Yves Leaute for pointing this out over an email. I'll resolve this by updating the PR and will also adjust the version setting in config.py to prevent artifact resolution problems

[TiagoPerez]

Hi Robert, great work with HiBench and Spark-Perf... Ever though of doing something similar for SparkBench? https://github.com/SparkTC/spark-bench

[harschware]

My environment is a little customized by the time I run this code (so line numbers don't match), but I am seeing this:

Loading configuration from <snip>/spark-perf/config/config.py
Traceback (most recent call last):
  File "./bin/../lib/sparkperf/main.py", line 40, in <module>
    config = imp.load_source("config", "", cf)
  File "", line 410
    MLLIB_SPARK_VERSION = 2.0.0

[zhang051]

I would like to run spark-perf with spark-2.0.0 support. Is there a beta- version of spark-perf (except updating individual files) that I can download?

Thanks,

Shuxia