Powderkeg: teaching Clojure to Spark
Christophe Grand, Igor Ges at Clojure/conj 2016
While there’s no shortage of libraries that let one interact with Spark using Clojure they treat the problem as providing a more fluid experience of using Java interfaces. In doing so they overlook the REPL experience (by relying on AOT) and create code duplication (having their own fn, map, reduce, filter, mapcat…). But there is another way: ignore most of the Java/Scala interfaces and consider Spark artifacts (RDDs and streams) as a transducing context. Through doing this we have achieved a true REPL experience, a small API and overall made interactions with Spark more Clojuresque.