What is the best functional language to do Hadoop Map-Reduce

Question

I'm doing an assignment for a course, which requires me to implement a parallel MapReduce engine in a functional language and then use it solve certain simple problems.

Which functional language do you think I should use?

Here are my requirements:

• Should be relatively easy to learn, since I have only about 2 weeks for this assignment.
• Has existing MapReduce implementations which can be found on the web - my course does not forbid me from using open-sourced code or internet resources in general.
• Should fit the problem, and be an overall worthwhile language to learn (a relatively popular language).

I am currently considering Haskell and Clojure, but both these languages are new to me - I have no idea if any of these languages are actually appropriate for the situation.

Answer 1

down voteacceptedBoth Clojure and Haskell are definitely worth learning, for different reasons. If you get a chance, I would try both. I'd also suggest adding Scala to your list.

If you have to pick one, I would choose Clojure, for the following reasons:

• It's a Lisp - everyone should learn a Lisp. See http://www.paulgraham.com/avg.html
• It has a unique approach to concurrency - see http://www.infoq.com/presentations/Value-Identity-State-Rich-Hickey
• It's a JVM language, which makes it immediately useful from a practical perspective: the library & tool ecosystem on the JVM is extremely good, better than any other platform IMHO. If you want to do serious tech. work in the enterprise or startup space, it is very helpful to gain a good knowledge of the JVM. FWIW, Scala also falls into this category of "interesting JVM languages".

Also, Clojure makes parallel map-reduce very easy. Here's one to start with:

(reduce + (pmap inc (range 1000)))
=> 500500

Using ratherpmap than map is enough to give you a parallel mapping operation. There are also parallel reducers if you use Clojure 1.5, see the reducers framework for more details.

Apart from that, you can also use  Scalding, which is a Scala abstraction on top of Cascading to abstract low-level Hadoop details. It was developed at Twitter, and seems mature enough today so you can start actually using it without too much trouble.

Here is an example how you would do a Wordcount in Scalding:

package com.twitter.scalding.examples

import com.twitter.scalding._

class WordCountJob(args : Args) extends Job(args) {
  TextLine( args("input") )
    .flatMap('line -> 'word) { line : String => tokenize(line) }
    .groupBy('word) { _.size }
    .write( Tsv( args("output") ) )

  // Split a piece of text into individual words.
  def tokenize(text : String) : Array[String] = {
    // Lowercase each word and remove punctuation.
    text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+")
  }
}

I think it's a good candidate since because it's using Scala it's not too far from regular Map/Reduce Java programs, and even if you don't know Scala it's not too hard to pick up.