Is there a NET equivalent to Apache Hadoop

Question

So, I've been looking at Hadoop with keen interest, and to be honest I'm fascinated, things don't get much cooler.

My only minor issue is I'm a C# developer and it's in Java.

It's not that I don't understand the Java as much as I'm looking for the Hadoop.net or NHadoop or the .NET project that embraces the Google MapReduce approach. Does anyone know of one?

Answer 1

Microsoft dropped its alternative (Dryad) in favor of Hadoop. Next year they will release MS SQL Server 2012 with Hadoop integration. Azure and Windows Sever support is being developed even as we speak.

It will be available in the first half in 2012.

Hadoop is the #1 BigData platform and is going to be supported by opensource and proprietary source (Java, .Net, Python, ...) even Oracle is adopting it.

If you were developing something, you should wait if you're on the .Net platform.

More information about what is possible will be available here

Recently, MySpace released their .NET MapReduce framework, Qizmt, as Open Source, so this is also a potential contender in this space.

Also see this - See http://research.microsoft.com/en-us/projects/dryadlinq/default.aspx or http://msdn.microsoft.com/en-us/library/dd179423.aspx

Answer 2

Hadoop is a Java-based platform. So, to execute a MapReduce job using a .NET language cannot be done directly as we don't have an exact equivalent of .NET for Hadoop. But the job can be done using Hadoop Streaming.

Hadoop streaming is a utility that comes with the Hadoop distribution. This utility allows you to create and run Map/Reduce jobs with any executable or script as the mapper and/or the reducer.

Here is a link that would help you help with creating/executing a simple MapReduce job with C#: https://martin.atlassian.net/wiki/spaces/lestermartin/blog/2014/07/28/27328520/hadoop+streaming+with+.net+map+reduce+api+executing+on+hdp+for+windows

Also, another option is HDInsight.

Azure HDInsight enables you to use rich productive tools for Hadoop and Spark with your preferred development environments. These development environments include Visual Studio, VSCode, Eclipse, and IntelliJ for Scala, Python, R, Java, and .NET support.

You can go try HDInsight as there are .NET interfaces to HDInsight.