Make sure that you have removed the comment tag and changed your JAVA_HOME in the hadoop-env.sh as well as the appropriate .bashrc and/or .profile:
# export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk
You can set your JAVA_HOME and PATH for all users (make sure you haven't previously set this to the wrong path) in /etc/profile.
Also, don't forget to activate the new change by logging-out/in or by executing source /etc/profile.
You should set JAVA_HOME in the hadoop-env.sh file also which is in the Hadoop configuration directory. By default, the JAVA_HOME setting line is commented.