Hadoop Installation on ubuntu

Can anyone help me in installing hadoop on my ubuntu vm . I am struck in formatting the namenode step I have tried all possible steps but I'm not able to understand where I have made the mistake. The following is the error I am getting while formatting .

user@ubuntu:/lib/hadoop/hadoop-2.7.1/bin$ hadoop namenode -format

DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
/lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: /lib/jvm/jdk1.8.0_111/bin/java: Permission denied
/lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: exec: /lib/jvm/jdk1.8.0_111/bin/java: cannot execute: Permission denied

And when i am going to start the namenodes it is telling me the following error and repetedly asking me the password.

user@ubuntu:/lib/hadoop/hadoop-2.7.1/sbin$ start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
/lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: /lib/jvm/jdk1.8.0_111/bin/java: Permission denied
/lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: exec: /lib/jvm/jdk1.8.0_111/bin/java: cannot execute: Permission denied
Starting namenodes on []
user@localhost's password:
localhost: starting namenode, logging to /lib/hadoop/hadoop-2.7.1/logs/hadoop-user-namenode-ubuntu.out
localhost: /lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: /lib/jvm/jdk1.8.0_111/bin/java: Permission denied
localhost: /lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: exec: /lib/jvm/jdk1.8.0_111/bin/java: cannot execute: Permission denied
user@localhost's password:
localhost: starting datanode, logging to /lib/hadoop/hadoop-2.7.1/logs/hadoop-user-datanode-ubuntu.out
localhost: /lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: /lib/jvm/jdk1.8.0_111/bin/java: Permission denied
localhost: /lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: exec: /lib/jvm/jdk1.8.0_111/bin/java: cannot execute: Permission denied
/lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: /lib/jvm/jdk1.8.0_111/bin/java: Permission denied
/lib/hadoop/hadoop-2.7.1/bin/hdfs: line 304: exec: /lib/jvm/jdk1.8.0_111/bin/java: cannot execute: Permission denied
starting yarn daemons
starting resourcemanager, logging to /lib/hadoop/hadoop-2.7.1/logs/yarn-user-resourcemanager-ubuntu.out
/lib/hadoop/hadoop-2.7.1/bin/yarn: line 330: /lib/jvm/jdk1.8.0_111/bin/java: Permission denied
/lib/hadoop/hadoop-2.7.1/bin/yarn: line 330: exec: /lib/jvm/jdk1.8.0_111/bin/java: cannot execute: Permission denied
user@localhost's password: 

2 Answers

I just ran this on a VM (Kubuntu 16.04) with hadoop version 2.7.3.

Prerequisites:

1. Ubuntu OS
2. Non-root user with sudo privileges
3. Installed java

Steps:

1. Download hadoop version from here by

   • Clicking the binary option under the tarball column for desired version

   • Click on the link under We suggest the following mirror site for your download: or right click and save link

     • if the link was saved then download using:

       wget

2. Check the integrity of the download:

   • Download .md5 file from download page

   • Run shasum -a 256 hadoop-2.7.3.tar.gz and compare result with cat hadoop-2.7.3.tar.gz.mds, at this line

     ... hadoop-2.7.3.tar.gz: SHA256 = D489DF38 08244B90 6EB38F4D 081BA49E 50C4603D B03EFD5E 594A1E98 B09259C2
     ...

3. Install hadoop:

   • Decompress and move:

     tar -xzvf hadoop-2.7.3.tar.gz && sudo mv hadoop-2.7.3 /usr/local/hadoop

   • locate java on your system with:

     readlink -f /usr/bin/java | sed "s:bin/java::"
     # result
     /usr/lib/jvm/java-8-openjdk-amd64/jre/

   • edit hadoop-env.sh file sudo nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh:

     • Option 1 (static value):

       #export JAVA_HOME=${JAVA_HOME}
       export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/

     • Option 2 (dynamic value changes with change in java on system):

       #export JAVA_HOME=${JAVA_HOME}
       export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")

4. Run hadoop:

   • /usr/local/hadoop/bin/hadoop

     • Output:

       Usage: hadoop [--config confdir] [COMMAND | CLASSNAME]
       CLASSNAME run the class named CLASSNAME
       or
       where COMMAND is one of:
       fs run a generic filesystem user client
       version print the version
       jar <jar> run a jar file note: please use "yarn jar" to launch YARN applications, not this command.
       checknative [-a|-h] check native hadoop and compression libraries availability
       distcp <srcurl> <desturl> copy file or directories recursively
       archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
       classpath prints the class path needed to get the
       credential interact with credential providers Hadoop jar and the required libraries
       daemonlog get/set the log level for each daemon

5. Test:

   mkdir ~/input
   cp /usr/local/hadoop/etc/hadoop/*.xml ~/input
   /usr/local/hadoop/bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep ~/input ~/grep_example 'principal[.]*';

   • Check for errors in the result:

      File System Counters FILE: Number of bytes read=1247674 FILE: Number of bytes written=2324248 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=2 Map output records=2 Map output bytes=37 Map output materialized bytes=47 Input split bytes=114 Combine input records=0 Combine output records=0 Reduce input groups=2 Reduce shuffle bytes=47 Reduce input records=2 Reduce output records=2 Spilled Records=4 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=61 Total committed heap usage (bytes)=263520256 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=151 File Output Format Counters Bytes Written=37

Source:

Add path to .bashrc:

# hadoop executable
export PATH="/usr/local/hadoop/bin:$PATH"

It seems you have missed some steps related to over riding the permission setting in hadoop configuration. But it shouldn't be a problem. It will ask for password for each component in Hadoop, so from your screenshot, 3times paswword have been asked, one for namenode, then the datanode and then the resource manager. It will ask another 2 times, one for secondary namenode and the other for Node Manager. Once you are done with this, the rest should follow smoothly.

Please refer the below link:

Detailed Step-by-Step Installation of Hadoop on Ubuntu, with detailed screenshots and Explanation:

This document has been created by me after installing several versions of Hadoop on several versions of Ubuntu. So, most of the common problems/errors/queries that a person faces while installing Hadoop on Ubuntu has been discussed in the above document.