Hadoop2.6.5单机安装

2023-12-02 09:08
文章标签 安装 单机 hadoop2.6

本文主要是介绍Hadoop2.6.5单机安装，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

 

 

Hadoop2.6.5单机安装

 

 

JDK的安装

 

配置JDK环境变量

 

[root@spark1 soft]# vim /etc/profile
#JDK环境变量配置
#export JAVA_HOME=/application/jdk1.7.0_79
export JAVA_HOME=/application/jdk1.8.0_172
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib/rt.jar
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

环境变量生效

[root@spark1 soft]# source /etc/profile[root@spark1 soft]# java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
[root@spark1 soft]#

 

配置SSH无密码登陆

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

验证ssh，# ssh localhost 
不需要输入密码即可登录。

 

 

 

Hadoop安装

 

下载

下载地址：

https://www.apache.org/dyn/closer.cgi/hadoop/common/

https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.6.5/hadoop-2.6.5.tar.gz

 

解压安装 

[root@spark1 soft]# tar -zxvf hadoop-2.6.5.tar.gz -C /application/

 

创建hadoop安装所需目录

在/root /hadoop/目录下，建立tmp、hdfs/name、hdfs/data目录，执行如下命令 

#mkdir /root/hadoop/tmp 
#mkdir /root/hadoop/hdfs 
#mkdir /root/hadoop/hdfs/data 
#mkdir /root/hadoop/hdfs/name

 

设置Hadoop环境变量

#Hadoop环境变量配置
export HADOOP_HOME=/application/hadoop-2.6.5
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
[root@spark1 soft]# source /etc/profile

 

 

Hadoop配置 

进入$HADOOP_HOME/etc/hadoop目录，配置 hadoop-env.sh等。涉及的配置文件如下： 
1）hadoop-2.6.5/etc/hadoop/hadoop-env.sh 
2）hadoop-2.6.5/etc/hadoop/yarn-env.sh 
3）hadoop-2.6.5/etc/hadoop/core-site.xml 
4）hadoop-2.6.5/etc/hadoop/hdfs-site.xml 
5）hadoop-2.6.5/etc/hadoop/mapred-site.xml 
6）hadoop-2.6.5/etc/hadoop/yarn-site.xml

 

1）配置hadoop-env.sh

# The java implementation to use.
#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/application/jdk1.8.0_172

 

2）配置yarn-env.sh

# some Java parameters
# export JAVA_HOME=/home/y/libexec/jdk1.6.0/
export JAVA_HOME=/application/jdk1.8.0_172

 

 

3）配置core-site.xml 


添加如下配置：

[root@spark1 hadoop]# cat core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration><property><name>fs.default.name</name><value>hdfs://spark1:9000</value><description>HDFS的URI，文件系统://namenode标识:端口号</description>
</property><property><name>hadoop.tmp.dir</name><value>/root/hadoop/tmp</value><description>namenode上本地的hadoop临时文件夹</description>
</property>
</configuration>[root@spark1 hadoop]#

 

4）配置hdfs-site.xml 

[root@spark1 hadoop]# cat hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration>
<!--hdfs-site.xml-->
<property><name>dfs.name.dir</name><value>/root/hadoop/hdfs/name</value><description>namenode上存储hdfs名字空间元数据 </description> 
</property><property><name>dfs.data.dir</name><value>/root/hadoop/hdfs/data</value><description>datanode上数据块的物理存储位置</description>
</property><property><name>dfs.replication</name><value>1</value><description>副本个数，配置默认是3,应小于datanode机器数量</description>
</property>
</configuration>
[root@spark1 hadoop]#

 

5）配置mapred-site.xml 

[root@spark1 hadoop]# cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property><name>mapreduce.framework.name</name><value>yarn</value>
</property>
</configuration>
[root@spark1 hadoop]#

 

6）配置yarn-site.xml 


[root@spark1 hadoop]# cat yarn-site.xml
<?xml version="1.0"?>
<configuration><!-- Site specific YARN configuration properties -->
<property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value>
</property>
<property><name>yarn.resourcemanager.webapp.address</name><value>${yarn.resourcemanager.hostname}:8099</value>
</property>
</configuration>
[root@spark1 hadoop]#

说明：

    1）默认端口是8088；

    2）这里我设置了yarn.resourcemanager.webapp.address为：${yarn.resourcemanager.hostname}:8099；

 

 

Hadoop启动 

 

1）格式化namenode

hadoop namenode -format

 

2）启动NameNode 和 DataNode 守护进程

start-dfs.sh

 

3）启动ResourceManager 和 NodeManager 守护进程

start-yarn.sh

 

启动验证 

1）执行jps命令，有如下进程，说明Hadoop正常启动

[root@spark1 soft]# jps
5649 DataNode
6631 ResourceManager
5815 SecondaryNameNode
5527 NameNode
6728 NodeManager
7981 Jps
[root@spark1 soft]#

2）访问hdfs

http://192.168.2.191:50070

 

3）在浏览器中输入：http://192.168.2.191:8099/cluster 即可看到YARN的ResourceManager的界面。

注意：默认端口是8088，这里我设置了yarn.resourcemanager.webapp.address为：${yarn.resourcemanager.hostname}:8099

 

运行Hadoop的一个例子

[root@spark1 hadoop]# hadoop jar /application/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar pi 2 100
Number of Maps  = 2
Samples per Map = 100
19/04/13 13:46:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0
Wrote input for Map #1
Starting Job
19/04/13 13:46:51 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/04/13 13:46:52 INFO input.FileInputFormat: Total input paths to process : 2
19/04/13 13:46:52 INFO mapreduce.JobSubmitter: number of splits:2
19/04/13 13:46:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1555134174372_0001
19/04/13 13:46:53 INFO impl.YarnClientImpl: Submitted application application_1555134174372_0001
19/04/13 13:46:53 INFO mapreduce.Job: The url to track the job: http://spark1:8099/proxy/application_1555134174372_0001/
19/04/13 13:46:53 INFO mapreduce.Job: Running job: job_1555134174372_0001
19/04/13 13:47:00 INFO mapreduce.Job: Job job_1555134174372_0001 running in uber mode : false
19/04/13 13:47:00 INFO mapreduce.Job:  map 0% reduce 0%
19/04/13 13:47:14 INFO mapreduce.Job:  map 100% reduce 0%
19/04/13 13:47:19 INFO mapreduce.Job:  map 100% reduce 100%
19/04/13 13:47:19 INFO mapreduce.Job: Job job_1555134174372_0001 completed successfully
19/04/13 13:47:19 INFO mapreduce.Job: Counters: 49File System CountersFILE: Number of bytes read=50FILE: Number of bytes written=322803FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=522HDFS: Number of bytes written=215HDFS: Number of read operations=11HDFS: Number of large read operations=0HDFS: Number of write operations=3Job Counters Launched map tasks=2Launched reduce tasks=1Data-local map tasks=2Total time spent by all maps in occupied slots (ms)=23209Total time spent by all reduces in occupied slots (ms)=2996Total time spent by all map tasks (ms)=23209Total time spent by all reduce tasks (ms)=2996Total vcore-milliseconds taken by all map tasks=23209Total vcore-milliseconds taken by all reduce tasks=2996Total megabyte-milliseconds taken by all map tasks=23766016Total megabyte-milliseconds taken by all reduce tasks=3067904Map-Reduce FrameworkMap input records=2Map output records=4Map output bytes=36Map output materialized bytes=56Input split bytes=286Combine input records=0Combine output records=0Reduce input groups=2Reduce shuffle bytes=56Reduce input records=4Reduce output records=0Spilled Records=8Shuffled Maps =2Failed Shuffles=0Merged Map outputs=2GC time elapsed (ms)=2514CPU time spent (ms)=12980Physical memory (bytes) snapshot=697511936Virtual memory (bytes) snapshot=6333603840Total committed heap usage (bytes)=499646464Shuffle ErrorsBAD_ID=0CONNECTION=0IO_ERROR=0WRONG_LENGTH=0WRONG_MAP=0WRONG_REDUCE=0File Input Format Counters Bytes Read=236File Output Format Counters Bytes Written=97
Job Finished in 28.254 seconds
Estimated value of Pi is 3.12000000000000000000
[root@spark1 hadoop]#

 

 


==============================
QQ群：143522604
群里有相关资源
欢迎和大家一起学习、交流、提升！
==============================

这篇关于Hadoop2.6.5单机安装的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！


http://www.chinasem.cn/article/444727。

相关文章

Win安装MySQL8全过程

Win安装MySQL8全过程

《Win安装MySQL8全过程》：本文主要介绍Win安装MySQL8全过程,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录Win安装mysql81、下载MySQL2、解压文件3、新建文件夹data，用于保存数据库数据文件4、在mysql根目录下新建文件my.ini
阅读更多...
最详细安装 PostgreSQL方法及常见问题解决

最详细安装 PostgreSQL方法及常见问题解决

《最详细安装PostgreSQL方法及常见问题解决》：本文主要介绍最详细安装PostgreSQL方法及常见问题解决,介绍了在Windows系统上安装PostgreSQL及Linux系统上安装Po... 目录一、在 Windows 系统上安装 PostgreSQL1. 下载 PostgreSQL 安装包2.
阅读更多...
Maven如何手动安装依赖到本地仓库

Maven如何手动安装依赖到本地仓库

《Maven如何手动安装依赖到本地仓库》：本文主要介绍Maven如何手动安装依赖到本地仓库问题,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录一、下载依赖二、安装 JAR 文件到本地仓库三、验证安装四、在项目中使用该依赖1、注意事项2、额外提示总结一、下载依赖登
阅读更多...
如何在Mac上安装并配置JDK环境变量详细步骤

如何在Mac上安装并配置JDK环境变量详细步骤

《如何在Mac上安装并配置JDK环境变量详细步骤》：本文主要介绍如何在Mac上安装并配置JDK环境变量详细步骤,包括下载JDK、安装JDK、配置环境变量、验证JDK配置以及可选地设置PowerSh... 目录步骤 1：下载JDK步骤 2：安装JDK步骤 3：配置环境变量1. 编辑~/.zshrc（对于zsh
阅读更多...
如何在pycharm安装torch包

如何在pycharm安装torch包

《如何在pycharm安装torch包》：本文主要介绍如何在pycharm安装torch包方式,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录在pycharm安装torch包适http://www.chinasem.cn配于我电脑的指令为适用的torch包为总结在p
阅读更多...
在PyCharm中安装PyTorch、torchvision和OpenCV详解

在PyCharm中安装PyTorch、torchvision和OpenCV详解

《在PyCharm中安装PyTorch、torchvision和OpenCV详解》：本文主要介绍在PyCharm中安装PyTorch、torchvision和OpenCV方式,具有很好的参考价值,... 目录PyCharm安装PyTorch、torchvision和OpenCV安装python安装PyTor
阅读更多...
Python Transformer 库安装配置及使用方法

Python Transformer 库安装配置及使用方法

《PythonTransformer库安装配置及使用方法》HuggingFaceTransformers是自然语言处理（NLP）领域最流行的开源库之一,支持基于Transformer架构的预训练模... 目录python 中的 Transformer 库及使用方法一、库的概述二、安装与配置三、基础使用：Pi
阅读更多...
如何解决mmcv无法安装或安装之后报错问题

如何解决mmcv无法安装或安装之后报错问题

《如何解决mmcv无法安装或安装之后报错问题》：本文主要介绍如何解决mmcv无法安装或安装之后报错问题,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录mmcv无法安装或安装之后报错问题1.当我们运行YOwww.chinasem.cnLO时遇到2.找到下图所示这里3.
阅读更多...
Python 安装和配置flask, flask_cors的图文教程

Python 安装和配置flask, flask_cors的图文教程

《Python安装和配置flask,flask_cors的图文教程》：本文主要介绍Python安装和配置flask,flask_cors的图文教程,本文通过图文并茂的形式给大家介绍的非常详细,... 目录一.python安装:二,配置环境变量,三:检查Python安装和环境变量,四:安装flask和flas
阅读更多...
Win11安装PostgreSQL数据库的两种方式详细步骤

Win11安装PostgreSQL数据库的两种方式详细步骤

《Win11安装PostgreSQL数据库的两种方式详细步骤》PostgreSQL是备受业界青睐的关系型数据库,尤其是在地理空间和移动领域,：本文主要介绍Win11安装PostgreSQL数据库的... 目录一、exe文件安装 （推荐）下载安装包1. 选择操作系统2. 跳转到EDB（PostgreSQL 的
阅读更多...

Copyright China编程 23002807@qq.com

沪ICP备2023033072号-2