Hadoop大数据应用:HDFS 集群节点缩容

2024-03-14 21:20

本文主要是介绍Hadoop大数据应用:HDFS 集群节点缩容,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!

目录

一、实验

1.环境

2.HDFS 集群节点缩容

二、问题

1.数据迁移有哪些状态

2.数据迁移失败


一、实验

1.环境

(1)主机

表1  主机

主机架构软件版本IP备注
hadoop

NameNode (已部署)

SecondaryNameNode (已部署)

ResourceManager(已部署)

hadoop

2.7.7192.168.204.50

node01

DataNode(已部署)

NodeManager(已部署)

hadoop

2.7.7192.168.204.51
node02

DataNode(已部署)

NodeManager(已部署)

hadoop

2.7.7192.168.204.52
node03

DataNode(已部署)

NodeManager(已部署)

hadoop

2.7.7192.168.204.53
node04

DataNode(已部署)

hadoop

2.7.7192.168.204.54需求迁移主机

(2)查看jps

hadoop节点

[root@hadoop ~]# jps

node01节点

node02节点

node03节点

node04节点

2.HDFS 集群节点缩容

(1)查看节点状态 (已使用)

node01: 176KB (约等于0.18M )

node02: 280KB  (约等于0.28M )

node03: 260KB  (约等于0.26M )

node04:  8KB (约等于0.008M)

[root@hadoop hadoop]# ./bin/hdfs dfsadmin -report
Configured Capacity: 822126559232 (765.67 GB)
Present Capacity: 798788423680 (743.93 GB)
DFS Remaining: 798787682304 (743.93 GB)
DFS Used: 741376 (724 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0-------------------------------------------------
Live datanodes (4):Name: 192.168.204.54:50010 (node04)
Hostname: node04
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 8192 (8 KB)
Non DFS Used: 5658730496 (5.27 GB)
DFS Remaining: 199872901120 (186.15 GB)
DFS Used%: 0.00%
DFS Remaining%: 97.25%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:16:33 CST 2024Name: 192.168.204.53:50010 (node03)
Hostname: node03
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 266240 (260 KB)
Non DFS Used: 5620936704 (5.23 GB)
DFS Remaining: 199910436864 (186.18 GB)
DFS Used%: 0.00%
DFS Remaining%: 97.27%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:16:33 CST 2024Name: 192.168.204.51:50010 (node01)
Hostname: node01
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 180224 (176 KB)
Non DFS Used: 6029176832 (5.62 GB)
DFS Remaining: 199502282752 (185.80 GB)
DFS Used%: 0.00%
DFS Remaining%: 97.07%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:16:34 CST 2024Name: 192.168.204.52:50010 (node02)
Hostname: node02
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 286720 (280 KB)
Non DFS Used: 6029291520 (5.62 GB)
DFS Remaining: 199502061568 (185.80 GB)
DFS Used%: 0.00%
DFS Remaining%: 97.07%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:16:34 CST 2024

(2)查看

[root@hadoop ~]# cd /usr/local/hadoop/
[root@hadoop hadoop]# ./bin/hadoop fs -ls /

(3)上传第一个测试文件

[root@hadoop ~]# /usr/local/hadoop/bin/hadoop fs -put /root/jenkins.tar.gz /jenkins.tar.gz

观察上传变化

[root@hadoop hadoop]# ./bin/hadoop fs -ls /

(4)上传第二个测试文件

[root@hadoop ~]# /usr/local/hadoop/bin/hadoop fs -put /root/pinpoint-image.tar.gz /pinpoint.tar.gz

 观察上传变化

[root@hadoop hadoop]# ./bin/hadoop fs -ls /

(5)web查看

http://192.168.204.50:50070/

(6)清空本地文件

[root@hadoop ~]# rm -rf jenkins.tar.gz pinpoint-image.tar.gz

(7)再次查看节点状态 (已使用)

node01: 2.41GB 

node02: 2.27GB 

node03: 2.15GB  

node04:  2.26GB 

[root@hadoop hadoop]#  ./bin/hdfs dfsadmin -report
Configured Capacity: 822126559232 (765.67 GB)
Present Capacity: 798784053248 (743.93 GB)
DFS Remaining: 789019389952 (734.83 GB)
DFS Used: 9764663296 (9.09 GB)
DFS Used%: 1.22%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0-------------------------------------------------
Live datanodes (4):Name: 192.168.204.54:50010 (node04)
Hostname: node04
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2424492032 (2.26 GB)
Non DFS Used: 5662638080 (5.27 GB)
DFS Remaining: 197444509696 (183.88 GB)
DFS Used%: 1.18%
DFS Remaining%: 96.07%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:57:57 CST 2024Name: 192.168.204.53:50010 (node03)
Hostname: node03
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2312085504 (2.15 GB)
Non DFS Used: 5620936704 (5.23 GB)
DFS Remaining: 197598617600 (184.03 GB)
DFS Used%: 1.12%
DFS Remaining%: 96.14%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:57:58 CST 2024Name: 192.168.204.51:50010 (node01)
Hostname: node01
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2592927744 (2.41 GB)
Non DFS Used: 6029148160 (5.62 GB)
DFS Remaining: 196909563904 (183.39 GB)
DFS Used%: 1.26%
DFS Remaining%: 95.80%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:57:57 CST 2024Name: 192.168.204.52:50010 (node02)
Hostname: node02
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2435158016 (2.27 GB)
Non DFS Used: 6029783040 (5.62 GB)
DFS Remaining: 197066698752 (183.53 GB)
DFS Used%: 1.18%
DFS Remaining%: 95.88%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 15:57:59 CST 2024

(8)计算总存储使用量

9.09 G (jenkins 1.24 GB,pinpoint 3.27 GB ,存储副本量为2)

(9)修改配置文件(hadoop节点)

#添加代码[root@hadoop hadoop]# vim hdfs-site.xml

修改前:

修改后:

……<property><name>dfs.hosts.exclude</name><value>/usr/local/hadoop/etc/hadoop/exclude</value></property>
……

(10)配置移除主机

[root@hadoop hadoop]# vim /usr/local/hadoop/etc/hadoop/exclude

node04

(11)数据刷新

[root@hadoop hadoop]# ./bin/hdfs dfsadmin -refreshNodes

(12)查看节点状态 

node04节点状态变为Decommissioned in Program

[root@hadoop hadoop]#  ./bin/hdfs dfsadmin -report
Configured Capacity: 619019411456 (576.51 GB)
Present Capacity: 600866701319 (559.60 GB)
DFS Remaining: 590966771712 (550.38 GB)
DFS Used: 9899929607 (9.22 GB)
DFS Used%: 1.65%
Under replicated blocks: 2
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0-------------------------------------------------
Live datanodes (4):Name: 192.168.204.54:50010 (node04)
Hostname: node04
Decommission Status : Decommission in progress
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2424492032 (2.26 GB)
Non DFS Used: 5663162368 (5.27 GB)
DFS Remaining: 197443985408 (183.88 GB)
DFS Used%: 1.18%
DFS Remaining%: 96.06%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 16:34:33 CST 2024Name: 192.168.204.53:50010 (node03)
Hostname: node03
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2312085504 (2.15 GB)
Non DFS Used: 5756112896 (5.36 GB)
DFS Remaining: 197463441408 (183.90 GB)
DFS Used%: 1.12%
DFS Remaining%: 96.07%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Thu Mar 14 16:34:31 CST 2024Name: 192.168.204.51:50010 (node01)
Hostname: node01
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2592927744 (2.41 GB)
Non DFS Used: 6232051712 (5.80 GB)
DFS Remaining: 196706660352 (183.20 GB)
DFS Used%: 1.26%
DFS Remaining%: 95.71%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 3
Last contact: Thu Mar 14 16:34:33 CST 2024Name: 192.168.204.52:50010 (node02)
Hostname: node02
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2570424327 (2.39 GB)
Non DFS Used: 6164545529 (5.74 GB)
DFS Remaining: 196796669952 (183.28 GB)
DFS Used%: 1.25%
DFS Remaining%: 95.75%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Thu Mar 14 16:34:32 CST 2024Decommissioning datanodes (1):Name: 192.168.204.54:50010 (node04)
Hostname: node04
Decommission Status : Decommission in progress
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2424492032 (2.26 GB)
Non DFS Used: 5663162368 (5.27 GB)
DFS Remaining: 197443985408 (183.88 GB)
DFS Used%: 1.18%
DFS Remaining%: 96.06%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 16:34:33 CST 2024

(13) 再次查看节点

node04节点状态变为Decommissioned

[root@hadoop hadoop]#  ./bin/hdfs dfsadmin -report
Configured Capacity: 619019411456 (576.51 GB)
Present Capacity: 601338761216 (560.04 GB)
DFS Remaining: 589149618176 (548.69 GB)
DFS Used: 12189143040 (11.35 GB)
DFS Used%: 2.03%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0-------------------------------------------------
Live datanodes (4):Name: 192.168.204.54:50010 (node04)
Hostname: node04
Decommission Status : Decommissioned
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2424492032 (2.26 GB)
Non DFS Used: 5662666752 (5.27 GB)
DFS Remaining: 197444481024 (183.88 GB)
DFS Used%: 1.18%
DFS Remaining%: 96.07%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 16:42:33 CST 2024Name: 192.168.204.53:50010 (node03)
Hostname: node03
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 3123707904 (2.91 GB)
Non DFS Used: 5620969472 (5.23 GB)
DFS Remaining: 196786962432 (183.27 GB)
DFS Used%: 1.52%
DFS Remaining%: 95.75%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 16:42:31 CST 2024Name: 192.168.204.51:50010 (node01)
Hostname: node01
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2998738944 (2.79 GB)
Non DFS Used: 6029717504 (5.62 GB)
DFS Remaining: 196503183360 (183.01 GB)
DFS Used%: 1.46%
DFS Remaining%: 95.61%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 16:42:33 CST 2024Name: 192.168.204.52:50010 (node02)
Hostname: node02
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 3642204160 (3.39 GB)
Non DFS Used: 6029963264 (5.62 GB)
DFS Remaining: 195859472384 (182.41 GB)
DFS Used%: 1.77%
DFS Remaining%: 95.29%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 16:42:32 CST 2024

(14) 重启

[root@hadoop hadoop]# ssh node04[root@node04 ~]# reboot

(15)查看节点(已使用)

node01: 2.79GB 

node02: 3.39GB 

node03: 2.91GB  

[root@hadoop hadoop]#  ./bin/hdfs dfsadmin -report
Configured Capacity: 619019411456 (576.51 GB)
Present Capacity: 601338118144 (560.04 GB)
DFS Remaining: 589148975104 (548.69 GB)
DFS Used: 12189143040 (11.35 GB)
DFS Used%: 2.03%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0-------------------------------------------------
Live datanodes (3):Name: 192.168.204.53:50010 (node03)
Hostname: node03
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 3123707904 (2.91 GB)
Non DFS Used: 5621551104 (5.24 GB)
DFS Remaining: 196786380800 (183.27 GB)
DFS Used%: 1.52%
DFS Remaining%: 95.75%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 17:00:14 CST 2024Name: 192.168.204.51:50010 (node01)
Hostname: node01
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2998738944 (2.79 GB)
Non DFS Used: 6029742080 (5.62 GB)
DFS Remaining: 196503158784 (183.01 GB)
DFS Used%: 1.46%
DFS Remaining%: 95.61%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 17:00:12 CST 2024Name: 192.168.204.52:50010 (node02)
Hostname: node02
Decommission Status : Normal
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 3642204160 (3.39 GB)
Non DFS Used: 6030000128 (5.62 GB)
DFS Remaining: 195859435520 (182.41 GB)
DFS Used%: 1.77%
DFS Remaining%: 95.29%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 17:00:12 CST 2024Dead datanodes (1):Name: 192.168.204.54:50010 (node04)
Hostname: node04
Decommission Status : Decommissioned
Configured Capacity: 205531639808 (191.42 GB)
DFS Used: 2424492032 (2.26 GB)
Non DFS Used: 5663240192 (5.27 GB)
DFS Remaining: 197443907584 (183.88 GB)
DFS Used%: 1.18%
DFS Remaining%: 96.06%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Mar 14 16:47:27 CST 2024

计算总使用量

(16)回顾使用量变化

表2  数据迁移

主机初始上传迁移前迁移后IP备注
node0176KB 

jenkins 1.24 GB

pinpoint 3.27 GB 

2.41GB2.79GB 192.168.204.51

replication 2

(副本备份量)

node02280KB  2.27GB 3.39GB 192.168.204.52
node03260KB  2.15GB  2.91GB  192.168.204.53
node048KB 2.26GB /192.168.204.54迁移主机
总大小9.09GB9.09GB

二、问题

1.数据迁移有哪些状态

(1)状态

1)正常状态
Normal2)数据正在迁移
Decommissioned in Progress3)数据迁移完成 (仅变成当前状态才能down机下线)
Decommissioned

2.数据迁移失败

(1)失败

迁移node04状态依然为Normal

(2)原因分析

配置错误。

(3)解决方法

修改配置。

修改前:

修改后:

成功:

这篇关于Hadoop大数据应用:HDFS 集群节点缩容的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!



http://www.chinasem.cn/article/809759

相关文章

使用Java将各种数据写入Excel表格的操作示例

《使用Java将各种数据写入Excel表格的操作示例》在数据处理与管理领域,Excel凭借其强大的功能和广泛的应用,成为了数据存储与展示的重要工具,在Java开发过程中,常常需要将不同类型的数据,本文... 目录前言安装免费Java库1. 写入文本、或数值到 Excel单元格2. 写入数组到 Excel表格

python处理带有时区的日期和时间数据

《python处理带有时区的日期和时间数据》这篇文章主要为大家详细介绍了如何在Python中使用pytz库处理时区信息,包括获取当前UTC时间,转换为特定时区等,有需要的小伙伴可以参考一下... 目录时区基本信息python datetime使用timezonepandas处理时区数据知识延展时区基本信息

Qt实现网络数据解析的方法总结

《Qt实现网络数据解析的方法总结》在Qt中解析网络数据通常涉及接收原始字节流,并将其转换为有意义的应用层数据,这篇文章为大家介绍了详细步骤和示例,感兴趣的小伙伴可以了解下... 目录1. 网络数据接收2. 缓冲区管理(处理粘包/拆包)3. 常见数据格式解析3.1 jsON解析3.2 XML解析3.3 自定义

SpringMVC 通过ajax 前后端数据交互的实现方法

《SpringMVC通过ajax前后端数据交互的实现方法》:本文主要介绍SpringMVC通过ajax前后端数据交互的实现方法,本文给大家介绍的非常详细,对大家的学习或工作具有一定的参考借鉴价... 在前端的开发过程中,经常在html页面通过AJAX进行前后端数据的交互,SpringMVC的controll

Pandas统计每行数据中的空值的方法示例

《Pandas统计每行数据中的空值的方法示例》处理缺失数据(NaN值)是一个非常常见的问题,本文主要介绍了Pandas统计每行数据中的空值的方法示例,具有一定的参考价值,感兴趣的可以了解一下... 目录什么是空值?为什么要统计空值?准备工作创建示例数据统计每行空值数量进一步分析www.chinasem.cn处

C语言中位操作的实际应用举例

《C语言中位操作的实际应用举例》:本文主要介绍C语言中位操作的实际应用,总结了位操作的使用场景,并指出了需要注意的问题,如可读性、平台依赖性和溢出风险,文中通过代码介绍的非常详细,需要的朋友可以参... 目录1. 嵌入式系统与硬件寄存器操作2. 网络协议解析3. 图像处理与颜色编码4. 高效处理布尔标志集合

如何使用 Python 读取 Excel 数据

《如何使用Python读取Excel数据》:本文主要介绍使用Python读取Excel数据的详细教程,通过pandas和openpyxl,你可以轻松读取Excel文件,并进行各种数据处理操... 目录使用 python 读取 Excel 数据的详细教程1. 安装必要的依赖2. 读取 Excel 文件3. 读

Spring 请求之传递 JSON 数据的操作方法

《Spring请求之传递JSON数据的操作方法》JSON就是一种数据格式,有自己的格式和语法,使用文本表示一个对象或数组的信息,因此JSON本质是字符串,主要负责在不同的语言中数据传递和交换,这... 目录jsON 概念JSON 语法JSON 的语法JSON 的两种结构JSON 字符串和 Java 对象互转

C++如何通过Qt反射机制实现数据类序列化

《C++如何通过Qt反射机制实现数据类序列化》在C++工程中经常需要使用数据类,并对数据类进行存储、打印、调试等操作,所以本文就来聊聊C++如何通过Qt反射机制实现数据类序列化吧... 目录设计预期设计思路代码实现使用方法在 C++ 工程中经常需要使用数据类,并对数据类进行存储、打印、调试等操作。由于数据类

Java中的Lambda表达式及其应用小结

《Java中的Lambda表达式及其应用小结》Java中的Lambda表达式是一项极具创新性的特性,它使得Java代码更加简洁和高效,尤其是在集合操作和并行处理方面,:本文主要介绍Java中的La... 目录前言1. 什么是Lambda表达式?2. Lambda表达式的基本语法例子1:最简单的Lambda表