使用pt-table-checksum校验MySQL主从复制

2023-10-10 20:38

本文主要是介绍使用pt-table-checksum校验MySQL主从复制,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!

pt-table-checksum是一个基于MySQL数据库主从架构在线数据一致性校验工具。其工作原理在主库上运行, 通过对同步的表在主从段执行checksum, 从而判断数据是否一致。在校验完毕时,该工具将列出与主库存在差异的对象结果。

一、主从不一致的情形

    Master端使用了不确定的语句(如:CURRENT_USER(), UUID())不正确的故障转移(failover)流程误操作或直接在Slave进行DML操作持续的升级更新(Rolling upgrades)混合使用事务引擎和非事务引擎的表跳过了复制事件 (SET GLOBAL SQL_SLAVE_SKIP_COUNTER = N)使用临时表(Temporary Tables)复制过滤(Replication Filters)使用含LIMIT且没有order by的更新语句(update/delete with LIMIT clause without order by)

二、pt-table-checksum特性

    pt-table-checksum connects to the server you specify, and finds databases and tables that match the filters you specify  (if any). It works one table at a time, so it does not accumulatelarge amounts of memory or do a lot of work before beginning to checksum. This makes it usableon very large servers. We have used it on servers with hundreds of thousands of databases and tables, and trillions of rows. No matter how large the server is, pt-table-checksum works equally well.One reason it can work on very large tables is that it divides each table into chunks of rows, and checksums each chunk with a single REPLACE..SELECT query. It varies the chunk size to make the checksum queries run in the desired amount of time. The goal of chunking the tables, instead of doing each table with a single big query, is to ensure that checksums are unintrusive and don’t cause toomuch replication lag or load on the server. That’s why the target time for each chunk is 0.5 seconds by default.The tool keeps track of how quickly the server is able to execute the queries, and adjusts the chunks as it learns more about the server’s performance. It uses an exponentially decaying weighted average to keep the chunk size stable, yet remain responsive if the server’s performance changes during checksumming for any reason. This means that the tool will quickly throttle itself if your server becomes heavily loaded during a trafficc spike or a background task, for example.After pt-table-checksum finishes checksumming all of the chunks in a table, it pauses and waits for all detected replicas to finish executing the checksum queries. Once that is finished, it checks all of the replicas to see if they have the same data as the master, and then prints a line of output with the results. 

三、演示pt-table-checksum

-- 环境:Master 192.168.1.8, Slave 192.168.1.12,主从已构建
-- 演示中,mysql提示符为:用户名@主机名[库名]
-- 如master@localhost[test],表示master用户表示在主,slave表示用户在slave上
-- 复制过滤器如下:
[root@vdbsrv4 ~]# mysql -uroot -p -e "show slave status\G"|grep "Replicate
Enter password: 
              Replicate_Do_DB: sakila,test
          Replicate_Ignore_DB: mysql
a、环境准备
--对用于执行checksum的用户授权,注,如果主从复制未开启mysql系统库复制,则从库也同样执行用户创建
master@localhost[test]> grant select, process, super, replication slave on *.* to
 ->  'checksums'@'192.168.1.%' identified by 'xxx';
Query OK, 0 rows affected (0.00 sec)--主库建表及插入记录
master@localhost[test]> create table t(id tinyint primary key auto_increment,ename varchar(20));
Query OK, 0 rows affected (0.01 sec)master@localhost[test]> insert into t(ename) values('Leshami'),('Henry'),('Jack');
Query OK, 3 rows affected (0.01 sec)
Records: 3  Duplicates: 0  Warnings: 0--从库查询结果
slave@localhost[test]> select * from t;
+----+---------+
| id | ename  |
+----+---------+
|  1 | Leshami |
|  2 | Henry  |
|  3 | Jack    |
+----+---------+--模拟数据不一致,slave端删除记录
slave@localhost[test]> delete from t where id=2;b、单表校验
-- 执行pt-table-checksum
[root@vdbsrv3 ~]# pt-table-checksum h='192.168.1.8',u='checksums',p='xxx',P=3306 \
> -dtest -tt --nocheck-replication-filters \
> --no-check-binlog-format  --replicate=test.checksum
            TS ERRORS  DIFFS    ROWS  CHUNKS SKIPPED    TIME TABLE
08-06T10:14:32      0      1        3      1      0  0.031 test.tTS            :完成检查的时间。
ERRORS        :检查时候发生错误和警告的数量。
DIFFS        :0表示一致,1表示不一致。当指定--no-replicate-check时,
                会一直为0,当指定--replicate-check-only会显示不同的信息。
ROWS          :表的行数。
CHUNKS        :被划分到表中的块的数目。
SKIPPED      :由于错误或警告或过大,则跳过块的数目。
TIME          :执行的时间。
TABLE        :被检查的表名。--基于从库端SQL脚本查看checksum结果
slave@localhost[test]> system more check_sync_stat.sql;
SELECT 
    db, tbl, SUM(this_cnt) AS total_rows, COUNT(*) AS chunks
FROM
    test.checksum
WHERE
    (master_cnt <> this_cnt
        OR master_crc <> this_crc
        OR ISNULL(master_crc) <> ISNULL(this_crc))
GROUP BY db , tbl;slave@localhost[test]> source check_sync_stat.sql;
+------+-----+------------+--------+
| db  | tbl | total_rows | chunks |
+------+-----+------------+--------+
| test | t  |          2 |      1 |
+------+-----+------------+--------+--从库端插入记录
slave@localhost[test]> insert into t(ename) values('Robin');
Query OK, 1 row affected (0.00 sec)slave@localhost[test]> select * from t;
+----+---------+
| id | ename  |
+----+---------+
|  1 | Leshami |   #Author : Leshami
|  3 | Jack    |   #Blog     : http://blog.csdn.net/leshami
|  4 | Robin  |
+----+---------+-- 再次在master端执行pt-table-checksum(此处略),后查看结果如下
slave@localhost[test]> source check_sync_stat.sql;
+------+-----+------------+--------+
| db  | tbl | total_rows | chunks |
+------+-----+------------+--------+
| test | t  |          3 |      1 |
+------+-----+------------+--------+b、查看pt-table-checksum工作原理
-- 使用--explain参数,不执行checksum,列出checksum时真正执行的SQL语句
Show, but do not execute, checksum queries (disables --[no]empty-replicate-table). If specifed
twice, the tool actually iterates through the chunking algorithm, printing the upper and lower boundary values
for each chunk, but not executing the checksum queries.[root@vdbsrv3 ~]# pt-table-checksum h='192.168.1.8',u='checksums',p='xxx',P=3306 \
> -dtest -tt --nocheck-replication-filters \
> --no-check-binlog-format  --replicate=test.checksum --explain
--
-- test.t
--REPLACE INTO `test`.`checksum` (db, tbl, chunk, chunk_index, lower_boundary,
upper_boundary, this_cnt, this_crc) SELECT ?, ?, ?, ?, ?, ?, COUNT(*) AS cnt, 
COALESCE(LOWER(CONV(BIT_XOR(CAST(CRC32(CONCAT_WS('#', `id`, `ename`, 
CONCAT(ISNULL(`ename`)))) AS UNSIGNED)), 10, 16)), 0) AS crc FROM `test`.`t`/*checksum table*/c、库级别校验
[root@vdbsrv3 ~]# pt-table-checksum h='192.168.1.8',u='checksums',p='xxx',P=3306 \
> --databases=sakila --nocheck-replication-filters --no-check-binlog-format \
> --replicate=test.checksum 
            TS ERRORS  DIFFS    ROWS  CHUNKS SKIPPED    TIME TABLE
08-06T13:52:17      0      0      200      1      0  0.083 sakila.actor
08-06T13:52:17      0      0      603      1      0  0.024 sakila.address
08-06T13:52:17      0      0      16      1      0  0.012 sakila.category
08-06T13:52:17      0      0      600      1      0  0.025 sakila.city
08-06T13:52:17      0      0      109      1      0  0.019 sakila.country
08-06T13:52:17      0      0      599      1      0  0.019 sakila.customer
08-06T13:52:17      0      0    1000      1      0  0.035 sakila.film
08-06T13:52:17      0      0    5462      1      0  0.295 sakila.film_actor
08-06T13:52:17      0      0    1000      1      0  0.019 sakila.film_category
08-06T13:52:17      0      0    1000      1      0  0.015 sakila.film_text
08-06T13:52:17      0      0    4581      1      0  0.041 sakila.inventory
08-06T13:52:17      0      0        6      1      0  0.012 sakila.language
08-06T13:52:18      0      0    16049      1      0  0.367 sakila.payment
08-06T13:52:18      0      0    16044      1      0  0.357 sakila.rental
08-06T13:52:18      0      0        2      1      0  0.013 sakila.staff
08-06T13:52:18      0      0        2      1      0  0.012 sakila.store--在从库删除一张表
slave@localhost[test]> drop table sakila.payment;
Query OK, 0 rows affected (0.01 sec)-- 再次执行pt-table-checksum,收到如下提示
08-06T13:56:42 Skipping table sakila.payment because it has problems on these replicas:
Table sakila.payment does not exist on replica vdbsrv4
This can break replication.  If you understand the risks, specify --no-check-slave-tables to disable this check.
08-06T13:56:42 Error checksumming table sakila.payment: DBD::mysql::db selectrow_hashref failed:
Table 'sakila.payment' doesn't exist 
[for Statement "EXPLAIN SELECT * FROM `sakila`.`payment` WHERE 1=1"] at /usr/bin/pt-table-checksum line 6530.d、多从校验
-- 下面演示多个从库时主从一致性校验
-- 缺省情况下
-- 参数:--recursion-method ; type: array; default: processlist,hosts.
--            Preferred recursion method for discovering replicas.
--  pt-table-checksum performs several “REPLICACHECKS” before and while running.master@localhost[(none)]> show slave hosts;
+-----------+------+------+-----------+--------------------------------------+
| Server_id | Host | Port | Master_id | Slave_UUID                          |
+-----------+------+------+-----------+--------------------------------------+
|        11 |      | 3307 |      1002 | 69fc46b6-3c06-11e5-94f0-000c29a05f26 |
|        1 |      | 3306 |      1002 | f2824060-e2cb-11e4-8f18-000c2926f457 |
+-----------+------+------+-----------+--------------------------------------+root@localhost[(none)]> show variables like 'port';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| port          | 3307  |
+---------------+-------+root@localhost[(none)]> delete from test.t where id=1;
Query OK, 1 row affected (0.00 sec)[root@vdbsrv3 ~]# pt-table-checksum h='192.168.1.8',u='checksums',p='xxx',P=3306 -dtest \
> -tt --nocheck-replication-filters --no-check-binlog-format --replicate=test.checksum \
> --recursion-method=hosts# A software update is available:
#  * The current version for Percona::Toolkit is 2.2.14.            TS ERRORS  DIFFS    ROWS  CHUNKS SKIPPED    TIME TABLE
08-06T16:12:52      0      1        3      1      0  0.034 test.t

四、参数描述

–nocheck-replication-filters
  不检查复制过滤器,建议启用。后面可以用–databases来指定需要检查的数据库。
–no-check-binlog-format
  不检查复制的binlog模式,要是binlog模式是ROW,则会报错。
–replicate-check-only
  只显示不同步的信息。
–replicate=
  把checksum的信息写入到指定表中,建议直接写到被检查的数据库当中。
–databases=
  指定需要被检查的数据库,多个则用逗号隔开。
–tables=
  指定需要被检查的表,多个用逗号隔开
  h=127.0.0.1 :Master的地址
  u=root :用户名
  p=123456 :密码
  P=3306 :端口

五、常见问题

[root@vdbsrv3 ~]# pt-table-checksum h='192.168.1.8',u='checksums',p='xxx',P=3306 -d mysql \
> --nocheck-replication-filters --replicate=test.checksums
Replica vdbsrv4 has binlog_format MIXED which could cause pt-table-checksum to break replication. 
Please read "Replicas using row-based replication" in the LIMITATIONS section of the tool's documentation.If you understand the risks, specify --no-check-binlog-format to disable this check.
上面描述的是关于使用mixed日志格式时的问题  [root@vdbsrv3 ~]# pt-table-checksum h='192.168.1.8',u='checksums',p='xxx',P=3306 -d mysql \
> --nocheck-replication-filters --no-check-binlog-format
DBD::mysql::db do failed: Access denied for user 'checksums'@'192.168.1.%' to database 'percona' 
[for Statement "CREATE DATABASE IF NOT EXISTS `percona` /* pt-table-checksum */"] 
at /usr/bin/pt-table-checksum line 10743.
07-29T08:42:03 --replicate database percona does not exist and it cannot be created automatically. 
You need to create the database.[root@vdbsrv3 ~]# pt-table-checksum h='192.168.1.8',u='checksums',p='xxx',P=3306 -dtest -tt \
> --nocheck-replication-filters --no-check-binlog-format  --replicate=test.checksum
Cannot connect to P=3306,h=vdbsrv4,p=...,u=checksums
Diffs cannot be detected because no slaves were found.  
Please read the --recursion-method documentation for information.TS ERRORS  DIFFS    ROWS  CHUNKS SKIPPED    TIME TABLE
08-06T10:03:10      0      0        3      1      0  0.023 test.t[root@vdbsrv3 ~]# pt-table-checksum h='192.168.1.8',u='checksums',p='xxx',P=3306 -dtest -tt \
> --nocheck-replication-filters --no-check-binlog-format \
> --replicate=test.checksum --recursion-method=hosts
Cannot connect to P=3306,h=,p=...,u=checksums
Cannot connect to P=3307,h=,p=...,u=checksums
Diffs cannot be detected because no slaves were found.  
Please read the --recursion-method documentation for information.TS ERRORS  DIFFS    ROWS  CHUNKS SKIPPED    TIME TABLE
08-06T16:02:27      0      0        3      1      0  0.016 test.tmaster@localhost[(none)]> show slave hosts;
+-----------+------+------+-----------+--------------------------------------+
| Server_id | Host | Port | Master_id | Slave_UUID                          |
+-----------+------+------+-----------+--------------------------------------+
|        1 |      | 3306 |      1002 | f2824060-e2cb-11e4-8f18-000c2926f457 |
|        11 |      | 3307 |      1002 | 69fc46b6-3c06-11e5-94f0-000c29a05f26 |
+-----------+------+------+-----------+--------------------------------------+-- 增加参数report_host后重启从库
[root@vdbsrv4 ~]# grep report_host /etc/my.cnf
report_host='192.168.1.12'master@localhost[(none)]> show slave hosts;
+-----------+--------------+------+-----------+--------------------------------------+
| Server_id | Host        | Port | Master_id | Slave_UUID                          |
+-----------+--------------+------+-----------+--------------------------------------+
|        11 | 192.168.1.12 | 3307 |      1002 | 69fc46b6-3c06-11e5-94f0-000c29a05f26 |
|        1 | 192.168.1.12 | 3306 |      1002 | f2824060-e2cb-11e4-8f18-000c2926f457 |
+-----------+--------------+------+-----------+--------------------------------------+

DBA牛鹏社(SQL/NOSQL/LINUX)

这里写图片描述

这篇关于使用pt-table-checksum校验MySQL主从复制的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!


原文地址:
本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.chinasem.cn/article/182931

相关文章

C++中assign函数的使用

《C++中assign函数的使用》在C++标准模板库中,std::list等容器都提供了assign成员函数,它比操作符更灵活,支持多种初始化方式,下面就来介绍一下assign的用法,具有一定的参考价... 目录​1.assign的基本功能​​语法​2. 具体用法示例​​​(1) 填充n个相同值​​(2)

MySql基本查询之表的增删查改+聚合函数案例详解

《MySql基本查询之表的增删查改+聚合函数案例详解》本文详解SQL的CURD操作INSERT用于数据插入(单行/多行及冲突处理),SELECT实现数据检索(列选择、条件过滤、排序分页),UPDATE... 目录一、Create1.1 单行数据 + 全列插入1.2 多行数据 + 指定列插入1.3 插入否则更

MySQL深分页进行性能优化的常见方法

《MySQL深分页进行性能优化的常见方法》在Web应用中,分页查询是数据库操作中的常见需求,然而,在面对大型数据集时,深分页(deeppagination)却成为了性能优化的一个挑战,在本文中,我们将... 目录引言:深分页,真的只是“翻页慢”那么简单吗?一、背景介绍二、深分页的性能问题三、业务场景分析四、

MySQL 迁移至 Doris 最佳实践方案(最新整理)

《MySQL迁移至Doris最佳实践方案(最新整理)》本文将深入剖析三种经过实践验证的MySQL迁移至Doris的最佳方案,涵盖全量迁移、增量同步、混合迁移以及基于CDC(ChangeData... 目录一、China编程JDBC Catalog 联邦查询方案(适合跨库实时查询)1. 方案概述2. 环境要求3.

Spring StateMachine实现状态机使用示例详解

《SpringStateMachine实现状态机使用示例详解》本文介绍SpringStateMachine实现状态机的步骤,包括依赖导入、枚举定义、状态转移规则配置、上下文管理及服务调用示例,重点解... 目录什么是状态机使用示例什么是状态机状态机是计算机科学中的​​核心建模工具​​,用于描述对象在其生命

SQL server数据库如何下载和安装

《SQLserver数据库如何下载和安装》本文指导如何下载安装SQLServer2022评估版及SSMS工具,涵盖安装配置、连接字符串设置、C#连接数据库方法和安全注意事项,如混合验证、参数化查... 目录第一步:打开官网下载对应文件第二步:程序安装配置第三部:安装工具SQL Server Manageme

C#连接SQL server数据库命令的基本步骤

《C#连接SQLserver数据库命令的基本步骤》文章讲解了连接SQLServer数据库的步骤,包括引入命名空间、构建连接字符串、使用SqlConnection和SqlCommand执行SQL操作,... 目录建议配合使用:如何下载和安装SQL server数据库-CSDN博客1. 引入必要的命名空间2.

使用Python删除Excel中的行列和单元格示例详解

《使用Python删除Excel中的行列和单元格示例详解》在处理Excel数据时,删除不需要的行、列或单元格是一项常见且必要的操作,本文将使用Python脚本实现对Excel表格的高效自动化处理,感兴... 目录开发环境准备使用 python 删除 Excphpel 表格中的行删除特定行删除空白行删除含指定

全面掌握 SQL 中的 DATEDIFF函数及用法最佳实践

《全面掌握SQL中的DATEDIFF函数及用法最佳实践》本文解析DATEDIFF在不同数据库中的差异,强调其边界计算原理,探讨应用场景及陷阱,推荐根据需求选择TIMESTAMPDIFF或inte... 目录1. 核心概念:DATEDIFF 究竟在计算什么?2. 主流数据库中的 DATEDIFF 实现2.1

MySQL 多列 IN 查询之语法、性能与实战技巧(最新整理)

《MySQL多列IN查询之语法、性能与实战技巧(最新整理)》本文详解MySQL多列IN查询,对比传统OR写法,强调其简洁高效,适合批量匹配复合键,通过联合索引、分批次优化提升性能,兼容多种数据库... 目录一、基础语法:多列 IN 的两种写法1. 直接值列表2. 子查询二、对比传统 OR 的写法三、性能分析