前言：
今天遇到主从表不一致的情况，很奇怪为什么会出现不一致的情况，因为复制状态一直都是正常的。最后检查出现不一致的数据都是主键，原来是当时初始化数据的时候导致的。现在分析记录下这个问题，避免以后再遇到这个"坑"。
背景：
主从服务器，MIXED复制模式。
分析：
表：SPU
Table: SPUCreate Table: CREATE TABLE `SPU` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `trademark` varchar(255) NOT NULL COMMENT '品牌', `item_code` varchar(255) NOT NULL COMMENT '货号', `product_id` int(10) DEFAULT '0', PRIMARY KEY (`id`), KEY `trademark` (`trademark`), KEY `item_code` (`item_code`)) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='SPU'
当时的初始化操作的SQL：
INSERT INTO SPU(trademark, item_code) SELECT * FROM (SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2FROM product_property a LEFT JOIN product_property b ON a.product_id = b.product_id WHERE a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> '' AND b.value IS NOT NULL AND b.value <> '' ) as aa
上面的SQL执行完之后，主从表的COUNT数量一样，但SPU表有个自增主键，就在这里出现主从插入SPU的顺序不一样，即主从SELECT出来的结果顺序不一样。由于上面的结果集很大，所以就取前10条记录看看：
主：
现在表中数据的顺序：zjy@192.168.10.23 : tt 04:38:48>select id,trademark col1,item_code col2 from SPU limit 10;+----+-----------+-------------+| id | col1 | col2 |+----+-----------+-------------+| 1 | 鼓浪屿 | gd rg || 2 | 山松 | 10020122 || 3 | coulter | 无 || 4 | oricell | mubmd-01101 || 5 | oricell | huxma-01101 || 6 | oricell | huxmf-01001 || 7 | oricell | rawmx-01001 || 8 | oricell | rasmx-01001 || 9 | oricell | rafmx-01101 || 10 | oricell | rbxmx-01001 |+----+-----------+-------------+10 rows in set (0.01 sec)当时初始化的sql读出数据的顺序：zjy@192.168.10.23 : tt 04:40:25>SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2 -> FROM -> product_property a LEFT JOIN product_property b -> ON -> a.product_id = b.product_id -> WHERE -> a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> '' -> AND -> b.value IS NOT NULL AND b.value <> '' limit 10;+-----------+-------------+| col1 | col2 |+-----------+-------------+| 鼓浪屿 | gd rg || 山松 | 10020122 || coulter | 无 || oricell | mubmd-01101 || oricell | huxma-01101 || oricell | huxmf-01001 || oricell | rawmx-01001 || oricell | rasmx-01001 || oricell | rafmx-01101 || oricell | rbxmx-01001 |+-----------+-------------+10 rows in set (0.01 sec)
上面结果col1,col2的顺序一模一样。
从：
现在表中数据的顺序：zjy@192.168.10.8 : tt 04:38:03>select id,trademark col1,item_code col2 from SPU limit 10;+----+--------------------------------+-----------------+| id | col1 | col2 |+----+--------------------------------+-----------------+| 1 | bio-rad | 125-0140 || 2 | oricell(tm) | huxma-90011 || 3 | oricell(tm) | huxmf-90011 || 4 | oricell(tm) | rawmx-90011 || 5 | oricell | rasmx-90011 || 6 | oricell | rafmx-90011 || 7 | oricell | rbxmx-90011 || 8 | 上海科技有限公司 | bsm03011 || 9 | oricell | caxmx-90011 || 10 | oricell(tm) | tedta-10001-100 |+----+--------------------------------+-----------------+10 rows in set (0.00 sec)当时初始化的sql读出数据的顺序：zjy@192.168.10.8 : tt 04:38:58>SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2 -> FROM -> product_property a LEFT JOIN product_property b -> ON -> a.product_id = b.product_id -> WHERE -> a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> '' -> AND -> b.value IS NOT NULL AND b.value <> '' limit 10;+--------------------------------+-----------------+| col1 | col2 |+--------------------------------+-----------------+| bio-rad | 125-0140 || oricell(tm) | huxma-90011 || oricell(tm) | huxmf-90011 || oricell(tm) | rawmx-90011 || oricell | rasmx-90011 || oricell | rafmx-90011 || oricell | rbxmx-90011 || 上海科技有限公司 | bsm03011 || oricell | caxmx-90011 || oricell(tm) | tedta-10001-100 |+--------------------------------+-----------------+10 rows in set (0.01 sec)
上面结果col1,col2的顺序一模一样。
到此为止，大家就清楚为什么SPU表的数据不一致了，准确来说是主键对应的数据不一致。要是自增主键只是提升INNODB的性能，没有业务上的意义，那么对于产品来说是没有影响的，可以忽略这个问题。否则，就需要好好的处理这个问题了。从另一个方面来说，也是因为复制的模式是STATEMENT引发这个问题的，因为同一个QUERY在2个地方执行出的结果不一样；要是ROW的复制模式，主会把所有字段的记录全部传送给从，就不会出现这个问题。
进一步分析： 为什么一样的SQL在主从上跑出来的数据顺序不一样呢？
通过EXPLAIN 看到主上的QUERY 先读 b表，再读a表；而从上的则是先扫描a表，再读b表。出现这样的情况，就是数据在块里面分布不一致，导致索引利用的方式也不一样，最终影响优化器的选择。因为INNODB是索引组织表的，一旦走的索引不一样，就会导致数据以不同的顺序被扫描出来。上面的这些结果刚好被验证。SQL执行计划如下：
主：先b再a
zjy@192.168.10.23 : tt 09:18:04>explain SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2 FROM product_property a LEFT JOIN product_property b ON a.product_id = b.product_id WHERE a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> '' AND b.value IS NOT NULL AND b.value <> '';+----+-------------+-------+------+------------------------+-------------+---------+----------------------+---------+------------------------------+| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |+----+-------------+-------+------+------------------------+-------------+---------+----------------------+---------+------------------------------+| 1 | SIMPLE | b | ref | property_id,product_id | property_id | 4 | const | 4145932 | Using where; Using temporary || 1 | SIMPLE | a | ref | property_id,product_id | product_id | 4 | tt.b.product_id | 1 | Using where |+----+-------------+-------+------+------------------------+-------------+---------+----------------------+---------+------------------------------+2 rows in set (0.01 sec)
从：先a再b
zjy@192.168.10.8 : tt 09:18:13>explain SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2 FROM product_property a LEFT JOIN product_property b ON a.product_id = b.product_id WHERE a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> '' AND b.value IS NOT NULL AND b.value <> '';+----+-------------+-------+------+------------------------+-------------+---------+----------------------+----------+------------------------------+| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |+----+-------------+-------+------+------------------------+-------------+---------+----------------------+----------+------------------------------+| 1 | SIMPLE | a | ref | property_id,product_id | property_id | 4 | const | 17079464 | Using where; Using temporary || 1 | SIMPLE | b | ref | property_id,product_id | product_id | 4 | tt.a.product_id | 5 | Using where |+----+-------------+-------+------+------------------------+-------------+---------+----------------------+----------+------------------------------+2 rows in set (0.13 sec)
主从对比发现，他们的执行计划和各表走的索引都不一样，导致最后出来的顺序也不一样的（结果集是一样的），这就验证了分析说的情况。那要是执行计划和索引一致呢？接下来继续验证下：
进一步验证：
因为INNODB是索引组织表的，索引就是数据，要是主从的执行计划一样，则他们的结果会是？
主的执行计划：zjy@192.168.10.23 : tt 05:42:15>explain SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2 -> FROM -> product_property a LEFT JOIN product_property b -> ON -> a.product_id = b.product_id -> WHERE -> a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> '' -> AND -> b.value IS NOT NULL AND b.value <> '';+----+-------------+-------+------+------------------------+-------------+---------+----------------------+---------+------------------------------+| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |+----+-------------+-------+------+------------------------+-------------+---------+----------------------+---------+------------------------------+| 1 | SIMPLE | b | ref | property_id,product_id | property_id | 4 | const | 3771874 | Using where; Using temporary || 1 | SIMPLE | a | ref | property_id,product_id | product_id | 4 | tt.b.product_id | 1 | Using where |+----+-------------+-------+------+------------------------+-------------+---------+----------------------+---------+------------------------------+2 rows in set (0.01 sec)从的执行计划：zjy@192.168.10.8 : tt 05:42:07>explain SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2 -> FROM -> product_property a LEFT JOIN product_property b -> ON -> a.product_id = b.product_id -> WHERE -> a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> '' -> AND -> b.value IS NOT NULL AND b.value <> '';+----+-------------+-------+------+------------------------+-------------+---------+----------------------+----------+------------------------------+| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |+----+-------------+-------+------+------------------------+-------------+---------+----------------------+----------+------------------------------+| 1 | SIMPLE | b | ref | property_id,product_id | property_id | 4 | const | 17375012 | Using where; Using temporary || 1 | SIMPLE | a | ref | property_id,product_id | product_id | 4 | tt.b.product_id | 5 | Using where |+----+-------------+-------+------+------------------------+-------------+---------+----------------------+----------+------------------------------+2 rows in set (0.00 sec)
执行计划一样，都是先b表再a表，再重新执行初始化的SQL：
主：
zjy@192.168.10.23 : tt 05:42:26>SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2 -> FROM -> product_property a LEFT JOIN product_property b -> ON -> a.product_id = b.product_id -> WHERE -> a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> '' -> AND -> b.value IS NOT NULL AND b.value <> '' limit 10 -> ;+-----------+-------------+| col1 | col2 |+-----------+-------------+| 鼓浪屿 | gd rg || 山松 | 10020122 || coulter | 无 || oricell | mubmd-01101 || oricell | huxma-01101 || oricell | huxmf-01001 || oricell | rawmx-01001 || oricell | rasmx-01001 || oricell | rafmx-01101 || oricell | rbxmx-01001 |+-----------+-------------+10 rows in set (0.00 sec)
从：
zjy@192.168.10.8 : tt 05:42:32>SELECT distinct lower(TRIM(a.`value`)) col1, lower(TRIM(b.`value`)) col2 -> FROM -> product_property a LEFT JOIN product_property b -> ON -> a.product_id = b.product_id -> WHERE -> a.property_id = 14 AND b.property_id = 9 AND a.value IS NOT NULL AND a.value <> '' -> AND -> b.value IS NOT NULL AND b.value <> '' limit 10 -> ;+-----------+-------------+| col1 | col2 |+-----------+-------------+| 鼓浪屿 | gd rg || 山松 | 10020122 || coulter | 无 || oricell | mubmd-01101 || oricell | huxma-01101 || oricell | huxmf-01001 || oricell | rawmx-01001 || oricell | rasmx-01001 || oricell | rafmx-01101 || oricell | rbxmx-01001 |+-----------+-------------+10 rows in set (0.01 sec)
好了，要是执行计划一样，结果是：SQL在主从上跑出来的结果一致了。
PS：另一个方法就是用ROW模式，有兴趣的可以测试下。
总结：
主从复制在STATEMENT下面确实被忽略了一些问题，可以用ROW模式代替，但也要知道ROW模式有哪些问题，可以参考。也要清楚数据在磁盘块里面分布不一致，影响优化器的选择而导致索引利用的方式也不一样，最终也影响到数据的顺序。
总之，在主从上执行一些比较大的数据量的操作（批量、初始化）的时候，尽可能的先去主从上查看他们的执行计划是否一样，走的索引是否一致，确保查询出来的结果一样。另：这篇文章：(MySQL "replace into" 的坑) 也在一定程度上说明别用自增主键当成有意义的数据。