Bug 216790

Summary: dm: broken btree accessing if commit failed
Product: Drivers Reporter: Zhihao Cheng (chengzhihao1)
Component: OtherAssignee: drivers_other
Status: NEW ---    
Severity: normal    
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 6.1.0-rc5 Subsystem:
Regression: No Bisected commit-id:
Attachments: main_diff
test.sh

Description Zhihao Cheng 2022-12-08 13:11:55 UTC 
qemu config:
sda 10M
sdb 1G

1. Apply main_diff
2. ./test.sh

[   99.738936] [NEW TRANS] pwd root 2536
[  101.504095] update root 17
[  101.556202] [NEW TRANS] pwd root 17    // last transaction's pmd->root, corrusponding btree is on disk
[  101.557501] update root 23             // current transaction's pmd->root, corrusponding btree is in mem (not on disk)
[  101.566865] ALLOC3 new block 31
[  101.575285] ALLOC3 new block 34
[  101.579088] ALLOC3 new block 35
[  101.579918] ALLOC3 new block 38
[  101.638409] current
[  101.638726] flags 2 blocknr 31 nr_entries 167
[  101.639168] 0 key 9977 value 1236
[  101.639519] 1 key 9978 value 1237
[  101.639879] 2 key 9979 value 1238
[  101.640222] 3 key 9980 value 1239
[  101.640572] 4 key 9981 value 1240
[  101.640936] 5 key 9982 value 1241
[  101.641271] 6 key 9983 value 1242
[  101.641651] 7 key 9984 value 1243
[  101.641987] 8 key 9985 value 1244
[  101.642318] 9 key 9986 value 1245
[  101.642689] 10 key 9987 value 1246
[  101.643031] 11 key 9988 value 1247
[  101.643372] 12 key 9989 value 1248
[  101.643752] 13 key 9990 value 1249
[  101.644097] 14 key 9991 value 1250
[  101.644457] 15 key 9992 value 1251
[  101.644869] 16 key 9993 value 1252
[  101.645253] 17 key 9994 value 1253
[  101.645684] 18 key 9995 value 1254
[  101.646071] 19 key 9996 value 1255
[  101.646418] 20 key 9997 value 1256
[  101.646869] 21 key 9998 value 1257
[  101.647183] 22 key 9999 value 1258
[  101.647520] 23 key 10000 value 1259
[  101.647876] 24 key 10001 value 1260
[  101.648199] 25 key 10002 value 1261
[  101.648539] 26 key 10003 value 1262
[  101.648895] 27 key 10004 value 1263
[  101.649217] 28 key 10005 value 1264
[  101.649564] 29 key 10006 value 1265
[  101.649919] 30 key 10007 value 1266
[  101.650242] 31 key 10008 value 1267
[  101.650579] 32 key 10009 value 1268
[  101.650934] 33 key 10010 value 1269
[  101.651259] 34 key 10011 value 1270
[  101.651633] 35 key 10012 value 1271
[  101.652020] 36 key 10013 value 1272
[  101.652423] 37 key 10014 value 1273
[  101.652863] 38 key 10015 value 1274
[  101.653295] 39 key 10016 value 1275
[  101.653771] 40 key 10017 value 1276
[  101.654203] 41 key 10018 value 1277
[  101.654662] 42 key 10019 value 1278
[  101.655098] 43 key 10020 value 1279
[  101.655532] 44 key 10021 value 1280
[  101.655990] 45 key 10022 value 1281
[  101.656432] 46 key 10023 value 1282
[  101.656894] 47 key 10024 value 1283
[  101.657331] 48 key 10025 value 1284
[  101.657798] 49 key 10026 value 1285
[  101.658236] 50 key 10027 value 1286
[  101.658697] 51 key 10028 value 1287
[  101.659137] 52 key 10029 value 1288
[  101.659575] 53 key 10030 value 1289
[  101.660035] 54 key 10031 value 1290
[  101.660468] 55 key 10032 value 1291
[  101.660928] 56 key 10033 value 1292
[  101.661362] 57 key 10034 value 1293
[  101.661823] 58 key 10035 value 1294
[  101.662257] 59 key 10036 value 1295
[  101.662726] 60 key 10037 value 1296
[  101.663161] 61 key 10038 value 1297
[  101.663596] 62 key 10039 value 1298
[  101.664049] 63 key 10040 value 1299
[  101.664490] 64 key 10041 value 1300
[  101.664951] 65 key 10042 value 1301
[  101.665387] 66 key 10043 value 1302
[  101.665854] 67 key 10044 value 1303
[  101.666288] 68 key 10045 value 1304
[  101.666644] 69 key 10046 value 1305
[  101.666964] 70 key 10047 value 1306
[  101.667283] 71 key 10048 value 1307
[  101.667605] 72 key 10049 value 1308
[  101.667937] 73 key 10050 value 1309
[  101.668259] 74 key 10051 value 1310
[  101.668671] 75 key 10052 value 1311
[  101.669072] 76 key 10053 value 1312
[  101.669419] 77 key 10054 value 1313
[  101.669778] 78 key 10055 value 1314
[  101.670111] 79 key 10056 value 1315
[  101.670468] 80 key 10057 value 1316
[  101.670836] 81 key 10058 value 1317
[  101.671214] 82 key 10059 value 1318
[  101.671558] 83 key 10060 value 1319
[  101.671940] 84 key 10061 value 1320
[  101.672294] 85 key 10062 value 1321
[  101.672678] 86 key 10063 value 1322
[  101.673050] 87 key 10064 value 1323
[  101.673423] 88 key 10065 value 1324
[  101.673807] 89 key 10066 value 1325
[  101.674142] 90 key 10067 value 1326
[  101.674512] 91 key 10068 value 1327
[  101.674900] 92 key 10069 value 1328
[  101.675276] 93 key 10070 value 1329
[  101.675626] 94 key 10071 value 1330
[  101.676046] 95 key 10072 value 1331
[  101.676371] 96 key 10073 value 1332
[  101.676744] 97 key 10074 value 1333
[  101.677089] 98 key 10075 value 1334
[  101.677458] 99 key 10076 value 1335
[  101.677843] 100 key 10077 value 1336
[  101.678173] 101 key 10078 value 1337
[  101.678514] 102 key 10079 value 1338
[  101.678880] 103 key 10080 value 1339
[  101.679225] 104 key 10081 value 1340
[  101.679622] 105 key 10082 value 1341
[  101.679983] 106 key 10083 value 1342
[  101.680311] 107 key 10084 value 1343
[  101.680666] 108 key 10085 value 1344
[  101.681013] 109 key 10086 value 1345
[  101.681337] 110 key 10087 value 1346
[  101.681741] 111 key 10088 value 1347
[  101.682123] 112 key 10089 value 1348
[  101.682474] 113 key 10090 value 1349
[  101.682953] 114 key 10091 value 1350
[  101.683387] 115 key 10092 value 1351
[  101.683825] 116 key 10093 value 1352
[  101.684277] 117 key 10094 value 1353
[  101.684746] 118 key 10095 value 1354
[  101.685204] 119 key 10096 value 1355
[  101.685678] 120 key 10097 value 1356
[  101.686124] 121 key 10098 value 1357
[  101.686559] 122 key 10099 value 1358
[  101.687037] 123 key 10100 value 1359
[  101.687481] 124 key 10101 value 1360
[  101.687953] 125 key 10102 value 1361
[  101.688405] 126 key 10103 value 1362
[  101.688866] 127 key 10104 value 1363
[  101.689316] 128 key 10105 value 1364
[  101.689784] 129 key 10106 value 1365
[  101.690229] 130 key 10107 value 1366
[  101.690701] 131 key 10108 value 1367
[  101.691151] 132 key 10109 value 1368
[  101.691605] 133 key 10110 value 1369
[  101.692066] 134 key 10111 value 1370
[  101.692501] 135 key 10112 value 1371
[  101.692970] 136 key 10113 value 1372
[  101.693427] 137 key 10114 value 1373
[  101.693905] 138 key 10115 value 1374
[  101.694346] 139 key 10116 value 1375
[  101.694817] 140 key 10117 value 1376
[  101.695259] 141 key 10118 value 1377
[  101.695725] 142 key 10119 value 1378
[  101.696150] 143 key 10120 value 1379
[  101.696557] 144 key 10121 value 1380
[  101.697016] 145 key 10122 value 1381
[  101.697467] 146 key 10123 value 1382
[  101.697943] 147 key 10124 value 1383
[  101.698401] 148 key 10125 value 1384
[  101.698872] 149 key 10126 value 1385
[  101.699321] 150 key 10127 value 1386
[  101.699796] 151 key 10128 value 1387
[  101.700238] 152 key 10129 value 1388
[  101.700705] 153 key 10130 value 1389
[  101.701150] 154 key 10131 value 1390
[  101.701624] 155 key 10132 value 1391
[  101.702071] 156 key 10133 value 1392
[  101.702518] 157 key 10134 value 1393
[  101.702943] 158 key 10135 value 1394
[  101.703386] 159 key 10136 value 1395
[  101.703858] 160 key 10137 value 1396
[  101.704298] 161 key 10138 value 1397
[  101.704780] 162 key 10139 value 1398
[  101.705229] 163 key 10140 value 1399
[  101.705701] 164 key 10141 value 1400
[  101.706148] 165 key 10142 value 1401
[  101.706629] 166 key 10143 value 1402
[  101.707116] found block 31
[  101.707469] fail io
[  101.732366] device-mapper: thin: 252:0: metadata operation 'dm_pool_commit_metadata' failed: error = -5
[  101.734193] device-mapper: thin: 252:0: aborting current metadata transaction
[  101.740362] EXT4-fs warning (device dm-1): ext4_end_bio:343: I/O error 10 writing to inode 12 starting block 100336)
[  101.742133] EXT4-fs warning (device dm-1): ext4_end_bio:343: I/O error 10 writing to inode 12 starting block 231408)
[  101.742249] Buffer I/O error on device dm-1, logical block 99813
[  101.743819] __find_block: root 23
[  101.744543] Buffer I/O error on device dm-1, logical block 99814
[  101.745650] Buffer I/O error on device dm-1, logical block 99815
[  101.746461] Buffer I/O error on device dm-1, logical block 99816
[  101.747223] Buffer I/O error on device dm-1, logical block 99817
[  101.747968] Buffer I/O error on device dm-1, logical block 99818
[  101.748707] Buffer I/O error on device dm-1, logical block 99819
[  101.749383] Buffer I/O error on device dm-1, logical block 99820
[  101.750082] Buffer I/O error on device dm-1, logical block 99821
[  101.750808] Buffer I/O error on device dm-1, logical block 99822
[  101.939791] 
[  101.939791] 
[  101.939791] dump origin(pmd root 23):
[  101.939850] 
[  101.939850] 
[  101.939850] dump origin(pmd root 23):           // root 23 points to a broken btree
[  101.940497] flags 2 blocknr 31 nr_entries 154  // stale node in block 31
[  101.940501] 0 key 8553 value 6274678784
[  101.940503] 1 key 8554 value 6291456000
[  101.940504] 2 key 8555 value 6308233216
[  101.940506] 3 key 8556 value 6325010432
[  101.940507] 4 key 8557 value 6341787648
[  101.940508] 5 key 8558 value 6358564864
[  101.940510] 6 key 8559 value 6375342080
[  101.940511] 7 key 8560 value 6392119296
[  101.940513] 8 key 8561 value 6408896512
[  101.940515] 9 key 8562 value 6425673728
[  101.940517] 10 key 8563 value 6442450944
[  101.940519] 11 key 8564 value 6459228160
[  101.940521] 12 key 8565 value 6476005376
[  101.940523] 13 key 8566 value 6492782592
[  101.940525] 14 key 8567 value 6509559808
[  101.940526] 15 key 8568 value 6526337024
[  101.940528] 16 key 8569 value 6543114240
[  101.940530] 17 key 8570 value 6559891456
[  101.940532] 18 key 8571 value 6576668672
[  101.940533] 19 key 8572 value 6593445888
[  101.940534] 20 key 8573 value 6610223104
[  101.940536] 21 key 8574 value 6627000320
[  101.940538] 22 key 8575 value 6643777536
[  101.940540] 23 key 8576 value 6660554752
[  101.940541] 24 key 8577 value 6677331968
[  101.940543] 25 key 8578 value 6694109184
[  101.940544] 26 key 8579 value 6710886400
[  101.940546] 27 key 8580 value 6727663616
[  101.940547] 28 key 8581 value 6744440832
[  101.940549] 29 key 8582 value 6761218048
[  101.940551] 30 key 8583 value 6777995264
[  101.940552] 31 key 8584 value 6794772480
[  101.940554] 32 key 8585 value 6811549696
[  101.940555] 33 key 8586 value 6828326912
[  101.940557] 34 key 8587 value 6845104128
[  101.940559] 35 key 8588 value 6861881344
[  101.940560] 36 key 8589 value 6878658560
[  101.940561] 37 key 8590 value 6895435776
[  101.940564] 38 key 8591 value 6912212992
[  101.940566] 39 key 8592 value 6928990208
[  101.940567] 40 key 8593 value 6945767424
[  101.940569] 41 key 8594 value 6962544640
[  101.940570] 42 key 8595 value 6979321856
[  101.940572] 43 key 8596 value 6996099072
[  101.940574] 44 key 8597 value 7012876288
[  101.940576] 45 key 8598 value 7029653504
[  101.940578] 46 key 8599 value 7046430720
[  101.940580] 47 key 8600 value 7063207936
[  101.940590] 48 key 8601 value 7079985152
[  101.940592] 49 key 8602 value 7096762368
[  101.940594] 50 key 8603 value 7113539584
[  101.940595] 51 key 8604 value 7130316800
[  101.940597] 52 key 8605 value 7147094016
[  101.940598] 53 key 8606 value 7163871232
[  101.940599] 54 key 8607 value 7180648448
[  101.940601] 55 key 8608 value 7197425664
[  101.940602] 56 key 8609 value 7214202880
[  101.940604] 57 key 8610 value 7230980096
[  101.940605] 58 key 8611 value 7247757312
[  101.940607] 59 key 8612 value 7264534528
[  101.940630] 60 key 8613 value 7281311744
[  101.940632] 61 key 8614 value 7298088960
[  101.940634] 62 key 8615 value 7314866176
[  101.940635] 63 key 8616 value 7331643392
[  101.940637] 64 key 8617 value 7348420608
[  101.940639] 65 key 8618 value 7365197824
[  101.940641] 66 key 8619 value 7381975040
[  101.940643] 67 key 8620 value 7398752256
[  101.940645] 68 key 8621 value 7415529472
[  101.940647] 69 key 8622 value 7432306688
[  101.940650] 70 key 8623 value 7449083904
[  101.940652] 71 key 8624 value 7465861120
[  101.940654] 72 key 8625 value 7482638336
[  101.940655] 73 key 8626 value 7499415552
[  101.940658] 74 key 8627 value 7516192768
[  101.940660] 75 key 8628 value 7532969984
[  101.940662] 76 key 8629 value 7549747200
[  101.940663] 77 key 8630 value 7566524416
[  101.940665] 78 key 8631 value 7583301632
[  101.940667] 79 key 8632 value 7600078848
[  101.940669] 80 key 8633 value 7616856064
[  101.940671] 81 key 8634 value 7633633280
[  101.940673] 82 key 8635 value 7650410496
[  101.940675] 83 key 8636 value 7667187712
[  101.940676] 84 key 8637 value 7683964928
[  101.940678] 85 key 8638 value 7700742144
[  101.940680] 86 key 8639 value 7717519360
[  101.940681] 87 key 8640 value 7734296576
[  101.940683] 88 key 8641 value 7751073792
[  101.941430] flags 2 blocknr 31 nr_entries 154
[  101.941852] 89 key 8642 value 7767851008
[  101.941855] 90 key 8643 value 7784628224
[  101.941856] 91 key 8644 value 7801405440
[  101.941857] 92 key 8645 value 7818182656
[  101.942300] 0 key 8553 value 6274678784
[  101.942693] 93 key 8646 value 7834959872
[  101.943129] 1 key 8554 value 6291456000
[  101.943483] 94 key 8647 value 7851737088
[  101.943943] 2 key 8555 value 6308233216
[  101.944299] 95 key 8648 value 7868514304
[  101.944757] 3 key 8556 value 6325010432
[  101.945188] 96 key 8649 value 7885291520
[  101.945647] 4 key 8557 value 6341787648
[  101.946059] 97 key 8650 value 7902068736
[  101.946499] 5 key 8558 value 6358564864
[  101.946893] 98 key 8651 value 7918845952
[  101.947399] 6 key 8559 value 6375342080
[  101.947874] 99 key 8652 value 7935623168
[  101.948320] 7 key 8560 value 6392119296
[  101.948731] 100 key 8653 value 7952400384
[  101.948734] 101 key 8654 value 7969177600
[  101.948736] 102 key 8655 value 7985954816
[  101.949185] 8 key 8561 value 6408896512
[  101.949570] 103 key 8656 value 8002732032
[  101.950044] 9 key 8562 value 6425673728
[  101.950430] 104 key 8657 value 8019509248
[  101.950900] 10 key 8563 value 6442450944
[  101.951321] 105 key 8658 value 8036286464
[  101.951797] 11 key 8564 value 6459228160
[  101.952282] 106 key 8659 value 8053063680
[  101.952762] 12 key 8565 value 6476005376
[  101.953248] 107 key 8660 value 8069840896
[  101.953730] 13 key 8566 value 6492782592
[  101.954118] 108 key 8661 value 8086618112
[  101.954575] 14 key 8567 value 6509559808
[  101.955100] 109 key 8662 value 8103395328
[  101.955506] 15 key 8568 value 6526337024
[  101.956023] 110 key 8663 value 8120172544
[  101.956466] 16 key 8569 value 6543114240
[  101.956876] 111 key 8664 value 8136949760
[  101.957340] 17 key 8570 value 6559891456
[  101.957839] 112 key 8665 value 8153726976
[  101.957843] 113 key 8666 value 8170504192
[  101.958307] 18 key 8571 value 6576668672
[  101.958806] 114 key 8667 value 8187281408
[  101.959269] 19 key 8572 value 6593445888
[  101.959766] 115 key 8668 value 8204058624
[  101.960221] 20 key 8573 value 6610223104
[  101.960793] 116 key 8669 value 8220835840
[  101.961239] 21 key 8574 value 6627000320
[  101.961670] 117 key 8670 value 8237613056
[  101.962114] 22 key 8575 value 6643777536
[  101.962557] 118 key 8671 value 8254390272
[  101.963030] 23 key 8576 value 6660554752
[  101.963508] 119 key 8672 value 8271167488
[  101.963986] 24 key 8577 value 6677331968
[  101.963990] 25 key 8578 value 6694109184
[  101.964375] 120 key 8673 value 8287944704
[  101.964859] 26 key 8579 value 6710886400
[  101.965222] 121 key 8674 value 8304721920
[  101.965726] 27 key 8580 value 6727663616
[  101.966092] 122 key 8675 value 8321499136
[  101.966552] 28 key 8581 value 6744440832
[  101.966943] 123 key 8676 value 8338276352
[  101.966946] 124 key 8677 value 8355053568
[  101.967426] 29 key 8582 value 6761218048
[  101.967816] 125 key 8678 value 8371830784
[  101.968270] 30 key 8583 value 6777995264
[  101.968670] 126 key 8679 value 8388608000
[  101.969136] 31 key 8584 value 6794772480
[  101.969518] 127 key 8680 value 8405385216
[  101.969993] 32 key 8585 value 6811549696
[  101.970510] 128 key 8681 value 8422162432
[  101.971008] 33 key 8586 value 6828326912
[  101.971012] 34 key 8587 value 6845104128
[  101.971385] 129 key 8682 value 8438939648
[  101.971860] 35 key 8588 value 6861881344
[  101.971865] 36 key 8589 value 6878658560
[  101.972265] 130 key 8683 value 8455716864
[  101.972757] 37 key 8590 value 6895435776
[  101.973195] 131 key 8684 value 8472494080
[  101.973680] 38 key 8591 value 6912212992
[  101.974180] 132 key 8685 value 8489271296
[  101.974655] 39 key 8592 value 6928990208
[  101.975179] 133 key 8686 value 8506048512
[  101.975182] 134 key 8687 value 8522825728
[  101.975653] 40 key 8593 value 6945767424
[  101.975657] 41 key 8594 value 6962544640
[  101.975659] 42 key 8595 value 6979321856
[  101.975661] 43 key 8596 value 6996099072
[  101.976195] 135 key 8688 value 8539602944
[  101.976666] 44 key 8597 value 7012876288
[  101.977207] 136 key 8689 value 8556380160
[  101.979525] 45 key 8598 value 7029653504
[  101.980098] 137 key 8690 value 8573157376
[  101.980550] 46 key 8599 value 7046430720
[  101.981106] 138 key 8691 value 8589934592
[  101.981555] 47 key 8600 value 7063207936
[  101.982132] 139 key 8692 value 8606711808
[  101.982599] 48 key 8601 value 7079985152
[  101.983153] 140 key 8693 value 8623489024
[  101.983156] 141 key 8694 value 8640266240
[  101.983653] 49 key 8602 value 7096762368
[  101.983656] 50 key 8603 value 7113539584
[  101.983659] 51 key 8604 value 7130316800
[  101.983660] 52 key 8605 value 7147094016
[  101.983663] 53 key 8606 value 7163871232
[  101.983665] 54 key 8607 value 7180648448
[  101.983667] 55 key 8608 value 7197425664
[  101.984167] 142 key 8695 value 8657043456
[  101.984637] 56 key 8609 value 7214202880
[  101.985162] 143 key 8696 value 8673820672
[  101.985634] 57 key 8610 value 7230980096
[  101.985638] 58 key 8611 value 7247757312
[  101.986125] 144 key 8697 value 8690597888
[  101.986569] 59 key 8612 value 7264534528
[  101.986992] 145 key 8698 value 8707375104
[  101.987417] 60 key 8613 value 7281311744
[  101.987834] 146 key 8699 value 8724152320
[  101.988275] 61 key 8614 value 7298088960
[  101.988744] 147 key 8700 value 8740929536
[  101.989187] 62 key 8615 value 7314866176
[  101.989596] 148 key 8701 value 8757706752
[  101.990065] 63 key 8616 value 7331643392
[  101.990562] 149 key 8702 value 8774483968
[  101.991023] 64 key 8617 value 7348420608
[  101.991398] 150 key 8703 value 8791261184
[  101.991869] 65 key 8618 value 7365197824
[  101.992339] 151 key 10240 value 8808038400
[  101.992812] 66 key 8619 value 7381975040
[  101.993292] 152 key 14336 value 8824815616
[  101.993793] 67 key 8620 value 7398752256
[  101.994204] 153 key 16383 value 8841592832
[  101.994691] 68 key 8621 value 7415529472
[  101.995146] Kernel panic - not syncing: DONE
[  101.995154] CPU: 0 PID: 80 Comm: kworker/u8:7 Not tainted 6.1.0-rc8-00014-g833c547058e8-dirty #1004
[  101.995158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20190727_073836-buildvm-ppc64le-16.ppc.fedoraproject.org-3.fc314
[  101.995164] Workqueue: dm-thin do_worker
[  101.995188] Call Trace:
[  101.995191]  <TASK>
[  101.995198]  dump_stack_lvl+0x73/0x9f
[  101.995204]  dump_stack+0x13/0x1b
[  101.995206]  panic+0x177/0x3e9
[  101.995215]  btree_lookup_raw.constprop.0.cold+0x3b/0x3b
[  101.995223]  dm_btree_lookup+0xc5/0x160
[  101.995227]  __find_block+0x84/0x100
[  101.995232]  dm_thin_find_block+0x58/0xb0
[  101.995239]  __process_bio_read_only+0x60/0x2c0
[  101.995245]  process_cell_read_only+0x19/0x30
[  101.995249]  do_worker+0x2d4/0xd90
[  101.995256]  process_one_work+0x269/0x630
[  101.995264]  worker_thread+0x266/0x630
[  101.995267]  ? rescuer_thread+0x540/0x540
[  101.995271]  kthread+0x151/0x1b0
[  101.995274]  ? kthread_exit+0x50/0x50
[  101.995277]  ret_from_fork+0x1f/0x30
[  101.995284]  </TASK>
[  101.995662] Kernel Offset: disabled
[  102.049270] ---[ end Kernel panic - not syncing: DONE ]---
Comment 1 Zhihao Cheng 2022-12-08 13:12:55 UTC 
Created attachment 303378 [details]
main_diff
Comment 2 Zhihao Cheng 2022-12-08 13:13:21 UTC 
Created attachment 303379 [details]
test.sh