-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathatom.xml
More file actions
1493 lines (1205 loc) · 107 KB
/
atom.xml
File metadata and controls
1493 lines (1205 loc) · 107 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title><![CDATA[Ant-Veil]]></title>
<link href="http://www.casparzhang.com/atom.xml" rel="self"/>
<link href="http://www.casparzhang.com/"/>
<updated>2017-01-10T11:04:38+08:00</updated>
<id>http://www.casparzhang.com/</id>
<author>
<name><![CDATA[Caspar]]></name>
</author>
<generator uri="http://octopress.org/">Octopress</generator>
<entry>
<title type="html"><![CDATA[Analyse Kernel Structures With Crash Utility]]></title>
<link href="http://www.casparzhang.com/blog/2017/01/10/analyse-kernel-structures-with-crash-utility/"/>
<updated>2017-01-10T00:35:59+08:00</updated>
<id>http://www.casparzhang.com/blog/2017/01/10/analyse-kernel-structures-with-crash-utility</id>
<content type="html"><![CDATA[<p>关于 Kdump 和 Crash 的背景知识，不是本文的重点，可以参考下列文章：</p>
<ul>
<li><a href="http://www.ibm.com/developerworks/cn/views/linux/libraryview.jsp?search_by=%E6%B7%B1%E5%85%A5%E6%8E%A2%E7%B4%A2+Kdump">developerworks 社区深入探索 Kdump 系列文档</a></li>
<li><a href="https://people.redhat.com/anderson/crash_whitepaper/">Dave Anderson Crash 官方文档</a></li>
</ul>
<p>这次碰到的问题是，系统当时死锁了，抓了个 vmcore 来研究，现在要看层层包含的某个结构体的一个 <code>rw_semaphore</code> 结构体变量的信息。当时这一层层的基本结构大体是这样的：</p>
<p>有两个链表，分别叫做: <code>sets</code> 和 <code>set_devs</code>; 顾名思义，第一个 <code>sets</code> 是一堆 <code>set</code> 结构体组成的链表；第二个 <code>set_devs</code> 是一堆 <code>set_dev</code> 结构体组成的链表。两个结构体如下：</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>struct set {
</span><span class='line'> struct list_head *list;
</span><span class='line'> blah blah blah;
</span><span class='line'> struct list_head *set_devs;
</span><span class='line'>};
</span><span class='line'>
</span><span class='line'>struct set_dev {
</span><span class='line'> struct list_head *list;
</span><span class='line'> blah blah blah;
</span><span class='line'> struct rw_semaphore rw_sem;
</span><span class='line'>};</span></code></pre></td></tr></table></div></figure>
<p>最终任务是要找到 <code>set_devs</code> 这个列表里所有 <code>set_dev</code> 的 <code>lock</code> 这个信号量成员变量的信息。</p>
<p>首先，打开 vmcore (确保 kernel debuginfo 都在)：</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>sudo crash /usr/lib/debug/lib/modules/3.10.0/vmlinux /var/crash/vmcore</span></code></pre></td></tr></table></div></figure>
<p>如果要分析的结构体在内核模块中，还需要用 <code>mod -s/-S</code> 来加载内核模块的 debuginfo.</p>
<p>然后找到 <code>sets</code> 这个链表的入口地址。很幸运，模块里这个链表是全局的，直接找到 symbol 就可以用：</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>crash> p sets
</span><span class='line'>sets = $1 = {
</span><span class='line'> next = 0xffffc90056696030,
</span><span class='line'> prev = 0xffffc90056696030
</span><span class='line'>}
</span><span class='line'>crash> p &sets // 顺手确认一下链表是否为空，如果 &sets == sets->next 就为空
</span><span class='line'>$2 = (struct list_head *) 0xffffffffa04401f0 <sets></span></code></pre></td></tr></table></div></figure>
<p>我们就得到了一个不为空的链表的第一个元素（从上面也可以看到是唯一的一个）。由于内核里的链表只有指针没有元素内容，我们需要用类似 <code>container_of</code> 的机制来从成员反查链表元素的地址：</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>crash> set -l set.list 0xffffc90056696030
</span><span class='line'>struct set {
</span><span class='line'> <snip>
</span><span class='line'> list = {
</span><span class='line'> next = 0xffffffffa04401f0 <sets>,
</span><span class='line'> prev = 0xffffffffa04401f0 <sets>
</span><span class='line'> },
</span><span class='line'> <snip>
</span><span class='line'> set_devs = {
</span><span class='line'> next = 0xffffc90050f67000,
</span><span class='line'> prev = 0xffffc90056ee4000
</span><span class='line'> },
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>
<p>这样我们又得到了一个链表的 LIST_HEAD 地址：</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>crash> list_head 0xffffc90050f67000
</span><span class='line'>struct list_head {
</span><span class='line'> next = 0xffffc90065f2a000,
</span><span class='line'> prev = 0xffffc90056696da8
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>
<p>得到第一个元素地址后反查 <code>set_dev</code> 结构体：</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>crash> struct set_dev -l set_dev.list 0xffffc90065f2a000
</span><span class='line'>struct set_dev {
</span><span class='line'> list = {
</span><span class='line'> next = 0xffffc900583dc000,
</span><span class='line'> prev = 0xffffc90050f67000
</span><span class='line'> },
</span><span class='line'> <snip>
</span><span class='line'> rw_sem = {
</span><span class='line'> count = 0,
</span><span class='line'> wait_lock = {
</span><span class='line'> raw_lock = {
</span><span class='line'> {
</span><span class='line'> head_tail = 0,
</span><span class='line'> tickets = {
</span><span class='line'> head = 0,
</span><span class='line'> tail = 0
</span><span class='line'> }
</span><span class='line'> }
</span><span class='line'> }
</span><span class='line'> },
</span><span class='line'> wait_list = {
</span><span class='line'> next = 0xffffc90065f2ab48,
</span><span class='line'> prev = 0xffffc90065f2ab48
</span><span class='line'> }
</span><span class='line'> },
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>
<p>找到 <code>rw_sem</code> 结构体信息，这正是我们想要的。然后依次往下找即可。btw, 我忘记链表是循环链表了，所以我傻乎乎地手工查了半个小时最后才发现自己重复了……</p>
<p>所以问题来了，crash 里面有快捷的方法可以挨个链表元素查找么……</p>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[Block Device Plugging]]></title>
<link href="http://www.casparzhang.com/blog/2015/09/10/block-device-plugging/"/>
<updated>2015-09-10T18:31:25+08:00</updated>
<id>http://www.casparzhang.com/blog/2015/09/10/block-device-plugging</id>
<content type="html"><![CDATA[<p>关于 block 设备的 plug 和 unplug 机制，可以参考<a href="http://lxr.free-electrons.com/source/Documentation/block/biodoc.txt#L1011">这篇</a>内核文档。简单来说就是内核提供了一个 plug 机制，请求被放到一个空队列里之后，可以将队列处于 “plugged” 状态，此时队列并不真正向下层设备发射(issue)请求，而是攒够足够多的请求之后，将队列 “unplug” 之后才发射，在这期间，I/O 的调度算法将有机会对请求进行合并和排序。unplug 的时机既可以是一个手工的 <code>blk_unplug()</code> 操作，也可以由一个 <code>unplug_timer</code> 定时器超时触发。这整个过程听起来有点像在等机场大巴，到站停靠(plug)，装载乘客(攒 request)，到点发车(unplug_timeout)或者人满发车(blk_unplug)。</p>
<p>很容易想到，unplug_timer 这个定时器的启动会发生在 plug 阶段(见 block/blk-core.c 中的实现):</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>void blk_plug_device(struct request_queue *q)
</span><span class='line'>{
</span><span class='line'> WARN_ON(!irqs_disabled());
</span><span class='line'> /*
</span><span class='line'> * don't plug a stopped queue, it must be paired with blk_start_queue()
</span><span class='line'> * which will restart the queueing
</span><span class='line'> */
</span><span class='line'> if (blk_queue_stopped(q))
</span><span class='line'> return;
</span><span class='line'> if (!queue_flag_test_and_set(QUEUE_FLAG_PLUGGED, q)) {
</span><span class='line'> mod_timer(&q->unplug_timer, jiffies + q->unplug_delay);
</span><span class='line'> trace_block_plug(q);
</span><span class='line'> }
</span><span class='line'>}
</span><span class='line'>EXPORT_SYMBOL(blk_plug_device);</span></code></pre></td></tr></table></div></figure>
<p>而定时器超时函数见同一个文件下的实现：</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>void blk_unplug_timeout(unsigned long data)
</span><span class='line'>{
</span><span class='line'> struct request_queue *q = (struct request_queue *)data;
</span><span class='line'> trace_block_unplug_timer(q);
</span><span class='line'> kblockd_schedule_work(q, &q->unplug_work);
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>
<p>这个函数是在 <code>blk_queue_make_request</code> 定义的时候设置的(见 block/blk-settings.c)：</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>void blk_queue_make_request(struct request_queue *q, make_request_fn *mfn)
</span><span class='line'>{
</span><span class='line'> <snip>
</span><span class='line'> q->unplug_thresh = 4; /* hmm */
</span><span class='line'> q->unplug_delay = msecs_to_jiffies(3); /* 3 milliseconds */
</span><span class='line'> if (q->unplug_delay == 0)
</span><span class='line'> q->unplug_delay = 1;
</span><span class='line'> q->unplug_timer.function = blk_unplug_timeout;
</span><span class='line'> q->unplug_timer.data = (unsigned long)q;
</span><span class='line'> <snip>
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>
<p></p>
<p>可以看到除了定义了超时函数，还定义了超时的时间，是 3ms. 这个 3ms 可以用来排查一系列奇怪的 I/O 性能问题。比如说在 iodepth 为 1 的情况下，发现 IOPS 是很固定的 333，那么多半就是因为请求队列没有及时 unplug 而导致定时器超时自动 unplug 了。通过简单计算得知，因为每个队列需要 3ms 传输，那一秒钟只能传送 333 个队列，即 IOPS == 333.</p>
<p>如果要主动 unplug，就需要调用 <code>blk_unplug()</code> 函数，或者 <code>generic_unplug_device()</code> 函数，从通用性来说，调用前者会比较好，看如下代码(block/blk-core.c):</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>void blk_unplug(struct request_queue *q)
</span><span class='line'>{
</span><span class='line'> /*
</span><span class='line'> * devices don't necessarily have an ->unplug_fn defined
</span><span class='line'> */
</span><span class='line'> if (q->unplug_fn) {
</span><span class='line'> trace_block_unplug_io(q);
</span><span class='line'> q->unplug_fn(q);
</span><span class='line'> }
</span><span class='line'>}</span></code></pre></td></tr></table></div></figure>
<p> <code>q->unplug_fn</code> 一般不需要自行设置，如果没有定义，会使用默认的函数，即 <code>generic_unplug_device()</code>。</p>
<p>看完了这几个函数，可以知道，plug 完之后及时 unplug 通常是避免 IO 延迟过高的良好手段。</p>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[Use Non-export Symbol]]></title>
<link href="http://www.casparzhang.com/blog/2015/05/21/use-non-export-symbol/"/>
<updated>2015-05-21T18:38:25+08:00</updated>
<id>http://www.casparzhang.com/blog/2015/05/21/use-non-export-symbol</id>
<content type="html"><![CDATA[<p>最近在折腾 <a href="http://linux-iscsi.org">LIO</a>，里面的模块好多，有核心模块(<code>target_core_mod</code>)、前端驱动模块(<code>vhost-scsi</code>, <code>iscsi</code>, etc)、后端驱动模块(<code>target_core_iblock</code>, <code>target_core_file</code>, etc)。我想要实现某种程度的模块“热升级”，具体来说，比如<code>target_core_mod</code>这个核心模块升级了之后，想要和旧模块共存，新的前端和后端驱动在接入的时候直接使用新模块。</p>
<p>于是想当然地认为只要把编译出来的模块改个名字就好了。然而事情并没那么简单，中间碰到两个问题，首先就是本文要讲到的，导出符号(EXPORT_SYMBOL)重复的问题。其实问题很好理解，我改了个名字之后的模块(比如：<code>target_core_mod_new.ko</code>，在代码中还是会导出相同的符号，所以解决方法就是在新的模块中不导出这些符号，删掉所有的<code>EXPORT_SYMBOL()</code>宏。</p>
<p>可是问题又来了，我的后端驱动模块(比如：<code>target_core_iblock.ko</code>)需要依赖新的模块的函数，在无法导出符号的情况下，我没有办法使用后端驱动模块。稍微搜了一下，找到<code>kallsyms_lookup_name()</code>函数(<a href="http://lxr.free-electrons.com/source/kernel/kallsyms.c?v=2.6.33#L170">定义</a>)，这个函数传入一个类似 <strong><em>module:symbol</em></strong> 形式的参数，用于读取<code>/proc/kallsyms</code>中的符号表，找到对应的符号，然后返回符号的地址(即函数的入口)。</p>
<p>于是我拿了这个函数开开心心地去改代码了。以<code>target_complete_cmd()</code>这个函数为例，我在后端驱动模块的代码需要调用这个函数的地方：</p>
<!--more-->
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='C'><span class='line'><span class="n">target_complete_cmd</span><span class="p">(</span><span class="n">cmd</span><span class="p">,</span> <span class="n">status</span><span class="p">);</span>
</span><span class='line'><span class="n">kfree</span><span class="p">(</span><span class="n">ibr</span><span class="p">);</span>
</span></code></pre></td></tr></table></div></figure>
<p>把它改为这样：</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
</pre></td><td class='code'><pre><code class='C'><span class='line'><span class="cm">/* 参考 target_core_mod 里的函数原型，定义函数指针变量 */</span>
</span><span class='line'><span class="kt">void</span> <span class="p">(</span><span class="o">*</span><span class="n">target_complete_cmd_ptr</span><span class="p">)(</span><span class="k">struct</span> <span class="n">se_cmd</span> <span class="o">*</span><span class="p">,</span> <span class="n">u8</span><span class="p">);</span>
</span><span class='line'><span class="cm">/* 得到的是 target_complete_cmd 函数入口地址 */</span>
</span><span class='line'><span class="n">target_complete_cmd_ptr</span> <span class="o">=</span> <span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="p">)</span><span class="n">kallsyms_lookup_name</span><span class="p">(</span>
</span><span class='line'> <span class="s">"target_core_mod_new:target_complete_cmd"</span><span class="p">);</span>
</span><span class='line'><span class="p">(</span><span class="o">*</span><span class="n">target_complete_cmd_ptr</span><span class="p">)(</span><span class="n">cmd</span><span class="p">,</span> <span class="n">status</span><span class="p">);</span>
</span><span class='line'><span class="n">kfree</span><span class="p">(</span><span class="n">ibr</span><span class="p">);</span>
</span></code></pre></td></tr></table></div></figure>
<p>稍微验证了一下，发现可以用了，于是开始改其他的。</p>
<p>结果——WTF! 一共有 40+个被引用的导出符号！一个一个改的话，这后端驱动的代码还能看么？</p>
<p>思来想去，觉得，该改的还是得改。决定写个宏包装一下，尽量提高一下可读性。</p>
<p>以下省略我反复碰壁的过程，直接上成品(some ideas was inspired by <a href="http://constellation.github.io/blog/2014/12/07/importing-hidden-kernel-symbols/">this post</a>, thanks!):</p>
<p>首先，新建一个头文件，把宏和一些函数指针定义都给丢到里面，比如叫<code>import_sym.h</code>:</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
</pre></td><td class='code'><pre><code class='C'><span class='line'><span class="cp">#include <linux/kallsyms.h></span>
</span><span class='line'>
</span><span class='line'><span class="cm">/* 先定义一个结构体存放所有可能用到的函数指针 */</span>
</span><span class='line'><span class="k">struct</span> <span class="n">symbol_set</span> <span class="p">{</span>
</span><span class='line'> <span class="o"><</span><span class="n">snip</span><span class="p">...</span><span class="o">></span>
</span><span class='line'> <span class="kt">void</span> <span class="p">(</span><span class="o">*</span><span class="n">target_complete_cmd</span><span class="p">)(</span><span class="k">struct</span> <span class="n">se_cmd</span> <span class="o">*</span><span class="p">,</span> <span class="n">u8</span><span class="p">);</span>
</span><span class='line'> <span class="o"><</span><span class="n">snip</span><span class="p">...</span><span class="o">></span>
</span><span class='line'><span class="p">};</span>
</span><span class='line'>
</span><span class='line'><span class="cm">/* 这个宏用于查找符号地址，并且做一点基本的错误检查 */</span>
</span><span class='line'><span class="cp">#define IMPORT_SYMBOL(name) \</span>
</span><span class='line'><span class="cp"> sym.name = (typeof(name) *)kallsyms_lookup_name( \</span>
</span><span class='line'><span class="cp"> "target_core_mod_new:" #name); \</span>
</span><span class='line'><span class="cp"> pr_debug("symbol name: " #name ", symbol addr found: 0x%lx\n", \</span>
</span><span class='line'><span class="cp"> (unsigned long)sym.name); \</span>
</span><span class='line'><span class="cp"> if (!sym.name) \</span>
</span><span class='line'><span class="cp"> return -EINVAL;</span>
</span><span class='line'>
</span><span class='line'><span class="cm">/*</span>
</span><span class='line'><span class="cm"> * 下面这两个宏会拼成一个函数，放在前后端驱动中，前面这个宏还定义了一个</span>
</span><span class='line'><span class="cm"> * 全局静态结构用于存放 symbol 的集合</span>
</span><span class='line'><span class="cm"> */</span>
</span><span class='line'><span class="cp">#define IMPORT_SYMBOL_START \</span>
</span><span class='line'><span class="cp">static struct symbol_set sym; \</span>
</span><span class='line'><span class="cp">static int __init_syms_lookup(void) \</span>
</span><span class='line'><span class="cp">{</span>
</span><span class='line'>
</span><span class='line'><span class="cp">#define IMPORT_SYMBOL_END \</span>
</span><span class='line'><span class="cp"> return 0; \</span>
</span><span class='line'><span class="cp">}</span>
</span><span class='line'>
</span><span class='line'><span class="cm">/* 下面这个宏用来替换前后端驱动中每个要引用原导出符号的语句 */</span>
</span><span class='line'><span class="cp">#define IMPORTED(name) (*sym.name)</span>
</span></code></pre></td></tr></table></div></figure>
<p>接下来改前后端驱动的代码，还是以<code>target_core_iblock</code>为例：</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
</pre></td><td class='code'><pre><code class='C'><span class='line'><span class="cp">#include "import_sym.h"</span>
</span><span class='line'>
</span><span class='line'><span class="n">IMPORT_SYMBOL_START</span>
</span><span class='line'><span class="nf">IMPORT_SYMBOL</span><span class="p">(</span><span class="n">target_complete_cmd</span><span class="p">);</span>
</span><span class='line'><span class="cm">/* 这里还可以继续添加 IMPORT_SYMBOL 宏以查找更多需要用的符号 */</span>
</span><span class='line'><span class="n">IMPORT_SYMBOL_END</span>
</span><span class='line'>
</span><span class='line'><span class="o"><</span><span class="n">snip</span><span class="p">...</span><span class="o">></span>
</span><span class='line'><span class="cm">/* 把 target_complete_cmd(cmd, status); 换掉 */</span>
</span><span class='line'><span class="n">IMPORTED</span><span class="p">(</span><span class="n">target_complete_cmd</span><span class="p">)(</span><span class="n">cmd</span><span class="p">,</span> <span class="n">status</span><span class="p">);</span>
</span><span class='line'>
</span><span class='line'><span class="o"><</span><span class="n">snip</span><span class="p">...</span><span class="o">></span>
</span><span class='line'><span class="k">static</span> <span class="kt">int</span> <span class="n">__init</span> <span class="n">iblock_module_init</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
</span><span class='line'><span class="p">{</span>
</span><span class='line'> <span class="cm">/* 在模块初始化的时候就找齐所有需要的符号地址 */</span>
</span><span class='line'> <span class="kt">int</span> <span class="n">ret</span> <span class="o">=</span> <span class="n">__init_syms_lookup</span><span class="p">();</span>
</span><span class='line'> <span class="k">if</span> <span class="p">(</span><span class="n">ret</span><span class="p">)</span>
</span><span class='line'> <span class="k">return</span> <span class="n">ret</span><span class="p">;</span>
</span><span class='line'> <span class="k">return</span> <span class="nf">IMPORTED</span><span class="p">(</span><span class="n">transport_subsystem_register</span><span class="p">)(</span><span class="o">&</span><span class="n">iblock_template</span><span class="p">);</span>
</span><span class='line'><span class="p">}</span>
</span></code></pre></td></tr></table></div></figure>
<p>这样就可以了。</p>
<p>由于不再使用<code>EXPORT_SYMBOL()</code>，做出来的前后端驱动模块会不再显式依赖<code>target_core_mod</code>，必须得确保 core 模块先加载，才能让前后端驱动模块加载之后找到符号地址。卸载的时候 core 模块要最后卸载，否则会碰到空指针引发 Kernel Panic。这算是这个方案的一个不完美之处吧。</p>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[Linux Kernel 208.5d Panic Issue]]></title>
<link href="http://www.casparzhang.com/blog/2015/05/21/208-day-crash-issue/"/>
<updated>2015-05-21T15:39:33+08:00</updated>
<id>http://www.casparzhang.com/blog/2015/05/21/208-day-crash-issue</id>
<content type="html"><![CDATA[<p>最近线上碰到的问题，虽说早就有解决方案了，但是陆陆续续还是碰到很多没有升级内核的机器挂掉。记录一下以下内容，仅供参考。</p>
<p>现象是机器运行到一定天数(刚开始反馈集中在 208.5 天左右)就会主动挂掉，报告文章末尾的 CallTrace，所幸社区几年前就有了解决方案：</p>
<h2>Upstream Patch</h2>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>commit 4cecf6d401a01d054afc1e5f605bcbfe553cb9b9
</span><span class='line'>Author: Salman Qazi <sqazi@google.com>
</span><span class='line'>Date: Tue Nov 15 14:12:06 2011 -0800
</span><span class='line'>
</span><span class='line'> sched, x86: Avoid unnecessary overflow in sched_clock
</span><span class='line'>
</span><span class='line'> (Added the missing signed-off-by line)
</span><span class='line'>
</span><span class='line'> In hundreds of days, the __cycles_2_ns calculation in sched_clock
</span><span class='line'> has an overflow. cyc * per_cpu(cyc2ns, cpu) exceeds 64 bits, causing
</span><span class='line'> the final value to become zero. We can solve this without losing
</span><span class='line'> any precision.
</span><span class='line'>
</span><span class='line'> We can decompose TSC into quotient and remainder of division by the
</span><span class='line'> scale factor, and then use this to convert TSC into nanoseconds.
</span><span class='line'>
</span><span class='line'> Signed-off-by: Salman Qazi <sqazi@google.com>
</span><span class='line'> Acked-by: John Stultz <johnstul@us.ibm.com>
</span><span class='line'> Reviewed-by: Paul Turner <pjt@google.com>
</span><span class='line'> Cc: stable@kernel.org
</span><span class='line'> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
</span><span class='line'> Link: http://lkml.kernel.org/r/20111115221121.7262.88871.stgit@dungbeetle.mtv.corp.google.com
</span><span class='line'> Signed-off-by: Ingo Molnar <mingo@elte.hu></span></code></pre></td></tr></table></div></figure>
<!--more-->
<h2>Red Hat Fix</h2>
<ul>
<li>Z-stream since 2.6.32-220.5.1.el6</li>
<li>Y-stream Since 2.6.32-226.el6</li>
</ul>
<h2>Ref Links</h2>
<ul>
<li><a href="http://kenichiokuyama.blogspot.hk/2011/12/schedclock-overflow-after-2085-days-in.html">http://kenichiokuyama.blogspot.hk/2011/12/schedclock-overflow-after-2085-days-in.html</a></li>
</ul>
<h2>Call Trace</h2>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
<span class='line-number'>39</span>
<span class='line-number'>40</span>
<span class='line-number'>41</span>
<span class='line-number'>42</span>
<span class='line-number'>43</span>
<span class='line-number'>44</span>
<span class='line-number'>45</span>
<span class='line-number'>46</span>
<span class='line-number'>47</span>
<span class='line-number'>48</span>
<span class='line-number'>49</span>
<span class='line-number'>50</span>
<span class='line-number'>51</span>
<span class='line-number'>52</span>
<span class='line-number'>53</span>
<span class='line-number'>54</span>
<span class='line-number'>55</span>
<span class='line-number'>56</span>
<span class='line-number'>57</span>
<span class='line-number'>58</span>
<span class='line-number'>59</span>
<span class='line-number'>60</span>
<span class='line-number'>61</span>
<span class='line-number'>62</span>
<span class='line-number'>63</span>
<span class='line-number'>64</span>
<span class='line-number'>65</span>
<span class='line-number'>66</span>
<span class='line-number'>67</span>
<span class='line-number'>68</span>
<span class='line-number'>69</span>
<span class='line-number'>70</span>
<span class='line-number'>71</span>
<span class='line-number'>72</span>
<span class='line-number'>73</span>
<span class='line-number'>74</span>
<span class='line-number'>75</span>
<span class='line-number'>76</span>
<span class='line-number'>77</span>
<span class='line-number'>78</span>
<span class='line-number'>79</span>
<span class='line-number'>80</span>
<span class='line-number'>81</span>
<span class='line-number'>82</span>
<span class='line-number'>83</span>
<span class='line-number'>84</span>
<span class='line-number'>85</span>
<span class='line-number'>86</span>
<span class='line-number'>87</span>
<span class='line-number'>88</span>
<span class='line-number'>89</span>
<span class='line-number'>90</span>
<span class='line-number'>91</span>
<span class='line-number'>92</span>
<span class='line-number'>93</span>
<span class='line-number'>94</span>
<span class='line-number'>95</span>
<span class='line-number'>96</span>
<span class='line-number'>97</span>
<span class='line-number'>98</span>
<span class='line-number'>99</span>
<span class='line-number'>100</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>[776183.791021] divide error: 0000 [#1] SMP
</span><span class='line'>[776183.795055] last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
</span><span class='line'>[776183.803045] CPU 9
</span><span class='line'>[776183.804967] Modules linked in: <snip>
</span><span class='line'>[776183.844841]
</span><span class='line'>[776183.846415] Modules linked in: <snip>
</span><span class='line'>[776183.886217] Pid: 0, comm: swapper Not tainted <kernel>
</span><span class='line'>[776183.895358] RIP: 0010:[<ffffffff81059e15>] [<ffffffff81059e15>] find_busiest_group+0x485/0xb50
</span><span class='line'>[776183.904258] RSP: 0018:ffff880028283c80 EFLAGS: 00010246
</span><span class='line'>[776183.909703] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff88002828f900
</span><span class='line'>[776183.917021] RDX: 0000000000000000 RSI: ffc2c20d09ff2179 RDI: 000000001dcd6500
</span><span class='line'>[776183.924341] RBP: ffff880028283de0 R08: 0002c1ef5f24b5a6 R09: ffff88002828f910
</span><span class='line'>[776183.931658] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880028295f01
</span><span class='line'>[776183.938977] R13: ffff88002828f900 R14: 0000000000015f40 R15: ffffffffffffffff
</span><span class='line'>[776183.946296] FS: 0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
</span><span class='line'>[776183.954568] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
</span><span class='line'>[776183.960447] CR2: 00000000008cb028 CR3: 0000000c61a84000 CR4: 00000000000006e0
</span><span class='line'>[776183.967776] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
</span><span class='line'>[776183.975095] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
</span><span class='line'>[776183.982418] Process swapper (pid: 0, threadinfo ffff880664a52000, task ffff880c6412f4c0)
</span><span class='line'>[776183.990691] Stack:
</span><span class='line'>[776183.992839] ffff880028283e40 ffff880028283d80 ffff880c7fc29600 ffff880028283e64
</span><span class='line'>[776184.000247] <0> 0000000000000040 ffff880028283e58 0000000960cf4700 ffff88002828f5e0
</span><span class='line'>[776184.008157] <0> 0000000028283ce0 ffff88002828f910 00000001ffffffff 0000000000000009
</span><span class='line'>[776184.016303] Call Trace:
</span><span class='line'>[776184.018884] <IRQ>
</span><span class='line'>[776184.021138] [<ffffffff8105af96>] rebalance_domains+0x146/0x520
</span><span class='line'>[776184.027193] [<ffffffff8105b3bc>] run_rebalance_domains+0x4c/0x150
</span><span class='line'>[776184.033506] [<ffffffff8106b1e0>] __do_softirq+0xc0/0x1e0
</span><span class='line'>[776184.039038] [<ffffffff8100c18c>] call_softirq+0x1c/0x30
</span><span class='line'>[776184.044483] [<ffffffff8100dd55>] do_softirq+0x65/0xa0
</span><span class='line'>[776184.049755] [<ffffffff8106af2c>] irq_exit+0x7c/0x90
</span><span class='line'>[776184.054856] [<ffffffff814b0710>] smp_apic_timer_interrupt+0x70/0x9d
</span><span class='line'>[776184.061343] [<ffffffff8100bb53>] apic_timer_interrupt+0x13/0x20
</span><span class='line'>[776184.067485] <EOI>
</span><span class='line'>[776184.069738] [<ffffffff813bdc8a>] ? poll_idle+0x2a/0x70
</span><span class='line'>[776184.075097] [<ffffffff813bdc73>] ? poll_idle+0x13/0x70
</span><span class='line'>[776184.080454] [<ffffffff813bec86>] ? menu_select+0xa6/0x330
</span><span class='line'>[776184.086072] [<ffffffff813bdbc9>] cpuidle_idle_call+0x99/0x130
</span><span class='line'>[776184.092042] [<ffffffff81009888>] cpu_idle+0xa8/0xe0
</span><span class='line'>[776184.097143] [<ffffffff814a0d64>] ? start_secondary+0x1b4/0x2a0
</span><span class='line'>[776184.103194] [<ffffffff814a0d72>] start_secondary+0x1c2/0x2a0
</span><span class='line'>[776184.109072] Code: d8 b8 01 00 00 00 48 c1 eb 0a 48 85 db 0f 45 c3 41 89 45 08 48 8b 8d 00 ff ff ff 48 8b 45 a8 8b 51 08 48 c1 e0 0a 48 89 d3 31 d2 <48> f7 f3 48 8b 55 b0 48 89 45 a0 31 c0 48 85 d2 74 0f 48 8b 45
</span><span class='line'>[776184.129029] RIP [<ffffffff81059e15>] find_busiest_group+0x485/0xb50
</span><span class='line'>[776184.135529] RSP <ffff880028283c80>
</span><span class='line'>[776184.139477] BUG: scheduling while atomic: swapper/0/0x10000100
</span><span class='line'>[776184.139485] ---[ end trace 32313550323d1731 ]---
</span><span class='line'>[776184.139586] Kernel panic - not syncing: Fatal exception in interrupt
</span><span class='line'>[776184.139589] Pid: 0, comm: swapper Tainted: G D ---------------- <kernel>
</span><span class='line'>[776184.139591] Call Trace:
</span><span class='line'>[776184.139592] <IRQ> [<ffffffff81062e85>] ? panic+0xa5/0x1a0
</span><span class='line'>[776184.139598] [<ffffffff81352006>] ? netoops+0x1c6/0x2a0
</span><span class='line'>[776184.139601] [<ffffffff81064ac5>] ? kmsg_dump+0x135/0x180
</span><span class='line'>[776184.139604] [<ffffffff81062ae5>] ? oops_exit+0x25/0x30
</span><span class='line'>[776184.139606] [<ffffffff814abafb>] ? oops_end+0xeb/0x100
</span><span class='line'>[776184.139609] [<ffffffff8100f07b>] ? die+0x5b/0x90
</span><span class='line'>[776184.139612] [<ffffffff814ab226>] ? do_trap+0x136/0x150
</span><span class='line'>[776184.139616] [<ffffffff8100c80f>] ? do_divide_error+0x8f/0xb0
</span><span class='line'>[776184.139618] [<ffffffff81059e15>] ? find_busiest_group+0x485/0xb50
</span><span class='line'>[776184.139623] [<ffffffff8142124f>] ? ip_rcv+0x26f/0x380
</span><span class='line'>[776184.139626] [<ffffffff8100bdbb>] ? divide_error+0x1b/0x20
</span><span class='line'>[776184.139629] [<ffffffff81059e15>] ? find_busiest_group+0x485/0xb50
</span><span class='line'>[776184.139633] [<ffffffff8105af96>] ? rebalance_domains+0x146/0x520
</span><span class='line'>[776184.139636] [<ffffffff8105b3bc>] ? run_rebalance_domains+0x4c/0x150
</span><span class='line'>[776184.139639] [<ffffffff8106b1e0>] ? __do_softirq+0xc0/0x1e0
</span><span class='line'>[776184.139642] [<ffffffff8100c18c>] ? call_softirq+0x1c/0x30
</span><span class='line'>[776184.139644] [<ffffffff8100dd55>] ? do_softirq+0x65/0xa0
</span><span class='line'>[776184.139647] [<ffffffff8106af2c>] ? irq_exit+0x7c/0x90
</span><span class='line'>[776184.139650] [<ffffffff814b0710>] ? smp_apic_timer_interrupt+0x70/0x9d
</span><span class='line'>[776184.139653] [<ffffffff8100bb53>] ? apic_timer_interrupt+0x13/0x20
</span><span class='line'>[776184.139654] <EOI> [<ffffffff813bdc8a>] ? poll_idle+0x2a/0x70
</span><span class='line'>[776184.139660] [<ffffffff813bdc73>] ? poll_idle+0x13/0x70
</span><span class='line'>[776184.139663] [<ffffffff813bec86>] ? menu_select+0xa6/0x330
</span><span class='line'>[776184.139666] [<ffffffff813bdbc9>] ? cpuidle_idle_call+0x99/0x130
</span><span class='line'>[776184.139669] [<ffffffff81009888>] ? cpu_idle+0xa8/0xe0
</span><span class='line'>[776184.139672] [<ffffffff814a0d64>] ? start_secondary+0x1b4/0x2a0
</span><span class='line'>[776184.139676] [<ffffffff814a0d72>] ? start_secondary+0x1c2/0x2a0
</span><span class='line'>[776184.325813] Modules linked in: <snip>
</span><span class='line'>[776184.367794] CPU 1:
</span><span class='line'>[776184.370036] Modules linked in: <snip>
</span><span class='line'>[776184.412014] Pid: 0, comm: swapper Tainted: G D ---------------- <kernel>
</span><span class='line'>[776184.423621] RIP: 0010:[<ffffffff813bdc8a>] [<ffffffff813bdc8a>] poll_idle+0x2a/0x70
</span><span class='line'>[776184.431644] RSP: 0018:ffff88066495bea8 EFLAGS: 00000246
</span><span class='line'>[776184.437128] RAX: ffff88066495bfd8 RBX: ffff88066495bed8 RCX: 0042e00b649a5000
</span><span class='line'>[776184.444484] RDX: 000000000001f6a9 RSI: ffff88002821d610 RDI: ffffffff81a2fbc0
</span><span class='line'>[776184.451844] RBP: ffffffff8100bb4e R08: 0000000000000000 R09: 0000000000000000
</span><span class='line'>[776184.459201] R10: 0000000000000000 R11: 0000000000000000 R12: 0042dfee0e6ae400
</span><span class='line'>[776184.466567] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000561f5fe16
</span><span class='line'>[776184.473927] FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
</span><span class='line'>[776184.482241] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
</span><span class='line'>[776184.488159] CR2: 00000000427b7078 CR3: 0000000651611000 CR4: 00000000000006e0
</span><span class='line'>[776184.495520] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
</span><span class='line'>[776184.502878] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
</span><span class='line'>[776184.510236] Call Trace:
</span><span class='line'>[776184.512861] [<ffffffff813bdc73>] ? poll_idle+0x13/0x70
</span><span class='line'>[776184.518261] [<ffffffff813bec86>] ? menu_select+0xa6/0x330
</span><span class='line'>[776184.523918] [<ffffffff813bdbc9>] ? cpuidle_idle_call+0x99/0x130
</span><span class='line'>[776184.530096] [<ffffffff81009888>] ? cpu_idle+0xa8/0xe0
</span><span class='line'>[776184.535408] [<ffffffff814a0d64>] ? start_secondary+0x1b4/0x2a0
</span><span class='line'>[776184.541499] [<ffffffff814a0d72>] ? start_secondary+0x1c2/0x2a0</span></code></pre></td></tr></table></div></figure>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[Blog 恢复]]></title>
<link href="http://www.casparzhang.com/blog/2015/05/21/blog-revived/"/>
<updated>2015-05-21T15:17:37+08:00</updated>
<id>http://www.casparzhang.com/blog/2015/05/21/blog-revived</id>
<content type="html"><![CDATA[<p>Blog 已经迁移到了 <a href="http://www.github.com">GitHub</a>，准备恢复更新，主要更新一些内核相关的内容(瞬间变得高大上了有没有~lol)。</p>
<p>且看看我这会儿能坚持多久吧。</p>
<p>对了，坚持用 FeedBurner 订阅的用户，跳出来冒个泡呗…… 估计已经没有多少了吧？</p>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[Fix Razer Orochi Re-connect Issue]]></title>
<link href="http://www.casparzhang.com/blog/2014/10/10/fix-razer-orochi-re-connect-issue/"/>
<updated>2014-10-10T00:00:00+08:00</updated>
<id>http://www.casparzhang.com/blog/2014/10/10/fix-razer-orochi-re-connect-issue</id>
<content type="html"><![CDATA[<h2>Issue</h2>
<p>雷蛇八歧大蛇 2013 版鼠标，短时间内不用自动休眠，系统中蓝牙连接断(预期行为)，鼠标从休眠恢复后系统中的蓝牙连接却无法自动重连，只能将鼠标置为配对模式，通过系统中的蓝牙工具手工连接。</p>
<h2>Environment</h2>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>Razer Orochi 蓝牙鼠标
</span><span class='line'>Gentoo + KDE
</span><span class='line'>kernel 3.10 ~ 3.16
</span><span class='line'>bluez 4.101 ~ 5.23
</span><span class='line'>bluedevil 1.3.2 ~ 2.0_rc1</span></code></pre></td></tr></table></div></figure>
<h2>Solution</h2>
<!--more-->
<p>通过 <code>hcidump</code> 工具发现：</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>> HCI Event: Connect Request (0x04) plen 10
</span><span class='line'> bdaddr F0:65:DD:94:9F:BB class 0x002580 type ACL
</span><span class='line'>> HCI Event: Command Status (0x0f) plen 4
</span><span class='line'> Accept Connection Request (0x01|0x0009) status 0x00 ncmd 1
</span><span class='line'>> HCI Event: Connect Complete (0x03) plen 11
</span><span class='line'> status 0x00 handle 13 bdaddr F0:65:DD:94:9F:BB type ACL encrypt 0x00
</span><span class='line'>> HCI Event: Command Status (0x0f) plen 4
</span><span class='line'> Read Remote Supported Features (0x01|0x001b) status 0x00 ncmd 1
</span><span class='line'>> HCI Event: Read Remote Supported Features (0x0b) plen 11
</span><span class='line'> status 0x00 handle 13
</span><span class='line'> Features: 0xbf 0x00 0xa0 0x78 0x18 0x1e 0x59 0x83
</span><span class='line'>> HCI Event: Command Status (0x0f) plen 4
</span><span class='line'> Read Remote Extended Features (0x01|0x001c) status 0x00 ncmd 1
</span><span class='line'>> HCI Event: Read Remote Extended Features (0x23) plen 13
</span><span class='line'> status 0x00 handle 13 page 1 max 1
</span><span class='line'> Features: 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00
</span><span class='line'>> HCI Event: Command Status (0x0f) plen 4
</span><span class='line'> Remote Name Request (0x01|0x0019) status 0x00 ncmd 1
</span><span class='line'>> HCI Event: Remote Name Req Complete (0x07) plen 255
</span><span class='line'> status 0x00 bdaddr F0:65:DD:94:9F:BB name 'Razer Orochi 2013'
</span><span class='line'>> HCI Event: Command Complete (0x0e) plen 10
</span><span class='line'> Link Key Request Negative Reply (0x01|0x000c) ncmd 1
</span><span class='line'> status 0x00 bdaddr F0:65:DD:94:9F:BB
</span><span class='line'>> HCI Event: Disconn Complete (0x05) plen 4
</span><span class='line'> status 0x00 handle 13 reason 0x13
</span><span class='line'> Reason: Remote User Terminated Connection</span></code></pre></td></tr></table></div></figure>
<p>出现 <strong><em>Link Key Request Negative Reply</em></strong> 的字样，怀疑是配对的时候使用自动 key，而蓝牙工具没有保存这个自动 key，导致鼠标从休眠恢复时试图以空 key 连接，从而失败。</p>
<p>知道了问题，解决方法就简单了，配对的时候手工输入 key/PIN 为 <strong><em>1234</em></strong>，连接即可。</p>
<p>就这么一个破问题，折腾了我半天时间，内核、bluez、bluedevil 都尝试了一遍，还在人家的 Macbook 上测试了一番。差点就冲动下单换 Mac 了。 <code>#我不是土豪#</code></p>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[Gentoo KDE 下折腾 OpenConnect]]></title>
<link href="http://www.casparzhang.com/blog/2014/03/21/configure-openconnect-in-gentoo-kde/"/>
<updated>2014-03-21T00:00:00+08:00</updated>
<id>http://www.casparzhang.com/blog/2014/03/21/configure-openconnect-in-gentoo-kde</id>
<content type="html"><![CDATA[<p>最近呢，推上某奸商推出了 AnyConnect 套餐，这对广大翻墙群众来说绝对是个利好消息啊，可以自动配路由的东东。我毫不犹豫去把自己的绝版廉价套餐换成了 100 元的套餐，然后开始折腾 AnyConnect 配置。</p>
<p>iOS 上配置十分简单，找到 Cisco AnyConnect 这个 App 就搞定。Linux 下要把它配置得很舒服，着实花了一番功夫。</p>
<p>公司 VPN 用的也是 AnyConnect，我就继续用着公司的客户端(Cisco AnyConnect Secure Mobility Client), 直到今天早上我修复了笔记本上的无线网，机器有了两个 IP 为止。有了两个 IP 的 AnyConnect service 居然 segfault 了，看了一下 debug 信息，我觉得对这种闭源工具我还是别折腾了。直接换开源方案 OpenConnect.</p>
<p>Portage 里搜了一下，openconnect 有两个包，networkmanager-openconnect 有 libkeyring-gnome 依赖，我现在是个有洁癖的 KDE 党，果断不能装啊，于是用第二个， openconnect 命令行版…… 不行，不能这么罗里八嗦，反正最后折腾结果如下：</p>
<!--more-->
<ol>
<li>只用命令行的话，Portage 里的 <strong><em>openconnect-5.99*</em></strong> 版本可用，配好 <code>/etc/conf.d/openconnect</code> 就可以了；</li>
<li>需要配合 KDE plasma 插件 <strong><em>networkmanagement</em></strong> (0.9+版本) 的话，就必须得装 <strong>NetworkManager-openconnect</strong> 了。事实证明，<strong>libkeyring-gnome</strong> 依赖是可以去掉的。我修改了 Portage 文件，放到我<a href="https://github.com/casparant/caspar-gentoo/tree/master/net-misc/networkmanager-openconnect">自己的 Overlay</a> 里了。</li>
<li><del datetime="2014-03-21T09:54:44+00:00">另外 <strong><em>openconnect-5.99*</em></strong> 版本会导致 <strong><em>networkmanagement</em></strong> 带了 <code>openconnect</code> flag 之后编译失败，只能回退到 5.03 版本。</del> <strong><em>networkmanagement</em></strong> 已经 deprecated 了，用 <strong><em>plasma-nm</em></strong> 代替，最新版本没有编译问题。</li>
<li><del datetime="2014-03-21T09:54:44+00:00"><strong><em>openconnect-5.03</em></strong> 版本也是有问题的，<code>openconnect</code> 命令执行后，如果从终端输入密码，需要先按一个回车，再输入一遍。所以配置文件中的 <code>tmp_SERVER <<-E</code> 后面需要加一个空行，否则 daemon 永远启动失败，因为接收不到密码。</del> 换了最新的 <strong><em>openconnect-5.99</em></strong> 之后，没问题了。</li>
</ol>
<p>就这些。配好之后，既可以用 NM 的 GUI 连，也可以起服务连。从此过上了永不断网（某奸商广告语）的生活～</p>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[WizNote: 终于找到比较干净的 Qt 的笔记软件]]></title>
<link href="http://www.casparzhang.com/blog/2014/03/11/an-introduction-of-wiz-note/"/>
<updated>2014-03-11T00:00:00+08:00</updated>
<id>http://www.casparzhang.com/blog/2014/03/11/an-introduction-of-wiz-note</id>
<content type="html"><![CDATA[<p>Linux 上记笔记，向来很纠结。以前用过一段时间 <a href="http://zim-wiki.org/index.html">Zim</a>，后来变成了 KDE 党，只好果断抛弃了。找到 <a href="http://basket.kde.org/">BasKet</a>，功能还是很强大的，可是格式一团糟。无法以 Plain Text 存储文字，拷贝来拷贝去的时候经常格式混乱。也尝试在 Web 端存东西，结果还是发现不习惯，而且离线无法访问。</p>
<p>然后今天工作的时候有同事提到客户报告鄙厂<a href="http://www.aliyun.com/">产品</a>的问题，碰到了故障（当然在内核组大牛们的努力下该问题已经解决了），我就好奇去看了一下报告者信息，然后就发现了这个笔记软件：<a href="http://wiz.cn/index.html">Wiz</a>. 看起来还挺 Geek 的，有手机端有 Linux 端有 Mac 端。于是去 gentoo-zh 的 Overlay 找了一圈（是的，5 年过去了我还在 Gentoo 的不归路上慢慢走着），<a href="https://github.com/microcai/gentoo-zh/blob/master/app-text/wiznote/wiznote-2.0.64.ebuild">找到了</a>。</p>
<p>看起来样子还不错，还比较清爽，也可以格式化为 Plain Text。功能上比 BasKet 少一些，但是用着还是挺舒服的。</p>
<p>最不能忍受的问题是没有任务栏图标，于是接着搜。</p>
<p>找到了一年前某人的一个没有 Merge 的 <a href="https://github.com/WizTeam/WizQTClient/pull/80">Pull Request</a>，然后稍微修改了一下补丁，打到源代码里，发现可以用了。</p>
<p>修改后的 ebuild 请<a href="https://github.com/casparant/caspar-gentoo/blob/master/app-text/wiznote/wiznote-2.1.0_beta.ebuild">猛戳</a>。</p>
<p>最后，有更靠谱的笔记软件请推荐。</p>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[MCE 的一些零散记录]]></title>
<link href="http://www.casparzhang.com/blog/2013/06/06/some-misc-items-of-mce/"/>
<updated>2013-06-06T00:00:00+08:00</updated>
<id>http://www.casparzhang.com/blog/2013/06/06/some-misc-items-of-mce</id>
<content type="html"><![CDATA[<p>以前也测过 MCE 相关的东西，但是惭愧，一直不清不楚。这两天稍微整理了一下相关知识，感觉见识还是很浅，记些零碎的东西在这，权当博客复活第一篇。如有理解错误，欢迎指出。</p>
<p>CPU 检测到硬件错误时，内核会报 Machine-check，根据硬件错误是否可以修正(CE, Correctable Error; UCE/UE, Un-correctable Error)，内核做出不同反应。CE 的话内核只把相关信息写到一个字符设备 <code>/dev/mcelog</code> 中，UE 的话是会在记录相关信息之余，还会做出不同处理，比如中断当前遇错的应用程序，或者 Panic. <code>[TODO: 这块相关的内核代码还得再看几遍]</code></p>
<p><code>/dev/mcelog</code> 中的信息可以用用户空间工具 <strong><a href="https://github.com/andikleen/mcelog">mcelog</a></strong> 来读取。<strong>mcelog</strong> 比较有意思的一个参数是 <code>--dmi</code>，可以尝试解析 ADDR 字段以获取诸如内存出厂信息，DIMM 位置等有用的信息，可是很遗憾在实际环境中这些 DIMM 信息基本上都是错的。 (mcelog 的 man page 说了，不要怪 Linux，得怪那稀奇古怪的主板厂商……)</p>
<p>man page 还有一个地方提到：</p>
<blockquote><p>The kernel prefers old messages over new. If the log buffer overflows only old ones will be kept.</p></blockquote>
<!--more-->
<p>查看内核代码中 <code>arch/x86/include/asm/mce.h</code>，有：</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='C'><span class='line'><span class="cp">#define MCE_LOG_LEN 32</span>
</span><span class='line'>
</span><span class='line'><span class="k">struct</span> <span class="n">mce_log</span> <span class="p">{</span>
</span><span class='line'> <span class="kt">char</span> <span class="n">signature</span><span class="p">[</span><span class="mi">12</span><span class="p">];</span> <span class="cm">/* "MACHINECHECK" */</span>
</span><span class='line'> <span class="kt">unsigned</span> <span class="n">len</span><span class="p">;</span> <span class="cm">/* = MCE_LOG_LEN */</span>
</span><span class='line'> <span class="kt">unsigned</span> <span class="n">next</span><span class="p">;</span>
</span><span class='line'> <span class="kt">unsigned</span> <span class="n">flags</span><span class="p">;</span>
</span><span class='line'> <span class="kt">unsigned</span> <span class="n">recordlen</span><span class="p">;</span> <span class="cm">/* length of struct mce */</span>
</span><span class='line'> <span class="k">struct</span> <span class="n">mce</span> <span class="n">entry</span><span class="p">[</span><span class="n">MCE_LOG_LEN</span><span class="p">];</span>
</span><span class='line'><span class="p">};</span>
</span></code></pre></td></tr></table></div></figure>
<p>这里的 <code>MCE_LOG_LEN</code> 就是 man page 中提到的 log buffer 长度。</p>
<p>内核中还有个 <code>mce_inject</code> 模块（<code>CONFIG_X86_MCE_INJECT=m/y</code>）,可以用于产生一些假的 MCE 以测试相关功能。需要配合用户空间工具 <strong><a href="https://github.com/andikleen/mce-inject/">mce-inject</a></strong> 来使用。</p>
<p>如果要做一个比较完整的测试，<strong><a href="https://github.com/andikleen/mce-test">mce-test</a></strong> 工具可以帮忙，其中会涉及一些 RAS 特性，比如 <strong><em>hwpoison</em></strong>(<code>Documentation/vm/hwpoison.txt</code>)，不过我现在的活儿不怎么管测试，暂时就没去回顾这个测试工具了。</p>
<p>先记这些，回头把细节都搞懂了再补充……</p>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[Resolved: 如何更新 Git 补丁]]></title>
<link href="http://www.casparzhang.com/blog/2012/01/03/how-to-update-your-git-patches/"/>
<updated>2012-01-03T00:00:00+08:00</updated>
<id>http://www.casparzhang.com/blog/2012/01/03/how-to-update-your-git-patches</id>
<content type="html"><![CDATA[<p>使用 git 制作补丁时，经常发现补丁需要修改。如果只是最后一次 commit 需要修改，那就好办，用下面的方法就可以搞定：</p>
<figure class='code'><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='bash'><span class='line'>git reset HEAD^
</span><span class='line'><span class="c"># edit edit edit</span>
</span><span class='line'>git commit -a -s -c ORIG_HEAD
</span><span class='line'>git format-patch --subject-prefix<span class="o">=</span><span class="s2">"PATCH v2"</span>
</span></code></pre></td></tr></table></div></figure>
<p>但是如果是一系列补丁中的中间几个补丁需要修改，该怎么办呢？</p>
<p><del datetime="2012-01-03T16:12:57+00:00">笨办法已经被删掉>.<</del></p>
<!--more-->
<p><strong>Update 1:</strong> 非常感谢 wangcong 的指点！<code>git rebase -i</code> 可以出色地完成这个任务，方法在 man-page 里面有详述，见 <strong><em>INTERACTIVE MODE</em></strong> 和 <strong><em>SPLITTING COMMITS</em></strong> 部分。</p>
<p>假设当前所在分支名为 <strong>topic</strong> ，做了 3 个补丁，打算提交给邮件列表的时候发现中间一个补丁需要更新，此时使用下列命令：</p>