BUG: unable to handle kernel NULL pointer dereference at

[renton-]

Crashdump:

Feb 9 12:50:38 l28 [6437762.553311] BUG: unable to handle kernel NULL pointer dereference at (null)
Feb 9 12:50:38 l28 [6437762.553582] IP: [] netflow_target+0x8aa/0x1210 [ipt_NETFLOW]
Feb 9 12:50:38 l28 [6437762.553835] PGD a7af67067 PUD ab8150067 PMD 0
Feb 9 12:50:38 l28 [6437762.553972] Oops: 0000 [#1] SMP
Feb 9 12:50:38 l28 [6437762.554109] Modules linked in:
Feb 9 12:50:38 l28 tcm_loop
Feb 9 12:50:38 l28 iscsi_target_mod
Feb 9 12:50:38 l28 target_core_pscsi
Feb 9 12:50:38 l28 target_core_file
Feb 9 12:50:38 l28 target_core_iblock
Feb 9 12:50:38 l28 target_core_mod
Feb 9 12:50:38 l28 dm_thin_pool
Feb 9 12:50:38 l28 dm_persistent_data
Feb 9 12:50:38 l28 dm_bufio
Feb 9 12:50:38 l28 dm_bio_prison
Feb 9 12:50:38 l28 ipfw_mod(O)
Feb 9 12:50:38 l28 ipt_NETFLOW(O)
Feb 9 12:50:38 l28 crc32c_intel
Feb 9 12:50:38 l28 configfs
Feb 9 12:50:38 l28 iscsi_tcp
Feb 9 12:50:38 l28 libiscsi_tcp
Feb 9 12:50:38 l28 libiscsi
Feb 9 12:50:38 l28 scsi_transport_iscsi
Feb 9 12:50:38 l28 fuse
Feb 9 12:50:38 l28
Feb 9 12:50:38 l28 [6437762.554801] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G O 3.12.21-1gb-mb #1
Feb 9 12:50:38 l28 [6437762.555028] Hardware name: Intel Corporation S2600IP ........../S2600IP, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013
Feb 9 12:50:38 l28 [6437762.555263] task: ffff88081c57ea00 ti: ffff88081c5be000 task.ti: ffff88081c5be000
Feb 9 12:50:38 l28 [6437762.555493] RIP: 0010:[] [] netflow_target+0x8aa/0x1210 [ipt_NETFLOW]
Feb 9 12:50:38 l28 [6437762.555734] RSP: 0018:ffff88103fd835d0 EFLAGS: 00010282
Feb 9 12:50:38 l28 [6437762.555864] RAX: 00000000fffffff2 RBX: 0000000000000000 RCX: 0000000000000000
Feb 9 12:50:38 l28 [6437762.556083] RDX: 0000000000000010 RSI: 0000000000000014 RDI: ffff880fe4450200
Feb 9 12:50:38 l28 [6437762.556344] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff880fe4450200
Feb 9 12:50:38 l28 [6437762.556544] R10: ffff8810190c2600 R11: ffff88103fd83740 R12: ffff880fe4450200
Feb 9 12:50:38 l28 [6437762.556751] R13: 0000000000000005 R14: 0000000000000002 R15: 0000000000000014
Feb 9 12:50:38 l28 [6437762.556954] FS: 0000000000000000(0000) GS:ffff88103fd80000(0000) knlGS:0000000000000000
Feb 9 12:50:38 l28 [6437762.557168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 9 12:50:38 l28 [6437762.557282] CR2: 0000000000000000 CR3: 0000000aac099000 CR4: 00000000000407e0
Feb 9 12:50:38 l28 [6437762.557511] Stack:
Feb 9 12:50:38 l28 [6437762.557620] ffff880dc9535f00 ffff88103fd83740 ffff880fe4450b00 ffff880dc9535f00
Feb 9 12:50:38 l28 [6437762.559420] 00000000e4450b00 0000000000000000 ffff880dc95352a0 ffffffffa00ba593
Feb 9 12:50:38 l28 [6437762.559673] ffff880dc95352a0 00000048e4450b00 0000002000000002 ffff88101a289c60
Feb 9 12:50:38 l28 [6437762.559924] Call Trace:
Feb 9 12:50:38 l28 [6437762.560027]
Feb 9 12:50:38 l28 [6437762.560051] [] ? ipfw2_queue_handler+0xfc/0x108 [ipfw_mod]
Feb 9 12:50:38 l28 [6437762.560385] [] ? hash_net4_kadt+0x9a/0xd0
Feb 9 12:50:38 l28 [6437762.560513] [] ? ipt_do_table+0x29f/0x3a0
Feb 9 12:50:38 l28 [6437762.560636] [] ? br_nf_dev_queue_xmit+0x10/0x10
Feb 9 12:50:38 l28 [6437762.560763] [] ? nf_iterate+0x96/0xd0
Feb 9 12:50:38 l28 [6437762.560884] [] ? br_flood+0x140/0x140
Feb 9 12:50:38 l28 [6437762.561003] [] ? br_nf_dev_queue_xmit+0x10/0x10
Feb 9 12:50:38 l28 [6437762.561129] [] ? nf_hook_slow+0x77/0x150
Feb 9 12:50:38 l28 [6437762.561253] [] ? br_nf_dev_queue_xmit+0x10/0x10
Feb 9 12:50:38 l28 [6437762.561370] [] ? br_dev_queue_push_xmit+0xc0/0xc0
Feb 9 12:50:38 l28 [6437762.561490] [] ? br_nf_forward_ip+0x249/0x3d0
Feb 9 12:50:38 l28 [6437762.561617] [] ? nf_iterate+0x96/0xd0
Feb 9 12:50:38 l28 [6437762.561731] [] ? br_dev_queue_push_xmit+0xc0/0xc0
Feb 9 12:50:38 l28 [6437762.561846] [] ? nf_hook_slow+0x77/0x150
Feb 9 12:50:38 l28 [6437762.561963] [] ? br_dev_queue_push_xmit+0xc0/0xc0
Feb 9 12:50:38 l28 [6437762.562080] [] ? __br_forward+0x94/0xf0
Feb 9 12:50:38 l28 [6437762.562230] [] ? skb_clone+0x41/0xc0
Feb 9 12:50:38 l28 [6437762.562347] [] ? __skb_clone+0x24/0x100
Feb 9 12:50:38 l28 [6437762.562462] [] ? br_forward_finish+0x60/0x60
Feb 9 12:50:38 l28 [6437762.562585] [] ? deliver_clone+0x36/0x60
Feb 9 12:50:38 l28 [6437762.562701] [] ? br_handle_frame_finish+0x12c/0x2a0
Feb 9 12:50:38 l28 [6437762.562839] [] ? br_nf_pre_routing_finish+0x1c8/0x360
Feb 9 12:50:38 l28 [6437762.562961] [] ? nf_reinject+0x60/0x180
Feb 9 12:50:38 l28 [6437762.563091] [] ? br_nf_pre_routing_finish_bridge+0x150/0x150
Feb 9 12:50:38 l28 [6437762.563297] [] ? ipfw2_queue_handler+0xd2/0x108 [ipfw_mod]
Feb 9 12:50:38 l28 [6437762.563501] [] ? nf_queue+0x13d/0x180
Feb 9 12:50:38 l28 [6437762.563633] [] ? br_nf_pre_routing_finish_bridge+0x150/0x150
Feb 9 12:50:38 l28 [6437762.563835] [] ? br_nf_pre_routing_finish_bridge+0x150/0x150
Feb 9 12:50:38 l28 [6437762.564035] [] ? nf_hook_slow+0xc2/0x150
Feb 9 12:50:38 l28 [6437762.564149] [] ? br_nf_pre_routing_finish_bridge+0x150/0x150
Feb 9 12:50:38 l28 [6437762.564370] [] ? br_handle_local_finish+0x60/0x60
Feb 9 12:50:38 l28 [6437762.564507] [] ? br_handle_local_finish+0x60/0x60
Feb 9 12:50:38 l28 [6437762.564623] [] ? br_nf_pre_routing+0x446/0x710
Feb 9 12:50:38 l28 [6437762.564755] [] ? nf_iterate+0x96/0xd0
Feb 9 12:50:38 l28 [6437762.564877] [] ? nf_iterate+0x96/0xd0
Feb 9 12:50:38 l28 [6437762.564986] [] ? br_handle_local_finish+0x60/0x60
Feb 9 12:50:38 l28 [6437762.565121] [] ? br_handle_local_finish+0x60/0x60
Feb 9 12:50:38 l28 [6437762.565248] [] ? nf_hook_slow+0x77/0x150
Feb 9 12:50:38 l28 [6437762.565361] [] ? br_handle_local_finish+0x60/0x60
Feb 9 12:50:38 l28 [6437762.565475] [] ? br_handle_frame+0x1c1/0x290
Feb 9 12:50:38 l28 [6437762.565591] [] ? br_handle_frame_finish+0x2a0/0x2a0
Feb 9 12:50:38 l28 [6437762.565720] [] ? __netif_receive_skb_core+0x1e5/0x5e0
Feb 9 12:50:38 l28 [6437762.565840] [] ? netif_receive_skb+0x24/0x80
Feb 9 12:50:38 l28 [6437762.565971] [] ? napi_gro_receive+0x98/0x110
Feb 9 12:50:38 l28 [6437762.566098] [] ? igb_poll+0x6c6/0xfa0
Feb 9 12:50:38 l28 [6437762.566222] [] ? net_rx_action+0xf1/0x190
Feb 9 12:50:38 l28 [6437762.566338] [] ? __do_softirq+0xc8/0x190
Feb 9 12:50:38 l28 [6437762.566456] [] ? handle_irq_event_percpu+0x7e/0x140
Feb 9 12:50:38 l28 [6437762.566593] [] ? call_softirq+0x1c/0x30
Feb 9 12:50:38 l28 [6437762.566734] [] ? do_softirq+0x4d/0x80
Feb 9 12:50:38 l28 [6437762.566856] [] ? irq_exit+0x55/0x60
Feb 9 12:50:38 l28 [6437762.566967] [] ? do_IRQ+0x5c/0xd0
Feb 9 12:50:38 l28 [6437762.567079] [] ? common_interrupt+0x6a/0x6a
Feb 9 12:50:38 l28 [6437762.567212]
Feb 9 12:50:38 l28 [6437762.567225] [] ? arch_remove_reservations+0x130/0x130
Feb 9 12:50:38 l28 [6437762.567459] [] ? default_idle+0x2/0x10
Feb 9 12:50:38 l28 [6437762.567572] [] ? cpu_startup_entry+0xb1/0x190
Feb 9 12:50:38 l28 [6437762.567703] [] ? start_secondary+0x1d2/0x230
Feb 9 12:50:38 l28 [6437762.567839] Code:
Feb 9 12:50:38 l28 60 41 2b 44 24 64 29 d8 83 f8 03 0f 8e 2b 03 00 00 48 63 db 49 03 9c 24 c8 00 00 00 74 0d 8b 03 c1 e8 10 66 89 84 24 df 00 00 00
Feb 9 12:50:38 l28 syslog-ng[14140]: Error processing log message: <8b>
Feb 9 12:50:38 l28 03 c6 44 24 10 00 48 c7 c5 c0 22 01 00 c7 44 24 34 00 00 00
Feb 9 12:50:38 l28 [6437762.568840] RIP [] netflow_target+0x8aa/0x1210 [ipt_NETFLOW]
Feb 9 12:50:38 l28 [6437762.569089] RSP
Feb 9 12:50:38 l28 [6437762.569194] CR2: 0000000000000000
Feb 9 12:50:38 l28 [6437762.569891] ---[ end trace b21adcce70e97d64 ]---
Feb 9 12:50:38 l28 [6437762.585494] Kernel panic - not syncing: Fatal exception in interrupt
Feb 9 12:50:38 l28 [6437762.692769] Rebooting in 5 seconds..
Feb 9 12:50:43 l28 [6437767.691839] ACPI MEMORY or I/O RESET_REG.

modinfo ipt_NETFLOW

filename: /lib/modules/3.12.21-1gb-mb/extra/ipt_NETFLOW.ko
alias: ip6t_NETFLOW
version: 1.8.2
description: iptables NETFLOW target module
author: [email protected]
license: GPL
depends:
vermagic: 3.12.21-1gb-mb SMP mod_unload
parm: destination:export destination ipaddress:port (charp)
parm: inactive_timeout:inactive flows timeout in seconds (int)
parm: active_timeout:active flows timeout in seconds (int)
parm: debug:debug verbosity level (int)
parm: sndbuf:udp socket SNDBUF size (int)
parm: protocol:netflow protocol version (5, 9, 10) (int)
parm: refresh_rate:NetFlow v9/IPFIX refresh rate (packets) (uint)
parm: timeout_rate:NetFlow v9/IPFIX timeout rate (minutes) (uint)
parm: hashsize:hash table size (int)
parm: maxflows:maximum number of flows (int)
parm: aggregation:aggregation ruleset (charp)

[aabc]

Can you provide what real version of module it is? (Version 1.8.2 does not exists.)
Also, can you send me binary of your ipt_NETFLOW.ko that is crashed?

[renton-]

Thank you for your answer.

Version of muled:
filename: /lib/modules/3.12.21-1gb-mb/extra/ipt_NETFLOW.ko
alias: ip6t_NETFLOW
version: 1.8.2
description: iptables NETFLOW target module
author: [email protected]
license: GPL
depends:
vermagic: 3.12.21-1gb-mb SMP mod_unload
parm: destination:export destination ipaddress:port (charp)
parm: inactive_timeout:inactive flows timeout in seconds (int)
parm: active_timeout:active flows timeout in seconds (int)
parm: debug:debug verbosity level (int)
parm: sndbuf:udp socket SNDBUF size (int)
parm: protocol:netflow protocol version (5, 9, 10) (int)
parm: refresh_rate:NetFlow v9/IPFIX refresh rate (packets)
(uint)
parm: timeout_rate:NetFlow v9/IPFIX timeout rate (minutes)
(uint)
parm: hashsize:hash table size (int)
parm: maxflows:maximum number of flows (int)
parm: aggregation:aggregation ruleset (charp)

module:
http://81.177.174.8/ipt_NETFLOW.ko.gz

On Tue, Feb 10, 2015 at 07:21:28AM -0800, ABC wrote:

Can you provide what version of module it is?
Also, can you send me binary of your ipt_NETFLOW.ko that is crashed?

---

Reply to this email directly or view it on GitHub:
#27 (comment)

[aabc]

Thanks you! What is the source you installed it from?

[renton-]

Perhaps from github? unfortunately, I don't remember.

http://81.177.174.8/ipt-netflow-master.zip

On Tue, Feb 10, 2015 at 07:57:11AM -0800, ABC wrote:

Thanks you! What is the source you installed it from?

---

Reply to this email directly or view it on GitHub:
#27 (comment)

[aabc]

Thanks much for explanation! This looks like very old testing version from git, before release 2.0. I usually try to maintain stability of release versions, so it's preferred to use them if you don't want to experiment with latest features. Non-release git versions may contain experimental features which is not always very stable.

In your case, it will be wiser to just upgrade to 2.0.1 (latest stable of 2.0 branch) or 2.1 (very recent release), there is many bugfixes already in them. Both of these have no stability related bug reports for a long time. If you wish to continue using testing version, please install latest version from git.

[renton-]

I installed 2.0.1.
Thanks for help!

On Tue, Feb 10, 2015 at 08:27:10AM -0800, ABC wrote:

Thanks much for explanation! This looks like very old testing version from git, before release 2.0. I usually try to maintain stability of release versions, so it's preferred to use them if you don't want to experiment with latest features. Non-release git versions may contain experimental features which is not always very stable.

In your case, it will be wiser to just upgrade to 2.0.1 (latest stable of 2.0 branch) or 2.1 (very recent release), there is many bugfixes already in them. Both of these have no stability related bug reports for a long time. If you wish to continue using testing version, please install latest version from git.

---

Reply to this email directly or view it on GitHub:
#27 (comment)

[aabc]

About your crash. Unfortunately, your binary have stripped debug symbols (probably kernel is compiled without CONFIG_DEBUG_INFO=y), thus, it's very hard to identify code point of crash.

Report say RIP: ... netflow_target+0x8aa/0x1210 and Code: 60 41 2b 44 24 64 29 d8 83 f8 03 0f 8e 2b 03 00 00 48 63 db 49 03 9c 24 c8 00 00 00 74 0d 8b 03 c1 e8 10 66 89 84 24 df 00 00 00 <missed code 8b here> 03 c6 44 24 10 00 48 c7 c5 c0 22 01 00 c7 44 24 34 00 00 00. Missed code (which I identified as 8b) is where RIP should be, but it's address (+0x8fa) is very different from +0x8aa.

objdump -S ipt_NETFLOW.asm shows at corresponding locations:

     8a0:       0f c9                   bswap  %ecx
     8a2:       66 0f 1f 44 00 00       nopw   0x0(%eax,%eax,1)
     8a8:       8b 50 14                mov    0x14(%eax),%edx
  -> 8ab:       21 ca                   and    %ecx,%edx   <---- supposed RIP location
     8ad:       3b 50 18                cmp    0x18(%eax),%edx
     8b0:       0f 84 a2 04 00 00       je     d58 <netflow_target+0xd08>
     8b6:       48                      dec    %eax
     8b7:       8b 00                   mov    (%eax),%eax
     8b9:       48                      dec    %eax
     8ba:       3d 00 00 00 00          cmp    $0x0,%eax
     8bf:       75 e7                   jne    8a8 <netflow_target+0x858>
     8c1:       c6 44 24 32 00          movb   $0x0,0x32(%esp)
     8c6:       e9 16 f9 ff ff          jmp    1e1 <netflow_target+0x191>
     8cb:       41                      inc    %ecx
     8cc:       8b 44 24 60             mov    0x60(%esp),%eax
     8d0:       41                      inc    %ecx
     8d1:       2b 44 24 64             sub    0x64(%esp),%eax
     8d5:       29 d8                   sub    %ebx,%eax
     8d7:       83 f8 03                cmp    $0x3,%eax
     8da:       0f 8e 2b 03 00 00       jle    c0b <netflow_target+0xbbb>
     8e0:       48                      dec    %eax
     8e1:       63 db                   arpl   %bx,%bx
     8e3:       49                      dec    %ecx
     8e4:       03 9c 24 c8 00 00 00    add    0xc8(%esp),%ebx
     8eb:       74 0d                   je     8fa <netflow_target+0x8aa>
     8ed:       8b 03                   mov    (%ebx),%eax
     8ef:       c1 e8 10                shr    $0x10,%eax
     8f2:       66 89 84 24 df 00 00    mov    %ax,0xdf(%esp)
     8f9:       00
  -> 8fa:       8b 03                   mov    (%ebx),%eax   <----- RIP <point> in the Code
     8fc:       c6 44 24 10 00          movb   $0x0,0x10(%esp)
     901:       48                      dec    %eax
     902:       c7 c5 00 00 00 00       mov    $0x0,%ebp
     908:       c7 44 24 34 00 00 00    movl   $0x0,0x34(%esp)

As you can see, address 8aa does not point to valid instruction boundary. And <8b> is located at 8fa.

What we can do about this:

1. You may need to recompile same sources of the module with CONFIG_DEBUG_INFO=y on the same box - it will probably match already compiled version, and contain debug symbols, thus I will be able to identify crash location. This is not really hard: Just find in Makefile line make -C $(KDIR) M=$(CURDIR) modules and after space append to it CONFIG_DEBUG_INFO=y so it will look like make -C $(KDIR) M=$(CURDIR) modules CONFIG_DEBUG_INFO=y, then make (not make install), and send me resulting ipt_NETFLOW.ko that will be created in compilation dir. This should contain helpful debug info.
2. You may just install from latest git or release version, and we hope this bug is already fixed (which is very likely). If you choose this option I will close the ticket as 'obsoleted'.

[aabc]

Sorry, I'm updated previous post with additional text, in case you only read replies in email you'll not see them, pls check #27 (comment)

[aabc]

I assume you choose second option. Thus, I close the ticket.