Bug 11804

Reply-To: akpm@linux-foundation.org


(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

(I'll reassign this to scsi).

On Tue, 21 Oct 2008 17:17:03 -0700 (PDT)
bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=11804
> 
>            Summary: iscsi: LRO plus iSCSI causes panic
>            Product: Networking
>            Version: 2.5
>      KernelVersion: 2.6.26
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Other
>         AssignedTo: acme@ghostprotocols.net
>         ReportedBy: jesse.brandeburg@intel.com
> 
> 
> Distribution: SuSE SLES 10.2
> Hardware Environment:
> .config is available on request, we are using i686 arch, dell 2950, ixgbe 
> adapter, inet_lro module.
> 
> Software Environment: none/gnome-session running yast2
> 
> Problem Description:
> We found that just trying to connect with no authentication to an iSCSI 
> target over an adapter running either the in-kernel LRO or an in-driver 
> version will cause this panic.
> 
> Steps to reproduce:
> in suse, start yast2 utility, find iscsi module, connect to iSCSI target
> using
> ixgbe driver with CONFIG_INET_LRO enabled.
> 
> I did some debugging:
> I tried to debug down a ways but got lost in figuring out what the code 
> was trying to do.  I believe the bug is because the memcpy in 
> iscsi_tcp_segment_recv faults.
> 
> from looking at the debug messages below, 
> skb_seq_read (only called by iSCSI) returns a negative value for avail, and a
> pointer value of 5a8 into &ptr in the call at line 953: of iscsi_tcp.c
> 
> I didn't figure out where in skb_seq_read returns the bogus data, I wanted 
> to send this along now that I've found out this much.
> 
> skb_seq_read appears to have the logic inside that it needs to handle LRO 
> packets (data either in frags[] or frag_list) but something is wrong 
> still.
> 
> I turned on tcp_debug messages in iscsi_tcp.c, here is the log and panic.  
> I believe the offsets in the function are slightly different than normal 
> due to the inclusion of the debug printks.
> 
> The normal panic is at 
> BUG: unable to handle kernel NULL pointer dereference at 000005a8
> IP: [<f8de64b2>] :iscsi_tcp:iscsi_tcp_recv+0x161/0x473
> *pdpt = 0000000036533001 *pde = 0000000000000000
> Oops: 0000 [#1] SMP
> Modules linked in: crc32c libcrc32c iscsi_tcp libiscsi scsi_transport_iscsi
> ixgbe netconsole inet_lro ipv6 af_packet button battery ac loop usbhid
> ff_memless ehci_hcd uhci_hcd usbcore dm_mod bnx2 ext3 jbd edd fan thermal
> processor thermal_sys sg megaraid_sas ata_piix libata dock piix sd_mod
> scsi_mod
> ide_disk ide_core [last unloaded: iscsi_tcp]
> 
> Pid: 0, comm: swapper Not tainted (2.6.26-bigsmp #1)
> EIP: 0060:[<f8de64b2>] EFLAGS: 00010202 CPU: 3
> EIP is at iscsi_tcp_recv+0x161/0x473 [iscsi_tcp]
> EAX: 0000002b EBX: f747dd48 ECX: 00000038 EDX: 00000000
> ESI: 000005a8 EDI: f593db20 EBP: f751ca10 ESP: f747dd20
>  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> Process swapper (pid: 0, ti=f747c000 task=f745abe0 task.ti=f747c000)
> Stack: f8de78e7 000000e0 f446c0c0 f6c35544 f751ca00 000005a8 00000000
> 000000e0
>        000005a8 08745958 00000000 00000a88 00000000 000005a8 f446c0c0
>        f78ba0ac
>        00000000 c0289617 00000000 00000000 05a80001 00007fff f78ba040
>        000005a8
> Call Trace:
>  [<c0289617>] tcp_ack+0x15bd/0x1757
>  [<c028391e>] tcp_read_sock+0x8c/0x1e0
>  [<f8de6351>] iscsi_tcp_recv+0x0/0x473 [iscsi_tcp]
>  [<f8de716a>] iscsi_tcp_data_ready+0x36/0x80 [iscsi_tcp]
>  [<c028d1a2>] tcp_send_ack+0xab/0xaf
>  [<c028c02e>] tcp_rcv_established+0x3b3/0x639
>  [<c02909fb>] tcp_v4_do_rcv+0x22/0x16f
>  [<c0292294>] tcp_v4_rcv+0x512/0x562
>  [<c027b921>] ip_local_deliver_finish+0xb2/0x14a
>  [<c027b852>] ip_rcv_finish+0x286/0x2a3
>  [<f8ce9a93>] packet_rcv_spkt+0xb6/0xbd [af_packet]
>  [<c0261889>] netif_receive_skb+0x2d0/0x33b
>  [<f8afd5ca>] lro_flush+0x314/0x340 [inet_lro]
>  [<f8afd636>] lro_flush_all+0x1b/0x28 [inet_lro]
>  [<f8b410eb>] ixgbe_clean_rx_irq+0x73b/0x850 [ixgbe]
>  [<f8b44183>] ixgbe_clean_rxonly+0x53/0xd0 [ixgbe]
>  [<c0263521>] net_rx_action+0x8a/0x152
>  [<c0124c6e>] __do_softirq+0x5d/0xc1
>  [<c0124d04>] do_softirq+0x32/0x36
>  [<c010663a>] do_IRQ+0x73/0x85
>  [<c0109152>] mwait_idle+0x0/0x32
>  [<c0105143>] common_interrupt+0x23/0x28
>  [<c0109152>] mwait_idle+0x0/0x32
>  [<c0109181>] mwait_idle+0x2f/0x32
>  [<c0103535>] cpu_idle+0x88/0x9c
>  =======================
> Code: 24 14 0f 46 44 24 14 89 44 24 14 50 68 e7 78 de f8 e8 2e b3 33 c7 8b 7d
> 08 03 7d 00 8b 4c 24 1c 8b 74 24 20 03 74 24 18 c1 e9 02 <f3> a5 8b 4c 24 1c
> 83
> e1 03 74 02 f3 a4 8b 4c 24 1c 01 4c 24 18
> EIP: [<f8de64b2>] iscsi_tcp_recv+0x161/0x473 [iscsi_tcp] SS:ESP 0068:f747dd20
> Kernel panic - not syncing: Fatal exception in interrupt
> 
> 
> full dmesg with debug:
> 
> console [netcon0] enabled
> netconsole: network logging started
> ixgbe: eth6: ixgbe_remove: complete
> ACPI: PCI interrupt for device 0000:0c:00.0 disabled
> ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.41-NAPI
> Copyright (c) 1999-2008 Intel Corporation.
> ACPI: PCI Interrupt 0000:0c:00.0[A] -> GSI 16 (level, low) -> IRQ 16
> PCI: Setting latency timer of device 0000:0c:00.0 to 64
> ixgbe: 0000:0c:00.0: ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx
> Queue
> count = 4, Tx Queue count = 1
> ixgbe: eth0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x4) 00:1b:21:09:1b:44
> ixgbe: eth0: ixgbe_probe: MAC: 1, PHY: 2
> ixgbe: eth0: ixgbe_probe: PCI-Express bandwidth available for this card is
> not
> sufficient for optimal performance.
> ixgbe: eth0: ixgbe_probe: For optimal performance a x8 PCI-Express slot is
> required.
> ixgbe: eth0: ixgbe_probe: In-kernel LRO is enabled
> ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
> ADDRCONF(NETDEV_UP): eth6: link is not ready
> ixgbe: eth6: ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: None
> ADDRCONF(NETDEV_CHANGE): eth6: link becomes ready
> eth6: no IPv6 routers present
> Loading iSCSI transport class v2.0-869.
> iscsi: registered transport (tcp)
> iscsi: registered transport (tcp)
> scsi3 : iSCSI Initiator over TCP/IP
> tcp: iscsi_tcp_hdr_recv_prep(f751ca00)
> tcp: iscsi_tcp_send_hdr_prep(f751ca00)
> tcp: iscsi_tcp_send_linear_data_prepare(f751ca00, datalen=464)
> tcp: copied 0 0 size 48 xmit
> tcp: copied 0 48 size 48 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 48 total size 48
> tcp: Header done. Next segment size 464 total_size 464
> tcp: copied 0 0 size 464 xmit
> tcp: copied 0 464 size 464 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 464 total size 464
> tcp: copied 0 0 size 0 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 464 total size 464
> tcp: in 380 bytes
> tcp: skb f446cb40 ptr=f446f854 avail=380
> tcp: copied 0 0 size 48 recv
> tcp: iscsi_tcp_segment_recv copying 48
> tcp: copied 0 48 size 48 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 48 total size 48
> tcp: segment done
> tcp: opcode 0x23 ahslen 0 datalen 331
> tcp: skb f446cb40 ptr=f446f884 avail=332
> tcp: copied 0 0 size 331 recv
> tcp: iscsi_tcp_segment_recv copying 331
> tcp: copied 0 331 size 331 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 331 total size 331
> tcp: consume 1 pad bytes
> tcp: iscsi_tcp_segment_recv copying 1
> tcp: copied 0 1 size 1 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 332 total size 332
> tcp: segment done
> tcp: iscsi_tcp_hdr_recv_prep(f751ca00)
> tcp: no more data avail. Consumed 380
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: xmit 512 bytes
> tcp: iscsi_tcp_send_hdr_prep(f751ca00)
> tcp: copied 0 0 size 48 xmit
> tcp: copied 0 48 size 48 xmit
> tcp: in 84 bytes
> tcp: skb f446ca80 ptr=f446f054 avail=84
> tcp: copied 0 0 size 48 recv
> tcp: iscsi_tcp_segment_recv copying 48
> tcp: copied 0 48 size 48 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 48 total size 48
> tcp: segment done
> tcp: opcode 0x25 ahslen 0 datalen 36
> tcp: iscsi_tcp_begin_data_in(f751ca00, offset=0, datalen=36)
> tcp: skb f446ca80 ptr=f446f084 avail=36
> tcp: copied 0 0 size 36 recv
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 36
> tcp: copied 0 36 size 36 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 36 total size 36
> tcp: segment done
> tcp: iscsi_tcp_hdr_recv_prep(f751ca00)
> tcp: no more data avail. Consumed 84
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 48 total size 48
> tcp: Header done. Next segment size 0 total_size 0
> tcp: copied 0 0 size 0 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 0 total size 0
> tcp: xmit 48 bytes
> tcp: iscsi_tcp_send_hdr_prep(f751ca00)
> tcp: copied 0 0 size 48 xmit
> tcp: copied 0 48 size 48 xmit
> tcp: in 124 bytes
> tcp: skb f446c9c0 ptr=f6540854 avail=124
> tcp: copied 0 0 size 48 recv
> tcp: iscsi_tcp_segment_recv copying 48
> tcp: copied 0 48 size 48 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 48 total size 48
> tcp: segment done
> tcp: opcode 0x25 ahslen 0 datalen 74
> tcp: iscsi_tcp_begin_data_in(f751ca00, offset=0, datalen=74)
> tcp: skb f446c9c0 ptr=f6540884 avail=76
> tcp: copied 0 0 size 74 recv
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 74
> tcp: copied 0 74 size 74 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 74 total size 74
> tcp: consume 2 pad bytes
> tcp: iscsi_tcp_segment_recv copying 2
> tcp: copied 0 2 size 2 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 76 total size 76
> tcp: segment done
> tcp: iscsi_tcp_hdr_recv_prep(f751ca00)
> tcp: no more data avail. Consumed 124
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 48 total size 48
> tcp: Header done. Next segment size 0 total_size 0
> tcp: copied 0 0 size 0 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 0 total size 0
> tcp: xmit 48 bytes
> scsi 3:0:0:0: Direct-Access     SUN      LCSM100_I        0670 PQ: 0 ANSI: 5
> tcp: iscsi_tcp_send_hdr_prep(f751ca00)
> tcp: copied 0 0 size 48 xmit
> tcp: copied 0 48 size 48 xmit
> tcp: in 48 bytes
> tcp: skb f446c900 ptr=f6540054 avail=48
> tcp: copied 0 0 size 48 recv
> tcp: iscsi_tcp_segment_recv copying 48
> tcp: copied 0 48 size 48 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 48 total size 48
> tcp: segment done
> tcp: opcode 0x21 ahslen 0 datalen 0
> tcp: iscsi_tcp_hdr_recv_prep(f751ca00)
> tcp: no more data avail. Consumed 48
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 48 total size 48
> tcp: Header done. Next segment size 0 total_size 0
> tcp: copied 0 0 size 0 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 0 total size 0
> tcp: xmit 48 bytes
> tcp: iscsi_tcp_send_hdr_prep(f751ca00)
> tcp: copied 0 0 size 48 xmit
> tcp: copied 0 48 size 48 xmit
> tcp: in 56 bytes
> tcp: skb f446c840 ptr=f6541854 avail=56
> tcp: copied 0 0 size 48 recv
> tcp: iscsi_tcp_segment_recv copying 48
> tcp: copied 0 48 size 48 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 48 total size 48
> tcp: segment done
> tcp: opcode 0x25 ahslen 0 datalen 8
> tcp: iscsi_tcp_begin_data_in(f751ca00, offset=0, datalen=8)
> tcp: skb f446c840 ptr=f6541884 avail=8
> tcp: copied 0 0 size 8 recv
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 8
> tcp: copied 0 8 size 8 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 8 total size 8
> tcp: segment done
> tcp: iscsi_tcp_hdr_recv_prep(f751ca00)
> tcp: no more data avail. Consumed 56
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 48 total size 48
> tcp: Header done. Next segment size 0 total_size 0
> tcp: copied 0 0 size 0 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 0 total size 0
> tcp: xmit 48 bytes
> sd 3:0:0:0: [sdb] 190421401 512-byte hardware sectors (97496 MB)
> tcp: iscsi_tcp_send_hdr_prep(f751ca00)
> tcp: copied 0 0 size 48 xmit
> tcp: copied 0 48 size 48 xmit
> tcp: in 52 bytes
> tcp: skb f446c780 ptr=f6541054 avail=52
> tcp: copied 0 0 size 48 recv
> tcp: iscsi_tcp_segment_recv copying 48
> tcp: copied 0 48 size 48 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 48 total size 48
> tcp: segment done
> tcp: opcode 0x25 ahslen 0 datalen 4
> tcp: iscsi_tcp_begin_data_in(f751ca00, offset=0, datalen=4)
> tcp: skb f446c780 ptr=f6541084 avail=4
> tcp: copied 0 0 size 4 recv
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 4
> tcp: copied 0 4 size 4 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 4 total size 4
> tcp: segment done
> tcp: iscsi_tcp_hdr_recv_prep(f751ca00)
> tcp: no more data avail. Consumed 52
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 48 total size 48
> tcp: Header done. Next segment size 0 total_size 0
> tcp: copied 0 0 size 0 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 0 total size 0
> tcp: xmit 48 bytes
> sd 3:0:0:0: [sdb] Write Protect is off
> sd 3:0:0:0: [sdb] Mode Sense: 77 00 10 08
> tcp: iscsi_tcp_send_hdr_prep(f751ca00)
> tcp: copied 0 0 size 48 xmit
> tcp: copied 0 48 size 48 xmit
> tcp: in 52 bytes
> tcp: skb f446c6c0 ptr=f6542854 avail=52
> tcp: copied 0 0 size 48 recv
> tcp: iscsi_tcp_segment_recv copying 48
> tcp: copied 0 48 size 48 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 48 total size 48
> tcp: segment done
> tcp: opcode 0x25 ahslen 0 datalen 4
> tcp: iscsi_tcp_begin_data_in(f751ca00, offset=0, datalen=4)
> tcp: skb f446c6c0 ptr=f6542884 avail=4
> tcp: copied 0 0 size 4 recv
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 4
> tcp: copied 0 4 size 4 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 4 total size 4
> tcp: segment done
> tcp: iscsi_tcp_hdr_recv_prep(f751ca00)
> tcp: no more data avail. Consumed 52
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 48 total size 48
> tcp: Header done. Next segment size 0 total_size 0
> tcp: copied 0 0 size 0 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 0 total size 0
> tcp: xmit 48 bytes
> tcp: iscsi_tcp_send_hdr_prep(f751ca00)
> tcp: copied 0 0 size 48 xmit
> tcp: copied 0 48 size 48 xmit
> tcp: in 80 bytes
> tcp: skb f446c600 ptr=f6542054 avail=80
> tcp: copied 0 0 size 48 recv
> tcp: iscsi_tcp_segment_recv copying 48
> tcp: copied 0 48 size 48 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 48 total size 48
> tcp: segment done
> tcp: opcode 0x25 ahslen 0 datalen 32
> tcp: iscsi_tcp_begin_data_in(f751ca00, offset=0, datalen=32)
> tcp: skb f446c600 ptr=f6542084 avail=32
> tcp: copied 0 0 size 32 recv
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 32
> tcp: copied 0 32 size 32 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 32 total size 32
> tcp: segment done
> tcp: iscsi_tcp_hdr_recv_prep(f751ca00)
> tcp: no more data avail. Consumed 80
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 48 total size 48
> tcp: Header done. Next segment size 0 total_size 0
> tcp: copied 0 0 size 0 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 0 total size 0
> tcp: xmit 48 bytes
> sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and
> FUA
> tcp: iscsi_tcp_send_hdr_prep(f751ca00)
> tcp: copied 0 0 size 48 xmit
> tcp: copied 0 48 size 48 xmit
> tcp: in 48 bytes
> tcp: skb f446c540 ptr=f6543854 avail=48
> tcp: copied 0 0 size 48 recv
> tcp: iscsi_tcp_segment_recv copying 48
> tcp: copied 0 48 size 48 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 48 total size 48
> tcp: segment done
> tcp: opcode 0x21 ahslen 0 datalen 0
> tcp: iscsi_tcp_hdr_recv_prep(f751ca00)
> tcp: no more data avail. Consumed 48
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 48 total size 48
> tcp: Header done. Next segment size 0 total_size 0
> tcp: copied 0 0 size 0 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 0 total size 0
> tcp: xmit 48 bytes
> tcp: iscsi_tcp_send_hdr_prep(f751ca00)
> tcp: copied 0 0 size 48 xmit
> tcp: copied 0 48 size 48 xmit
> tcp: in 56 bytes
> tcp: skb f446c480 ptr=f6543054 avail=56
> tcp: copied 0 0 size 48 recv
> tcp: iscsi_tcp_segment_recv copying 48
> tcp: copied 0 48 size 48 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 48 total size 48
> tcp: segment done
> tcp: opcode 0x25 ahslen 0 datalen 8
> tcp: iscsi_tcp_begin_data_in(f751ca00, offset=0, datalen=8)
> tcp: skb f446c480 ptr=f6543084 avail=8
> tcp: copied 0 0 size 8 recv
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 8
> tcp: copied 0 8 size 8 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 8 total size 8
> tcp: segment done
> tcp: iscsi_tcp_hdr_recv_prep(f751ca00)
> tcp: no more data avail. Consumed 56
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 48 total size 48
> tcp: Header done. Next segment size 0 total_size 0
> tcp: copied 0 0 size 0 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 0 total size 0
> tcp: xmit 48 bytes
> sd 3:0:0:0: [sdb] 190421401 512-byte hardware sectors (97496 MB)
> tcp: iscsi_tcp_send_hdr_prep(f751ca00)
> tcp: copied 0 0 size 48 xmit
> tcp: copied 0 48 size 48 xmit
> tcp: in 52 bytes
> tcp: skb f446c3c0 ptr=f6544854 avail=52
> tcp: copied 0 0 size 48 recv
> tcp: iscsi_tcp_segment_recv copying 48
> tcp: copied 0 48 size 48 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 48 total size 48
> tcp: segment done
> tcp: opcode 0x25 ahslen 0 datalen 4
> tcp: iscsi_tcp_begin_data_in(f751ca00, offset=0, datalen=4)
> tcp: skb f446c3c0 ptr=f6544884 avail=4
> tcp: copied 0 0 size 4 recv
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 4
> tcp: copied 0 4 size 4 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 4 total size 4
> tcp: segment done
> tcp: iscsi_tcp_hdr_recv_prep(f751ca00)
> tcp: no more data avail. Consumed 52
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 48 total size 48
> tcp: Header done. Next segment size 0 total_size 0
> tcp: copied 0 0 size 0 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 0 total size 0
> tcp: xmit 48 bytes
> sd 3:0:0:0: [sdb] Write Protect is off
> sd 3:0:0:0: [sdb] Mode Sense: 77 00 10 08
> tcp: iscsi_tcp_send_hdr_prep(f751ca00)
> tcp: copied 0 0 size 48 xmit
> tcp: copied 0 48 size 48 xmit
> tcp: in 52 bytes
> tcp: skb f446c300 ptr=f6544054 avail=52
> tcp: copied 0 0 size 48 recv
> tcp: iscsi_tcp_segment_recv copying 48
> tcp: copied 0 48 size 48 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 48 total size 48
> tcp: segment done
> tcp: opcode 0x25 ahslen 0 datalen 4
> tcp: iscsi_tcp_begin_data_in(f751ca00, offset=0, datalen=4)
> tcp: skb f446c300 ptr=f6544084 avail=4
> tcp: copied 0 0 size 4 recv
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 4
> tcp: copied 0 4 size 4 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 4 total size 4
> tcp: segment done
> tcp: iscsi_tcp_hdr_recv_prep(f751ca00)
> tcp: no more data avail. Consumed 52
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 48 total size 48
> tcp: Header done. Next segment size 0 total_size 0
> tcp: copied 0 0 size 0 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 0 total size 0
> tcp: xmit 48 bytes
> tcp: iscsi_tcp_send_hdr_prep(f751ca00)
> tcp: copied 0 0 size 48 xmit
> tcp: copied 0 48 size 48 xmit
> tcp: in 80 bytes
> tcp: skb f446c240 ptr=f6545854 avail=80
> tcp: copied 0 0 size 48 recv
> tcp: iscsi_tcp_segment_recv copying 48
> tcp: copied 0 48 size 48 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 48 total size 48
> tcp: segment done
> tcp: opcode 0x25 ahslen 0 datalen 32
> tcp: iscsi_tcp_begin_data_in(f751ca00, offset=0, datalen=32)
> tcp: skb f446c240 ptr=f6545884 avail=32
> tcp: copied 0 0 size 32 recv
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 32
> tcp: copied 0 32 size 32 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 32 total size 32
> tcp: segment done
> tcp: iscsi_tcp_hdr_recv_prep(f751ca00)
> tcp: no more data avail. Consumed 80
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 48 total size 48
> tcp: Header done. Next segment size 0 total_size 0
> tcp: copied 0 0 size 0 xmit
> tcp: iscsi_tcp_segment_unmap f751cb4c
> tcp: total copied 0 total size 0
> tcp: xmit 48 bytes
> sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and
> FUA
>  sdb:<6>tcp: iscsi_tcp_send_hdr_prep(f751ca00)
> tcp: copied 0 0 size 48 xmit
> tcp: copied 0 48 size 48 xmit
> tcp: in 1448 bytes
> tcp: skb f446c180 ptr=f6545054 avail=1448
> tcp: copied 0 0 size 48 recv
> tcp: iscsi_tcp_segment_recv copying 48
> tcp: copied 0 48 size 48 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: total copied 48 total size 48
> tcp: segment done
> tcp: opcode 0x25 ahslen 0 datalen 4096
> tcp: iscsi_tcp_begin_data_in(f751ca00, offset=0, datalen=4096)
> tcp: skb f446c180 ptr=f6545084 avail=1400
> tcp: copied 0 0 size 512 recv
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 512
> tcp: copied 0 512 size 512 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 512 total size 4096
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 512
> tcp: copied 0 512 size 512 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 1024 total size 4096
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 376
> tcp: copied 0 376 size 512 recv
> tcp: iscsi_tcp_segment_recv copied 1400 bytes
> tcp: no more data avail. Consumed 1448
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: in 2696 bytes
> tcp: skb f446c0c0 ptr=f6546854 avail=1448
> tcp: copied 376 0 size 512 recv
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 136
> tcp: copied 376 136 size 512 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 1536 total size 4096
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 512
> tcp: copied 0 512 size 512 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 2048 total size 4096
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 512
> tcp: copied 0 512 size 512 recv
> tcp: iscsi_tcp_segment_unmap f751ca10
> tcp: iscsi_tcp_segment_unmap valid
> tcp: total copied 2560 total size 4096
> tcp: iscsi_tcp_segment_map recv f751ca10
> tcp: iscsi_tcp_segment_recv copying 288
> tcp: copied 0 288 size 512 recv
> tcp: iscsi_tcp_segment_recv copied 1448 bytes
> tcp: skb f446c0c0 ptr=000005a8 avail=141842776
> tcp: copied 288 0 size 512 recv
> tcp: iscsi_tcp_segment_recv copying 224
> BUG: unable to handle kernel NULL pointer dereference at 000005a8
> IP: [<f8de64b2>] :iscsi_tcp:iscsi_tcp_recv+0x161/0x473
> *pdpt = 0000000036533001 *pde = 0000000000000000
> Oops: 0000 [#1] SMP
> Modules linked in: crc32c libcrc32c iscsi_tcp libiscsi scsi_transport_iscsi
> ixgbe netconsole inet_lro ipv6 af_packet button battery ac loop usbhid
> ff_memless ehci_hcd uhci_hcd usbcore dm_mod bnx2 ext3 jbd edd fan thermal
> processor thermal_sys sg megaraid_sas ata_piix libata dock piix sd_mod
> scsi_mod
> ide_disk ide_core [last unloaded: iscsi_tcp]
> 
> Pid: 0, comm: swapper Not tainted (2.6.26-bigsmp #1)
> EIP: 0060:[<f8de64b2>] EFLAGS: 00010202 CPU: 3
> EIP is at iscsi_tcp_recv+0x161/0x473 [iscsi_tcp]
> EAX: 0000002b EBX: f747dd48 ECX: 00000038 EDX: 00000000
> ESI: 000005a8 EDI: f593db20 EBP: f751ca10 ESP: f747dd20
>  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> Process swapper (pid: 0, ti=f747c000 task=f745abe0 task.ti=f747c000)
> Stack: f8de78e7 000000e0 f446c0c0 f6c35544 f751ca00 000005a8 00000000
> 000000e0
>        000005a8 08745958 00000000 00000a88 00000000 000005a8 f446c0c0
>        f78ba0ac
>        00000000 c0289617 00000000 00000000 05a80001 00007fff f78ba040
>        000005a8
> Call Trace:
>  [<c0289617>] tcp_ack+0x15bd/0x1757
>  [<c028391e>] tcp_read_sock+0x8c/0x1e0
>  [<f8de6351>] iscsi_tcp_recv+0x0/0x473 [iscsi_tcp]
>  [<f8de716a>] iscsi_tcp_data_ready+0x36/0x80 [iscsi_tcp]
>  [<c028d1a2>] tcp_send_ack+0xab/0xaf
>  [<c028c02e>] tcp_rcv_established+0x3b3/0x639
>  [<c02909fb>] tcp_v4_do_rcv+0x22/0x16f
>  [<c0292294>] tcp_v4_rcv+0x512/0x562
>  [<c027b921>] ip_local_deliver_finish+0xb2/0x14a
>  [<c027b852>] ip_rcv_finish+0x286/0x2a3
>  [<f8ce9a93>] packet_rcv_spkt+0xb6/0xbd [af_packet]
>  [<c0261889>] netif_receive_skb+0x2d0/0x33b
>  [<f8afd5ca>] lro_flush+0x314/0x340 [inet_lro]
>  [<f8afd636>] lro_flush_all+0x1b/0x28 [inet_lro]
>  [<f8b410eb>] ixgbe_clean_rx_irq+0x73b/0x850 [ixgbe]
>  [<f8b44183>] ixgbe_clean_rxonly+0x53/0xd0 [ixgbe]
>  [<c0263521>] net_rx_action+0x8a/0x152
>  [<c0124c6e>] __do_softirq+0x5d/0xc1
>  [<c0124d04>] do_softirq+0x32/0x36
>  [<c010663a>] do_IRQ+0x73/0x85
>  [<c0109152>] mwait_idle+0x0/0x32
>  [<c0105143>] common_interrupt+0x23/0x28
>  [<c0109152>] mwait_idle+0x0/0x32
>  [<c0109181>] mwait_idle+0x2f/0x32
>  [<c0103535>] cpu_idle+0x88/0x9c
>  =======================
> Code: 24 14 0f 46 44 24 14 89 44 24 14 50 68 e7 78 de f8 e8 2e b3 33 c7 8b 7d
> 08 03 7d 00 8b 4c 24 1c 8b 74 24 20 03 74 24 18 c1 e9 02 <f3> a5 8b 4c 24 1c
> 83
> e1 03 74 02 f3 a4 8b 4c 24 1c 01 4c 24 18
> EIP: [<f8de64b2>] iscsi_tcp_recv+0x161/0x473 [iscsi_tcp] SS:ESP 0068:f747dd20
> Kernel panic - not syncing: Fatal exception in interrupt
> 

Andrew Morton wrote:
> (switched to email.  Please respond via emailed reply-to-all, not via
> the bugzilla web interface).

This panic is still unaddressed.  Can I provide more information?



 
> (I'll reassign this to scsi).
> 
> On Tue, 21 Oct 2008 17:17:03 -0700 (PDT)
> bugme-daemon@bugzilla.kernel.org wrote:
> 
>> http://bugzilla.kernel.org/show_bug.cgi?id=11804
>> 
>>            Summary: iscsi: LRO plus iSCSI causes panic
>>            Product: Networking
>>            Version: 2.5
>>      KernelVersion: 2.6.26
>>           Platform: All
>>         OS/Version: Linux
>>               Tree: Mainline
>>             Status: NEW
>>           Severity: normal
>>           Priority: P1
>>          Component: Other
>>         AssignedTo: acme@ghostprotocols.net
>>         ReportedBy: jesse.brandeburg@intel.com
>> 
>> 
>> Distribution: SuSE SLES 10.2
>> Hardware Environment:
>> .config is available on request, we are using i686 arch, dell 2950,
>> ixgbe adapter, inet_lro module. 
>> 
>> Software Environment: none/gnome-session running yast2
>> 
>> Problem Description:
>> We found that just trying to connect with no authentication to an
>> iSCSI 
>> target over an adapter running either the in-kernel LRO or an
>> in-driver version will cause this panic.

Reply-To: akpm@linux-foundation.org

On Thu, 6 Nov 2008 16:08:08 -0800
"Brandeburg, Jesse" <jesse.brandeburg@intel.com> wrote:

> Andrew Morton wrote:
> > (switched to email.  Please respond via emailed reply-to-all, not via
> > the bugzilla web interface).
> 
> This panic is still unaddressed.

It looks like we own it :(

>  Can I provide more information?
> 
> >> http://bugzilla.kernel.org/show_bug.cgi?id=11804

I guess it would be useful to use addr2line or gdb or whatever to work
out exactly which statement is faulting, which pointer contains the
garbage value, etc.

Reply-To: michaelc@cs.wisc.edu

Brandeburg, Jesse wrote:
> Andrew Morton wrote:
>> (switched to email.  Please respond via emailed reply-to-all, not via
>> the bugzilla web interface).
> 
> This panic is still unaddressed.  Can I provide more information?
> 

Sorry this was my fault. I misread your first mail. I am trying to 
replicate this now here.

also may be able to reproduce using the skb text search functionality (the only other caller of skb_seq_read)

several patches were committed to fix this issue.  Currently in net-next

http://git.kernel.org/?p=linux/kernel/git/davem/net-next-2.6.git;a=commitdiff;h=95e3b24cfb4ec0479d2c42f7a1780d68063a542a

http://git.kernel.org/?p=linux/kernel/git/davem/net-next-2.6.git;a=commitdiff;h=71b3346d182355f19509fadb8fe45114a35cc499

Thank you for everyone who worked on this