Bug 39742

Summary:	3.0.0 (2.6.39.3 also) crash BUG at include/linux/skbuff.h:1189! with igb-RSS and L2TP PPTP
Product:	Networking	Reporter:	cpulinker
Component:	IPV4	Assignee:	Stephen Hemminger (stephen)
Status:	RESOLVED OBSOLETE
Severity:	normal	CC:	alan, xeb
Priority:	P1
Hardware:	All
OS:	Linux
Kernel Version:	2.6.39.3-3.0.0	Subsystem:
Regression:	No	Bisected commit-id:
Attachments:	full trap log, dmesg, iptables, lsmod, ipset

Description cpulinker 2011-07-22 10:32:41 UTC

Created attachment 66342 [details]
full trap log, dmesg, iptables, lsmod, ipset

Hi!
I set up the server with the new software. It must serves users with VPN connections (l2tp,pptp - kernel mode).
As soon as I start the users it hangs. With latest drivers from Intel (igb) and the latest kernel (2.6.39.3)
it hangs (rarely) or a kernel panic (most often) in 1-2 minutes,
when users log on to the NAS server and starts to go traffic on VPN connections.
I tried to turn off the incoming shaper on the ifb, disable, conntrack(-j NOTRACK in the RAW table),
was trying to use only PPTP - the kernel still hangs or falls into a panic.
Hardare is ASUS RS100 - Intel P4 Q6600, x86_64, 4GB RAM, SATA HDD, Intel 82576 2 ports (igb with enabled RSS)
Main software:
Kernel is 2.6.39 upped to the 2.6.39.3 by patch from ftp://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/files/SRPMS/kernel-image-std-def-2.6.39-alt3.src.rpm

iproute2 2.6.39
ipset 4.4
igb 3.1.16
iptables 1.4.10
git://accel-ppp.git.sourceforge.net/gitroot/accel-ppp/accel-ppp

[ 6089.358502] BUG: unable to handle kernel NULL pointer dereference at           (null)
[ 6089.362482] IP: [<ffffffffa0264e24>] __nf_conntrack_confirm+0x2b4/0x480 [nf_conntrack]
[ 6089.362482] PGD 117822067 PUD 117823067 PMD 0
[ 6089.362482] Oops: 0002 [#1] SMP
[ 6089.362482] last sysfs file: /sys/devices/virtual/net/ppp6/uevent
[ 6089.362482] CPU 1
[ 6089.362482] Modules linked in: act_mirred act_skbedit cls_u32 sch_ingress arc4 ecb ppp_mppe l2tp_ppp l2tp_netlink l2tpd
[ 6089.362482]
[ 6089.362482] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-std-def-alt3 #1 ASUS RS100-E4/PI2/P5M2-M/RS100-E4
[ 6089.362482] RIP: 0010:[<ffffffffa0264e24>]  [<ffffffffa0264e24>] __nf_conntrack_confirm+0x2b4/0x480 [nf_conntrack]
[ 6089.362482] RSP: 0018:ffff88011fc83a80  EFLAGS: 00010202
[ 6089.362482] RAX: 00000000000047cd RBX: ffff88011fc94620 RCX: 0000000000000000
[ 6089.362482] RDX: 000000000000fa10 RSI: 00000000d00dd719 RDI: ffffffffa026f680
[ 6089.362482] RBP: ffff88011fc83ac0 R08: 00000000e753847a R09: ffff880117f00000
[ 6089.362482] R10: 0000000000004000 R11: 0000000000000001 R12: 0000000000000000
[ 6089.362482] R13: ffff880116a95080 R14: ffffffff81a4e1c0 R15: 00000000000114d8
[ 6089.362482] FS:  0000000000000000(0000) GS:ffff88011fc80000(0000) knlGS:0000000000000000
[ 6089.362482] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 6089.362482] CR2: 0000000000000000 CR3: 0000000117821000 CR4: 00000000000006e0
[ 6089.362482] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6089.362482] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 6089.362482] Process kworker/0:0 (pid: 0, threadinfo ffff880118ef6000, task ffff880118ef42c0)
[ 6089.362482] Stack:
[ 6089.362482]  000000007d09da80 0000400016bd4400 0000000116ea2000 ffff880116a95080
[ 6089.362482]  0000000000000001 ffff88011fc94620 0000000000000002 ffff880116ea2000
[ 6089.362482]  ffff88011fc83b30 ffffffffa02b1dd8 ffff880116ea2000 0000000000000000
[ 6089.362482] Call Trace:
[ 6089.362482]  <IRQ>
[ 6089.362482]  [<ffffffffa02b1dd8>] ipv4_confirm+0x188/0x1c0 [nf_conntrack_ipv4]
[ 6089.362482]  [<ffffffff81382004>] nf_iterate+0x84/0xa0
[ 6089.362482]  [<ffffffff81389390>] ? ip_rcv_finish+0x390/0x390
[ 6089.362482]  [<ffffffff81382096>] nf_hook_slow+0x76/0x130
[ 6089.362482]  [<ffffffff81389390>] ? ip_rcv_finish+0x390/0x390
[ 6089.362482]  [<ffffffff813897d7>] ip_local_deliver+0x67/0x90
[ 6089.362482]  [<ffffffff81389135>] ip_rcv_finish+0x135/0x390
[ 6089.362482]  [<ffffffff81389a1c>] ip_rcv+0x21c/0x2e0
[ 6089.362482]  [<ffffffff81356f5a>] __netif_receive_skb+0x52a/0x690
[ 6089.362482]  [<ffffffff813572d0>] netif_receive_skb+0x60/0x90
[ 6089.362482]  [<ffffffff8124de9c>] ? is_swiotlb_buffer+0x3c/0x50
[ 6089.362482]  [<ffffffff81357440>] napi_skb_finish+0x50/0x70
[ 6089.362482]  [<ffffffff813579bd>] napi_gro_receive+0xbd/0xd0
[ 6089.362482]  [<ffffffffa012129b>] igb_poll+0x6fb/0xae0 [igb]
[ 6089.362482]  [<ffffffff8107fb61>] ? enqueue_hrtimer+0x31/0x80
[ 6090.302391]  [<ffffffff81357be5>] net_rx_action+0x135/0x270
[ 6090.302391]  [<ffffffff81062705>] __do_softirq+0xa5/0x1d0
[ 6090.302391]  [<ffffffff8141301c>] call_softirq+0x1c/0x30
[ 6090.302391]  [<ffffffff8100d355>] do_softirq+0x65/0xa0
[ 6090.302391]  [<ffffffff81062a96>] irq_exit+0x86/0xa0
[ 6090.302391]  [<ffffffff8100cf01>] do_IRQ+0x61/0xe0
[ 6090.302391]  [<ffffffff8140a593>] common_interrupt+0x13/0x13
[ 6090.302391]  <EOI>
[ 6090.302391]  [<ffffffff81012deb>] ? mwait_idle+0x9b/0x1d0
[ 6090.302391]  [<ffffffff8140de35>] ? atomic_notifier_call_chain+0x15/0x20
[ 6090.302391]  [<ffffffff8100a1e6>] cpu_idle+0x56/0xa0
[ 6090.302391]  [<ffffffff81403729>] start_secondary+0x197/0x19c
[ 6090.302391] Code: ff 74 09 40 0f b6 ff 41 0f b7 0c 38 66 41 39 cc 0f 84 f1 fe ff ff 48 8b 00 a8 01 0f 84 51 ff ff ff 4

Comment 1 Andrew Morton 2011-07-22 21:42:54 UTC

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Fri, 22 Jul 2011 10:32:45 GMT
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=39742
> 
>            Summary: 2.6.39.3 crash and hangs in 1-2 minutes with igb-RSS
>                     and L2TP PPTP services on the NAS server
>            Product: Networking
>            Version: 2.5
>     Kernel Version: 2.6.39.3
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: IPV4
>         AssignedTo: shemminger@linux-foundation.org
>         ReportedBy: cpulinker@yandex.ru
>         Regression: No
> 
> 
> Created an attachment (id=66342)
>  --> (https://bugzilla.kernel.org/attachment.cgi?id=66342)
> full trap log, dmesg, iptables, lsmod, ipset
> 
> Hi!
> I set up the server with the new software. It must serves users with VPN
> connections (l2tp,pptp - kernel mode).
> As soon as I start the users it hangs. With latest drivers from Intel (igb)
> and
> the latest kernel (2.6.39.3)
> it hangs (rarely) or a kernel panic (most often) in 1-2 minutes,
> when users log on to the NAS server and starts to go traffic on VPN
> connections.
> I tried to turn off the incoming shaper on the ifb, disable, conntrack(-j
> NOTRACK in the RAW table),
> was trying to use only PPTP - the kernel still hangs or falls into a panic.
> Hardare is ASUS RS100 - Intel P4 Q6600, x86_64, 4GB RAM, SATA HDD, Intel
> 82576
> 2 ports (igb with enabled RSS)
> Main software:
> Kernel is 2.6.39 upped to the 2.6.39.3 by patch from
>
> ftp://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/files/SRPMS/kernel-image-std-def-2.6.39-alt3.src.rpm
> 
> iproute2 2.6.39
> ipset 4.4
> igb 3.1.16
> iptables 1.4.10
> git://accel-ppp.git.sourceforge.net/gitroot/accel-ppp/accel-ppp
> 
> [ 6089.358502] BUG: unable to handle kernel NULL pointer dereference at       
>   (null)
> [ 6089.362482] IP: [<ffffffffa0264e24>] __nf_conntrack_confirm+0x2b4/0x480
> [nf_conntrack]
> [ 6089.362482] PGD 117822067 PUD 117823067 PMD 0
> [ 6089.362482] Oops: 0002 [#1] SMP
> [ 6089.362482] last sysfs file: /sys/devices/virtual/net/ppp6/uevent
> [ 6089.362482] CPU 1
> [ 6089.362482] Modules linked in: act_mirred act_skbedit cls_u32 sch_ingress
> arc4 ecb ppp_mppe l2tp_ppp l2tp_netlink l2tpd
> [ 6089.362482]
> [ 6089.362482] Pid: 0, comm: kworker/0:0 Not tainted 2.6.39-std-def-alt3 #1
> ASUS RS100-E4/PI2/P5M2-M/RS100-E4
> [ 6089.362482] RIP: 0010:[<ffffffffa0264e24>]  [<ffffffffa0264e24>]
> __nf_conntrack_confirm+0x2b4/0x480 [nf_conntrack]
> [ 6089.362482] RSP: 0018:ffff88011fc83a80  EFLAGS: 00010202
> [ 6089.362482] RAX: 00000000000047cd RBX: ffff88011fc94620 RCX:
> 0000000000000000
> [ 6089.362482] RDX: 000000000000fa10 RSI: 00000000d00dd719 RDI:
> ffffffffa026f680
> [ 6089.362482] RBP: ffff88011fc83ac0 R08: 00000000e753847a R09:
> ffff880117f00000
> [ 6089.362482] R10: 0000000000004000 R11: 0000000000000001 R12:
> 0000000000000000
> [ 6089.362482] R13: ffff880116a95080 R14: ffffffff81a4e1c0 R15:
> 00000000000114d8
> [ 6089.362482] FS:  0000000000000000(0000) GS:ffff88011fc80000(0000)
> knlGS:0000000000000000
> [ 6089.362482] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 6089.362482] CR2: 0000000000000000 CR3: 0000000117821000 CR4:
> 00000000000006e0
> [ 6089.362482] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 6089.362482] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 6089.362482] Process kworker/0:0 (pid: 0, threadinfo ffff880118ef6000, task
> ffff880118ef42c0)
> [ 6089.362482] Stack:
> [ 6089.362482]  000000007d09da80 0000400016bd4400 0000000116ea2000
> ffff880116a95080
> [ 6089.362482]  0000000000000001 ffff88011fc94620 0000000000000002
> ffff880116ea2000
> [ 6089.362482]  ffff88011fc83b30 ffffffffa02b1dd8 ffff880116ea2000
> 0000000000000000
> [ 6089.362482] Call Trace:
> [ 6089.362482]  <IRQ>
> [ 6089.362482]  [<ffffffffa02b1dd8>] ipv4_confirm+0x188/0x1c0
> [nf_conntrack_ipv4]
> [ 6089.362482]  [<ffffffff81382004>] nf_iterate+0x84/0xa0
> [ 6089.362482]  [<ffffffff81389390>] ? ip_rcv_finish+0x390/0x390
> [ 6089.362482]  [<ffffffff81382096>] nf_hook_slow+0x76/0x130
> [ 6089.362482]  [<ffffffff81389390>] ? ip_rcv_finish+0x390/0x390
> [ 6089.362482]  [<ffffffff813897d7>] ip_local_deliver+0x67/0x90
> [ 6089.362482]  [<ffffffff81389135>] ip_rcv_finish+0x135/0x390
> [ 6089.362482]  [<ffffffff81389a1c>] ip_rcv+0x21c/0x2e0
> [ 6089.362482]  [<ffffffff81356f5a>] __netif_receive_skb+0x52a/0x690
> [ 6089.362482]  [<ffffffff813572d0>] netif_receive_skb+0x60/0x90
> [ 6089.362482]  [<ffffffff8124de9c>] ? is_swiotlb_buffer+0x3c/0x50
> [ 6089.362482]  [<ffffffff81357440>] napi_skb_finish+0x50/0x70
> [ 6089.362482]  [<ffffffff813579bd>] napi_gro_receive+0xbd/0xd0
> [ 6089.362482]  [<ffffffffa012129b>] igb_poll+0x6fb/0xae0 [igb]
> [ 6089.362482]  [<ffffffff8107fb61>] ? enqueue_hrtimer+0x31/0x80
> [ 6090.302391]  [<ffffffff81357be5>] net_rx_action+0x135/0x270
> [ 6090.302391]  [<ffffffff81062705>] __do_softirq+0xa5/0x1d0
> [ 6090.302391]  [<ffffffff8141301c>] call_softirq+0x1c/0x30
> [ 6090.302391]  [<ffffffff8100d355>] do_softirq+0x65/0xa0
> [ 6090.302391]  [<ffffffff81062a96>] irq_exit+0x86/0xa0
> [ 6090.302391]  [<ffffffff8100cf01>] do_IRQ+0x61/0xe0
> [ 6090.302391]  [<ffffffff8140a593>] common_interrupt+0x13/0x13
> [ 6090.302391]  <EOI>
> [ 6090.302391]  [<ffffffff81012deb>] ? mwait_idle+0x9b/0x1d0
> [ 6090.302391]  [<ffffffff8140de35>] ? atomic_notifier_call_chain+0x15/0x20
> [ 6090.302391]  [<ffffffff8100a1e6>] cpu_idle+0x56/0xa0
> [ 6090.302391]  [<ffffffff81403729>] start_secondary+0x197/0x19c
> [ 6090.302391] Code: ff 74 09 40 0f b6 ff 41 0f b7 0c 38 66 41 39 cc 0f 84 f1
> fe ff ff 48 8b 00 a8 01 0f 84 51 ff ff ff 4
>

Comment 2 xeb 2011-07-23 15:42:04 UTC

Please check this message http://www.spinics.net/lists/netdev/msg170766.html maybe it is the reason.

Comment 3 cpulinker 2011-07-24 20:18:23 UTC

Andrew Morton wrote:
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
I apply patch from this thread.
-----------------------------------------
--- Comment #2 from xeb@mail.ru  2011-07-23 15:42:04 ---
Please check this message http://www.spinics.net/lists/netdev/msg170766.html
maybe it is the reason.
-----------------------------------------
Now :
[root@nas3 ~]# uptime
  23:36:49 up 1 day, 41 min,  1 user,  load average: 0.00, 0.01, 0.05
[root@nas3 ~]#
Seems it work! There is now strange hangs and panic.

Comment 4 cpulinker 2011-08-01 11:58:23 UTC

With applied patch from xeb 
(http://www.spinics.net/lists/netdev/msg170766.html)
there isn't hangs any more. Panics is still occurs. They only occur much 
less frequently. And seems it's different place.
Here it is.
1 was like:
[ 1582.319303] ------------[ cut here ]------------
[ 1582.323205] kernel BUG at include/linux/skbuff.h:1189!
[ 1582.323205] invalid opcode: 0000 [#1] SMP
[ 1582.323205] last sysfs file: /sys/devices/virtual/net/ppp168/uevent
[ 1582.323205] CPU 0
[ 1582.323205] Modules linked in: cls_fw sch_sfq arc4 ecb ppp_mppe 
act_mirred act_skbedit cls_u32 sch_ingress l2tp_ppp l2d
[ 1582.323205]
[ 1582.598100] Pid: 0, comm: swapper Not tainted 2.6.39-std-def-alt3 #1 
ASUS RS100-E4/PI2/P5M2-M/RS100-E4
[ 1582.616208] RIP: 0010:[<ffffffff8134a94b>]  [<ffffffff8134a94b>] 
skb_pull+0x2b/0x30
[ 1582.616208] RSP: 0018:ffff88011fc03b20  EFLAGS: 00010293
[ 1582.616208] RAX: 0000000000000247 RBX: ffff880117ac2880 RCX: 
0000000000000027
[ 1582.616208] RDX: 0000000000000027 RSI: 0000000000000002 RDI: 
ffff880117ac2880
[ 1582.698595] RBP: ffff88011fc03b20 R08: 0000000000000228 R09: 
000000000000000c
[ 1582.698595] R10: 0000000000000240 R11: 0000000000000001 R12: 
ffff880118f00800
[ 1582.698595] R13: ffff8801176a086e R14: 000000000000002f R15: 
ffff880115346000
[ 1582.698595] FS:  0000000000000000(0000) GS:ffff88011fc00000(0000) 
knlGS:0000000000000000
[ 1582.698595] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1582.808009] CR2: 0000000000645f4c CR3: 000000011760b000 CR4: 
00000000000006f0
[ 1582.808009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[ 1582.808009] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[ 1582.808009] Process swapper (pid: 0, threadinfo ffffffff81800000, 
task ffffffff8180b020)
[ 1582.808009] Stack:
[ 1582.808009]  ffff88011fc03b50 ffffffffa034663a ffff88011fc03b80 
ffff880118f00800
[ 1582.808009]  ffff880117ac2880 0000000000000000 ffff88011fc03b90 
ffffffff81346ecc
[ 1582.808009]  ffff88011fc03bb0 ffffffff8134b4e9 0000000000000001 
ffff880117ac2880
[ 1582.808009] Call Trace:
[ 1582.808009]  <IRQ>
[ 1582.808009]  [<ffffffffa034663a>] pptp_rcv_core+0x21a/0x220 [pptp]
[ 1582.808009]  [<ffffffff81346ecc>] sk_receive_skb+0x10c/0x140
[ 1582.808009]  [<ffffffff8134b4e9>] ? __pskb_pull_tail+0x59/0x3e0
[ 1582.808009]  [<ffffffffa0346783>] pptp_rcv+0x143/0x190 [pptp]
[ 1582.808009]  [<ffffffffa03250ad>] gre_rcv+0x5d/0x80 [gre]
[ 1582.808009]  [<ffffffff8138946d>] ip_local_deliver_finish+0xdd/0x2a0
[ 1582.808009]  [<ffffffff813897f0>] ip_local_deliver+0x80/0x90
[ 1582.808009]  [<ffffffff81389135>] ip_rcv_finish+0x135/0x390
[ 1582.808009]  [<ffffffff81389a1c>] ip_rcv+0x21c/0x2e0
[ 1582.808009]  [<ffffffff81356f5a>] __netif_receive_skb+0x52a/0x690
[ 1582.808009]  [<ffffffff8100cf01>] ? do_IRQ+0x61/0xe0
[ 1582.808009]  [<ffffffff8140a593>] ? common_interrupt+0x13/0x13
[ 1582.808009]  [<ffffffff813572d0>] netif_receive_skb+0x60/0x90
[ 1582.808009]  [<ffffffff8124de9c>] ? is_swiotlb_buffer+0x3c/0x50
[ 1582.808009]  [<ffffffff81357440>] napi_skb_finish+0x50/0x70
[ 1582.808009]  [<ffffffff813579bd>] napi_gro_receive+0xbd/0xd0
[ 1582.808009]  [<ffffffffa01b229b>] igb_poll+0x6fb/0xae0 [igb]
[ 1582.808009]  [<ffffffff81356f5a>] ? __netif_receive_skb+0x52a/0x690
[ 1582.808009]  [<ffffffff81357be5>] net_rx_action+0x135/0x270
[ 1582.808009]  [<ffffffff81062705>] __do_softirq+0xa5/0x1d0
[ 1582.808009]  [<ffffffff8141301c>] call_softirq+0x1c/0x30
[ 1582.808009]  [<ffffffff8100d355>] do_softirq+0x65/0xa0
[ 1582.808009]  [<ffffffff81062a96>] irq_exit+0x86/0xa0
[ 1582.808009]  [<ffffffff8100cf01>] do_IRQ+0x61/0xe0
[ 1582.808009]  [<ffffffff8140a593>] common_interrupt+0x13/0x13
[ 1582.808009]  <EOI>
[ 1582.808009]  [<ffffffff81012deb>] ? mwait_idle+0x9b/0x1d0
[ 1582.808009]  [<ffffffff8140de35>] ? atomic_notifier_call_chain+0x15/0x20
[ 1582.808009]  [<ffffffff8100a1e6>] cpu_idle+0x56/0xa0
[ 1582.808009]  [<ffffffff813f396d>] rest_init+0x6d/0x80
[ 1582.808009]  [<ffffffff8187fbe6>] start_kernel+0x392/0x39d
[ 1582.808009]  [<ffffffff8187f347>] x86_64_start_reservations+0x132/0x136
[ 1582.808009]  [<ffffffff8187f44c>] x86_64_start_kernel+0x101/0x110
[ 1582.808009] Code: 8b 47 68 55 48 89 e5 39 c6 77 1c 29 f0 3b 47 6c 89 
47 68 72 16 89 f0 48 03 87 e0 00 00 00 48 89 87 e
[ 1582.808009] RIP  [<ffffffff8134a94b>] skb_pull+0x2b/0x30
[ 1582.808009]  RSP <ffff88011fc03b20>
[ 1583.628015] ---[ end trace 6c39f3d0e04ed229 ]---
[ 1583.641960] Kernel panic - not syncing: Fatal exception in interrupt
[ 1583.661088] Pid: 0, comm: swapper Tainted: G      D 
2.6.39-std-def-alt3 #1
[ 1583.682846] Call Trace:
[ 1583.690254]  <IRQ>  [<ffffffff81407284>] panic+0x8c/0x197
[ 1583.706576]  [<ffffffff8140b4d2>] oops_end+0xe2/0xf0
[ 1583.721540]  [<ffffffff8100e876>] die+0x56/0x90
[ 1583.735215]  [<ffffffff8140abf4>] do_trap+0xc4/0x170
[ 1583.750191]  [<ffffffff8100bf00>] do_invalid_op+0x90/0xb0
[ 1583.766466]  [<ffffffff8134a94b>] ? skb_pull+0x2b/0x30
[ 1583.781965]  [<ffffffffa01f31d6>] ? ipt_do_table+0x256/0x670 [ip_tables]

other was:
[  245.268856] ------------[ cut here ]------------
[  245.272797] kernel BUG at include/linux/skbuff.h:1189!
[  245.272797] invalid opcode: 0000 [#1] SMP
[  245.272797] last sysfs file: /sys/devices/virtual/net/ppp100/uevent
[  245.272797] CPU 3
[  245.272797] Modules linked in: cls_fw sch_sfq arc4 ecb ppp_mppe 
act_mirred act_skbedit cls_u32 sch_ingress l2tp_ppp l2d
[  245.272797]
[  245.272797] Pid: 0, comm: kworker/0:1 Not tainted 2.6.39-std-def-alt3 
#1 ASUS RS100-E4/PI2/P5M2-M/RS100-E4
[  245.272797] RIP: 0010:[<ffffffff8134a94b>]  [<ffffffff8134a94b>] 
skb_pull+0x2b/0x30
[  245.272797] RSP: 0018:ffff88011fd83b20  EFLAGS: 00010297
[  245.272797] RAX: 0000000000000518 RBX: ffff880116a0e0c0 RCX: 
0000000000000444
[  245.272797] RDX: 0000000000000444 RSI: 0000000000000010 RDI: 
ffff880116a0e0c0
[  245.272797] RBP: ffff88011fd83b20 R08: 0000000000000500 R09: 
000000000000000c
[  245.272797] R10: 0000000000000240 R11: 0000000000000001 R12: 
ffff8801171fb400
[  245.272797] R13: ffff880116a0cc72 R14: 000000000000002f R15: 
ffff8801168de000
[  245.272797] FS:  0000000000000000(0000) GS:ffff88011fd80000(0000) 
knlGS:0000000000000000
[  245.272797] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  245.272797] CR2: 00000000008faa2c CR3: 00000001151c2000 CR4: 
00000000000006e0
[  245.272797] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[  245.272797] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[  245.272797] Process kworker/0:1 (pid: 0, threadinfo ffff880118f46000, 
task ffff880118f444c0)
[  245.272797] Stack:
[  245.272797]  ffff88011fd83b50 ffffffffa03374e2 ffff88011fd83b80 
ffff8801171fb400
[  245.272797]  ffff880116a0e0c0 0000000000000000 ffff88011fd83b90 
ffffffff81346ecc
[  245.892667]  ffff88011fd83bb0 ffffffff8134b4e9 0000000000000001 
ffff880116a0e0c0
[  245.892667] Call Trace:
[  245.892667]  <IRQ>
[  245.892667]  [<ffffffffa03374e2>] pptp_rcv_core+0xc2/0x220 [pptp]
[  245.892667]  [<ffffffff81346ecc>] sk_receive_skb+0x10c/0x140
[  245.892667]  [<ffffffff8134b4e9>] ? __pskb_pull_tail+0x59/0x3e0
[  245.892667]  [<ffffffffa0337783>] pptp_rcv+0x143/0x190 [pptp]
[  245.892667]  [<ffffffffa03160ad>] gre_rcv+0x5d/0x80 [gre]
[  245.892667]  [<ffffffff8138946d>] ip_local_deliver_finish+0xdd/0x2a0
[  245.892667]  [<ffffffff813897f0>] ip_local_deliver+0x80/0x90
[  245.892667]  [<ffffffff81389135>] ip_rcv_finish+0x135/0x390
[  245.892667]  [<ffffffff81389a1c>] ip_rcv+0x21c/0x2e0
[  245.892667]  [<ffffffff81356f5a>] __netif_receive_skb+0x52a/0x690
[  245.892667]  [<ffffffff81137985>] ? __kmalloc_node_track_caller+0x55/0x60
[  245.892667]  [<ffffffff813572d0>] netif_receive_skb+0x60/0x90
[  245.892667]  [<ffffffff8124de9c>] ? is_swiotlb_buffer+0x3c/0x50
[  245.892667]  [<ffffffff81357440>] napi_skb_finish+0x50/0x70
[  245.892667]  [<ffffffff813579bd>] napi_gro_receive+0xbd/0xd0
[  245.892667]  [<ffffffffa01ea29b>] igb_poll+0x6fb/0xae0 [igb]
[  245.892667]  [<ffffffff81356f5a>] ? __netif_receive_skb+0x52a/0x690
[  245.892667]  [<ffffffff81357be5>] net_rx_action+0x135/0x270
[  245.892667]  [<ffffffff81062705>] __do_softirq+0xa5/0x1d0
[  245.892667]  [<ffffffff8141301c>] call_softirq+0x1c/0x30
[  245.892667]  [<ffffffff8100d355>] do_softirq+0x65/0xa0
[  245.892667]  [<ffffffff81062a96>] irq_exit+0x86/0xa0
[  245.892667]  [<ffffffff8100cf01>] do_IRQ+0x61/0xe0
[  245.892667]  [<ffffffff8140a593>] common_interrupt+0x13/0x13
[  245.892667]  <EOI>
[  245.892667]  [<ffffffff81012deb>] ? mwait_idle+0x9b/0x1d0
[  245.892667]  [<ffffffff8140de35>] ? atomic_notifier_call_chain+0x15/0x20
[  245.892667]  [<ffffffff8100a1e6>] cpu_idle+0x56/0xa0
[  245.892667]  [<ffffffff81403729>] start_secondary+0x197/0x19c
[  245.892667] Code: 8b 47 68 55 48 89 e5 39 c6 77 1c 29 f0 3b 47 6c 89 
47 68 72 16 89 f0 48 03 87 e0 00 00 00 48 89 87 e
[  245.892667] RIP  [<ffffffff8134a94b>] skb_pull+0x2b/0x30
[  245.892667]  RSP <ffff88011fd83b20>
[  246.520989] ---[ end trace ad4c118e9d8ac857 ]---
[  246.534912] Kernel panic - not syncing: Fatal exception in interrupt
[  246.534916] Pid: 0, comm: kworker/0:1 Tainted: G      D 
2.6.39-std-def-alt3 #1

Comment 5 cpulinker 2011-08-03 15:29:20 UTC

I was trying using new kernel, but i have same bug and trap from the kernel:
------------------------------------------------
[98592.361121] ------------[ cut here ]------------
[98592.365040] kernel BUG at include/linux/skbuff.h:1189!
[98592.365040] invalid opcode: 0000 [#1] PREEMPT SMP
[98592.365040] CPU 1
[98592.365040] Modules linked in: cls_fw sch_sfq ip_set_hash_net 
ip_set_hash_ip arc4 ecb ppp_mppe xt_mark nf_conntrack_ipv
4 nf_defrag_ipv4 xt_state xt_TCPMSS ipt_LOG xt_recent xt_NOTRACK 
nf_conntrack xt_statistic ts_kmp xt_tcpudp xt_string xt_m
ultiport xt_set iptable_raw iptable_mangle iptable_filter ip_tables 
act_mirred act_skbedit cls_u32 sch_ingress ip_set nfne
tlink l2tp_ppp l2tp_netlink l2tp_core pptp pppox ppp_generic slhc gre 
ipt_ULOG x_tables 8021q garp stp cls_flow sch_htb if
b dm_multipath scsi_dh dm_mod tg3 i2c_i801 rtc_cmos psmouse pcspkr 
sr_mod ehci_hcd cdrom igb uhci_hcd evdev i2c_core sg se
rio_raw usbcore dca i3000_edac edac_core processor button ext3 jbd 
mbcache sd_mod crc_t10dif ide_pci_generic ide_core pata
_acpi ata_generic ata_piix libata scsi_mod
[98592.365040]
[98592.365040] Pid: 0, comm: kworker/0:0 Not tainted 3.0.0-un-def-alt2 
#1 ASUS RS100-E4/PI2/P5M2-M/RS100-E4
[98592.365040] RIP: 0010:[<ffffffff8136ca2b>]  [<ffffffff8136ca2b>] 
skb_pull+0x2b/0x30
[98592.365040] RSP: 0018:ffff88011fc83ab0  EFLAGS: 00010283
[98592.365040] RAX: 000000000000057e RBX: ffff880117f5f080 RCX: 
000000000000011f
[98592.365040] RDX: 000000000000011f RSI: 0000000000000002 RDI: 
ffff880117f5f080
[98592.365040] RBP: ffff88011fc83ab0 R08: 0000000000000000 R09: 
0000000000000102
[98592.365040] R10: 0000000000000000 R11: 0000000000000001 R12: 
ffff8800378bb000
[98592.365040] R13: ffff880117834c6e R14: 000000000000002f R15: 
ffffffff8168dd80
[98592.365040] FS:  0000000000000000(0000) GS:ffff88011fc80000(0000) 
knlGS:0000000000000000
[98592.365040] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[98592.365040] CR2: 00000000007ced6c CR3: 0000000117faa000 CR4: 
00000000000006e0
[98592.365040] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[98592.365040] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[98592.365040] Process kworker/0:0 (pid: 0, threadinfo ffff880118f5c000, 
task ffff880118f5a480)
[98592.365040] Stack:
[98592.365040]  ffff88011fc83ae0 ffffffffa027b7fa ffff88011fc83ae0 
ffff8800378bb000
[98592.365040]  ffff880117f5f080 ffff8800378bb050 ffff88011fc83b10 
ffffffff8136bddc
[98592.365040]  ffff88011fc83b40 ffff880117f5f080 ffff8800378bb000 
000000000c1a000a
[98592.365040] Call Trace:
[98592.365040]  <IRQ>
[98592.365040]  [<ffffffffa027b7fa>] pptp_rcv_core+0x21a/0x220 [pptp]
[98592.365040]  [<ffffffff8136bddc>] sk_receive_skb+0x13c/0x160
[98592.365040]  [<ffffffffa027b1be>] pptp_rcv+0x15e/0x1b0 [pptp]
[98592.365040]  [<ffffffffa0256133>] gre_rcv+0x73/0xa0 [gre]
[98592.365040]  [<ffffffff813af10d>] ip_local_deliver_finish+0xed/0x2c0
[98592.365040]  [<ffffffff813af360>] ip_local_deliver+0x80/0x90
[98592.365040]  [<ffffffff813ae959>] ip_rcv_finish+0x119/0x3b0
[98592.365040]  [<ffffffff813aef4d>] ip_rcv+0x21d/0x2f0
[98592.365040]  [<ffffffff8137cffc>] __netif_receive_skb+0x20c/0x6c0
[98592.365040]  [<ffffffff81141e15>] ? __kmalloc_node_track_caller+0x55/0x60
[98592.365040]  [<ffffffff8137dbcd>] netif_receive_skb+0xbd/0xd0
[98592.365040]  [<ffffffff8125fcfc>] ? is_swiotlb_buffer+0x3c/0x50
[98593.222041]  [<ffffffff8137dd30>] napi_skb_finish+0x50/0x70
[98593.222041]  [<ffffffff8137e315>] napi_gro_receive+0xc5/0xd0
[98593.222041]  [<ffffffffa01a8e0f>] igb_poll+0x6cf/0xb50 [igb]
[98593.222041]  [<ffffffff8137d004>] ? __netif_receive_skb+0x214/0x6c0
[98593.222041]  [<ffffffff8137d004>] ? __netif_receive_skb+0x214/0x6c0
[98593.222041]  [<ffffffff8137f0d4>] net_rx_action+0x164/0x340
[98593.222041]  [<ffffffff810684b5>] __do_softirq+0xd5/0x270
[98593.222041]  [<ffffffff8143c89c>] call_softirq+0x1c/0x30
[98593.222041]  [<ffffffff8100e6c5>] do_softirq+0x95/0xe0
[98593.222041]  [<ffffffff81068305>] irq_exit+0xd5/0xf0
[98593.222041]  [<ffffffff8100dec1>] do_IRQ+0x61/0xd0
[98593.222041]  [<ffffffff81434753>] common_interrupt+0x13/0x13
[98593.222041]  <EOI>
[98593.222041]  [<ffffffff81015936>] ? mwait_idle+0xe6/0x2b0
[98593.222041]  [<ffffffff8101589a>] ? mwait_idle+0x4a/0x2b0
[98593.222041]  [<ffffffff8100bb76>] cpu_idle+0x66/0xd0
[98593.222041]  [<ffffffff8142c8f9>] start_secondary+0x1bf/0x1c4
[98593.222041] Code: 8b 47 68 55 48 89 e5 39 c6 77 1c 29 f0 3b 47 6c 89 
47 68 72 16 89 f0 48 03 87 e0 00 00 00 48 89 87 e0
  00 00 00 c9 c3 31 c0 c9 c3 <0f> 0b eb fe 90 55 39 77 68 48 89 e5 76 1c 
8b 47 6c 85 c0 75 17
[98593.532028] RIP  [<ffffffff8136ca2b>] skb_pull+0x2b/0x30
[98593.532028]  RSP <ffff88011fc83ab0>
[98593.582191] ---[ end trace 7eee6e2d8ae05a69 ]---
[98593.596115] Kernel panic - not syncing: Fatal exception in interrupt
[98593.596119] Pid: 0, comm: kworker/0:0 Tainted: G      D 
3.0.0-un-def-alt2 #1
------------------------------------------------
Have anybody some ideas?

Comment 6 cpulinker 2011-08-05 13:25:22 UTC

New kernel, with "gre: fix improper error handling" applied - result 
same. Please Help! Which new git tree you recommend me to testing?
------------------------------------------------
[ 4081.605966] ------------[ cut here ]------------
[ 4081.609919] kernel BUG at include/linux/skbuff.h:1189!
[ 4081.609919] invalid opcode: 0000 [#1] PREEMPT SMP
[ 4081.609919] CPU 1
[ 4081.609919] Modules linked in: cls_fw sch_sfq arc4 ecb ppp_mppe 
act_mirred act_skbedit cls_u32 sch_ingress l2tp_ppp l2t
p_netlink l2tp_core pptp pppox ppp_generic slhc gre 8021q garp stp 
cls_flow sch_htb ifb ipt_NETFLOW xt_mark nf_conntrack_i
pv4 nf_defrag_ipv4 xt_state xt_TCPMSS ipt_LOG xt_recent xt_NOTRACK 
nf_conntrack xt_statistic ts_kmp xt_tcpudp xt_string xt
_multiport xt_set iptable_raw iptable_mangle iptable_filter ip_tables 
x_tables ip_set_hash_ip ip_set_hash_net ip_set nfnet
link dm_multipath scsi_dh dm_mod psmouse sr_mod evdev pcspkr rtc_cmos 
cdrom sg serio_raw i2c_i801 ehci_hcd tg3 uhci_hcd i2
c_core i3000_edac igb usbcore edac_core dca processor button ext3 jbd 
mbcache sd_mod crc_t10dif ide_pci_generic ide_core p
ata_acpi ata_generic ata_piix libata scsi_mod
[ 4081.609919]
[ 4081.609919] Pid: 0, comm: kworker/0:0 Not tainted 3.0.0-un-def-alt2 
#1 ASUS RS100-E4/PI2/P5M2-M/RS100-E4
[ 4081.609919] RIP: 0010:[<ffffffff8136ca2b>]  [<ffffffff8136ca2b>] 
skb_pull+0x2b/0x30
[ 4081.609919] RSP: 0018:ffff88011fc83ab0  EFLAGS: 00010283
[ 4081.609919] RAX: 000000000000057e RBX: ffff88011378db80 RCX: 
0000000000000179
[ 4081.609919] RDX: 0000000000000179 RSI: 0000000000000002 RDI: 
ffff88011378db80
[ 4081.609919] RBP: ffff88011fc83ab0 R08: 0000000000000000 R09: 
0000000000000102
[ 4081.609919] R10: 0000000000000000 R11: 0000000000000001 R12: 
ffff88010d0a8800
[ 4081.609919] R13: ffff88010dc4186e R14: 000000000000002f R15: 
ffffffff8168dd80
[ 4081.609919] FS:  0000000000000000(0000) GS:ffff88011fc80000(0000) 
knlGS:0000000000000000
[ 4081.609919] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4081.609919] CR2: 0000000000fc10fc CR3: 0000000001625000 CR4: 
00000000000006e0
[ 4081.609919] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[ 4081.609919] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[ 4081.609919] Process kworker/0:0 (pid: 0, threadinfo ffff880118f5c000, 
task ffff880118f5a480)
[ 4081.609919] Stack:
[ 4081.609919]  ffff88011fc83ae0 ffffffffa03267fa ffff88011fc83ae0 
ffff88010d0a8800
[ 4081.609919]  ffff88011378db80 ffff88010d0a8850 ffff88011fc83b10 
ffffffff8136bddc
[ 4081.609919]  ffff88011fc83b40 ffff88011378db80 ffff88010d0a8800 
000000008b55000a
[ 4081.609919] Call Trace:
[ 4081.609919]  <IRQ>
[ 4081.609919]  [<ffffffffa03267fa>] pptp_rcv_core+0x21a/0x220 [pptp]
[ 4081.609919]  [<ffffffff8136bddc>] sk_receive_skb+0x13c/0x160
[ 4081.609919]  [<ffffffffa03261be>] pptp_rcv+0x15e/0x1b0 [pptp]
[ 4081.609919]  [<ffffffffa03080e3>] gre_rcv+0x73/0xa0 [gre]
[ 4081.609919]  [<ffffffff813af10d>] ip_local_deliver_finish+0xed/0x2c0
[ 4081.609919]  [<ffffffff813af360>] ip_local_deliver+0x80/0x90
[ 4081.609919]  [<ffffffff813ae959>] ip_rcv_finish+0x119/0x3b0
[ 4081.609919]  [<ffffffff813aef4d>] ip_rcv+0x21d/0x2f0
[ 4081.609919]  [<ffffffff8137cffc>] __netif_receive_skb+0x20c/0x6c0
[ 4081.609919]  [<ffffffff8137dbcd>] netif_receive_skb+0xbd/0xd0
[ 4081.609919]  [<ffffffff8125fcfc>] ? is_swiotlb_buffer+0x3c/0x50
[ 4081.609919]  [<ffffffff8137dd30>] napi_skb_finish+0x50/0x70
[ 4081.609919]  [<ffffffff8137e315>] napi_gro_receive+0xc5/0xd0
[ 4081.609919]  [<ffffffffa01dbe0f>] igb_poll+0x6cf/0xb50 [igb]
[ 4081.609919]  [<ffffffff8137d004>] ? __netif_receive_skb+0x214/0x6c0
[ 4081.609919]  [<ffffffff8137d004>] ? __netif_receive_skb+0x214/0x6c0
[ 4081.609919]  [<ffffffff8137f0d4>] net_rx_action+0x164/0x340
[ 4081.609919]  [<ffffffff810684b5>] __do_softirq+0xd5/0x270
[ 4081.609919]  [<ffffffff8143c89c>] call_softirq+0x1c/0x30
[ 4081.609919]  [<ffffffff8100e6c5>] do_softirq+0x95/0xe0
[ 4081.609919]  [<ffffffff81068305>] irq_exit+0xd5/0xf0
[ 4081.609919]  [<ffffffff8100dec1>] do_IRQ+0x61/0xd0
[ 4081.609919]  [<ffffffff81434753>] common_interrupt+0x13/0x13
[ 4081.609919]  <EOI>
[ 4081.609919]  [<ffffffff8108af60>] ? sched_clock_cpu+0xd0/0x140
[ 4081.609919]  [<ffffffff81015936>] ? mwait_idle+0xe6/0x2b0
[ 4081.609919]  [<ffffffff8101589a>] ? mwait_idle+0x4a/0x2b0
[ 4081.609919]  [<ffffffff8100bb76>] cpu_idle+0x66/0xd0
[ 4081.609919]  [<ffffffff8142c8f9>] start_secondary+0x1bf/0x1c4
[ 4081.609919] Code: 8b 47 68 55 48 89 e5 39 c6 77 1c 29 f0 3b 47 6c 89 
47 68 72 16 89 f0 48 03 87 e0 00 00 00 48 89 87 e0
  00 00 00 c9 c3 31 c0 c9 c3 <0f> 0b eb fe 90 55 39 77 68 48 89 e5 76 1c 
8b 47 6c 85 c0 75 17
[ 4081.609919] RIP  [<ffffffff8136ca2b>] skb_pull+0x2b/0x30
[ 4081.609919]  RSP <ffff88011fc83ab0>
[ 4082.825070] ---[ end trace d6baf8611ae0f359 ]---
[ 4082.838992] Kernel panic - not syncing: Fatal exception in interrupt
[ 4082.858102] Pid: 0, comm: kworker/0:0 Tainted: G      D 
3.0.0-un-def-alt2 #1
[ 4082.881241] Call Trace:
[ 4082.888651]  <IRQ>  [<ffffffff81430b2a>] panic+0x96/0x1bb
[ 4082.904981]  [<ffffffff8143577a>] oops_end+0x10a/0x120
[ 4082.920451]  [<ffffffff8100fbf6>] die+0x56/0x90
[ 4082.934102]  [<ffffffff81434e64>] do_trap+0xc4/0x170
[ 4082.949051]  [<ffffffff8100d801>] do_invalid_op+0xa1/0xb0
[ 4082.965301]  [<ffffffff8136ca2b>] ? skb_pull+0x2b/0x30
[ 4082.980772]  [<ffffffff8106837c>] ? local_bh_enable+0x5c/0xc0
[ 4082.998067]  [<ffffffffa026935a>] ? ipt_do_table+0x22a/0x640 [ip_tables]
[ 4083.018212]  [<ffffffff8143c61b>] invalid_op+0x1b/0x20
[ 4083.033683]  [<ffffffff8136ca2b>] ? skb_pull+0x2b/0x30
[ 4083.049153]  [<ffffffffa03267fa>] pptp_rcv_core+0x21a/0x220 [pptp]
[ 4083.067742]  [<ffffffff8136bddc>] sk_receive_skb+0x13c/0x160
[ 4083.084774]  [<ffffffffa03261be>] pptp_rcv+0x15e/0x1b0 [pptp]
[ 4083.102064]  [<ffffffffa03080e3>] gre_rcv+0x73/0xa0 [gre]
[ 4083.118314]  [<ffffffff813af10d>] ip_local_deliver_finish+0xed/0x2c0
[ 4083.137425]  [<ffffffff813af360>] ip_local_deliver+0x80/0x90
[ 4083.154453]  [<ffffffff813ae959>] ip_rcv_finish+0x119/0x3b0
[ 4083.171223]  [<ffffffff813aef4d>] ip_rcv+0x21d/0x2f0
[ 4083.186174]  [<ffffffff8137cffc>] __netif_receive_skb+0x20c/0x6c0
[ 4083.204504]  [<ffffffff8137dbcd>] netif_receive_skb+0xbd/0xd0
[ 4083.221795]  [<ffffffff8125fcfc>] ? is_swiotlb_buffer+0x3c/0x50
[ 4083.239605]  [<ffffffff8137dd30>] napi_skb_finish+0x50/0x70
[ 4083.256374]  [<ffffffff8137e315>] napi_gro_receive+0xc5/0xd0
[ 4083.273412]  [<ffffffffa01dbe0f>] igb_poll+0x6cf/0xb50 [igb]
[ 4083.290436]  [<ffffffff8137d004>] ? __netif_receive_skb+0x214/0x6c0
[ 4083.309286]  [<ffffffff8137d004>] ? __netif_receive_skb+0x214/0x6c0
[ 4083.328136]  [<ffffffff8137f0d4>] net_rx_action+0x164/0x340
[ 4083.344906]  [<ffffffff810684b5>] __do_softirq+0xd5/0x270
[ 4083.361157]  [<ffffffff8143c89c>] call_softirq+0x1c/0x30
[ 4083.377147]  [<ffffffff8100e6c5>] do_softirq+0x95/0xe0
[ 4083.392618]  [<ffffffff81068305>] irq_exit+0xd5/0xf0
[ 4083.407565]  [<ffffffff8100dec1>] do_IRQ+0x61/0xd0
[ 4083.421998]  [<ffffffff81434753>] common_interrupt+0x13/0x13
[ 4083.439027]  <EOI>  [<ffffffff8108af60>] ? sched_clock_cpu+0xd0/0x140
[ 4083.458477]  [<ffffffff81015936>] ? mwait_idle+0xe6/0x2b0
[ 4083.474725]  [<ffffffff8101589a>] ? mwait_idle+0x4a/0x2b0
[ 4083.490976]  [<ffffffff8100bb76>] cpu_idle+0x66/0xd0
[ 4083.505927]  [<ffffffff8142c8f9>] start_secondary+0x1bf/0x1c4
[ 4083.523225] Rebooting in 30 seconds..
------------------------------------------------