(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Wed, 3 Jun 2009 03:16:35 GMT bugzilla-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=13432
> 
>            Summary: dma mapping bug -> crash .Some where between the
>                     2.6.29 and the 2.6.29.4 kernel a bug has appeared.

A post-2.6.29 regression in the stable tree.

>            Product: IO/Storage
>            Version: 2.5
>     Kernel Version: 2.26.9.4
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: high
>           Priority: P1
>          Component: Serial ATA
>         AssignedTo: jgarzik@pobox.com
>         ReportedBy: db.pub.mail@gmail.com
>         Regression: Yes
> 
> 
> Some where between the 2.6.29 and the 2.6.29.4 kernel a bug has appeared.
> This
> system is a linkstation pro v2. Both kernel were compiled on the device. To
> reproduce this bug i turn the device on and use samba to copy files to it.
> Below is one report re: the bug. This bug can crash the system completely. 
> 
> 
> 
> 
> 
> 
> May 28 00:01:34 linkstation kernel: [82380.150000] kernel BUG at
> arch/arm/mm/dma-mapping.c:496!
> May 28 00:01:34 linkstation kernel: [82380.150000] Unable to handle kernel
> NULL
> pointer dereference at virtual address 00000000
> May 28 00:01:34 linkstation kernel: [82380.160000] pgd = c6188000
> May 28 00:01:34 linkstation kernel: [82380.160000] [00000000] *pgd=05825031,
> *pte=00000000, *ppte=00000000
> May 28 00:01:34 linkstation kernel: [82380.170000] Internal error: Oops: 817
> [#1] PREEMPT
> May 28 00:01:34 linkstation kernel: [82380.170000] Modules linked in: ipv6
> nfsd
> exportfs nfs lockd nfs_acl auth_rpcgss sunrpc fuse usblp
> May 28 00:01:34 linkstation kernel: [82380.170000] CPU: 0 Not tainted
> (2.6.29.4
> #1)
> May 28 00:01:34 linkstation kernel: [82380.170000] PC is at __bug+0x20/0x2c
> May 28 00:01:34 linkstation kernel: [82380.170000] LR is at
> release_console_sem+0x1d8/0x260
> May 28 00:01:34 linkstation kernel: [82380.170000] pc : [<c002dda0>] lr :
> [<c004ef80>] psr: 60000013
> May 28 00:01:34 linkstation kernel: [82380.170000] sp : c5a07c68 ip :
> c5a07ba8
> fp : c5a07c74
> May 28 00:01:34 linkstation kernel: [82380.170000] r10: c799f258 r9 :
> c79e31e0
> r8 : c64b0100
> May 28 00:01:34 linkstation kernel: [82380.170000] r7 : c591f858 r6 :
> c79e31e0
> r5 : c03f49a4 r4 : c591f858
> May 28 00:01:34 linkstation kernel: [82380.170000] r3 : 00000000 r2 :
> 00000000
> r1 : 00002e67 r0 : 0000003f
> May 28 00:01:34 linkstation kernel: [82380.170000] Flags: nZCv IRQs on FIQs
> on
> Mode SVC_32 ISA ARM Segment user
> May 28 00:01:34 linkstation kernel: [82380.170000] Control: a005317f Table:
> 06188000 DAC: 00000015
> May 28 00:01:34 linkstation kernel: [82380.170000] Process smbd (pid: 26454,
> stack limit = 0xc5a06268)
> May 28 00:01:34 linkstation kernel: [82380.170000] Stack: (0xc5a07c68 to
> 0xc5a08000)
> May 28 00:01:34 linkstation kernel: [82380.170000] 7c60: c5a07c94 c5a07c78
> c002fe74 c002dd90 c006a9bc ffffffff
> May 28 00:01:34 linkstation kernel: [82380.170000] 7c80: c03f49a4 00000e38
> c5a07ccc c5a07c98 c025e078 c002fe00 c006aa08 c006a99c
> May 28 00:01:34 linkstation kernel: [82380.170000] 7ca0: 00000000 c79e31e0
> c5a07f20 00000001 00002b48 00000072 c79e9b14 c591f858
> May 28 00:01:34 linkstation kernel: [82380.170000] 7cc0: c5a07d14 c5a07cd0
> c025f100 c025e04c c79e31e0 00000000 000022aa c79e9b00
> May 28 00:01:34 linkstation kernel: [82380.170000] 7ce0: c5a07f18 c799f258
> c0264308 00000072 00000072 00000000 00000072 00002238
> May 28 00:01:34 linkstation kernel: [82380.170000] 7d00: 00000072 c5a06000
> c5a07d5c c5a07d18 c02818b8 c025f040 00000072 c5a07d28
> May 28 00:01:34 linkstation kernel: [82380.170000] 7d20: c02648c0 c5a07f18
> c7850bc0 c799f258 c64f0820 c7850bc0 c6a58b00 00000072
> May 28 00:01:34 linkstation kernel: [82380.170000] 7d40: 00000000 00002238
> 00000072 c5a06000 c5a07dd4 c5a07d60 c02a06a8 c0281864
> May 28 00:01:34 linkstation kernel: [82380.170000] 7d60: 00000072 c79e9b00
> 00000000 c5a07e30 00000000 c6a58da4 00000001 c64f0820
> May 28 00:01:34 linkstation kernel: [82380.170000] 7d80: 00000000 c6a58b64
> c6a58dc0 c6a58ba4 c6a58e90 c6a58b7c 0000002e 0000002e
> May 28 00:01:34 linkstation kernel: [82380.170000] 7da0: 7fffffff c02a10d8
> c0102e94 bf12d430 c5a07e30 000022aa c74c68c0 c5a07e30
> May 28 00:01:34 linkstation kernel: [82380.170000] 7dc0: c5a07f18 00000001
> c5a07e04 c5a07dd8 c02634e0 c02a007c 00000000 00000000
> May 28 00:01:34 linkstation kernel: [82380.170000] 7de0: c5a07dec be9e79c8
> be9e7948 00000000 bf119190 c5a07e88 c5a07e7c c5a07e08
> May 28 00:01:34 linkstation kernel: [82380.170000] 7e00: c0260abc c02634a4
> 00000000 00000040 00000000 00000000 00000000 000022aa
> May 28 00:01:34 linkstation kernel: [82380.170000] 7e20: c74c68c0 c79e31e0
> 00000000 c5a07e30 00000000 00000000 c5a07f18 00000001
> May 28 00:01:34 linkstation kernel: [82380.170000] 7e40: 00000000 00000000
> 00000000 c5a07e88 c006556c c5a07e88 c79e31e0 c5a07f18
> May 28 00:01:34 linkstation kernel: [82380.170000] 7e60: c5a07f70 fffffdee
> c5a06000 00000000 c5a07f44 c5a07e80 c00aa8bc c02609c0
> May 28 00:01:34 linkstation kernel: [82380.170000] 7e80: 00000000 00000000
> 00000001 be9e7510 00000000 00000001 ffffffff c79e31e0
> May 28 00:01:34 linkstation kernel: [82380.170000] 7ea0: 00000000 00000000
> 00000000 00000000 c64f0820 c00342a0 00000000 00000000
> May 28 00:01:34 linkstation kernel: [82380.170000] 7ec0: 0011aa06 c64f0820
> c006556c c5a07ecc c5a07ecc 00000000 00000000 00000000
> May 28 00:01:34 linkstation kernel: [82380.170000] 7ee0: c5a07e10 c5a07ef0
> c006e0c8 c00342a0 000022aa c5a07f60 0011aa06 c03f1580
> May 28 00:01:34 linkstation kernel: [82380.170000] 7f00: c03f1560 be9e7948
> 00000015 00000000 00000000 c74c69f4 00664606 00000000
> May 28 00:01:34 linkstation kernel: [82380.170000] 7f20: c0ee9e38 c79e31e0
> 0066235c c5a07f70 000022aa 0066235c c5a07f6c c5a07f48
> May 28 00:01:34 linkstation kernel: [82380.170000] 7f40: c00ab544 c00aa810
> 00000000 00000000 00000000 00000000 c79e31e0 000022aa
> May 28 00:01:34 linkstation kernel: [82380.170000] 7f60: c5a07fa4 c5a07f70
> c00ab6c8 c00ab488 00000000 00000000 00000000 00000000
> May 28 00:01:34 linkstation kernel: [82380.170000] 7f80: 4a1bf63d 00000000
> 000022aa 0066235c 00000003 c002a048 00000000 c5a07fa8
> May 28 00:01:34 linkstation kernel: [82380.170000] 7fa0: c0029ea0 c00ab694
> 00000000 000022aa 00000006 0066235c 000022aa 0000839a
> May 28 00:01:34 linkstation kernel: [82380.170000] 7fc0: 00000000 000022aa
> 0066235c 00000003 0000839a 0000ea60 0000839a 00530000
> May 28 00:01:34 linkstation kernel: [82380.170000] 7fe0: 00000006 be9e7798
> 001a1fcc 40267eac 40000010 00000006 00000000 00000000
> May 28 00:01:34 linkstation kernel: [82380.170000] Backtrace:
> May 28 00:01:34 linkstation kernel: [82380.170000] [<c002dd80>]
> (__bug+0x0/0x2c) from [<c002fe74>] (dma_cache_maint+0x84/0xec)
> May 28 00:01:34 linkstation kernel: [82380.170000] [<c002fdf0>]
> (dma_cache_maint+0x0/0xec) from [<c025e078>]
> (dma_async_memcpy_buf_to_pg+0x3c/0x108)
> May 28 00:01:34 linkstation kernel: [82380.170000] r6:00000e38 r5:c03f49a4
> r4:ffffffff
> May 28 00:01:34 linkstation kernel: [82380.170000] [<c025e03c>]
> (dma_async_memcpy_buf_to_pg+0x0/0x108) from [<c025f100>]
> (dma_memcpy_to_iovec+0xd0/0x134)
> May 28 00:01:34 linkstation kernel: [82380.170000] [<c025f030>]
> (dma_memcpy_to_iovec+0x0/0x134) from [<c02818b8>]
> (dma_skb_copy_datagram_iovec+0x64/0x1c8)
> May 28 00:01:34 linkstation kernel: [82380.170000] [<c0281854>]
> (dma_skb_copy_datagram_iovec+0x0/0x1c8) from [<c02a06a8>]
> (tcp_recvmsg+0x63c/0xb94)
> May 28 00:01:34 linkstation kernel: [82380.170000] [<c02a006c>]
> (tcp_recvmsg+0x0/0xb94) from [<c02634e0>] (sock_common_recvmsg+0x4c/0x60)
> May 28 00:01:34 linkstation kernel: [82380.170000] [<c0263494>]
> (sock_common_recvmsg+0x0/0x60) from [<c0260abc>] (sock_aio_read+0x10c/0x11c)
> May 28 00:01:34 linkstation kernel: [82380.170000] r5:c5a07e88 r4:bf119190
> May 28 00:01:34 linkstation kernel: [82380.170000] [<c02609b0>]
> (sock_aio_read+0x0/0x11c) from [<c00aa8bc>] (do_sync_read+0xbc/0x108)
> May 28 00:01:34 linkstation kernel: [82380.170000] [<c00aa800>]
> (do_sync_read+0x0/0x108) from [<c00ab544>] (vfs_read+0xcc/0x18c)
> May 28 00:01:34 linkstation kernel: [82380.170000] r8:0066235c r7:000022aa
> r6:c5a07f70 r5:0066235c r4:c79e31e0
> May 28 00:01:34 linkstation kernel: [82380.170000] [<c00ab478>]
> (vfs_read+0x0/0x18c) from [<c00ab6c8>] (sys_read+0x44/0x70)
> May 28 00:01:34 linkstation kernel: [82380.170000] r7:000022aa r6:c79e31e0
> r5:00000000 r4:00000000
> May 28 00:01:34 linkstation kernel: [82380.170000] [<c00ab684>]
> (sys_read+0x0/0x70) from [<c0029ea0>] (ret_fast_syscall+0x0/0x2c)
> May 28 00:01:34 linkstation kernel: [82380.170000] r8:c002a048 r7:00000003
> r6:0066235c r5:000022aa r4:00000000
> May 28 00:01:34 linkstation kernel: [82380.170000] Code: e1a01000 e59f000c
> eb0af149 e3a03000 (e5833000)
> May 28 00:01:34 linkstation kernel: [82380.640000] ---[ end trace
> c8d58d057de0b990 ]---

On Tue, 2009-06-02 at 20:48 -0700, Andrew Morton wrote:
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Wed, 3 Jun 2009 03:16:35 GMT bugzilla-daemon@bugzilla.kernel.org wrote:
> 
> > http://bugzilla.kernel.org/show_bug.cgi?id=13432
> > 
> >            Summary: dma mapping bug -> crash .Some where between the
> >                     2.6.29 and the 2.6.29.4 kernel a bug has appeared.
> 
> A post-2.6.29 regression in the stable tree.
> 
> >            Product: IO/Storage
> >            Version: 2.5
> >     Kernel Version: 2.26.9.4
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: high
> >           Priority: P1
> >          Component: Serial ATA
> >         AssignedTo: jgarzik@pobox.com
> >         ReportedBy: db.pub.mail@gmail.com
> >         Regression: Yes
> > 
> > 
> > Some where between the 2.6.29 and the 2.6.29.4 kernel a bug has appeared.
> This
> > system is a linkstation pro v2. Both kernel were compiled on the device. To
> > reproduce this bug i turn the device on and use samba to copy files to it.
> > Below is one report re: the bug. This bug can crash the system completely. 
> > 

This is not a regression.  I suspect CONFIG_NET_DMA has been enabled in
the configuration between the 2.6.29 and 2.6.29.4 builds?  The general
problem is that x86 and Power are more permissive in the addresses that
can be passed to the dma mapping api.  It seems that changes in the
network stack and or ARM are leading to attempts to copy from
ARM-invalid virtual addresses in this case.

NET_DMA is only a performance gain on I/O-coherent architectures.  On
ARM it is a performance loss because the cache maintenance is overkill
for these sub-page-size copies.

The Kconfig entry for NET_DMA already recommends not turning it on, but
perhaps this should be more forceful?

Regards,
Dan

diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig
index 3b3c01b..f31ae83 100644
--- a/drivers/dma/Kconfig
+++ b/drivers/dma/Kconfig
@@ -90,14 +90,11 @@ comment "DMA Clients"
 config NET_DMA
 	bool "Network: TCP receive copy offload"
 	depends on DMA_ENGINE && NET
-	default (INTEL_IOATDMA || FSL_DMA)
+	depends on (INTEL_IOATDMA || FSL_DMA)
 	help
 	  This enables the use of DMA engines in the network stack to
 	  offload receive copy-to-user operations, freeing CPU cycles.
 
-	  Say Y here if you enabled INTEL_IOATDMA or FSL_DMA, otherwise
-	  say N.
-
 config ASYNC_TX_DMA
 	bool "Async_tx: Offload support for the async_tx api"
 	depends on DMA_ENGINE

On Wed, 03 Jun 2009 10:44:41 -0700, Dan Williams <dan.j.williams@intel.com> wrote:
> This is not a regression.  I suspect CONFIG_NET_DMA has been enabled in
> the configuration between the 2.6.29 and 2.6.29.4 builds?  The general
> problem is that x86 and Power are more permissive in the addresses that
> can be passed to the dma mapping api.  It seems that changes in the
> network stack and or ARM are leading to attempts to copy from
> ARM-invalid virtual addresses in this case.

It looks a source address of dma_async_memcpy_buf_to_pg() triggers the
BUG_ON().  And the address comes from skb passed to
dma_skb_copy_datagram_iovec().  I don't know how skb can has invalid
address though.

Even if disabling NET_DMA obscure the crash, there should be real bug
somewhere.

---
Atsushi Nemoto

Reply-To: maciej.sosnowski@intel.com

Williams, Dan J wrote:
> On Tue, 2009-06-02 at 20:48 -0700, Andrew Morton wrote:
>> (switched to email.  Please respond via emailed reply-to-all, not via the
>> bugzilla web
>> interface). 
>> 
>> On Wed, 3 Jun 2009 03:16:35 GMT bugzilla-daemon@bugzilla.kernel.org wrote:
>> 
>>> http://bugzilla.kernel.org/show_bug.cgi?id=13432
>>> 
>>>            Summary: dma mapping bug -> crash .Some where between the
>>>                     2.6.29 and the 2.6.29.4 kernel a bug has appeared.
>> 
>> A post-2.6.29 regression in the stable tree.
>> 
>>>            Product: IO/Storage
>>>            Version: 2.5
>>>     Kernel Version: 2.26.9.4
>>>           Platform: All
>>>         OS/Version: Linux
>>>               Tree: Mainline
>>>             Status: NEW
>>>           Severity: high
>>>           Priority: P1
>>>          Component: Serial ATA
>>>         AssignedTo: jgarzik@pobox.com
>>>         ReportedBy: db.pub.mail@gmail.com
>>>         Regression: Yes
>>> 
>>> 
>>> Some where between the 2.6.29 and the 2.6.29.4 kernel a bug has appeared.
>>> This
>>> system is a linkstation pro v2. Both kernel were compiled on the device. To
>>> reproduce this bug i turn the device on and use samba to copy files to it.
>>> Below is one report re: the bug. This bug can crash the system completely.
>>> 
> 
> This is not a regression.  I suspect CONFIG_NET_DMA has been enabled in
> the configuration between the 2.6.29 and 2.6.29.4 builds?  The general
> problem is that x86 and Power are more permissive in the addresses that
> can be passed to the dma mapping api.  It seems that changes in the
> network stack and or ARM are leading to attempts to copy from
> ARM-invalid virtual addresses in this case.
> 
> NET_DMA is only a performance gain on I/O-coherent architectures.  On
> ARM it is a performance loss because the cache maintenance is overkill
> for these sub-page-size copies.
> 
> The Kconfig entry for NET_DMA already recommends not turning it on, but
> perhaps this should be more forceful?
> 
> Regards,
> Dan
> 
> diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig
> index 3b3c01b..f31ae83 100644
> --- a/drivers/dma/Kconfig
> +++ b/drivers/dma/Kconfig
> @@ -90,14 +90,11 @@ comment "DMA Clients"
>  config NET_DMA
>       bool "Network: TCP receive copy offload"
>       depends on DMA_ENGINE && NET
> -     default (INTEL_IOATDMA || FSL_DMA)
> +     depends on (INTEL_IOATDMA || FSL_DMA)
>       help
>         This enables the use of DMA engines in the network stack to
>         offload receive copy-to-user operations, freeing CPU cycles.
> 
> -       Say Y here if you enabled INTEL_IOATDMA or FSL_DMA, otherwise
> -       say N.
> -
>  config ASYNC_TX_DMA
>       bool "Async_tx: Offload support for the async_tx api"
>       depends on DMA_ENGINE

I agree with Dan. Let's do net_dma depending on ioatdma or fsldma.

Regards,
Maciej

Here are the config for the two kernels (2.6.29 and the 2.6.29.4).
.... So i should disable the net dma option?


2009/6/4 Sosnowski, Maciej <maciej.sosnowski@intel.com>:
> Williams, Dan J wrote:
>> On Tue, 2009-06-02 at 20:48 -0700, Andrew Morton wrote:
>>> (switched to email.  Please respond via emailed reply-to-all, not via the
>>> bugzilla web
>>> interface).
>>>
>>> On Wed, 3 Jun 2009 03:16:35 GMT bugzilla-daemon@bugzilla.kernel.org wrote:
>>>
>>>> http://bugzilla.kernel.org/show_bug.cgi?id=13432
>>>>
>>>>            Summary: dma mapping bug -> crash .Some where between the
>>>>                     2.6.29 and the 2.6.29.4 kernel a bug has appeared.
>>>
>>> A post-2.6.29 regression in the stable tree.
>>>
>>>>            Product: IO/Storage
>>>>            Version: 2.5
>>>>     Kernel Version: 2.26.9.4
>>>>           Platform: All
>>>>         OS/Version: Linux
>>>>               Tree: Mainline
>>>>             Status: NEW
>>>>           Severity: high
>>>>           Priority: P1
>>>>          Component: Serial ATA
>>>>         AssignedTo: jgarzik@pobox.com
>>>>         ReportedBy: db.pub.mail@gmail.com
>>>>         Regression: Yes
>>>>
>>>>
>>>> Some where between the 2.6.29 and the 2.6.29.4 kernel a bug has appeared.
>>>> This
>>>> system is a linkstation pro v2. Both kernel were compiled on the device.
>>>> To
>>>> reproduce this bug i turn the device on and use samba to copy files to it.
>>>> Below is one report re: the bug. This bug can crash the system completely.
>>>>
>>
>> This is not a regression.  I suspect CONFIG_NET_DMA has been enabled in
>> the configuration between the 2.6.29 and 2.6.29.4 builds?  The general
>> problem is that x86 and Power are more permissive in the addresses that
>> can be passed to the dma mapping api.  It seems that changes in the
>> network stack and or ARM are leading to attempts to copy from
>> ARM-invalid virtual addresses in this case.
>>
>> NET_DMA is only a performance gain on I/O-coherent architectures.  On
>> ARM it is a performance loss because the cache maintenance is overkill
>> for these sub-page-size copies.
>>
>> The Kconfig entry for NET_DMA already recommends not turning it on, but
>> perhaps this should be more forceful?
>>
>> Regards,
>> Dan
>>
>> diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig
>> index 3b3c01b..f31ae83 100644
>> --- a/drivers/dma/Kconfig
>> +++ b/drivers/dma/Kconfig
>> @@ -90,14 +90,11 @@ comment "DMA Clients"
>>  config NET_DMA
>>       bool "Network: TCP receive copy offload"
>>       depends on DMA_ENGINE && NET
>> -     default (INTEL_IOATDMA || FSL_DMA)
>> +     depends on (INTEL_IOATDMA || FSL_DMA)
>>       help
>>         This enables the use of DMA engines in the network stack to
>>         offload receive copy-to-user operations, freeing CPU cycles.
>>
>> -       Say Y here if you enabled INTEL_IOATDMA or FSL_DMA, otherwise
>> -       say N.
>> -
>>  config ASYNC_TX_DMA
>>       bool "Async_tx: Offload support for the async_tx api"
>>       depends on DMA_ENGINE
>
> I agree with Dan. Let's do net_dma depending on ioatdma or fsldma.
>
> Regards,
> Maciej
>

Reply-To: maciej.sosnowski@intel.com

db wrote:
> Here are the config for the two kernels (2.6.29 and the 2.6.29.4).
> .... So i should disable the net dma option?

Well, it's clear now that it's not a bug that appeared between 2.6.29 and 2.6.29.4 but it's a matter of enabling NET_DMA in 2.6.29.4 (it's disabled in your config for 2.6.29).

Yes, as you do not use ioatdma nor fsldma you should keep NET_DMA disabled.

Regards,
Maciej