Most recent kernel where this bug did not occur: unknown Distribution: CentOS 4.3 Hardware Environment: amd64, 2 CPUs Problem Description: It's a Xen Dom0 machine. Although the guest OSes get the same lockups (unsurprisingly, they tend to coincide with the Dom0 lockups), it seems to be more common in Dom0. The machine hangs and is totally unresponsive for a short while (anything from 5 to 60 seconds or so), and then continues as if nothing had happened. Although the problem isn't explicitly reproducible, it happens reasonably often, and so it's usually just a case of waiting for a bit, and it'll hang again. The only patterns I can see are that it seems to almost always be mid way through logging packets dropped by iptables, and that ipt_LOG features in the call trace. It seems to follow a burst of traffic being dropped, too (100+ packets/second). Although it's not purely a mainline kernel, there was no option to log it as a Xen kernel in the alternate trees category. The Xen install grabbed the 2.6.16 mainline kernel, applied the Xen patches and compiled it from there. No other patches were applied. Call traces follow: Jul 10 11:19:50 shelbyville kernel: ** Dropped: IN=xenbr0 OUT=xenbr0 PHYSIN=peth0 PHYSOUT=vif10.0 SRC=87.74.242.134 DST=212.69.38.65 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=0 DF PROTO=TCP SPT=80 DPT=35634 WINDOW=0 RES=0x00 RST URGP=0 Jul 10 11:19:50 shelbyville kernel: ** Dropped: IN=xenbr0 OUT=xenbr0 PHYSIN=peth0 PHYSOUT=vif10.0 SRC=87.74.242.134 DST=212.69.38.65 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=0 DF <3>BUG: soft lockup detected on CPU#0! Jul 10 11:19:50 shelbyville kernel: CPU 0: Jul 10 11:19:50 shelbyville kernel: Modules linked in: vfat fat loop bridge nfsd exportfs lockd sunrpc ipv6 xt_state ip_conntrack nfnetlink xt_tcpudp ipt_LOG ipt_REJECT xt_physdev iptable_filter ip_tables x_tables video thermal processor fan container bu tton battery ac ohci_hcd ehci_hcd i2c_nforce2 i2c_core tg3 e100 mii floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod sata_nv 3w_9xxx Jul 10 11:19:50 shelbyville kernel: Pid: 0, comm: swapper Not tainted 2.6.16-xen #1 Jul 10 11:19:50 shelbyville kernel: RIP: e030:[<ffffffff8010722a>] <ffffffff8010722a>{hypercall_page+554} Jul 10 11:19:50 shelbyville kernel: RSP: e02b:ffffffff80436cc8 EFLAGS: 00000246 Jul 10 11:19:50 shelbyville kernel: RAX: 0000000000030000 RBX: 00000000016cbf03 RCX: ffffffff8010722a Jul 10 11:19:50 shelbyville kernel: RDX: ffffffff804488e0 RSI: 0000000000000000 RDI: 0000000000000000 Jul 10 11:19:50 shelbyville kernel: RBP: 00000000016cbf03 R08: 00000000fffffffb R09: 0000000000000003 Jul 10 11:19:50 shelbyville kernel: R10: 00000000ffffffff R11: 0000000000000246 R12: 0000000000000000 Jul 10 11:19:50 shelbyville kernel: R13: ffffffffffffabeb R14: 0000000000000003 R15: 0000000000000000 Jul 10 11:19:50 shelbyville kernel: FS: 00002ad193661b00(0000) GS:ffffffff8049d000(0000) knlGS:0000000000000000 Jul 10 11:19:50 shelbyville kernel: CS: e033 DS: 0000 ES: 0000 Jul 10 11:19:50 shelbyville kernel: Jul 10 11:19:50 shelbyville kernel: Call Trace: <IRQ> <ffffffff8012efac>{__call_console_drivers+76} Jul 10 11:19:50 shelbyville kernel: <ffffffff80250eda>{force_evtchn_callback+10} <ffffffff8012f2ea>{release_console_sem+378} Jul 10 11:19:51 shelbyville kernel: <ffffffff8012f621>{vprintk+689} <ffffffff8010ba0a>{do_hypervisor_callback+30} Jul 10 11:19:51 shelbyville kernel: <ffffffff8012f6dd>{printk+141} <ffffffff8010722a>{hypercall_page+554} Jul 10 11:19:51 shelbyville kernel: <ffffffff80250eda>{force_evtchn_callback+10} <ffffffff8012f327>{release_console_sem+439} Jul 10 11:19:51 shelbyville kernel: <ffffffff8012f621>{vprintk+689} <ffffffff880db158>{:ipt_LOG:dump_packet+344} Jul 10 11:19:51 shelbyville kernel: <ffffffff8012f6dd>{printk+141} <ffffffff880d80bd>{:ipt_REJECT:reject+173} Jul 10 11:19:51 shelbyville kernel: <ffffffff80112ddb>{dma_map_single+123} <ffffffff8806c59a>{:e100:e100_exec_cmd+170} Jul 10 11:19:51 shelbyville kernel: <ffffffff880dbadd>{:ipt_LOG:ipt_log_packet+381} <ffffffff880dbb8b>{:ipt_LOG:ipt_log_target+107} Jul 10 11:19:51 shelbyville kernel: <ffffffff880cf334>{:ip_tables:ipt_do_table+772} <ffffffff802fdc9a>{nf_iterate+90} Jul 10 11:19:51 shelbyville kernel: <ffffffff881b5050>{:bridge:br_nf_forward_finish+0} <ffffffff802fdd5d>{nf_hook_slow+125} Jul 10 11:19:51 shelbyville kernel: <ffffffff881b5050>{:bridge:br_nf_forward_finish+0} <ffffffff881b52ca>{:bridge:br_nf_forward_ip+346} Jul 10 11:19:51 shelbyville kernel: <ffffffff802deeb7>{skb_checksum+87} <ffffffff802fdc9a>{nf_iterate+90} Jul 10 11:19:51 shelbyville kernel: <ffffffff881afff0>{:bridge:br_forward_finish+0} <ffffffff802fdd5d>{nf_hook_slow+125} Jul 10 11:19:51 shelbyville kernel: <ffffffff881afff0>{:bridge:br_forward_finish+0} <ffffffff881b011c>{:bridge:__br_forward+92} Jul 10 11:19:51 shelbyville kernel: <ffffffff881b0e9b>{:bridge:br_handle_frame_finish+235} Jul 10 11:19:51 shelbyville kernel: <ffffffff881b4711>{:bridge:br_nf_pre_routing_finish+913} Jul 10 11:19:51 shelbyville kernel: <ffffffff802fdc9a>{nf_iterate+90} <ffffffff881b4380>{:bridge:br_nf_pre_routing_finish+0} Jul 10 11:19:51 shelbyville kernel: <ffffffff802fdd5d>{nf_hook_slow+125} <ffffffff881b4380>{:bridge:br_nf_pre_routing_finish+0} Jul 10 11:19:51 shelbyville kernel: <ffffffff881b4fbd>{:bridge:br_nf_pre_routing+2109} <ffffffff802fdc9a>{nf_iterate+90} Jul 10 11:19:51 shelbyville kernel: <ffffffff881b0db0>{:bridge:br_handle_frame_finish+0} Jul 10 11:19:51 shelbyville kernel: <ffffffff802fdd5d>{nf_hook_slow+125} <ffffffff881b0db0>{:bridge:br_handle_frame_finish+0} Jul 10 11:19:51 shelbyville kernel: <ffffffff881b1076>{:bridge:br_handle_frame+374} <ffffffff802e3d52>{netif_receive_skb+578} Jul 10 11:19:51 shelbyville kernel: <ffffffff8806e4b0>{:e100:e100_poll+784} <ffffffff802e40ef>{net_rx_action+239} Jul 10 11:19:51 shelbyville kernel: <ffffffff801345a3>{__do_softirq+131} <ffffffff8010beda>{call_softirq+30} Jul 10 11:19:51 shelbyville kernel: <ffffffff8010dc97>{do_softirq+71} <ffffffff8010dae9>{do_IRQ+73} Jul 10 11:19:51 shelbyville kernel: <ffffffff80250fad>{evtchn_do_upcall+205} <ffffffff8010ba0a>{do_hypervisor_callback+30} <EOI> Jul 10 11:19:51 shelbyville kernel: <ffffffff801073aa>{hypercall_page+938} <ffffffff801073aa>{hypercall_page+938} Jul 10 11:19:51 shelbyville kernel: <ffffffff80108f02>{xen_idle+130} <ffffffff8010902a>{cpu_idle+234} Jul 10 11:19:51 shelbyville kernel: <ffffffff804b575a>{start_kernel+506} <ffffffff804b51b2>{_sinittext+434} Jul 10 11:19:51 shelbyville kernel: PROTO=TCP SPT=80 DPT=35605 WINDOW=0 RES=0x00 RST URGP=0 Jul 10 11:55:06 shelbyville kernel: ** Dropped: IN=xenbr0 OUT=xenbr0 PHYSIN=peth0 PHYSOUT=vif10.0 SRC=87.74.242.134 DST=212.69.38.65 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=0 DF PROTO=TCP SPT=80 DPT=46561 WINDOW=0 RES=0x00 RST URGP=0 Jul 10 11:55:06 shelbyville kernel: ** Dropped: IN=xenbr0 OUT=xenbr0 PHYSIN=peth0 PHYSOUT=vif10.0 <3>BUG: soft lockup detected on CPU#0! Jul 10 11:55:06 shelbyville kernel: CPU 0: Jul 10 11:55:06 shelbyville kernel: Modules linked in: vfat fat loop bridge nfsd exportfs lockd sunrpc ipv6 xt_state ip_conntrack nfnetlink xt_tcpudp ipt_LOG ipt_REJECT xt_physdev iptable_filter ip_tables x_tables video thermal processor fan container bu tton battery ac ohci_hcd ehci_hcd i2c_nforce2 i2c_core tg3 e100 mii floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod sata_nv 3w_9xxx Jul 10 11:55:06 shelbyville kernel: Pid: 0, comm: swapper Not tainted 2.6.16-xen #1 Jul 10 11:55:06 shelbyville kernel: RIP: e030:[<ffffffff8010722a>] <ffffffff8010722a>{hypercall_page+554} Jul 10 11:55:06 shelbyville kernel: RSP: e02b:ffffffff80436df8 EFLAGS: 00000246 Jul 10 11:55:06 shelbyville kernel: RAX: 0000000000030000 RBX: 00000000016fb53a RCX: ffffffff8010722a Jul 10 11:55:06 shelbyville kernel: RDX: ffffffff80447f0a RSI: 0000000000000000 RDI: 0000000000000000 Jul 10 11:55:06 shelbyville kernel: RBP: 00000000016fb53a R08: 00000000ffffffff R09: 0000000000000010 Jul 10 11:55:06 shelbyville kernel: R10: 00000000ffffffff R11: 0000000000000246 R12: 0000000000000000 Jul 10 11:55:06 shelbyville kernel: R13: ffffffffffffa4c3 R14: 0000000000000010 R15: 0000000000000000 Jul 10 11:55:06 shelbyville kernel: FS: 00002af805865b00(0000) GS:ffffffff8049d000(0000) knlGS:0000000000000000 Jul 10 11:55:06 shelbyville kernel: CS: e033 DS: 0000 ES: 0000 Jul 10 11:55:06 shelbyville kernel: Jul 10 11:55:06 shelbyville kernel: Call Trace: <IRQ> <ffffffff8012efac>{__call_console_drivers+76} Jul 10 11:55:06 shelbyville kernel: <ffffffff80250eda>{force_evtchn_callback+10} <ffffffff8012f2ea>{release_console_sem+378} Jul 10 11:55:06 shelbyville kernel: <ffffffff8012f621>{vprintk+689} <ffffffff8012f6dd>{printk+141} Jul 10 11:55:06 shelbyville kernel: <ffffffff801389ca>{__mod_timer+186} <ffffffff80112ddb>{dma_map_single+123} Jul 10 11:55:06 shelbyville kernel: <ffffffff8806c59a>{:e100:e100_exec_cmd+170} <ffffffff8806d530>{:e100:e100_xmit_prepare+0} Jul 10 11:55:06 shelbyville kernel: <ffffffff880dba33>{:ipt_LOG:ipt_log_packet+211} <ffffffff880dbb8b>{:ipt_LOG:ipt_log_target+107} Jul 10 11:55:06 shelbyville kernel: <ffffffff880cf334>{:ip_tables:ipt_do_table+772} <ffffffff802fdc9a>{nf_iterate+90} Jul 10 11:55:06 shelbyville kernel: <ffffffff881b5050>{:bridge:br_nf_forward_finish+0} <ffffffff802fdd5d>{nf_hook_slow+125} Jul 10 11:55:06 shelbyville kernel: <ffffffff881b5050>{:bridge:br_nf_forward_finish+0} <ffffffff881b52ca>{:bridge:br_nf_forward_ip+346} Jul 10 11:55:06 shelbyville kernel: <ffffffff802deeb7>{skb_checksum+87} <ffffffff802fdc9a>{nf_iterate+90} Jul 10 11:55:06 shelbyville kernel: <ffffffff881afff0>{:bridge:br_forward_finish+0} <ffffffff802fdd5d>{nf_hook_slow+125} Jul 10 11:55:06 shelbyville kernel: <ffffffff881afff0>{:bridge:br_forward_finish+0} <ffffffff881b011c>{:bridge:__br_forward+92} Jul 10 11:55:06 shelbyville kernel: <ffffffff881b0e9b>{:bridge:br_handle_frame_finish+235} Jul 10 11:55:06 shelbyville kernel: <ffffffff881b4711>{:bridge:br_nf_pre_routing_finish+913} Jul 10 11:55:06 shelbyville kernel: <ffffffff802fdc9a>{nf_iterate+90} <ffffffff881b4380>{:bridge:br_nf_pre_routing_finish+0} Jul 10 11:55:06 shelbyville kernel: <ffffffff802fdd5d>{nf_hook_slow+125} <ffffffff881b4380>{:bridge:br_nf_pre_routing_finish+0} Jul 10 11:55:06 shelbyville kernel: <ffffffff881b4fbd>{:bridge:br_nf_pre_routing+2109} <ffffffff802fdc9a>{nf_iterate+90} Jul 10 11:55:06 shelbyville kernel: <ffffffff881b0db0>{:bridge:br_handle_frame_finish+0} Jul 10 11:55:06 shelbyville kernel: <ffffffff802fdd5d>{nf_hook_slow+125} <ffffffff881b0db0>{:bridge:br_handle_frame_finish+0} Jul 10 11:55:06 shelbyville kernel: <ffffffff881b1076>{:bridge:br_handle_frame+374} <ffffffff802e3d52>{netif_receive_skb+578} Jul 10 11:55:06 shelbyville kernel: <ffffffff8806e4b0>{:e100:e100_poll+784} <ffffffff802e40ef>{net_rx_action+239} Jul 10 11:55:06 shelbyville kernel: <ffffffff801345a3>{__do_softirq+131} <ffffffff8010beda>{call_softirq+30} Jul 10 11:55:06 shelbyville kernel: <ffffffff8010dc97>{do_softirq+71} <ffffffff8010dae9>{do_IRQ+73} Jul 10 11:55:06 shelbyville kernel: <ffffffff80250fad>{evtchn_do_upcall+205} <ffffffff8010ba0a>{do_hypervisor_callback+30} <EOI> Jul 10 11:55:06 shelbyville kernel: <ffffffff801073aa>{hypercall_page+938} <ffffffff801073aa>{hypercall_page+938} Jul 10 11:55:06 shelbyville kernel: <ffffffff80108f02>{xen_idle+130} <ffffffff8010902a>{cpu_idle+234} Jul 10 11:55:06 shelbyville kernel: <ffffffff804b575a>{start_kernel+506} <ffffffff804b51b2>{_sinittext+434} Jul 10 11:55:06 shelbyville kernel: SRC=87.74.242.134 DST=212.69.38.65 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=0 DF PROTO=TCP SPT=80 DPT=46633 WINDOW=0 RES=0x00 RST URGP=0 Jul 10 18:20:28 shelbyville kernel: ** Dropped: IN=xenbr0 OUT=xenbr0 PHYSIN=peth0 PHYSOUT=vif10.0 SRC=87.74.242.134 DST=212.69.38.65 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=0 DF PROTO=TCP SPT=80 DPT=42944 WINDOW=0 RES=0x00 RST URGP=0 Jul 10 18:20:28 shelbyville kernel: ** Dropped: IN=xenbr0 OUT=xenbr0 PHYSIN=peth0 PHYSOUT=vif10.0 SRC=87.74.242.134 DST=212.69.38.65 LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=0 <3>BUG: soft lockup detected on CPU#0! Jul 10 18:20:28 shelbyville kernel: CPU 0: Jul 10 18:20:28 shelbyville kernel: Modules linked in: vfat fat loop bridge nfsd exportfs lockd sunrpc ipv6 xt_state ip_conntrack nfnetlink xt_tcpudp ipt_LOG ipt_REJECT xt_physdev iptable_filter ip_tables x_tables video thermal processor fan container bu tton battery ac ohci_hcd ehci_hcd i2c_nforce2 i2c_core tg3 e100 mii floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod sata_nv 3w_9xxx Jul 10 18:20:28 shelbyville kernel: Pid: 0, comm: swapper Not tainted 2.6.16-xen #1 Jul 10 18:20:28 shelbyville kernel: RIP: e030:[<ffffffff8010722a>] <ffffffff8010722a>{hypercall_page+554} Jul 10 18:20:28 shelbyville kernel: RSP: e02b:ffffffff80436cc8 EFLAGS: 00000246 Jul 10 18:20:28 shelbyville kernel: RAX: 0000000000030000 RBX: 00000000017fac7b RCX: ffffffff8010722a Jul 10 18:20:28 shelbyville kernel: RDX: ffffffff80447635 RSI: 0000000000000000 RDI: 0000000000000000 Jul 10 18:20:28 shelbyville kernel: RBP: 00000000017fac7b R08: 00000000fffffffb R09: 0000000000000026 Jul 10 18:20:28 shelbyville kernel: R10: 00000000ffffffff R11: 0000000000000246 R12: 0000000000000000 Jul 10 18:20:28 shelbyville kernel: R13: ffffffffffff9163 R14: 0000000000000026 R15: 0000000000000000 Jul 10 18:20:28 shelbyville kernel: FS: 00002ad193661b00(0000) GS:ffffffff8049d000(0000) knlGS:0000000000000000 Jul 10 18:20:28 shelbyville kernel: CS: e033 DS: 0000 ES: 0000 Jul 10 18:20:28 shelbyville kernel: Jul 10 18:20:28 shelbyville kernel: Call Trace: <IRQ> <ffffffff8012efac>{__call_console_drivers+76} Jul 10 18:20:28 shelbyville kernel: <ffffffff80250eda>{force_evtchn_callback+10} <ffffffff8012f2ea>{release_console_sem+378} Jul 10 18:20:28 shelbyville kernel: <ffffffff8012f621>{vprintk+689} <ffffffff8010ba0a>{do_hypervisor_callback+30} Jul 10 18:20:28 shelbyville kernel: <ffffffff8012f6dd>{printk+141} <ffffffff8010722a>{hypercall_page+554} Jul 10 18:20:28 shelbyville kernel: <ffffffff80250eda>{force_evtchn_callback+10} <ffffffff8012f327>{release_console_sem+439} Jul 10 18:20:28 shelbyville kernel: <ffffffff8012f621>{vprintk+689} <ffffffff880db12c>{:ipt_LOG:dump_packet+300} Jul 10 18:20:28 shelbyville kernel: <ffffffff8012f6dd>{printk+141} <ffffffff801f90dc>{map_single+396} Jul 10 18:20:28 shelbyville kernel: <ffffffff880d80bd>{:ipt_REJECT:reject+173} <ffffffff80112ddb>{dma_map_single+123} Jul 10 18:20:28 shelbyville kernel: <ffffffff8806c59a>{:e100:e100_exec_cmd+170} <ffffffff880dbadd>{:ipt_LOG:ipt_log_packet+381} Jul 10 18:20:28 shelbyville kernel: <ffffffff880dbb8b>{:ipt_LOG:ipt_log_target+107} <ffffffff880cf334>{:ip_tables:ipt_do_table+772} Jul 10 18:20:28 shelbyville kernel: <ffffffff802fdc9a>{nf_iterate+90} <ffffffff881b5050>{:bridge:br_nf_forward_finish+0} Jul 10 18:20:28 shelbyville kernel: <ffffffff802fdd5d>{nf_hook_slow+125} <ffffffff881b5050>{:bridge:br_nf_forward_finish+0} Jul 10 18:20:28 shelbyville kernel: <ffffffff881b52ca>{:bridge:br_nf_forward_ip+346} <ffffffff802deeb7>{skb_checksum+87} Jul 10 18:20:28 shelbyville kernel: <ffffffff802fdc9a>{nf_iterate+90} <ffffffff881afff0>{:bridge:br_forward_finish+0} Jul 10 18:20:28 shelbyville kernel: <ffffffff802fdd5d>{nf_hook_slow+125} <ffffffff881afff0>{:bridge:br_forward_finish+0} Jul 10 18:20:29 shelbyville kernel: <ffffffff881b011c>{:bridge:__br_forward+92} <ffffffff881b0e9b>{:bridge:br_handle_frame_finish+235} Jul 10 18:20:29 shelbyville kernel: <ffffffff881b4711>{:bridge:br_nf_pre_routing_finish+913} Jul 10 18:20:29 shelbyville kernel: <ffffffff802fdc9a>{nf_iterate+90} <ffffffff881b4380>{:bridge:br_nf_pre_routing_finish+0} Jul 10 18:20:29 shelbyville kernel: <ffffffff802fdd5d>{nf_hook_slow+125} <ffffffff881b4380>{:bridge:br_nf_pre_routing_finish+0} Jul 10 18:20:29 shelbyville kernel: <ffffffff881b4fbd>{:bridge:br_nf_pre_routing+2109} <ffffffff802fdc9a>{nf_iterate+90} Jul 10 18:20:29 shelbyville kernel: <ffffffff881b0db0>{:bridge:br_handle_frame_finish+0} Jul 10 18:20:29 shelbyville kernel: <ffffffff802fdd5d>{nf_hook_slow+125} <ffffffff881b0db0>{:bridge:br_handle_frame_finish+0} Jul 10 18:20:29 shelbyville kernel: <ffffffff881b1076>{:bridge:br_handle_frame+374} <ffffffff802e3d52>{netif_receive_skb+578} Jul 10 18:20:29 shelbyville kernel: <ffffffff8806e4b0>{:e100:e100_poll+784} <ffffffff802e40ef>{net_rx_action+239} Jul 10 18:20:29 shelbyville kernel: <ffffffff801345a3>{__do_softirq+131} <ffffffff8010beda>{call_softirq+30} Jul 10 18:20:29 shelbyville kernel: <ffffffff8010dc97>{do_softirq+71} <ffffffff8010dae9>{do_IRQ+73} Jul 10 18:20:29 shelbyville kernel: <ffffffff80250fad>{evtchn_do_upcall+205} <ffffffff8010ba0a>{do_hypervisor_callback+30} <EOI> Jul 10 18:20:29 shelbyville kernel: <ffffffff801073aa>{hypercall_page+938} <ffffffff801073aa>{hypercall_page+938} Jul 10 18:20:29 shelbyville kernel: <ffffffff80108f02>{xen_idle+130} <ffffffff8010902a>{cpu_idle+234} Jul 10 18:20:29 shelbyville kernel: <ffffffff804b575a>{start_kernel+506} <ffffffff804b51b2>{_sinittext+434} Jul 10 18:20:29 shelbyville kernel: DF PROTO=TCP SPT=80 DPT=42978 WINDOW=0 RES=0x00 RST URGP=0
Iptables LOG is just a regular printk user, although it can be a heavy one. Stalls are usually related to a slow serial console or something similar, the backtrace supports that theory. I'm not familiar with Xen, is there something like a virtual serial console?
Ahhh... that probably explains it. Yes, the Xen guests have a virtual serial console, but this is the Dom0 instance, and the output goes to a 19200 real serial console. It may be that it just can't keep up. I've tried rate limiting the iptables logging to see if that helps.
The LOG target shouldn't be used with serial consoles since it might stall the entire machine. I suggest to use ULOG and the ulog daemon instead.