LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-13  3:45                   ` Tino Keitel
@ 2000-01-17  4:43                     ` Tino Keitel
  2007-01-13 10:18                     ` Pavel Machek
  1 sibling, 0 replies; 16+ messages in thread
From: Tino Keitel @ 2000-01-17  4:43 UTC (permalink / raw)
  To: Pavel Machek, Luming Yu, Adrian Bunk, Lee Revell, linux-kernel

On Sat, Jan 13, 2007 at 04:45:12 +0100, Tino Keitel wrote:
> On Sat, Jan 13, 2007 at 04:05:28 +0100, Tino Keitel wrote:
> 
> [...]
> 
> > I think I found the problem. In 2.6.18, I had a slightly different
> > config. With 2.6.20-rc4, I had sucessful suspend/resume cycles without
> > the USB DVB-T box attached. I tweaked the USB options a bit and
> > activated some options (CONFIG_USB_SUSPEND,
> > CONFIG_USB_MULTITHREAD_PROBE, CONFIG_USB_EHCI_SPLIT_ISO,
> > CONFIG_USB_EHCI_ROOT_HUB_TT, CONFIG_USB_EHCI_TT_NEWSCHED) and now I can
> > suspend/resume without hangs. At least I haven't seen one until now.
> 
> Just after I sent the mail, I had 2 failures again. :-(

PM_TRACE revealed that the Ethernet driver (sky2) failed to resume. I
removed the patches for wake on LAN and hope that it works now.

Regards,
Tino

^ permalink raw reply	[flat|nested] 16+ messages in thread

* 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
@ 2007-01-07 15:17 Tino Keitel
  2007-01-07 18:23 ` Lee Revell
  0 siblings, 1 reply; 16+ messages in thread
From: Tino Keitel @ 2007-01-07 15:17 UTC (permalink / raw)
  To: linux-kernel

Hi folks,

I tried 2.6.20-rc3 and suspend to RAM is now broken. The screen stays
dark after resume, the same with the network link. It worked with
2.6.18 (I skipped 2.6.19 because of a regression in the sky2 driver).

I enabled pm_trace and did a echo mem > /sys/power/state in single user
mode.

After the reboot, all I got from pm_trace is this:

ACPI: (supports S0 S3 S4 S5)
  Magic number: 0:798:636
  hash matches drivers/base/power/resume.c:46
Freeing unused kernel memory: 228k freed

This is line 46 in resume.c:

	TRACE_RESUME(error);

No information about the device/driver that refuses to resume.

I think that this is a regression, as it worked with 2.6.18 and the
kernel config is the same. The hardare is a Mac mini Core Duo.

Regards,
Tino

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-07 15:17 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo Tino Keitel
@ 2007-01-07 18:23 ` Lee Revell
  2007-01-07 20:04   ` Tino Keitel
  0 siblings, 1 reply; 16+ messages in thread
From: Lee Revell @ 2007-01-07 18:23 UTC (permalink / raw)
  To: Tino Keitel; +Cc: linux-kernel

On Sun, 2007-01-07 at 16:17 +0100, Tino Keitel wrote:
> No information about the device/driver that refuses to resume.

You should be able to identify the problematic driver by removing each
driver manually before suspending.

Lee


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-07 18:23 ` Lee Revell
@ 2007-01-07 20:04   ` Tino Keitel
  2007-01-07 22:27     ` Tino Keitel
  0 siblings, 1 reply; 16+ messages in thread
From: Tino Keitel @ 2007-01-07 20:04 UTC (permalink / raw)
  To: Lee Revell; +Cc: linux-kernel

On Sun, Jan 07, 2007 at 13:23:13 -0500, Lee Revell wrote:
> On Sun, 2007-01-07 at 16:17 +0100, Tino Keitel wrote:
> > No information about the device/driver that refuses to resume.
> 
> You should be able to identify the problematic driver by removing each
> driver manually before suspending.

I can not reproduce it anymore, resume now works. I really hope that it
will stay so.

Regards,
Tino

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-07 20:04   ` Tino Keitel
@ 2007-01-07 22:27     ` Tino Keitel
  2007-01-07 23:44       ` Adrian Bunk
  2007-01-08 16:17       ` Pavel Machek
  0 siblings, 2 replies; 16+ messages in thread
From: Tino Keitel @ 2007-01-07 22:27 UTC (permalink / raw)
  To: Lee Revell, linux-kernel

On Sun, Jan 07, 2007 at 21:04:53 +0100, Tino Keitel wrote:
> On Sun, Jan 07, 2007 at 13:23:13 -0500, Lee Revell wrote:
> > On Sun, 2007-01-07 at 16:17 +0100, Tino Keitel wrote:
> > > No information about the device/driver that refuses to resume.
> > 
> > You should be able to identify the problematic driver by removing each
> > driver manually before suspending.
> 
> I can not reproduce it anymore, resume now works. I really hope that it
> will stay so.

It didn't. It looks like it is unusable, becuase it isn't reliable in
2.6.20-rc3.

Regards,
Tino

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-07 22:27     ` Tino Keitel
@ 2007-01-07 23:44       ` Adrian Bunk
  2007-01-08 21:04         ` Tino Keitel
  2007-01-08 16:17       ` Pavel Machek
  1 sibling, 1 reply; 16+ messages in thread
From: Adrian Bunk @ 2007-01-07 23:44 UTC (permalink / raw)
  To: Lee Revell, linux-kernel

On Sun, Jan 07, 2007 at 11:27:06PM +0100, Tino Keitel wrote:
> On Sun, Jan 07, 2007 at 21:04:53 +0100, Tino Keitel wrote:
> > On Sun, Jan 07, 2007 at 13:23:13 -0500, Lee Revell wrote:
> > > On Sun, 2007-01-07 at 16:17 +0100, Tino Keitel wrote:
> > > > No information about the device/driver that refuses to resume.
> > > 
> > > You should be able to identify the problematic driver by removing each
> > > driver manually before suspending.
> > 
> > I can not reproduce it anymore, resume now works. I really hope that it
> > will stay so.
> 
> It didn't. It looks like it is unusable, becuase it isn't reliable in
> 2.6.20-rc3.

Is this issue still present in -rc4?

> Regards,
> Tino

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-07 22:27     ` Tino Keitel
  2007-01-07 23:44       ` Adrian Bunk
@ 2007-01-08 16:17       ` Pavel Machek
  2007-01-08 22:02         ` Tino Keitel
  1 sibling, 1 reply; 16+ messages in thread
From: Pavel Machek @ 2007-01-08 16:17 UTC (permalink / raw)
  To: Lee Revell, linux-kernel

On Sun 2007-01-07 23:27:06, Tino Keitel wrote:
> On Sun, Jan 07, 2007 at 21:04:53 +0100, Tino Keitel wrote:
> > On Sun, Jan 07, 2007 at 13:23:13 -0500, Lee Revell wrote:
> > > On Sun, 2007-01-07 at 16:17 +0100, Tino Keitel wrote:
> > > > No information about the device/driver that refuses to resume.
> > > 
> > > You should be able to identify the problematic driver by removing each
> > > driver manually before suspending.
> > 
> > I can not reproduce it anymore, resume now works. I really hope that it
> > will stay so.
> 
> It didn't. It looks like it is unusable, becuase it isn't reliable in
> 2.6.20-rc3.

What was last working version? Can you pinpoint driver breaking it?

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-07 23:44       ` Adrian Bunk
@ 2007-01-08 21:04         ` Tino Keitel
  2007-01-09 14:51           ` Luming Yu
  0 siblings, 1 reply; 16+ messages in thread
From: Tino Keitel @ 2007-01-08 21:04 UTC (permalink / raw)
  To: Adrian Bunk; +Cc: Lee Revell, linux-kernel

On Mon, Jan 08, 2007 at 00:44:45 +0100, Adrian Bunk wrote:
> On Sun, Jan 07, 2007 at 11:27:06PM +0100, Tino Keitel wrote:
> > On Sun, Jan 07, 2007 at 21:04:53 +0100, Tino Keitel wrote:
> > > On Sun, Jan 07, 2007 at 13:23:13 -0500, Lee Revell wrote:
> > > > On Sun, 2007-01-07 at 16:17 +0100, Tino Keitel wrote:
> > > > > No information about the device/driver that refuses to resume.
> > > > 
> > > > You should be able to identify the problematic driver by removing each
> > > > driver manually before suspending.
> > > 
> > > I can not reproduce it anymore, resume now works. I really hope that it
> > > will stay so.
> > 
> > It didn't. It looks like it is unusable, becuase it isn't reliable in
> > 2.6.20-rc3.
> 
> Is this issue still present in -rc4?

I used 2.6.20-rc4 in single user mode, and applied 2 patches from
netdev to get wake on LAN support. This way I was able to set up an
automatic suspend/resume loop. It looked good, but after e.g. 20
minutes, the resume hang. So it is reproduceable with 2.6.20-rc4.
Unfortunately, I can not test the same with 2.6.18, as the wake on LAN
patches need 2.6.20-rc.

Regards,
Tino

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-08 16:17       ` Pavel Machek
@ 2007-01-08 22:02         ` Tino Keitel
  2007-01-08 23:29           ` Rafael J. Wysocki
  0 siblings, 1 reply; 16+ messages in thread
From: Tino Keitel @ 2007-01-08 22:02 UTC (permalink / raw)
  To: linux-kernel

On Mon, Jan 08, 2007 at 17:17:19 +0100, Pavel Machek wrote:
> On Sun 2007-01-07 23:27:06, Tino Keitel wrote:
> > On Sun, Jan 07, 2007 at 21:04:53 +0100, Tino Keitel wrote:
> > > On Sun, Jan 07, 2007 at 13:23:13 -0500, Lee Revell wrote:
> > > > On Sun, 2007-01-07 at 16:17 +0100, Tino Keitel wrote:
> > > > > No information about the device/driver that refuses to resume.
> > > > 
> > > > You should be able to identify the problematic driver by removing each
> > > > driver manually before suspending.
> > > 
> > > I can not reproduce it anymore, resume now works. I really hope that it
> > > will stay so.
> > 
> > It didn't. It looks like it is unusable, becuase it isn't reliable in
> > 2.6.20-rc3.
> 
> What was last working version? Can you pinpoint driver breaking it?

I just used 2.6.18.2 with a manual driven suspend/resume loop and fully
loaded userspace for ca. 40 minutes, without a failure.

I tried to pinpoint the driver with pm_trace, without success (see my
original posting).

Regards,
Tino

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-08 22:02         ` Tino Keitel
@ 2007-01-08 23:29           ` Rafael J. Wysocki
  0 siblings, 0 replies; 16+ messages in thread
From: Rafael J. Wysocki @ 2007-01-08 23:29 UTC (permalink / raw)
  To: Tino Keitel; +Cc: linux-kernel

On Monday, 8 January 2007 23:02, Tino Keitel wrote:
> On Mon, Jan 08, 2007 at 17:17:19 +0100, Pavel Machek wrote:
> > On Sun 2007-01-07 23:27:06, Tino Keitel wrote:
> > > On Sun, Jan 07, 2007 at 21:04:53 +0100, Tino Keitel wrote:
> > > > On Sun, Jan 07, 2007 at 13:23:13 -0500, Lee Revell wrote:
> > > > > On Sun, 2007-01-07 at 16:17 +0100, Tino Keitel wrote:
> > > > > > No information about the device/driver that refuses to resume.
> > > > > 
> > > > > You should be able to identify the problematic driver by removing each
> > > > > driver manually before suspending.
> > > > 
> > > > I can not reproduce it anymore, resume now works. I really hope that it
> > > > will stay so.
> > > 
> > > It didn't. It looks like it is unusable, becuase it isn't reliable in
> > > 2.6.20-rc3.
> > 
> > What was last working version? Can you pinpoint driver breaking it?
> 
> I just used 2.6.18.2 with a manual driven suspend/resume loop and fully
> loaded userspace for ca. 40 minutes, without a failure.
> 
> I tried to pinpoint the driver with pm_trace, without success (see my
> original posting).

Could you please verify if the issue is reproducible when you boot with
init=/bin/bash and suspend with the minimal list of modules loaded?

Rafael


-- 
If you don't have the time to read,
you don't have the time or the tools to write.
		- Stephen King

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-08 21:04         ` Tino Keitel
@ 2007-01-09 14:51           ` Luming Yu
  2007-01-09 23:16             ` Tino Keitel
  0 siblings, 1 reply; 16+ messages in thread
From: Luming Yu @ 2007-01-09 14:51 UTC (permalink / raw)
  To: Adrian Bunk, Lee Revell, linux-kernel

> > > It didn't. It looks like it is unusable, becuase it isn't reliable in
> > > 2.6.20-rc3.
> >
> > Is this issue still present in -rc4?
>
> I used 2.6.20-rc4 in single user mode, and applied 2 patches from
> netdev to get wake on LAN support. This way I was able to set up an
> automatic suspend/resume loop. It looked good, but after e.g. 20
> minutes, the resume hang. So it is reproduceable with 2.6.20-rc4.
> Unfortunately, I can not test the same with 2.6.18, as the wake on LAN
> patches need 2.6.20-rc.

Hmm, do you mean this is the first time of this kind of testing?
Is this issue related to LAN driver?
I guess you should be able to set up an automatic suspend/resume loop
with /proc/acpi/alarm, and test similar with 2.6.18.

--Luming

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-09 14:51           ` Luming Yu
@ 2007-01-09 23:16             ` Tino Keitel
  2007-01-12 14:50               ` Pavel Machek
  0 siblings, 1 reply; 16+ messages in thread
From: Tino Keitel @ 2007-01-09 23:16 UTC (permalink / raw)
  To: Luming Yu; +Cc: Adrian Bunk, Lee Revell, linux-kernel

On Tue, Jan 09, 2007 at 22:51:04 +0800, Luming Yu wrote:
> >> > It didn't. It looks like it is unusable, becuase it isn't reliable in
> >> > 2.6.20-rc3.
> >>
> >> Is this issue still present in -rc4?
> >
> >I used 2.6.20-rc4 in single user mode, and applied 2 patches from
> >netdev to get wake on LAN support. This way I was able to set up an
> >automatic suspend/resume loop. It looked good, but after e.g. 20
> >minutes, the resume hang. So it is reproduceable with 2.6.20-rc4.
> >Unfortunately, I can not test the same with 2.6.18, as the wake on LAN
> >patches need 2.6.20-rc.
> 
> Hmm, do you mean this is the first time of this kind of testing?
> Is this issue related to LAN driver?
> I guess you should be able to set up an automatic suspend/resume loop
> with /proc/acpi/alarm, and test similar with 2.6.18.

Thanks for the hint. I just used /proc/acpi/alarm to set up a
suspend/resume loop and did ca. 100 cycles in a row with 2.6.18.2 in
single user mode, without a failure.

Regards,
Tino

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-09 23:16             ` Tino Keitel
@ 2007-01-12 14:50               ` Pavel Machek
  2007-01-13  3:05                 ` Tino Keitel
  0 siblings, 1 reply; 16+ messages in thread
From: Pavel Machek @ 2007-01-12 14:50 UTC (permalink / raw)
  To: Luming Yu, Adrian Bunk, Lee Revell, linux-kernel

Hi!

> > >> > It didn't. It looks like it is unusable, becuase it isn't reliable in
> > >> > 2.6.20-rc3.
> > >>
> > >> Is this issue still present in -rc4?
> > >
> > >I used 2.6.20-rc4 in single user mode, and applied 2 patches from
> > >netdev to get wake on LAN support. This way I was able to set up an
> > >automatic suspend/resume loop. It looked good, but after e.g. 20
> > >minutes, the resume hang. So it is reproduceable with 2.6.20-rc4.
> > >Unfortunately, I can not test the same with 2.6.18, as the wake on LAN
> > >patches need 2.6.20-rc.
> > 
> > Hmm, do you mean this is the first time of this kind of testing?
> > Is this issue related to LAN driver?
> > I guess you should be able to set up an automatic suspend/resume loop
> > with /proc/acpi/alarm, and test similar with 2.6.18.
> 
> Thanks for the hint. I just used /proc/acpi/alarm to set up a
> suspend/resume loop and did ca. 100 cycles in a row with 2.6.18.2 in
> single user mode, without a failure.

Can you do similar test on 2.6.20 -- w/o network driver loaded (and
generaly minimum drivers?)
							Pavel

-- 
Thanks for all the (sleeping) penguins.

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-12 14:50               ` Pavel Machek
@ 2007-01-13  3:05                 ` Tino Keitel
  2007-01-13  3:45                   ` Tino Keitel
  0 siblings, 1 reply; 16+ messages in thread
From: Tino Keitel @ 2007-01-13  3:05 UTC (permalink / raw)
  To: Pavel Machek; +Cc: Luming Yu, Adrian Bunk, Lee Revell, linux-kernel

On Fri, Jan 12, 2007 at 14:50:25 +0000, Pavel Machek wrote:
> Hi!
> 
> > > >> > It didn't. It looks like it is unusable, becuase it isn't reliable in
> > > >> > 2.6.20-rc3.
> > > >>
> > > >> Is this issue still present in -rc4?
> > > >
> > > >I used 2.6.20-rc4 in single user mode, and applied 2 patches from
> > > >netdev to get wake on LAN support. This way I was able to set up an
> > > >automatic suspend/resume loop. It looked good, but after e.g. 20
> > > >minutes, the resume hang. So it is reproduceable with 2.6.20-rc4.
> > > >Unfortunately, I can not test the same with 2.6.18, as the wake on LAN
> > > >patches need 2.6.20-rc.
> > > 
> > > Hmm, do you mean this is the first time of this kind of testing?
> > > Is this issue related to LAN driver?
> > > I guess you should be able to set up an automatic suspend/resume loop
> > > with /proc/acpi/alarm, and test similar with 2.6.18.
> > 
> > Thanks for the hint. I just used /proc/acpi/alarm to set up a
> > suspend/resume loop and did ca. 100 cycles in a row with 2.6.18.2 in
> > single user mode, without a failure.
> 
> Can you do similar test on 2.6.20 -- w/o network driver loaded (and
> generaly minimum drivers?)

I think I found the problem. In 2.6.18, I had a slightly different
config. With 2.6.20-rc4, I had sucessful suspend/resume cycles without
the USB DVB-T box attached. I tweaked the USB options a bit and
activated some options (CONFIG_USB_SUSPEND,
CONFIG_USB_MULTITHREAD_PROBE, CONFIG_USB_EHCI_SPLIT_ISO,
CONFIG_USB_EHCI_ROOT_HUB_TT, CONFIG_USB_EHCI_TT_NEWSCHED) and now I can
suspend/resume without hangs. At least I haven't seen one until now.

Thanks for you patience and regards,
Tino


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-13  3:05                 ` Tino Keitel
@ 2007-01-13  3:45                   ` Tino Keitel
  2000-01-17  4:43                     ` Tino Keitel
  2007-01-13 10:18                     ` Pavel Machek
  0 siblings, 2 replies; 16+ messages in thread
From: Tino Keitel @ 2007-01-13  3:45 UTC (permalink / raw)
  To: Pavel Machek, Luming Yu, Adrian Bunk, Lee Revell, linux-kernel

On Sat, Jan 13, 2007 at 04:05:28 +0100, Tino Keitel wrote:

[...]

> I think I found the problem. In 2.6.18, I had a slightly different
> config. With 2.6.20-rc4, I had sucessful suspend/resume cycles without
> the USB DVB-T box attached. I tweaked the USB options a bit and
> activated some options (CONFIG_USB_SUSPEND,
> CONFIG_USB_MULTITHREAD_PROBE, CONFIG_USB_EHCI_SPLIT_ISO,
> CONFIG_USB_EHCI_ROOT_HUB_TT, CONFIG_USB_EHCI_TT_NEWSCHED) and now I can
> suspend/resume without hangs. At least I haven't seen one until now.

Just after I sent the mail, I had 2 failures again. :-(

Regards,
Tino

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo
  2007-01-13  3:45                   ` Tino Keitel
  2000-01-17  4:43                     ` Tino Keitel
@ 2007-01-13 10:18                     ` Pavel Machek
  1 sibling, 0 replies; 16+ messages in thread
From: Pavel Machek @ 2007-01-13 10:18 UTC (permalink / raw)
  To: Luming Yu, Adrian Bunk, Lee Revell, linux-kernel

On Sat 2007-01-13 04:45:12, Tino Keitel wrote:
> On Sat, Jan 13, 2007 at 04:05:28 +0100, Tino Keitel wrote:
> 
> [...]
> 
> > I think I found the problem. In 2.6.18, I had a slightly different
> > config. With 2.6.20-rc4, I had sucessful suspend/resume cycles without
> > the USB DVB-T box attached. I tweaked the USB options a bit and
> > activated some options (CONFIG_USB_SUSPEND,
> > CONFIG_USB_MULTITHREAD_PROBE, CONFIG_USB_EHCI_SPLIT_ISO,
> > CONFIG_USB_EHCI_ROOT_HUB_TT, CONFIG_USB_EHCI_TT_NEWSCHED) and now I can
> > suspend/resume without hangs. At least I haven't seen one until now.
> 
> Just after I sent the mail, I had 2 failures again. :-(

CONFIG_USB_MULTITHREAD_PROBE is known bad.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2007-01-13 11:02 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2007-01-07 15:17 2.6.20-rc3 regression: suspend to RAM broken on Mac mini Core Duo Tino Keitel
2007-01-07 18:23 ` Lee Revell
2007-01-07 20:04   ` Tino Keitel
2007-01-07 22:27     ` Tino Keitel
2007-01-07 23:44       ` Adrian Bunk
2007-01-08 21:04         ` Tino Keitel
2007-01-09 14:51           ` Luming Yu
2007-01-09 23:16             ` Tino Keitel
2007-01-12 14:50               ` Pavel Machek
2007-01-13  3:05                 ` Tino Keitel
2007-01-13  3:45                   ` Tino Keitel
2000-01-17  4:43                     ` Tino Keitel
2007-01-13 10:18                     ` Pavel Machek
2007-01-08 16:17       ` Pavel Machek
2007-01-08 22:02         ` Tino Keitel
2007-01-08 23:29           ` Rafael J. Wysocki

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).