Re: Vmware crashes if compress/misc.c scrolls?

Re: Vmware crashes if compress/misc.c scrolls?

[Zachary Amsden]

H. Peter Anvin wrote:
> I just got the following message on the syslinux mailing list:
>
>   
>> 2. On some platforms (vmware for example :), READING from the video memory
>>    in the 32bit mode is impossible (causes an exeption). Taking in to account
>>    that the scroll function in ilinux/arch/i386/boot/compressed/misc.c
>>    works using a memcpy of the video memory, when the linux bootsector is given
>>    control and the cursor is at the end of the screen (-1-2 lines) an exception is
>>    arisen and in the presence of it's handler in the bootloader the system halts.
>>    Both lilo and grub handle this situation - the cursor is always somewhere at the
>>    middle of the screen when the control is given to the linux bootsector.
>>    When using pxelinux taking in to account that the PXE bios prints out the
>>    presence of the second initramfs which causes the screen to scroll and the bug
>>    to appear :)
>>    A very simple solution - the screen is cleared before giving control to the
>>    linux bootsector (in the patch).
>>     
>
> First of all, is this actually true?  This seems like a monumental screwup.
>
> Second, assuming it is true, what should be done about it?  The notion
> that the bootloader should erase the screen is ridiculous since we spend
> a lot of effort maintaining the screen context from 16-bit code.
>
> I see a couple of options:
>
> a. Do nothing.  It's a Vmware bug, let the users bug Vmware and get an
>    updated version.
> b. Detect Vmware versions that are affected (how?) and suppress output.
> c. Don't attempt to scroll in this code, and instead wrap around the
>    screen (presumably clearing the rest of the line.)
> d. Blindly assume hardware scrolling works, instead of doing
>    copy-to-scroll.  Although I have seen craptops on which hardware
>    scrolling is broken, it's hardly critical messages we're dealing with
>    here.  A bug for a bug...
>   

Sorry it took me so long to follow up on this.

It's not a VMware bug.  Video memory operates just fine under VMware.  
This led me to put the bug on the back-burner.

Today it came back to the front, and I did some investigation.  It's 
also not a syslinux bug, although it is triggered by syslinux.

Depending on how much information is on the screen, you can trigger a 
Linux bug caused by a compiler problem.

In the boot decompressor for the kernel in the image Iouri provided, I 
find the following code present in memory before the first kernel 
instruction executes:

Here is Linux function scroll, which calls memcpy to scroll vga memory:

0x104131 :     mov    %edx,0x4(%esp,1)
0x104135 :     mov    %ecx,(%esp,1)
0x104138 :     mov    %eax,0x8(%esp,1)
0x10413c :     call   0x1040d0

Here is Linux function memcpy:

(gdb) x/40i 0x1040d0
0x1040d0 :     push   %ebp
0x1040d1 :     test   %ecx,%ecx
0x1040d3 :     mov    %esp,%ebp
0x1040d5 :     push   %edi
0x1040d6 :     mov    %eax,%edi
0x1040d8 :     push   %esi
0x1040d9 :     mov    %edx,%esi
0x1040db :     push   %ebx
0x1040dc :     mov    %ecx,%ebx
0x1040de :     je     0x1040f4
0x1040e0 :     xor    %ecx,%ecx
0x1040e2 :     xor    %edx,%edx
0x1040e4 :     movzbl (%esi,%edx,1),%eax
0x1040e8 :     add    $0x1,%ecx
0x1040eb :     cmp    %ebx,%ecx
0x1040ed :     mov    %al,(%edi,%edx,1)
0x1040f0 :     mov    %ecx,%edx
0x1040f2 :     jne    0x1040e4
0x1040f4 :     mov    %edi,%eax
0x1040f6 :     pop    %ebx
0x1040f7 :     pop    %esi
0x1040f8 :     pop    %edi
0x1040f9 :     pop    %ebp
0x1040fa :     ret


As you can plainly see, the call to memcpy (which is redefined in 
boot/compressed/misc.c) is made using stack calling convention.  
Unfortunately, the compiler generated the memcpy function itself using 
regparm(3) convention.  I am guessing this happened because a leftover 
prototype declaring memcpy as regparm got pulled in from somewhere, but 
gcc ignored it in the callsite because the function was declared static 
in the same file.

The net result of this is the parameters to memcpy get swapped, causing 
a memcpy from 0xb80a0 to destination (in register edx, 80x24 * 2) 0xa00, 
of size 0xb8000.

This corrupts all usable memory below 640k, including the GDT which is 
currently loaded.

But since the compressed kernel is running in high memory, above 1Mb, 
the kernel is fine.  The compress loader head.S then proceeds to copy 
the move routine to address 0x11000, and executes the following instruction:

        ljmp $(__KERNEL_CS), $0x1000 # and jump to the move routine

This reloads CS descriptor from the GDT, which is now garbage, and 
contains an invalid segment.  The ljmp then generates a #GP.  Since 
there is no IDT setup to handle the GDT (and even if there were, the 
destination CS would again be reloaded, causing a fault), a double fault 
is generated.  Since the double-fault also hits the same problem, the 
machine then triple faults and proceeds to reboot.

Since the SCREEN_INFO passed to the kernel in the bad case has orig_y is 
set to 22, when we print 3 newlines:

        putstr(".\nDecompressing Linux...");
        gunzip();
        putstr("done.\nBooting the kernel.\n");

We cause the VGA to scroll, and trigger the bug.

Does anyone recall compiler problems (I noticed this VM was booting a 
64-bit kernel, but the decompressor was 32-bit code, so perhaps at some 
time the 64-bit makefile or toolchain for Linux had some kind of 
compiler bug).

Zach

Re: Vmware crashes if compress/misc.c scrolls?

[Andi Kleen]

> In the boot decompressor for the kernel in the image Iouri provided, I

32bit or 64bit image?

> As you can plainly see, the call to memcpy (which is redefined in
> boot/compressed/misc.c) is made using stack calling convention.
> Unfortunately, the compiler generated the memcpy function itself using
> regparm(3) convention.  I am guessing this happened because a leftover

I can't find any here grepping .i files and includes. None of the memcpy
prototypes in include have a __fastcall

Are you sure this still happens in mainline?

When I look at my scroll it also looks correct:

      c0:       0f af f8                imul   %eax,%edi
      c3:       01 c0                   add    %eax,%eax
      c5:       89 c2                   mov    %eax,%edx
      c7:       01 f2                   add    %esi,%edx
      c9:       89 45 f0                mov    %eax,0xfffffff0(%ebp)
      cc:       89 f0                   mov    %esi,%eax
      ce:       89 f9                   mov    %edi,%ecx
      d0:       e8 7b ff ff ff          call   50 <memcpy>


> Does anyone recall compiler problems (I noticed this VM was booting a
> 64-bit kernel, 

Ok 64bit, that was 32bit above. 

Since some time the decompressor is 64bit and 64bit always uses 
regparms. 64bit Scroll is ok too:

     92:       0f af eb                imul   %ebx,%ebp
      95:       01 db                   add    %ebx,%ebx
      97:       48 63 f3                movslq %ebx,%rsi
      9a:       4c 01 ee                add    %r13,%rsi
      9d:       89 ea                   mov    %ebp,%edx
      9f:       e8 9c ff ff ff          callq  40 <memcpy>


> but the decompressor was 32-bit code, 

That must be a old kernel.

Can you people please verify this all still happens with a mainline or
2.6.22 kernel? 

> so perhaps at some  
> time the 64-bit makefile or toolchain for Linux had some kind of
> compiler bug).

I'm not aware of any such bugs.

-Andi

Re: Vmware crashes if compress/misc.c scrolls?

[Zachary Amsden]

Andi Kleen wrote:
>> In the boot decompressor for the kernel in the image Iouri provided, I
>>     
>
> 32bit or 64bit image?
>
>   
>> As you can plainly see, the call to memcpy (which is redefined in
>> boot/compressed/misc.c) is made using stack calling convention.
>> Unfortunately, the compiler generated the memcpy function itself using
>> regparm(3) convention.  I am guessing this happened because a leftover
>>     
>
> I can't find any here grepping .i files and includes. None of the memcpy
> prototypes in include have a __fastcall
>
> Are you sure this still happens in mainline?
>
> When I look at my scroll it also looks correct:
>
>       c0:       0f af f8                imul   %eax,%edi
>       c3:       01 c0                   add    %eax,%eax
>       c5:       89 c2                   mov    %eax,%edx
>       c7:       01 f2                   add    %esi,%edx
>       c9:       89 45 f0                mov    %eax,0xfffffff0(%ebp)
>       cc:       89 f0                   mov    %esi,%eax
>       ce:       89 f9                   mov    %edi,%ecx
>       d0:       e8 7b ff ff ff          call   50 <memcpy>
>
>
>   
>> Does anyone recall compiler problems (I noticed this VM was booting a
>> 64-bit kernel, 
>>     
>
> Ok 64bit, that was 32bit above. 
>
> Since some time the decompressor is 64bit and 64bit always uses 
> regparms. 64bit Scroll is ok too:
>
>      92:       0f af eb                imul   %ebx,%ebp
>       95:       01 db                   add    %ebx,%ebx
>       97:       48 63 f3                movslq %ebx,%rsi
>       9a:       4c 01 ee                add    %r13,%rsi
>       9d:       89 ea                   mov    %ebp,%edx
>       9f:       e8 9c ff ff ff          callq  40 <memcpy>
>
>
>   
>> but the decompressor was 32-bit code, 
>>     
>
> That must be a old kernel.
>
> Can you people please verify this all still happens with a mainline or
> 2.6.22 kernel? 
>   

It doesn't.

>   
>> so perhaps at some  
>> time the 64-bit makefile or toolchain for Linux had some kind of
>> compiler bug).
>>     
>
> I'm not aware of any such bugs.
>
>   

Me neither, nor do any of my current 32-bit or 64-bit kernels have this 
problem.  I was just wondering if it got fixed deliberately, or if some 
old gcc bug out there could still cause such problems with specific 
toolchains.

I'll look up what version the kernel was; Iouri, can you provide us with 
the binutils / gcc versions used to build your kernel?

Zach

Re: Vmware crashes if compress/misc.c scrolls?

[Zachary Amsden]

Andi Kleen wrote:
>> In the boot decompressor for the kernel in the image Iouri provided, I
>>     
>
> 32bit or 64bit image?
>
>   
>> As you can plainly see, the call to memcpy (which is redefined in
>> boot/compressed/misc.c) is made using stack calling convention.
>> Unfortunately, the compiler generated the memcpy function itself using
>> regparm(3) convention.  I am guessing this happened because a leftover
>>     
>
> I can't find any here grepping .i files and includes. None of the memcpy
> prototypes in include have a __fastcall
>
> Are you sure this still happens in mainline?
>
> When I look at my scroll it also looks correct:
>
>       c0:       0f af f8                imul   %eax,%edi
>       c3:       01 c0                   add    %eax,%eax
>       c5:       89 c2                   mov    %eax,%edx
>       c7:       01 f2                   add    %esi,%edx
>       c9:       89 45 f0                mov    %eax,0xfffffff0(%ebp)
>       cc:       89 f0                   mov    %esi,%eax
>       ce:       89 f9                   mov    %edi,%ecx
>       d0:       e8 7b ff ff ff          call   50 <memcpy>
>
>
>   
>> Does anyone recall compiler problems (I noticed this VM was booting a
>> 64-bit kernel, 
>>     
>
> Ok 64bit, that was 32bit above. 
>
> Since some time the decompressor is 64bit and 64bit always uses 
> regparms. 64bit Scroll is ok too:
>
>      92:       0f af eb                imul   %ebx,%ebp
>       95:       01 db                   add    %ebx,%ebx
>       97:       48 63 f3                movslq %ebx,%rsi
>       9a:       4c 01 ee                add    %r13,%rsi
>       9d:       89 ea                   mov    %ebp,%edx
>       9f:       e8 9c ff ff ff          callq  40 <memcpy>
>
>
>   
>> but the decompressor was 32-bit code, 
>>     
>
> That must be a old kernel.
>
> Can you people please verify this all still happens with a mainline or
> 2.6.22 kernel? 
>
>   
>> so perhaps at some  
>> time the 64-bit makefile or toolchain for Linux had some kind of
>> compiler bug).
>>     
>
> I'm not aware of any such bugs.
>   

It was 2.6.21.4, so it is fairly recent, but with 32-bit decompress 
stage.  GCC was 4.2.0.

Sounds like a gcc bug, to me.

Zach

Re: [syslinux] Vmware crashes if compress/misc.c scrolls?

[H. Peter Anvin]

Zachary Amsden wrote:
> 
> It was 2.6.21.4, so it is fairly recent, but with 32-bit decompress 
> stage.  GCC was 4.2.0.
> 
> Sounds like a gcc bug, to me.
> 

I would argue that anything that causes a function and its callsite to
diverge, when they're part of the same file, is a gcc bug almost by
definition.

That doesn't mean it's not a problem and that there is not a solution --
if nothing else there should be a patch for -stable.

	-hpa