BUG: Files corrupt after moving LVM volume to USB disk

BUG: Files corrupt after moving LVM volume to USB disk

[Marti Raudsepp]

Hello LKML,

Second repost of "BUG: Killing and reviving files with USB disks", this time
also destined to linux-fsdevel.

This is a reproducible demonstration of the problem initially reported in my
first e-mail, titled "PROBLEM: 'bio too big device' after moving to a USB
disk" (http://lkml.org/lkml/2007/3/7/657)

Summary:
01. Create LVM volume; initialize dm-crypt; initialize reiserfs; mount
02. Populate the disk with files; sync; flush caches
03. Confirm that the files are still readable and non-corrupt (sha1sum)
04. Migrate logical volume to USB disk; sync; flush caches
05. Confirm that ALL PREVIOUSLY VERIFIED FILES ARE CORRUPT
06. Observe "bio too big device dm-0 (256 > 240)" messages in dmesg
07. Run reiserfsck to check for corruptions -- none.
08. Migrate logical volume back to SATA disk; sync; flush caches
09. Confirm that files are readable and non-corrupt again
10. Clean up


Environment/configuration:
* Linux non 2.6.19-gentoo-r6 #1 Wed Feb 28 20:55:24 EET 2007 x86_64 AMD
  Athlon(tm) 64 Processor 3000+ AuthenticAMD GNU/Linux
* USB disk 120GB Western Digital, model 00UE-00KVT0 (according to udev),
  serial DEF10CD7F64C
* Older SATA disk 200GB Seagate 7200.7, model ST3200822AS
* Motherboard Asus A8N5X, nForce4 chipset
* Offending file system: reiserfs v3.6, mounted with noatime,barrier=flush
* dm-crypt using aes-256 with cbc-essiv:sha256; using assembly-optimized AES
  on x86_64 (CONFIG_CRYPTO_AES_X86_64)
* Test partition is 68 extents times 16MiB = 1088 MiB large (that's the
  largest I could allocate while remaining in one segment -- otherwise lvmove
  wouldn't move the partition back to /dev/sda5 after "defragmenting" it.)
* LVM utilities version: 2.02.17 (2006-12-14)
* LVM library version: 1.02.12 (2006-10-13)
* LVM driver version: 4.10.0
* cryptsetup-luks 1.0.4 (user space interface to dm-crypt)

Involved block devices:
* /dev/sda: My old SATA disk.
* /dev/sda5: The LVM partition on the old disk.
* /dev/sdb: The new offending USB disk; whole disk is used as an LVM physical
            volume.
* /dev/primary: LVM volume group 'primary', consisting of /dev/sdb and
                /dev/sda5
* /dev/primary/punchbag: LVM volume 'punchbag' for demonstration purposes.
* /dev/mapper/crypt-punchbag: The dm-crypt "decrypted" loopback device.


----
00. PV information

[non]# pvdisplay /dev/sda5
  --- Physical volume ---
  PV Name               /dev/sda5
  VG Name               primary
  PV Size               145.83 GB / not usable 2.73 MB
  Allocatable           yes
  PE Size (KByte)       16384
  Total PE              9333
  Free PE               117
  Allocated PE          9216
  PV UUID               YdrDuL-jw5z-J0SA-EEKU-NOC4-6gGR-T90YCA

[non]# pvdisplay /dev/sdb
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               primary
  PV Size               111.79 GB / not usable 9.46 MB
  Allocatable           yes
  PE Size (KByte)       16384
  Total PE              7154
  Free PE               7129
  Allocated PE          25
  PV UUID               nE8C5L-lfI1-VNgs-Q7ei-1Zz3-GeGT-UNhH4p

----
01. Create LVM volume; initialize dm-crypt; initialize reiserfs; mount

[non]# lvcreate -n punchbag --extents 68 primary /dev/sda5
  Logical volume "punchbag" created
[non]# lvs --segments -o +devices
  LV       VG      Attr   #Str Type   SSize   Devices
  [...]
  punchbag primary -wi-a-    1 linear   1.06G /dev/sda5(0)
  [...]
[non]# cryptsetup luksFormat /dev/primary/punchbag -c
aes-cbc-essiv:sha256 -h sha1
  [...]
Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase:
Verify passphrase:
Command successful.
[non]# cryptsetup luksOpen /dev/primary/punchbag crypt-punchbag
Enter LUKS passphrase:
key slot 0 unlocked.
Command successful.
[non]# mkfs.reiserfs /dev/mapper/crypt-punchbag
  [...]
Guessing about desired format.. Kernel 2.6.19-gentoo-r6 is running.
Format 3.6 with standard journal
Count of blocks on the device: 343920
Number of blocks consumed by mkreiserfs formatting process: 8222
Blocksize: 4096
Hash function used to sort names: "r5"
Journal Size 8193 blocks (first block 18)
Journal Max transaction length 1024
inode generation number: 0
UUID: c1857208-5090-49dd-9163-9fb002d96a88
ATTENTION: YOU SHOULD REBOOT AFTER FDISK!
        ALL DATA WILL BE LOST ON '/dev/mapper/crypt-punchbag'!
Continue (y/n):y

Initializing journal - 0%....20%....40%....60%....80%....100%
Syncing..ok
  [...]

ReiserFS is successfully created on /dev/mapper/crypt-punchbag.
[non]# mkdir /mnt/punchbag
[non]# mount /dev/mapper/crypt-punchbag /mnt/punchbag -o noatime,barrier=flush

----
02. Populate the disk with files; sync; flush caches

[non]# cp -r ~marti/mp3 /mnt/punchbag
  [... yes, these are legal mp3s ;)]
cp: writing `/mnt/punchbag/mp3/...': No space left on device
^C
[non]# sync
[non]# echo 3 > /proc/sys/vm/drop_caches
[non]# cd /mnt/punchbag/mp3/mr\ Epic\ -\ Sideways

----
03. Confirm that the files are still readable and non-corrupt (sha1sum)

[non]# sha1sum -c *.sha1
mr Epic - Sideways - 01. Down Low.flac: OK
mr Epic - Sideways - 02. Blue Days.flac: OK
mr Epic - Sideways - 03. Sift.flac: OK
mr Epic - Sideways - 04. Slipping.flac: OK
mr Epic - Sideways - 05. Ruff and Tumble.flac: OK
mr Epic - Sideways - 06. In.flac: OK
mr Epic - Sideways - 07. Out.flac: OK

----
04. Migrate logical volume to USB disk; sync; flush caches

[non]# pvmove -n punchbag /dev/sda5 /dev/sdb
  [...]
  /dev/sda5: Moved: 100.0%
[non]# lvs --segments -o +devices
  LV       VG      Attr   #Str Type   SSize   Devices
  [...]
  punchbag primary -wi-ao    1 linear   1.06G /dev/sdb(25)
[non]# sync
[non]# echo 3 > /proc/sys/vm/drop_caches

----
05. Confirm that all previously verified files are corrupt

[non]# sha1sum -c *.sha1
sha1sum: mr Epic - Sideways - 01. Down Low.flac: Input/output error
mr Epic - Sideways - 01. Down Low.flac: FAILED open or read
sha1sum: mr Epic - Sideways - 02. Blue Days.flac: Input/output error
mr Epic - Sideways - 02. Blue Days.flac: FAILED open or read
sha1sum: mr Epic - Sideways - 03. Sift.flac: Input/output error
mr Epic - Sideways - 03. Sift.flac: FAILED open or read
sha1sum: mr Epic - Sideways - 04. Slipping.flac: Input/output error
mr Epic - Sideways - 04. Slipping.flac: FAILED open or read
sha1sum: mr Epic - Sideways - 05. Ruff and Tumble.flac: Input/output error
mr Epic - Sideways - 05. Ruff and Tumble.flac: FAILED open or read
sha1sum: mr Epic - Sideways - 06. In.flac: Input/output error
mr Epic - Sideways - 06. In.flac: FAILED open or read
sha1sum: mr Epic - Sideways - 07. Out.flac: Input/output error
mr Epic - Sideways - 07. Out.flac: FAILED open or read
sha1sum: WARNING: 7 of 7 listed files could not be read

----
06. Observe "bio too big device dm-0 (256 > 240)" messages in dmesg

[non]# dmesg |tail -n7
[101403.106825] bio too big device dm-0 (256 > 240)
[101403.115228] bio too big device dm-0 (256 > 240)
[101403.115912] bio too big device dm-0 (256 > 240)
[101403.130529] bio too big device dm-0 (256 > 240)
[101403.131220] bio too big device dm-0 (256 > 240)
[101403.141577] bio too big device dm-0 (256 > 240)
[101403.142255] bio too big device dm-0 (256 > 240)

----
07. Run reiserfsck to check for corruptions -- none.

[non]# cd
[non]# umount /mnt/punchbag
[non]# reiserfsck --check /dev/mapper/crypt-punchbag
  [...]
Will read-only check consistency of the filesystem on
/dev/mapper/crypt-punchbag
Will put log info to 'stdout'

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes
###########
reiserfsck --check started at Sat Mar 10 19:00:08 2007
###########
Replaying journal..
Reiserfs journal '/dev/mapper/crypt-punchbag' in blocks [18..8211]: 0
transactions replayed
Checking Internal tree...
0%....20%....40%....60%....80%....100%
Checking Semantic tree...
There are on the filesystem:
        Leaves 282
        Internal nodes 3
        Directories 18
        Other files 196
        Data block pointers 268374 (0 of them are zero)
        Safe links 0
No corruptions found
###########
reiserfsck finished at Sat Mar 10 19:00:10 2007
###########

----
08. Migrate logical volume back to SATA disk; sync; flush caches

[non]# mount /dev/mapper/crypt-punchbag /mnt/punchbag -o noatime,barrier=flush
[non]# cd /mnt/punchbag/mp3/mr\ Epic\ -\ Sideways
[non]# pvmove -n punchbag /dev/sdb /dev/sda5
  [...]
  /dev/sdb: Moved: 100.0%
[non]# lvs --segments -o +devices
  LV       VG      Attr   #Str Type   SSize   Devices
  [...]
  punchbag primary -wi-a-    1 linear   1.06G /dev/sda5(0)
  [...]
[non]# sync
[non]# echo 3 > /proc/sys/vm/drop_caches

----
09. Confirm that files are readable and non-corrupt again

[non]# sha1sum -c *.sha1
mr Epic - Sideways - 01. Down Low.flac: OK
mr Epic - Sideways - 02. Blue Days.flac: OK
mr Epic - Sideways - 03. Sift.flac: OK
mr Epic - Sideways - 04. Slipping.flac: OK
mr Epic - Sideways - 05. Ruff and Tumble.flac: OK
mr Epic - Sideways - 06. In.flac: OK
mr Epic - Sideways - 07. Out.flac: OK

----
10. Clean up

[non]# cd
[non]# umount /mnt/punchbag
[non]# cryptsetup remove crypt-punchbag
[non]# lvremove primary/punchbag
Do you really want to remove active logical volume "punchbag"? [y/n]: y
  Logical volume "punchbag" successfully removed

I repeated this five times and each time was able to reproduce the problem.
More details on the symptoms of the problem in my previous problem report:
http://lkml.org/lkml/2007/3/7/657

I'm not subscribed to the list, so please keep me in Cc:.


Regards,
Marti Raudsepp

Re: BUG: Files corrupt after moving LVM volume to USB disk

[Lennart Sorensen]

On Tue, Mar 20, 2007 at 12:34:08PM +0200, Marti Raudsepp wrote:
> Second repost of "BUG: Killing and reviving files with USB disks", this time
> also destined to linux-fsdevel.
> 
> This is a reproducible demonstration of the problem initially reported in my
> first e-mail, titled "PROBLEM: 'bio too big device' after moving to a USB
> disk" (http://lkml.org/lkml/2007/3/7/657)
> 
> Summary:
> 01. Create LVM volume; initialize dm-crypt; initialize reiserfs; mount
> 02. Populate the disk with files; sync; flush caches
> 03. Confirm that the files are still readable and non-corrupt (sha1sum)
> 04. Migrate logical volume to USB disk; sync; flush caches
> 05. Confirm that ALL PREVIOUSLY VERIFIED FILES ARE CORRUPT
> 06. Observe "bio too big device dm-0 (256 > 240)" messages in dmesg
> 07. Run reiserfsck to check for corruptions -- none.
> 08. Migrate logical volume back to SATA disk; sync; flush caches
> 09. Confirm that files are readable and non-corrupt again
> 10. Clean up

Not that I really have a clue, but some questions anyhow:

(I noticed someone else complained about lvm/dm-crypt/reiserfs problems
a few months ago here: http://lkml.org/lkml/2007/1/18/146)

Does this happen only with this combination, or can you elliminate
something as the cause? 

Does it happen with ext3 or only reiserfs?

Does it require dm-crypt or is just moving the LVM enough to cause the
problem?

Do you ever unmount and shutdown the lvm during this test?

What happens if you were to build the LVM and filesystem, then issue the
move, and then reboot.  Can you then read them from the USB device or is
it still corrupt?

> Environment/configuration:
> * Linux non 2.6.19-gentoo-r6 #1 Wed Feb 28 20:55:24 EET 2007 x86_64 AMD
>  Athlon(tm) 64 Processor 3000+ AuthenticAMD GNU/Linux

Using any insane gentoo compiler options?

> * USB disk 120GB Western Digital, model 00UE-00KVT0 (according to udev),
>  serial DEF10CD7F64C
> * Older SATA disk 200GB Seagate 7200.7, model ST3200822AS
> * Motherboard Asus A8N5X, nForce4 chipset
> * Offending file system: reiserfs v3.6, mounted with noatime,barrier=flush
> * dm-crypt using aes-256 with cbc-essiv:sha256; using assembly-optimized AES
>  on x86_64 (CONFIG_CRYPTO_AES_X86_64)
> * Test partition is 68 extents times 16MiB = 1088 MiB large (that's the
>  largest I could allocate while remaining in one segment -- otherwise lvmove
>  wouldn't move the partition back to /dev/sda5 after "defragmenting" it.)

That sounds strange.

> * LVM utilities version: 2.02.17 (2006-12-14)
> * LVM library version: 1.02.12 (2006-10-13)
> * LVM driver version: 4.10.0
> * cryptsetup-luks 1.0.4 (user space interface to dm-crypt)
> 
> Involved block devices:
> * /dev/sda: My old SATA disk.
> * /dev/sda5: The LVM partition on the old disk.
> * /dev/sdb: The new offending USB disk; whole disk is used as an LVM 
> physical
>            volume.
> * /dev/primary: LVM volume group 'primary', consisting of /dev/sdb and
>                /dev/sda5
> * /dev/primary/punchbag: LVM volume 'punchbag' for demonstration purposes.
> * /dev/mapper/crypt-punchbag: The dm-crypt "decrypted" loopback device.
> 
> 
> ----
> 00. PV information
> 
> [non]# pvdisplay /dev/sda5
>  --- Physical volume ---
>  PV Name               /dev/sda5
>  VG Name               primary
>  PV Size               145.83 GB / not usable 2.73 MB
>  Allocatable           yes
>  PE Size (KByte)       16384
>  Total PE              9333
>  Free PE               117
>  Allocated PE          9216
>  PV UUID               YdrDuL-jw5z-J0SA-EEKU-NOC4-6gGR-T90YCA
> 
> [non]# pvdisplay /dev/sdb
>  --- Physical volume ---
>  PV Name               /dev/sdb
>  VG Name               primary
>  PV Size               111.79 GB / not usable 9.46 MB
>  Allocatable           yes
>  PE Size (KByte)       16384
>  Total PE              7154
>  Free PE               7129
>  Allocated PE          25
>  PV UUID               nE8C5L-lfI1-VNgs-Q7ei-1Zz3-GeGT-UNhH4p
> 
> ----
> 01. Create LVM volume; initialize dm-crypt; initialize reiserfs; mount
> 
> [non]# lvcreate -n punchbag --extents 68 primary /dev/sda5
>  Logical volume "punchbag" created
> [non]# lvs --segments -o +devices
>  LV       VG      Attr   #Str Type   SSize   Devices
>  [...]
>  punchbag primary -wi-a-    1 linear   1.06G /dev/sda5(0)
>  [...]
> [non]# cryptsetup luksFormat /dev/primary/punchbag -c
> aes-cbc-essiv:sha256 -h sha1
>  [...]
> Are you sure? (Type uppercase yes): YES
> Enter LUKS passphrase:
> Verify passphrase:
> Command successful.
> [non]# cryptsetup luksOpen /dev/primary/punchbag crypt-punchbag
> Enter LUKS passphrase:
> key slot 0 unlocked.
> Command successful.
> [non]# mkfs.reiserfs /dev/mapper/crypt-punchbag
>  [...]
> Guessing about desired format.. Kernel 2.6.19-gentoo-r6 is running.
> Format 3.6 with standard journal
> Count of blocks on the device: 343920
> Number of blocks consumed by mkreiserfs formatting process: 8222
> Blocksize: 4096
> Hash function used to sort names: "r5"
> Journal Size 8193 blocks (first block 18)
> Journal Max transaction length 1024
> inode generation number: 0
> UUID: c1857208-5090-49dd-9163-9fb002d96a88
> ATTENTION: YOU SHOULD REBOOT AFTER FDISK!
>        ALL DATA WILL BE LOST ON '/dev/mapper/crypt-punchbag'!
> Continue (y/n):y
> 
> Initializing journal - 0%....20%....40%....60%....80%....100%
> Syncing..ok
>  [...]
> 
> ReiserFS is successfully created on /dev/mapper/crypt-punchbag.
> [non]# mkdir /mnt/punchbag
> [non]# mount /dev/mapper/crypt-punchbag /mnt/punchbag -o 
> noatime,barrier=flush
> 
> ----
> 02. Populate the disk with files; sync; flush caches
> 
> [non]# cp -r ~marti/mp3 /mnt/punchbag
>  [... yes, these are legal mp3s ;)]
> cp: writing `/mnt/punchbag/mp3/...': No space left on device
> ^C
> [non]# sync
> [non]# echo 3 > /proc/sys/vm/drop_caches
> [non]# cd /mnt/punchbag/mp3/mr\ Epic\ -\ Sideways
> 
> ----
> 03. Confirm that the files are still readable and non-corrupt (sha1sum)
> 
> [non]# sha1sum -c *.sha1
> mr Epic - Sideways - 01. Down Low.flac: OK
> mr Epic - Sideways - 02. Blue Days.flac: OK
> mr Epic - Sideways - 03. Sift.flac: OK
> mr Epic - Sideways - 04. Slipping.flac: OK
> mr Epic - Sideways - 05. Ruff and Tumble.flac: OK
> mr Epic - Sideways - 06. In.flac: OK
> mr Epic - Sideways - 07. Out.flac: OK
> 
> ----
> 04. Migrate logical volume to USB disk; sync; flush caches
> 
> [non]# pvmove -n punchbag /dev/sda5 /dev/sdb
>  [...]
>  /dev/sda5: Moved: 100.0%
> [non]# lvs --segments -o +devices
>  LV       VG      Attr   #Str Type   SSize   Devices
>  [...]
>  punchbag primary -wi-ao    1 linear   1.06G /dev/sdb(25)
> [non]# sync
> [non]# echo 3 > /proc/sys/vm/drop_caches
> 
> ----
> 05. Confirm that all previously verified files are corrupt
> 
> [non]# sha1sum -c *.sha1
> sha1sum: mr Epic - Sideways - 01. Down Low.flac: Input/output error
> mr Epic - Sideways - 01. Down Low.flac: FAILED open or read
> sha1sum: mr Epic - Sideways - 02. Blue Days.flac: Input/output error
> mr Epic - Sideways - 02. Blue Days.flac: FAILED open or read
> sha1sum: mr Epic - Sideways - 03. Sift.flac: Input/output error
> mr Epic - Sideways - 03. Sift.flac: FAILED open or read
> sha1sum: mr Epic - Sideways - 04. Slipping.flac: Input/output error
> mr Epic - Sideways - 04. Slipping.flac: FAILED open or read
> sha1sum: mr Epic - Sideways - 05. Ruff and Tumble.flac: Input/output error
> mr Epic - Sideways - 05. Ruff and Tumble.flac: FAILED open or read
> sha1sum: mr Epic - Sideways - 06. In.flac: Input/output error
> mr Epic - Sideways - 06. In.flac: FAILED open or read
> sha1sum: mr Epic - Sideways - 07. Out.flac: Input/output error
> mr Epic - Sideways - 07. Out.flac: FAILED open or read
> sha1sum: WARNING: 7 of 7 listed files could not be read
> 
> ----
> 06. Observe "bio too big device dm-0 (256 > 240)" messages in dmesg
> 
> [non]# dmesg |tail -n7
> [101403.106825] bio too big device dm-0 (256 > 240)
> [101403.115228] bio too big device dm-0 (256 > 240)
> [101403.115912] bio too big device dm-0 (256 > 240)
> [101403.130529] bio too big device dm-0 (256 > 240)
> [101403.131220] bio too big device dm-0 (256 > 240)
> [101403.141577] bio too big device dm-0 (256 > 240)
> [101403.142255] bio too big device dm-0 (256 > 240)
> 
> ----
> 07. Run reiserfsck to check for corruptions -- none.
> 
> [non]# cd
> [non]# umount /mnt/punchbag
> [non]# reiserfsck --check /dev/mapper/crypt-punchbag
>  [...]
> Will read-only check consistency of the filesystem on
> /dev/mapper/crypt-punchbag
> Will put log info to 'stdout'
> 
> Do you want to run this program?[N/Yes] (note need to type Yes if you 
> do):Yes
> ###########
> reiserfsck --check started at Sat Mar 10 19:00:08 2007
> ###########
> Replaying journal..
> Reiserfs journal '/dev/mapper/crypt-punchbag' in blocks [18..8211]: 0
> transactions replayed
> Checking Internal tree...
> 0%....20%....40%....60%....80%....100%
> Checking Semantic tree...
> There are on the filesystem:
>        Leaves 282
>        Internal nodes 3
>        Directories 18
>        Other files 196
>        Data block pointers 268374 (0 of them are zero)
>        Safe links 0
> No corruptions found
> ###########
> reiserfsck finished at Sat Mar 10 19:00:10 2007
> ###########
> 
> ----
> 08. Migrate logical volume back to SATA disk; sync; flush caches
> 
> [non]# mount /dev/mapper/crypt-punchbag /mnt/punchbag -o 
> noatime,barrier=flush
> [non]# cd /mnt/punchbag/mp3/mr\ Epic\ -\ Sideways
> [non]# pvmove -n punchbag /dev/sdb /dev/sda5
>  [...]
>  /dev/sdb: Moved: 100.0%
> [non]# lvs --segments -o +devices
>  LV       VG      Attr   #Str Type   SSize   Devices
>  [...]
>  punchbag primary -wi-a-    1 linear   1.06G /dev/sda5(0)
>  [...]
> [non]# sync
> [non]# echo 3 > /proc/sys/vm/drop_caches
> 
> ----
> 09. Confirm that files are readable and non-corrupt again
> 
> [non]# sha1sum -c *.sha1
> mr Epic - Sideways - 01. Down Low.flac: OK
> mr Epic - Sideways - 02. Blue Days.flac: OK
> mr Epic - Sideways - 03. Sift.flac: OK
> mr Epic - Sideways - 04. Slipping.flac: OK
> mr Epic - Sideways - 05. Ruff and Tumble.flac: OK
> mr Epic - Sideways - 06. In.flac: OK
> mr Epic - Sideways - 07. Out.flac: OK
> 
> ----
> 10. Clean up
> 
> [non]# cd
> [non]# umount /mnt/punchbag
> [non]# cryptsetup remove crypt-punchbag
> [non]# lvremove primary/punchbag
> Do you really want to remove active logical volume "punchbag"? [y/n]: y
>  Logical volume "punchbag" successfully removed
> 
> I repeated this five times and each time was able to reproduce the problem.
> More details on the symptoms of the problem in my previous problem report:
> http://lkml.org/lkml/2007/3/7/657
> 
> I'm not subscribed to the list, so please keep me in Cc:.

--
Len Sorensen

Re: BUG: Files corrupt after moving LVM volume to USB disk

[Marti Raudsepp]

On 3/21/07, Lennart Sorensen <lsorense@csclub.uwaterloo.ca> wrote:
> Not that I really have a clue, but some questions anyhow:
Thanks for your response; I will answer the remaining questions after I've
freed up some space on the USB disk and done some additional tests. Possibly
later today, or maybe tomorrow.

> Do you ever unmount and shutdown the lvm during this test?
> What happens if you were to build the LVM and filesystem, then issue the
> move, and then reboot.  Can you then read them from the USB device or is
> it still corrupt?
Yes, when I first came across this problem, I had done several reboots, prior
to and following the lvmove; the filesystem-level corruptions remained
afterwards (though, as reported, everything worked fine again after moving it
back).

> Using any insane gentoo compiler options?
Nothing insane, just -fstack-protector -march=athlon64 -fomit-frame-pointer

> On Tue, Mar 20, 2007 at 12:34:08PM +0200, Marti Raudsepp wrote:
> > * Test partition is 68 extents times 16MiB = 1088 MiB large (that's the
> >  largest I could allocate while remaining in one segment -- otherwise lvmove
> >  wouldn't move the partition back to /dev/sda5 after "defragmenting" it.)
>
> That sounds strange.
I suppose it's a shortcoming of the current LVM mirror target -- when
allocating a larger LV on the primary disk, it initially creates the volume in
two segments; after done moving them to the secondary disk, it merges them into
one, as the two segments had been allocated continuously.

However, when attempting to move back the LV, the allocater can create two
extents, but the "mirror" LVM target does not appear to support regions that
are continuous on one copy and fragmented on another.

I even tried explicitly specifying --alloc= to lvmove, and it didn't have any
effect, confirming that it's not about allocation, but likely the "mirror"
stage of the move.

Regards,
Marti Raudsepp

Re: BUG: Files corrupt after moving LVM volume to USB disk

[Milan Broz]

Marti Raudsepp wrote:
> This is a reproducible demonstration of the problem initially reported in my
> first e-mail, titled "PROBLEM: 'bio too big device' after moving to a USB
> disk" (http://lkml.org/lkml/2007/3/7/657)
...
> 06. Observe "bio too big device dm-0 (256 > 240)" messages in dmesg
> 
> [non]# dmesg |tail -n7
> [101403.106825] bio too big device dm-0 (256 > 240)
> [101403.115228] bio too big device dm-0 (256 > 240)
> [101403.115912] bio too big device dm-0 (256 > 240)
> [101403.130529] bio too big device dm-0 (256 > 240)
> [101403.131220] bio too big device dm-0 (256 > 240)
> [101403.141577] bio too big device dm-0 (256 > 240)
> [101403.142255] bio too big device dm-0 (256 > 240)

if (unlikely(bio_sectors(bio) > q->max_hw_sectors)) {
	printk("bio too big device %s (%u > %u)\n", 
...

Please could you try if this patch helps ?
http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/dm-merge-max_hw_sector.patch


Milan
--
mbroz@redhat.com

Re: BUG: Files corrupt after moving LVM volume to USB disk

[Alasdair G Kergon]

On Wed, Mar 21, 2007 at 06:15:47PM +0200, Marti Raudsepp wrote:
> >On Tue, Mar 20, 2007 at 12:34:08PM +0200, Marti Raudsepp wrote:
> >lvmove
> >>  wouldn't move the partition back to /dev/sda5 after "defragmenting" it.)

> However, when attempting to move back the LV, the allocater can create two
> extents, but the "mirror" LVM target does not appear to support regions that
> are continuous on one copy and fragmented on another.
 
This is entirely a user-space restriction, and a fix is finally on the
cards.

> I even tried explicitly specifying --alloc= to lvmove, and it didn't have 
> any
> effect, confirming that it's not about allocation, but likely the "mirror"
> stage of the move.

The workaround is to instruct 'pvmove' how to break up the move into two
or more pieces by specifying ranges of Physical Extents on the command
line (see man pages).  E.g. /dev/sdc:10000-19999:30000-
 
Alasdair
-- 
agk@redhat.com