Raid 10 Problems?

Raid 10 Problems?

[Marc Perkel]

Running into a problem and not sure what I'm doing
wrong. Created a software raid 10 array. Everything
seems to be normal except that if you take the array
down and run e2fsck on it there are always errors,
mostly all little stuff and it recovers without losing
any data.

I'm running the latest OpenVZ kernel 2.6.18. I'm not
sure if this is a factor or not as the problem occurs
without starting any VEs.

The problem will happen most all the time. I can
format the partition, put the data on it then shut it
down and run 2efsck and there will be some errors to
fix.

In the past after running for weeks the raid array
will unexpectedly go inti read only mode due to the
errors. After running e2fsck it will run normal again.

I've never used raid 10 before (stripes on top of 2
mirrors) so I don't have anything to compare this
with. I'm just wondering if I'm doing something wrong.
I am using ACL and dir_index attributes.

If I run e2fsck twice in a row it comes up with no
errors he second time. However if I mount the /dev/md2
device and then immediately umount it and run e2fsck
again I get some errors. Here's a run I did just
mounting and unmounting immediately.

e2fsck 1.39 (29-May-2006)
Pass 1: Checking inodes, blocks, and sizes
Inode 17172774, i_blocks is 8304, should be 19272.
Fix? yes

Inode 37126149, i_blocks is 872, should be 2192. Fix?
yes

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: +(39500534--39501904)
+(74267806--74267970)
Fix? yes

Free blocks count wrong for group #1205 (1371,
counted=0).
Fix? yes

Free blocks count wrong for group #2266 (17206,
counted=17041).
Fix? yes

Free blocks count wrong (218321825,
counted=218320289).
Fix? yes


/vz: ***** FILE SYSTEM WAS MODIFIED *****

125791 inodes used (0.11%)
451 non-contiguous inodes (0.4%)
# of inodes with ind/dind/tind blocks: 7086/648/0
15831007 blocks used (6.76%)
0 bad blocks
3 large files

107783 regular files
11529 directories
1584 character device files
10 block device files
3 fifos
1 link
4872 symbolic links (4864 fast symbolic links)
1 socket
--------
125783 files


 
____________________________________________________________________________________
Get your own web address.  
Have a HUGE year through Yahoo! Small Business.
http://smallbusiness.yahoo.com/domains/?p=BESTDEAL

Re: Raid 10 Problems?

[Jan Engelhardt]

On Mar 4 2007 08:25, Marc Perkel wrote:
>I'm running the latest OpenVZ kernel 2.6.18. I'm not
>sure if this is a factor or not as the problem occurs
>without starting any VEs.
>
>I've never used raid 10 before (stripes on top of 2
>mirrors) so I don't have anything to compare this
>with. I'm just wondering if I'm doing something wrong.

Are you using raid1+0 (3 md devices) or raid10 (1 raid device)?
Depending on which, you might want to try the other.


Jan
-- 

Re: Raid 10 Problems?

[Marc Perkel]

--- Jan Engelhardt <jengelh@linux01.gwdg.de> wrote:

> 
> On Mar 4 2007 08:25, Marc Perkel wrote:
> >I'm running the latest OpenVZ kernel 2.6.18. I'm
> not
> >sure if this is a factor or not as the problem
> occurs
> >without starting any VEs.
> >
> >I've never used raid 10 before (stripes on top of 2
> >mirrors) so I don't have anything to compare this
> >with. I'm just wondering if I'm doing something
> wrong.
> 
> Are you using raid1+0 (3 md devices) or raid10 (1
> raid device)?
> Depending on which, you might want to try the other.
> 
> 
> Jan
> -- 


I'm using 3 devices. Can you use just one? If so -
how?
How do I create a raid 10 using mdadm?



 
____________________________________________________________________________________
Want to start your own business?
Learn how on Yahoo! Small Business.
http://smallbusiness.yahoo.com/r-index

Re: Raid 10 Problems?

[Jan Engelhardt]

On Mar 4 2007 15:10, Marc Perkel wrote:
>> On Mar 4 2007 08:25, Marc Perkel wrote:
>> >I'm running the latest OpenVZ kernel 2.6.18. I'm
>> not
>> >sure if this is a factor or not as the problem
>> occurs
>> >without starting any VEs.
>> >
>> >I've never used raid 10 before (stripes on top of 2
>> >mirrors) so I don't have anything to compare this
>> >with. I'm just wondering if I'm doing something
>> wrong.
>> 
>> Are you using raid1+0 (3 md devices) or raid10 (1
>> raid device)?
>> Depending on which, you might want to try the other.
>
>
>I'm using 3 devices. Can you use just one? If so -
>how?
>How do I create a raid 10 using mdadm?

mdadm -C /dev/md0 -N nicearray -b internal -e 1.0 -l 10 -n 4 /dev/sd[abcd]1

for example. (See the manpage for details.)


Jan
-- 

Re: Raid 10 Problems?

[Marc Perkel]

--- Jan Engelhardt <jengelh@linux01.gwdg.de> wrote:

> 
> On Mar 4 2007 15:10, Marc Perkel wrote:
> >> On Mar 4 2007 08:25, Marc Perkel wrote:
> >> >I'm running the latest OpenVZ kernel 2.6.18. I'm
> >> not
> >> >sure if this is a factor or not as the problem
> >> occurs
> >> >without starting any VEs.
> >> >
> >> >I've never used raid 10 before (stripes on top
> of 2
> >> >mirrors) so I don't have anything to compare
> this
> >> >with. I'm just wondering if I'm doing something
> >> wrong.
> >> 
> >> Are you using raid1+0 (3 md devices) or raid10 (1
> >> raid device)?
> >> Depending on which, you might want to try the
> other.
> >
> >
> >I'm using 3 devices. Can you use just one? If so -
> >how?
> >How do I create a raid 10 using mdadm?
> 
> mdadm -C /dev/md0 -N nicearray -b internal -e 1.0 -l
> 10 -n 4 /dev/sd[abcd]1
> 
> for example. (See the manpage for details.)
> 

Thanks - because of your suggestion I had found the
instructions. But you have some interesting options
set. 

-N nicearray -b internal -e 1.0

Are these important? I can restart the process. I
think that I found my original problem. I forgot to
use:

pci=nommconf iommu=soft




 
____________________________________________________________________________________
We won't tell. Get more on shows you hate to love 
(and love to hate): Yahoo! TV's Guilty Pleasures list.
http://tv.yahoo.com/collections/265 

Re: Raid 10 Problems?

[Jan Engelhardt]

On Mar 4 2007 19:17, Marc Perkel wrote:
>Thanks - because of your suggestion I had found the
>instructions. But you have some interesting options
>set. 
>
>-N nicearray -b internal -e 1.0
>
>Are these important?

  -N? What's in a name? I suppose, it's not so important.
  (Arrays are identified by their UUID anyway. But maybe
  udev can do something with the name someday as it does
  today with /dev/disk/*.)

  -b internal -- seems like a good idea to speed up
  resynchronization.

  -e 1.0 -- I wonder why the new superblock format is
  not default in mdadm (0.90 is still used).


>pci=nommconf iommu=soft

The nvidia chipset corruption problem?



Jan
-- 

Re: Raid 10 Problems?

[Marc Perkel]

--- Jan Engelhardt <jengelh@linux01.gwdg.de> wrote:

> 
> On Mar 4 2007 19:17, Marc Perkel wrote:
> >Thanks - because of your suggestion I had found the
> >instructions. But you have some interesting options
> >set. 
> >
> >-N nicearray -b internal -e 1.0
> >
> >Are these important?
> 
>   -N? What's in a name? I suppose, it's not so
> important.
>   (Arrays are identified by their UUID anyway. But
> maybe
>   udev can do something with the name someday as it
> does
>   today with /dev/disk/*.)

Not worth starting over for.

> 
>   -b internal -- seems like a good idea to speed up
>   resynchronization.

I'm trying to figure out what the default is. 

> 
>   -e 1.0 -- I wonder why the new superblock format
> is
>   not default in mdadm (0.90 is still used).
> 

Looks interesting for big arrays but not sure it's
worth starting over for. Trying to get through a 2
hour sync using 4 500gb sata 2 drives.

> 
> >pci=nommconf iommu=soft
> 
> The nvidia chipset corruption problem?
> 

Yep - that's the one.



 
____________________________________________________________________________________
It's here! Your new message!  
Get new email alerts with the free Yahoo! Toolbar.
http://tools.search.yahoo.com/toolbar/features/mail/

Re: Raid 10 Problems?

[Jan Engelhardt]

On Mar 4 2007 19:37, Marc Perkel wrote:
>> 
>>   -b internal -- seems like a good idea to speed up
>>   resynchronization.
>
>I'm trying to figure out what the default is. 

-b none, meaning the whole drive will be resynchronized when the
even counters don't match.

http://gentoo-wiki.com/HOWTO_Install_on_Software_RAID#Write-intent_bitmap


Jan
-- 

Re: Raid 10 Problems?

[Marc Perkel]

--- Jan Engelhardt <jengelh@linux01.gwdg.de> wrote:

> 
> On Mar 4 2007 19:37, Marc Perkel wrote:
> >> 
> >>   -b internal -- seems like a good idea to speed
> up
> >>   resynchronization.
> >
> >I'm trying to figure out what the default is. 
> 
> -b none, meaning the whole drive will be
> resynchronized when the
> even counters don't match.
> 
>
http://gentoo-wiki.com/HOWTO_Install_on_Software_RAID#Write-intent_bitmap
> 
> 

That information has been extremely useful. Thanks a
lot. I fund a command to do the bitmap internal after
the array was made so I added that. Seems like some of
these features should be default. Maybe it's time for
the raid folks to update what is default?



 
____________________________________________________________________________________
8:00? 8:25? 8:40? Find a flick in no time 
with the Yahoo! Search movie showtime shortcut.
http://tools.search.yahoo.com/shortcuts/#news

Re: Raid 10 Problems?

[Jan Engelhardt]

On Mar 5 2007 07:59, Marc Perkel wrote:
>
>That information has been extremely useful. Thanks a
>lot. I fund a command to do the bitmap internal after
>the array was made so I added that. Seems like some of
>these features should be default. Maybe it's time for
>the raid folks to update what is default?

So write to linux-raid@vger


Jan
-- 

Re: Raid 10 Problems?

[Bill Davidsen]

Marc Perkel wrote:
> --- Jan Engelhardt <jengelh@linux01.gwdg.de> wrote:
> 
>> On Mar 4 2007 19:17, Marc Perkel wrote:
>>> Thanks - because of your suggestion I had found the
>>> instructions. But you have some interesting options
>>> set. 
>>>
>>> -N nicearray -b internal -e 1.0
>>>
>>> Are these important?
>>   -N? What's in a name? I suppose, it's not so
>> important.
>>   (Arrays are identified by their UUID anyway. But
>> maybe
>>   udev can do something with the name someday as it
>> does
>>   today with /dev/disk/*.)
> 
> Not worth starting over for.
> 
>>   -b internal -- seems like a good idea to speed up
>>   resynchronization.
> 
> I'm trying to figure out what the default is. 
> 
>>   -e 1.0 -- I wonder why the new superblock format
>> is
>>   not default in mdadm (0.90 is still used).
>>
> 
> Looks interesting for big arrays but not sure it's
> worth starting over for. Trying to get through a 2
> hour sync using 4 500gb sata 2 drives.

That's exactly why you want the bitmap. Fortunately you can add it after 
the array is created. Now the bad news, you should read and understand 
the meaning of the "far" layout. Part of the information is in the mdadm 
man page under -p, some in the md man page. Use of "far" layout will 
effect the performance of the array, the balance of read vs. write 
performance, and (maybe) the reliability.

Two hours is a pretty short time to invest if you find that you have 
your layout wrong and would be better off for the life of the array with 
some other data layout. And the time to do the reading is worth if if 
you wind up convinced that the default settings are fine for you.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Raid 10 Problems?

[Jan Engelhardt]

>> > -b internal -- seems like a good idea to speed up
>> > resynchronization.
>> 
>> I'm trying to figure out what the default is. 
>> >  -e 1.0 -- I wonder why the new superblock format
>> > is
>> > not default in mdadm (0.90 is still used).
>> > 
>> 
>> Looks interesting for big arrays but not sure it's
>> worth starting over for. Trying to get through a 2
>> hour sync using 4 500gb sata 2 drives.
>
> That's exactly why you want the bitmap. Fortunately you can add it after the
> array is created. Now the bad news, you should read and understand the meaning
> of the "far" layout. Part of the information is in the mdadm man page under -p,
> some in the md man page. Use of "far" layout will effect the performance of the
> array, the balance of read vs. write performance, and (maybe) the reliability.

So to keep it simple, don't use far unless you have a reason to.


Jan
-- 

Re: Raid 10 Problems?

[dean gaudet]

On Mon, 5 Mar 2007, Marc Perkel wrote:

> 
> --- Jan Engelhardt <jengelh@linux01.gwdg.de> wrote:
> 
> > 
> > On Mar 4 2007 19:37, Marc Perkel wrote:
> > >> 
> > >>   -b internal -- seems like a good idea to speed
> > up
> > >>   resynchronization.
> > >
> > >I'm trying to figure out what the default is. 
> > 
> > -b none, meaning the whole drive will be
> > resynchronized when the
> > even counters don't match.
> > 
> >
> http://gentoo-wiki.com/HOWTO_Install_on_Software_RAID#Write-intent_bitmap
> > 
> > 
> 
> That information has been extremely useful. Thanks a
> lot. I fund a command to do the bitmap internal after
> the array was made so I added that. Seems like some of
> these features should be default. Maybe it's time for
> the raid folks to update what is default?

the bitmap has performance implications... for example:

http://www.mail-archive.com/linux-raid@vger.kernel.org/msg07229.html

i still prefer to use the bitmap -- i just make it external for devices 
which are busy.

note that unless you tweak your init scripts you'll need to put external 
bitmaps on your root partition, see this thread:

http://www.mail-archive.com/linux-raid@vger.kernel.org/msg06441.html

-dean

Re: Raid 10 Problems?

[Jan Engelhardt]

On Mar 7 2007 10:20, dean gaudet wrote:
>>> http://gentoo-wiki.com/HOWTO_Install_on_Software_RAID#Write-intent_bitmap
>> 
>> That information has been extremely useful. Thanks a
>> lot. I fund a command to do the bitmap internal after
>> the array was made so I added that. Seems like some of
>> these features should be default. Maybe it's time for
>> the raid folks to update what is default?
>
>the bitmap has performance implications... for example:
>http://www.mail-archive.com/linux-raid@vger.kernel.org/msg07229.html

I wonder if bitmapping a raid1 volume is faster than bmp.ing raid5.

The other thing is, the bitmap is supposed to be written out at intervals,
not at every write, so the extra head movement for bitmap updates should
be really low, and not making the tar -xjf process slower by half a
minute.
Is there a way to tweak the write-bitmap-to-disk interval? Perhaps 
something in /sys or ye olde /proc. Maybe linux-raid@ knows 8)


>note that unless you tweak your init scripts you'll need to put external 
>bitmaps on your root partition, see this thread:

Huh? That statement does not make sense. But I think you meant: when using
external bitmaps, adjust the init scripts. Because internal bitmaps are good
for one thing: you don't need to change anything.


Jan
-- 

Re: Raid 10 Problems?

[Michael Tokarev]

Jan Engelhardt wrote:
[]
> The other thing is, the bitmap is supposed to be written out at intervals,
> not at every write, so the extra head movement for bitmap updates should
> be really low, and not making the tar -xjf process slower by half a minute.
> Is there a way to tweak the write-bitmap-to-disk interval? Perhaps 
> something in /sys or ye olde /proc. Maybe linux-raid@ knows 8)

Hmm.  Bitmap is supposed to be written before actual data write, to mark
the to-be-written areas of the array as "being written", so that those
areas can be detected and recovered in case of power failure during
actual write.

So in case of writing to a clean array, head movement always takes place -
first got to bitmap area, and second to the actual data area.

That "written at intervals" is about clearing the bitmaps after some idle
time.

In other words, dirtying bitmap bits occurs right before actual write,
and clearing bits occurs at intervals.

Sure, if you write to (or near) the same place again and again, without
giving a chance to md subsystem to actually clean the bitmap, there will
be no additional head movement.  And that means, for example, tar -xjf
sometimes, since filesystem will place the files being extracted close to
each other, thus hitting the same bit in the bitmap, hence md will skip
repeated bitmap updates in this case.

/mjt

Re: Raid 10 Problems?

[Marc Perkel]

--- Michael Tokarev <mjt@tls.msk.ru> wrote:

> Jan Engelhardt wrote:
> []
> > The other thing is, the bitmap is supposed to be
> written out at intervals,
> > not at every write, so the extra head movement for
> bitmap updates should
> > be really low, and not making the tar -xjf process
> slower by half a minute.
> > Is there a way to tweak the write-bitmap-to-disk
> interval? Perhaps 
> > something in /sys or ye olde /proc. Maybe
> linux-raid@ knows 8)
> 
> Hmm.  Bitmap is supposed to be written before actual
> data write, to mark
> the to-be-written areas of the array as "being
> written", so that those
> areas can be detected and recovered in case of power
> failure during
> actual write.
> 
> So in case of writing to a clean array, head
> movement always takes place -
> first got to bitmap area, and second to the actual
> data area.
> 
> That "written at intervals" is about clearing the
> bitmaps after some idle
> time.
> 
> In other words, dirtying bitmap bits occurs right
> before actual write,
> and clearing bits occurs at intervals.
> 
> Sure, if you write to (or near) the same place again
> and again, without
> giving a chance to md subsystem to actually clean
> the bitmap, there will
> be no additional head movement.  And that means, for
> example, tar -xjf
> sometimes, since filesystem will place the files
> being extracted close to
> each other, thus hitting the same bit in the bitmap,
> hence md will skip
> repeated bitmap updates in this case.
> 
> /mjt
> 

I assume that if a block is already dirty then that is
cached somewhere in memory so you aren't writing to
the bitmap unless you're changing it for clean to
dirty? If that's the case then I would think that
writing to the map wouldn't be that expensive?



 
____________________________________________________________________________________
Now that's room service!  Choose from over 150,000 hotels
in 45,000 destinations on Yahoo! Travel to find your fit.
http://farechase.yahoo.com/promo-generic-14795097