nfs_update_inode: inode X mode changed, Y to Z

nfs_update_inode: inode X mode changed, Y to Z

[Yaroslav Halchenko]

Dear Kernel People,

I am running a tiny cluster (27 nodes). Setup is as follows:

* NFS server:
   kernel: vanilla 2.6.18.2 + areca drivers
   mounted partition is XFS
   nfs-kernel-server: 1.0.10-1~bpo.1 (from backports.org)

* clients:
   kernel: 2.6.17-2-amd64 (Debian stock from etch installed few months ago)

Today under heavy load which consisted of the nodes manipulating tons of
small files, I started to mention that software reported weird errors
like:

EOFError: EOF read where object expected
mv: cannot stat <FILENAME> : No such file or directory
etc

Looking at the client's dmesg I found nasty:

node17.ravana.rutgers.edu: Feb 23 13:32:06 node17 kernel: nfs_update_inode: inode 680223263 mode changed, 0042755 to 0100644
node17.ravana.rutgers.edu: Feb 23 13:41:19 node17 kernel: nfs_update_inode: inode 681427742 mode changed, 0042755 to 0100644
node22.ravana.rutgers.edu: Feb 22 22:58:15 node22 kernel: nfs_update_inode: inode 677306152 mode changed, 0042755 to 0100644
node22.ravana.rutgers.edu: Feb 23 13:48:33 node22 kernel: nfs_update_inode: inode 681695507 mode changed, 0100644 to 0042755
node12.ravana.rutgers.edu: Feb 23 13:31:01 node12 kernel: nfs_update_inode: inode 680150798 mode changed, 0100644 to 0042755
node12.ravana.rutgers.edu: Feb 23 13:34:57 node12 kernel: nfs_update_inode: inode 680418141 mode changed, 0100644 to 0042755
node12.ravana.rutgers.edu: Feb 23 13:37:01 node12 kernel: nfs_update_inode: inode 680637478 mode changed, 0100644 to 0042755
node12.ravana.rutgers.edu: Feb 23 13:39:25 node12 kernel: nfs_update_inode: inode 681034087 mode changed, 0100644 to 0042755
node12.ravana.rutgers.edu: Feb 23 13:40:06 node12 kernel: nfs_update_inode: inode 681225056 mode changed, 0042755 to 0100644
node12.ravana.rutgers.edu: Feb 23 13:43:26 node12 kernel: nfs_update_inode: inode 681474682 mode changed, 0100644 to 0042755
......

server logs didn't show any abnormal things... where should I look for the
source of the problem? googling up seems to be of no interesting result...
is there way to eliminate cause may be by tuning some performance
parameters (ie sacrificing performance for stability)?

Thanks everyone in advance for hints
-- 
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik        

Re: nfs_update_inode: inode X mode changed, Y to Z

[Yaroslav Halchenko]

Dear Kernel People,

Problem persists and it seems that I am not alone who hits this problem:
http://lkml.org/lkml/2007/3/1/53

in my case it happens if there is a lot (>50) nfs clients which in a
fast pace create a small file, close it and then open it for reading.
Some times it fails saying that the file is a directory, some times it
just fails to open it for writing, etc

I would really appreciate any comments and/or ideas on how to
troubleshoot/eliminate this issue.

Cheers
Yarik


On Fri, 23 Feb 2007, Yaroslav Halchenko wrote:

> Dear Kernel People,

> I am running a tiny cluster (27 nodes). Setup is as follows:

> * NFS server:
>    kernel: vanilla 2.6.18.2 + areca drivers
>    mounted partition is XFS
>    nfs-kernel-server: 1.0.10-1~bpo.1 (from backports.org)

> * clients:
>    kernel: 2.6.17-2-amd64 (Debian stock from etch installed few months ago)

> Today under heavy load which consisted of the nodes manipulating tons of
> small files, I started to mention that software reported weird errors
> like:

> EOFError: EOF read where object expected
> mv: cannot stat <FILENAME> : No such file or directory
> etc

> Looking at the client's dmesg I found nasty:

> node17.ravana.rutgers.edu: Feb 23 13:32:06 node17 kernel: nfs_update_inode: inode 680223263 mode changed, 0042755 to 0100644
> node17.ravana.rutgers.edu: Feb 23 13:41:19 node17 kernel: nfs_update_inode: inode 681427742 mode changed, 0042755 to 0100644
> node22.ravana.rutgers.edu: Feb 22 22:58:15 node22 kernel: nfs_update_inode: inode 677306152 mode changed, 0042755 to 0100644
> node22.ravana.rutgers.edu: Feb 23 13:48:33 node22 kernel: nfs_update_inode: inode 681695507 mode changed, 0100644 to 0042755
> node12.ravana.rutgers.edu: Feb 23 13:31:01 node12 kernel: nfs_update_inode: inode 680150798 mode changed, 0100644 to 0042755
> node12.ravana.rutgers.edu: Feb 23 13:34:57 node12 kernel: nfs_update_inode: inode 680418141 mode changed, 0100644 to 0042755
> node12.ravana.rutgers.edu: Feb 23 13:37:01 node12 kernel: nfs_update_inode: inode 680637478 mode changed, 0100644 to 0042755
> node12.ravana.rutgers.edu: Feb 23 13:39:25 node12 kernel: nfs_update_inode: inode 681034087 mode changed, 0100644 to 0042755
> node12.ravana.rutgers.edu: Feb 23 13:40:06 node12 kernel: nfs_update_inode: inode 681225056 mode changed, 0042755 to 0100644
> node12.ravana.rutgers.edu: Feb 23 13:43:26 node12 kernel: nfs_update_inode: inode 681474682 mode changed, 0100644 to 0042755
> ......

> server logs didn't show any abnormal things... where should I look for the
> source of the problem? googling up seems to be of no interesting result...
> is there way to eliminate cause may be by tuning some performance
> parameters (ie sacrificing performance for stability)?

> Thanks everyone in advance for hints
-- 
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik