LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Alexey Dobriyan <adobriyan@gmail.com>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: Waiman Long <longman@redhat.com>,
	Kate Stewart <kstewart@linuxfoundation.org>,
	Thomas Gleixner <tglx@linutronix.de>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH] proc/stat: Separate out individual irq counts into /proc/stat_irqs
Date: Thu, 19 Apr 2018 23:02:17 +0300	[thread overview]
Message-ID: <20180419200217.GA4467@avx2> (raw)
In-Reply-To: <20180419124319.8e329eb25234c045bf161cd5@linux-foundation.org>

On Thu, Apr 19, 2018 at 12:43:19PM -0700, Andrew Morton wrote:
> On Thu, 19 Apr 2018 13:09:29 -0400 Waiman Long <longman@redhat.com> wrote:
> 
> > It was found that reading /proc/stat could be time consuming on
> > systems with a lot of irqs. For example, reading /proc/stat in a
> > certain 2-socket Skylake server took about 4.6ms because it had over
> > 5k irqs. In that particular case, the majority of the CPU cycles for
> > reading /proc/stat was spent in the kstat_irqs() function.  Therefore,
> > application performance can be impacted if the application reads
> > /proc/stat rather frequently.
> > 
> > The "intr" line within /proc/stat contains a sum total of all the irqs
> > that have happened followed by a list of irq counts for each individual
> > irq number. In many cases, the first number is good enough. The
> > individual irq counts may not provide that much more information.
> > 
> > In order to avoid this kind of performance issue, all these individual
> > irq counts are now separated into a new /proc/stat_irqs file. The
> > sum total irq count will stay in /proc/stat and be duplicated in
> > /proc/stat_irqs. Applications that need to look up individual irq counts
> > will now have to look into /proc/stat_irqs instead of /proc/stat.
> > 
> 
> (cc /proc maintainer)
> 
> It's a non-backward-compatible change.  For something which has
> existing for so long, it would be a mighty task to demonstrate that no
> existing userspace will be disrupted by this change.
> 
> So we need to think again.  A new interface which omits the per-IRQ
> stats might be acceptable.

Here is profile of open+read+close /proc/stat

30% is taking mutex only to print "0".

+   98.80%     0.04%  a.out    [kernel.vmlinux]  [k] entry_SYSCALL_64                                                                            ▒
+   98.75%     0.10%  a.out    [kernel.vmlinux]  [k] do_syscall_64                                                                               ▒
+   95.56%     0.04%  a.out    libc-2.25.so      [.] __GI___libc_read                                                                            ◆
+   95.09%     0.01%  a.out    [kernel.vmlinux]  [k] sys_read                                                                                    ▒
+   95.04%     0.03%  a.out    [kernel.vmlinux]  [k] vfs_read                                                                                    ▒
+   94.98%     0.05%  a.out    [kernel.vmlinux]  [k] proc_reg_read                                                                               ▒
+   94.98%     0.00%  a.out    [kernel.vmlinux]  [k] __vfs_read                                                                                  ▒
+   94.92%     0.06%  a.out    [kernel.vmlinux]  [k] seq_read                                                                                    ▒
+   94.52%     3.65%  a.out    [kernel.vmlinux]  [k] show_stat                                                                                   ▒
+   48.62%     2.59%  a.out    [kernel.vmlinux]  [k] kstat_irqs_usr                                                                              ▒
+   33.52%     9.55%  a.out    [kernel.vmlinux]  [k] seq_put_decimal_ull                                                                         ▒
+   19.63%    19.59%  a.out    [kernel.vmlinux]  [k] memcpy_erms                                                                                 ▒
+   17.34%     9.53%  a.out    [kernel.vmlinux]  [k] kstat_irqs                                                                                  ▒
-   15.45%    15.43%  a.out    [kernel.vmlinux]  [k] mutex_lock                                                                                  ▒
     15.43% __GI___libc_read                                                                                                                     ▒
        entry_SYSCALL_64                                                                                                                         ▒
        do_syscall_64                                                                                                                            ▒
        sys_read                                                                                                                                 ▒
        vfs_read                                                                                                                                 ▒
        __vfs_read                                                                                                                               ▒
        proc_reg_read                                                                                                                            ▒
      - seq_read                                                                                                                                 ▒
         - 15.41% show_stat                                                                                                                      ▒
              kstat_irqs_usr                                                                                                                     ▒
              mutex_lock                                                                                                                         ▒
+   13.32%    13.27%  a.out    [kernel.vmlinux]  [k] mutex_unlock                                                                                ▒
+    4.60%     1.35%  a.out    [kernel.vmlinux]  [k] cpumask_next                                                                                ▒
+    3.03%     3.03%  a.out    [kernel.vmlinux]  [k] __radix_tree_lookup                                                                         ▒
+    2.96%     0.08%  a.out    [kernel.vmlinux]  [k] seq_printf                                                                                  ▒
+    2.92%     0.02%  a.out    libc-2.25.so      [.] __GI___libc_open                                                                            ▒
+    2.89%     0.07%  a.out    [kernel.vmlinux]  [k] seq_vprintf                                                                                 ▒
+    2.81%     0.70%  a.out    [kernel.vmlinux]  [k] vsnprintf                                                                                   ▒
+    2.66%     2.66%  a.out    [kernel.vmlinux]  [k] _find_next_bit                                                                              ▒
+    2.42%     1.36%  a.out    [kernel.vmlinux]  [k] num_to_str                                                                                  ▒
+    2.41%     0.19%  a.out    [kernel.vmlinux]  [k] get_idle_time                                                                               ▒
+    2.39%     0.02%  a.out    [kernel.vmlinux]  [k] do_sys_open

  parent reply	other threads:[~2018-04-19 20:02 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-04-19 17:09 Waiman Long
2018-04-19 17:38 ` Randy Dunlap
2018-04-19 18:44   ` Waiman Long
2018-04-19 19:43 ` Andrew Morton
2018-04-19 19:57   ` Waiman Long
2018-04-19 20:02   ` Alexey Dobriyan [this message]
2018-04-19 19:08 Alexey Dobriyan
2018-04-19 19:28 ` Waiman Long
2018-04-19 19:55   ` Alexey Dobriyan
     [not found]     ` <eb7c1569-e445-5cbb-6d10-2694b625232a@redhat.com>
2018-04-19 20:39       ` Alexey Dobriyan
2018-04-19 20:58         ` Waiman Long
2018-04-19 23:23           ` Joel Fernandes (Google)
2018-04-21 20:34             ` Alexey Dobriyan
2018-04-21 20:36               ` Alexey Dobriyan
2018-04-24  5:54                 ` David Rientjes
2018-04-24  6:18                   ` Alexey Dobriyan
2018-05-02  0:02                     ` Andrew Morton
2018-04-19 21:05         ` Waiman Long

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180419200217.GA4467@avx2 \
    --to=adobriyan@gmail.com \
    --cc=akpm@linux-foundation.org \
    --cc=gregkh@linuxfoundation.org \
    --cc=kstewart@linuxfoundation.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=longman@redhat.com \
    --cc=tglx@linutronix.de \
    --subject='Re: [PATCH] proc/stat: Separate out individual irq counts into /proc/stat_irqs' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).