LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Masahiro Yamada <masahiroy@kernel.org>
To: Kees Cook <keescook@chromium.org>
Cc: Linux Kbuild mailing list <linux-kbuild@vger.kernel.org>,
	Sami Tolvanen <samitolvanen@google.com>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Michal Marek <michal.lkml@markovi.net>,
	Nathan Chancellor <nathan@kernel.org>,
	Nick Desaulniers <ndesaulniers@google.com>,
	clang-built-linux <clang-built-linux@googlegroups.com>
Subject: Re: [PATCH 10/13] kbuild: build modules in the same way with/without Clang LTO
Date: Thu, 19 Aug 2021 17:11:49 +0900	[thread overview]
Message-ID: <CAK7LNATXYSdGx2rg=-mHd0_RAiJLUSKob8ybCD6koGDGsXhSZQ@mail.gmail.com> (raw)
In-Reply-To: <202108182348.715797A@keescook>

On Thu, Aug 19, 2021 at 3:59 PM Kees Cook <keescook@chromium.org> wrote:
>
> On Thu, Aug 19, 2021 at 09:57:41AM +0900, Masahiro Yamada wrote:
> > When Clang LTO is enabled, additional intermediate files *.lto.o are
> > created because LLVM bitcode must be converted to ELF before modpost.
> >
> > For non-LTO builds:
> >
> >          $(LD)             $(LD)
> >   objects ---> <modname>.o -----> <modname>.ko
> >                              |
> >           <modname>.mod.o ---/
> >
> > For Clang LTO builds:
> >
> >          $(AR)            $(LD)                 $(LD)
> >   objects ---> <modname>.o ---> <modname>.lto.o -----> <modname>.ko
> >                                                   |
> >                                 <modname>.mod.o --/
> >
> > Since the Clang LTO introduction, ugly CONFIG_LTO_CLANG conditionals
> > are sprinkled everywhere in the kbuild code.
> >
> > Another confusion for Clang LTO builds is, <modname>.o is an archive
> > that contains LLVM bitcode files. The suffix should have been .a
> > instead of .o
> >
> > To clean up the code, unify the build process of modules, as follows:
> >
> >          $(AR)            $(LD)                     $(LD)
> >   objects ---> <modname>.a ---> <modname>.prelink.o -----> <modname>.ko
> >                                                       |
> >                                 <modname>.mod.o ------/
> >
> > Here, 'objects' are either ELF or LLVM bitcode. <modname>.a is an archive,
> > <modname>.prelink.o is ELF.
>
> I like this design, but I do see that it has a small but measurable
> impact on build times:
>
> allmodconfig build, GCC:
>
> make -j72 allmodconfig
> make -j72 -s clean && time make -j72
>
>     kbuild/for-next:
>         6m16.140s
>         6m19.742s
>         6m15.848s
>
>     +this-series:
>         6m22.742s
>         6m20.589s
>         6m19.911s
>
> Thought with not so many modules, it's within the noise:
>
> defconfig build, GCC:
>
> make -j72 defconfig
> make -j72 -s clean && time make -j72
>
>     kbuild/for-next:
>         0m41.579s
>         0m41.214s
>         0m41.370s
>
>     +series:
>         0m41.423s
>         0m41.434s
>         0m41.384s
>
>
> However, I do see that even LTO builds are slightly slower now, so
> perhaps the above numbers aren't due to the added $(AR) step:
>
> allmodconfig + Clang ThinLTO:
>
> make -j72 LLVM=1 LLVM_IAS=1 allmodconfig
> ./scripts/config -d GCOV_KERNEL -d KASAN -d LTO_NONE -e LTO_CLANG_THIN
> make -j72 LLVM=1 LLVM_IAS=1 olddefconfig
> make -j72 -s LLVM=1 LLVM_IAS=1 clean && time make -j72 LLVM=1 LLVM_IAS=1
>
>     kbuild/for-next:
>         9m53.927s
>         9m45.874s
>         9m47.722s
>
>     +series:
>         9m58.395s
>         9m53.201s
>         9m56.387s



I have not tested this closely, but
perhaps this might be the cost of $(AR) t $<)

In Sami's implementation, *.symversions are merged
by shell command.
Presumably, it runs faster than llvm-ar.
Instead, it has a risk of Argument list too long
as reported in [1].

[1] https://lore.kernel.org/lkml/20210614094948.30023-1-lecopzer.chen@mediatek.com/


Anyway, when I find a time,
I will look into some bench mark.




>
>
> I haven't been able to isolate where the changes in build times are
> coming from (nor have I done link-phase-only timings -- I realize those
> are really the most important).
>
> I did notice some warnings from this patch, though, in the
> $(modules-single) target:
>
> scripts/Makefile.build:434: target 'drivers/scsi/libiscsi.a' given more than once in the same rule
> scripts/Makefile.build:434: target 'drivers/atm/suni.a' given more than once in the same rule


Ah, right.

I also noticed needless rebuilds of prelink.symversions.

In v2, I will fix as follows:


index 957addea830b..cf6b79dff5f9 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -438,6 +438,8 @@ cmd_merge_symver =                                  \
 $(obj)/%.prelink.symversions: $(obj)/%.a FORCE
        $(call if_changed,merge_symver)

+targets += $(patsubst %.a, %.prelink.symversions, $(modules))
+
 $(obj)/%.prelink.o: ld_flags += --script=$(filter %.symversions,$^)
 module-symver = $(obj)/%.prelink.symversions

diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index f604d2d01cad..5074922db82d 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -107,8 +107,8 @@ real-dtb-y  := $(addprefix $(obj)/, $(real-dtb-y))
 subdir-ym      := $(addprefix $(obj)/,$(subdir-ym))

 modules                := $(patsubst %.o, %.a, $(obj-m))
-modules-multi  := $(patsubst %.o, %.a, $(multi-obj-m))
-modules-single := $(filter-out $(modules-multi), $(filter %.a, $(modules)))
+modules-multi  := $(sort $(patsubst %.o, %.a, $(multi-obj-m)))
+modules-single := $(sort $(filter-out $(modules-multi), $(filter %.a,
$(modules))))

 # Finds the multi-part object the current object will be linked into.
 # If the object belongs to two or more multi-part objects, list them all.








-- 
Best Regards
Masahiro Yamada

  reply	other threads:[~2021-08-19  8:12 UTC|newest]

Thread overview: 31+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-08-19  0:57 [PATCH 00/13] kbuild: refactoring after " Masahiro Yamada
2021-08-19  0:57 ` [PATCH 01/13] kbuild: move objtool_args back to scripts/Makefile.build Masahiro Yamada
2021-08-19  2:45   ` Kees Cook
2021-08-19  0:57 ` [PATCH 02/13] gen_compile_commands: extract compiler command from a series of commands Masahiro Yamada
2021-08-19  6:48   ` Kees Cook
2021-08-19  0:57 ` [PATCH 03/13] kbuild: detect objtool changes correctly and remove .SECONDEXPANSION Masahiro Yamada
2021-08-19  2:55   ` Kees Cook
2021-08-19  3:18     ` Masahiro Yamada
2021-08-19  0:57 ` [PATCH 04/13] kbuild: remove unused quiet_cmd_update_lto_symversions Masahiro Yamada
2021-08-19  6:19   ` Kees Cook
2021-08-19  0:57 ` [PATCH 05/13] kbuild: remove stale *.symversions Masahiro Yamada
2021-08-19  6:29   ` Kees Cook
2021-08-19  0:57 ` [PATCH 06/13] kbuild: merge vmlinux_link() between the ordinary link and Clang LTO Masahiro Yamada
2021-08-19  2:42   ` Kees Cook
2021-08-19  0:57 ` [PATCH 07/13] kbuild: do not remove 'linux' link in scripts/link-vmlinux.sh Masahiro Yamada
2021-08-19  6:32   ` Kees Cook
2021-08-19  0:57 ` [PATCH 08/13] kbuild: merge vmlinux_link() between ARCH=um and other architectures Masahiro Yamada
2021-08-19  2:43   ` Kees Cook
2021-08-19  0:57 ` [PATCH 09/13] kbuild: do not create built-in.a.symversions or lib.a.symversions Masahiro Yamada
2021-08-19  6:41   ` Kees Cook
2021-08-28 11:57     ` Masahiro Yamada
2021-08-19  0:57 ` [PATCH 10/13] kbuild: build modules in the same way with/without Clang LTO Masahiro Yamada
2021-08-19  6:59   ` Kees Cook
2021-08-19  8:11     ` Masahiro Yamada [this message]
2021-08-19  0:57 ` [PATCH 11/13] kbuild: always postpone CRC links for module versioning Masahiro Yamada
2021-08-19  6:43   ` Kees Cook
2021-08-19  0:57 ` [PATCH 12/13] kbuild: merge cmd_modversions_c and cmd_modversions_S Masahiro Yamada
2021-08-19  3:06   ` Kees Cook
2021-08-19  0:57 ` [PATCH 13/13] kbuild: merge cmd_ar_builtin and cmd_ar_module Masahiro Yamada
2021-08-19  3:01   ` Kees Cook
2021-08-25  4:57 ` [PATCH 00/13] kbuild: refactoring after Clang LTO Masahiro Yamada

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAK7LNATXYSdGx2rg=-mHd0_RAiJLUSKob8ybCD6koGDGsXhSZQ@mail.gmail.com' \
    --to=masahiroy@kernel.org \
    --cc=clang-built-linux@googlegroups.com \
    --cc=keescook@chromium.org \
    --cc=linux-kbuild@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=michal.lkml@markovi.net \
    --cc=nathan@kernel.org \
    --cc=ndesaulniers@google.com \
    --cc=samitolvanen@google.com \
    --subject='Re: [PATCH 10/13] kbuild: build modules in the same way with/without Clang LTO' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).