LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Kees Cook <keescook@chromium.org>
To: Masahiro Yamada <masahiroy@kernel.org>
Cc: linux-kbuild@vger.kernel.org,
	Michal Marek <michal.lkml@markovi.net>,
	Nick Desaulniers <ndesaulniers@google.com>,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 07/13] kbuild: do not create built-in.a.symversions or lib.a.symversions
Date: Tue, 31 Aug 2021 10:36:28 -0700	[thread overview]
Message-ID: <202108311035.5390256E@keescook> (raw)
In-Reply-To: <20210831074004.3195284-8-masahiroy@kernel.org>

On Tue, Aug 31, 2021 at 04:39:58PM +0900, Masahiro Yamada wrote:
> Merge all *.o.symversions in scripts/link-vmlinux.sh instead of
> incrementally merging them in the unit of built-in.a or lib.a.
> 
> This is a preparation for further code cleanups.
> 
> The initial patch version was implemented in a shell script, but it
> was slow due to the slowness of the 'cat' command [1]. This version
> was implemented in Perl.
> 
> [1]: https://lore.kernel.org/lkml/CAK7LNATyNAu6sa-TT9JXy=BXr5d2Q5K-sp-mVXXtJDuJyi6_bA@mail.gmail.com/
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

My earlier questions about speed here were actually with regard to
non-LTO builds. But it sounds like it's not a problem there either.

Reviewed-by: Kees Cook <keescook@chromium.org>


> ---
> 
>  scripts/Makefile.build   | 10 ++------
>  scripts/link-vmlinux.sh  |  9 ++-----
>  scripts/merge-symvers.pl | 52 ++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 56 insertions(+), 15 deletions(-)
>  create mode 100644 scripts/merge-symvers.pl
> 
> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
> index afc906cd7256..3ad1b1227371 100644
> --- a/scripts/Makefile.build
> +++ b/scripts/Makefile.build
> @@ -434,11 +434,8 @@ endif
>  quiet_cmd_ar_builtin = AR      $@
>        cmd_ar_builtin = rm -f $@; $(AR) cDPrST $@ $(real-prereqs)
>  
> -quiet_cmd_ar_and_symver = AR      $@
> -      cmd_ar_and_symver = $(cmd_update_lto_symversions); $(cmd_ar_builtin)
> -
>  $(obj)/built-in.a: $(real-obj-y) FORCE
> -	$(call if_changed,ar_and_symver)
> +	$(call if_changed,ar_builtin)
>  
>  #
>  # Rule to create modules.order file
> @@ -458,11 +455,8 @@ $(obj)/modules.order: $(obj-m) FORCE
>  #
>  # Rule to compile a set of .o files into one .a file (with symbol table)
>  #
> -quiet_cmd_ar_lib = AR      $@
> -      cmd_ar_lib = $(cmd_update_lto_symversions); $(cmd_ar)
> -
>  $(obj)/lib.a: $(lib-y) FORCE
> -	$(call if_changed,ar_lib)
> +	$(call if_changed,ar)
>  
>  # NOTE:
>  # Do not replace $(filter %.o,^) with $(real-prereqs). When a single object
> diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
> index d74cee5c4326..0cc6a03f2cb1 100755
> --- a/scripts/link-vmlinux.sh
> +++ b/scripts/link-vmlinux.sh
> @@ -57,13 +57,8 @@ gen_initcalls()
>  gen_symversions()
>  {
>  	info GEN .tmp_symversions.lds
> -	rm -f .tmp_symversions.lds
> -
> -	for o in ${KBUILD_VMLINUX_OBJS} ${KBUILD_VMLINUX_LIBS}; do
> -		if [ -f ${o}.symversions ]; then
> -			cat ${o}.symversions >> .tmp_symversions.lds
> -		fi
> -	done
> +	${PERL} scripts/merge-symvers.pl -a ${AR} -o .tmp_symversions.lds \
> +		${KBUILD_VMLINUX_OBJS} ${KBUILD_VMLINUX_LIBS}
>  }
>  
>  # Link of vmlinux.o used for section mismatch analysis
> diff --git a/scripts/merge-symvers.pl b/scripts/merge-symvers.pl
> new file mode 100644
> index 000000000000..0bd092d24eff
> --- /dev/null
> +++ b/scripts/merge-symvers.pl
> @@ -0,0 +1,52 @@
> +#!/usr/bin/env perl
> +# SPDX-License-Identifier: GPL-2.0-only
> +
> +use autodie;
> +use strict;
> +use warnings;
> +use Getopt::Long 'GetOptions';
> +
> +my $ar;
> +my $output;
> +
> +GetOptions(
> +	'a|ar=s' => \$ar,
> +	'o|output=s'  => \$output,
> +);
> +
> +# Collect all objects
> +my @objects;
> +
> +foreach (@ARGV) {
> +	if (/\.o$/) {
> +		# Some objects (head-y) are linked to vmlinux directly.
> +		push(@objects, $_);
> +	} elsif (/\.a$/) {
> +		# Most of built-in objects are contained in built-in.a or lib.a.
> +		# Use 'ar -t' to get the list of the contained objects.
> +		$_ = `$ar -t $_`;
> +		push(@objects, split(/\n/));
> +	} else {
> +		die "$_: unknown file type\n";
> +	}
> +}
> +
> +open(my $out_fh, '>', "$output");
> +
> +foreach (@objects) {
> +	# The symbol CRCs for foo/bar/baz.o is output to foo/bar/baz.o.symversions
> +	s/(.*)/$1.symversions/;
> +
> +	if (! -e $_) {
> +		# .symversions does not exist if the object does not contain
> +		# EXPORT_SYMBOL at all. Skip it.
> +		next;
> +	}
> +
> +	open(my $in_fh, '<', "$_");
> +	# Concatenate all the existing *.symversions files.
> +	print $out_fh do { local $/; <$in_fh> };
> +	close $in_fh;
> +}
> +
> +close $out_fh;
> -- 
> 2.30.2
> 

-- 
Kees Cook

  reply	other threads:[~2021-08-31 17:36 UTC|newest]

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-08-31  7:39 [PATCH v2 00/13] kbuild: second round of Clang LTO refactoring Masahiro Yamada
2021-08-31  7:39 ` [PATCH v2 01/13] kbuild: move objtool_args back to scripts/Makefile.build Masahiro Yamada
2021-09-04 18:18   ` Josh Poimboeuf
2021-08-31  7:39 ` [PATCH v2 02/13] kbuild: rename __objtool_obj to objtool Masahiro Yamada
2021-08-31 17:16   ` Nick Desaulniers
2021-08-31 17:29   ` Kees Cook
2021-09-04 18:19   ` Josh Poimboeuf
2021-08-31  7:39 ` [PATCH v2 03/13] kbuild: store the objtool command in *.cmd files Masahiro Yamada
2021-08-31 17:23   ` Nick Desaulniers
2021-08-31 17:30   ` Kees Cook
2021-09-04 18:04   ` Josh Poimboeuf
2021-09-04 18:45     ` Josh Poimboeuf
2021-09-08 16:12       ` Masahiro Yamada
2021-08-31  7:39 ` [PATCH v2 04/13] kbuild: factor out OBJECT_FILES_NON_STANDARD check into a macro Masahiro Yamada
2021-08-31 17:25   ` Nick Desaulniers
2021-08-31 17:31   ` Kees Cook
2021-09-04 18:59   ` Josh Poimboeuf
2021-08-31  7:39 ` [PATCH v2 05/13] kbuild: detect objtool update without using .SECONDEXPANSION Masahiro Yamada
2021-08-31 17:32   ` Kees Cook
2021-09-03  0:43     ` Masahiro Yamada
2021-08-31 17:33   ` Nick Desaulniers
2021-09-02 23:42     ` Masahiro Yamada
2021-09-04 19:04   ` Josh Poimboeuf
2021-08-31  7:39 ` [PATCH v2 06/13] kbuild: reuse $(cmd_objtool) for cmd_cc_lto_link_modules Masahiro Yamada
2021-08-31 17:35   ` Kees Cook
2021-09-03  0:39     ` Masahiro Yamada
2021-09-03  1:49       ` Kees Cook
2021-09-04 19:11   ` Josh Poimboeuf
2021-08-31  7:39 ` [PATCH v2 07/13] kbuild: do not create built-in.a.symversions or lib.a.symversions Masahiro Yamada
2021-08-31 17:36   ` Kees Cook [this message]
2021-08-31  7:39 ` [PATCH v2 08/13] kbuild: build modules in the same way with/without Clang LTO Masahiro Yamada
2021-08-31 17:39   ` Kees Cook
2021-08-31 17:46   ` Nick Desaulniers
2021-08-31  7:40 ` [PATCH v2 09/13] kbuild: add cmd_and_savecmd macro Masahiro Yamada
2021-08-31 17:39   ` Kees Cook
2021-08-31  7:40 ` [PATCH v2 10/13] kbuild: rebuild modules when objtool is updated for CONFIG_LTO_CLANG Masahiro Yamada
2021-08-31 17:40   ` Kees Cook
2021-09-04 19:13   ` Josh Poimboeuf
2021-08-31  7:40 ` [PATCH v2 11/13] kbuild: always postpone CRC links for module versioning Masahiro Yamada
2021-08-31  7:40 ` [PATCH v2 12/13] kbuild: merge cmd_modversions_c and cmd_modversions_S Masahiro Yamada
2021-08-31  7:40 ` [PATCH v2 13/13] kbuild: merge cmd_ar_builtin and cmd_ar_module Masahiro Yamada

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=202108311035.5390256E@keescook \
    --to=keescook@chromium.org \
    --cc=linux-kbuild@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=masahiroy@kernel.org \
    --cc=michal.lkml@markovi.net \
    --cc=ndesaulniers@google.com \
    --subject='Re: [PATCH v2 07/13] kbuild: do not create built-in.a.symversions or lib.a.symversions' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).