[RFC/PATCH] Documentation of kernel messages

[RFC/PATCH] Documentation of kernel messages

[holzheu]

Greetings,

The operation of a Linux system sometimes requires to decode the
meaning of a specific kernel message, e.g. an error message of a
driver. Especially on our mainframe zSeries platform system
administrators want to have descriptions for Linux kernel messages.
They are used to that, because all other operating systems on that
platform like z/OS, z/VM or z/VSE have message catalogs with detailed
descriptions about the semantics of the messages.

In general we think, that also for Linux it is a good thing to have
documentation for the most important kernel/driver messages. Even
kernel hackers not always are aware of the meaning of kernel messages
for components, which they don't know in detail. Most of the messages
are self explaining but sometimes you get something like "Clocksource
tsc unstable (delta = 7304132729 ns)" and you wonder if your system is
going to explode.

Unfortunately currently there is no general infrastructure in the Linux
kernel for the documentation of messages. I worked on a proposal, how
that could be implemented in an easy way using the already existing
kernel-doc infrastructure and using printk. The proposal is as follows

1. We use message identifiers in order to map messages to message
descriptions. A message identifier consists out of a component name and
within that component unique message number.

2. Messages and message descriptions are maintained together in the
kernel sources in order to keep them up to date. Messages catalog are
generated automatically for exactly one kernel level.

3. A special tool checks, if messages are not documented or if there
are message descriptions without corresponding messages.

4. We use the already existing kernel-doc tool to generate an online
message catalog or e.g. a pdf for offline documentation.

Current prototype implementation:
================================= 

The structure of a kernel message is: <component>.<msg number>: <msg>

* component: Name of the kernel or driver component e.g. "pci", "ide", 
  etc.
* msg number: Within the component unique number of a kernel message.
* msg: printk message

New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be
used in printk. These macros have as parameter the message number and
are using a per c-file defined macro KMSG_COMPONENT.

Example: Define message 2 in component "kmsgtest":

#define KMSG_COMPONENT "kmsgtest"

void f(void)
{
        printk(KMSG_ERR(1) "device %x not online\n", devno);
}

The output of that kernel message would be:
"kmsgtest.1: device 4711 not online"

The messages have to be documented within the C source file
in the following way:

/**
 * message
 * @0: device number of device.
 *
 * Description:
 * An operation has been performed on the msgtest device, but the
 * device has not been set online. Therefore the operation failed
 *
 * User Response:
 * Operator should set device online.
 * Issue "chccwdev -e <device number>".
 *
 */

KMSG_DOC(kmsgtest, 2,  "Device %x not online");

I created a patch for the kernel-doc tool so it can be used to generate
a catalog of all kernel messages:

>> kernel-doc -man kmsgtest.c > kmsgtest.2
>> man ./kmsgtest.2 

# Kernel API(9)               Linux Messages               Kernel API(9)
# 
# MESSAGE:
#       kmsgtest.2: "device %x not online"
#
# Parameter:
#       1           Device number of device.
#
# Description:
#       An  operation  has been performed on the msgtest device, but
#       the device has not been set online. Therefore the operation failed.
#
# User Response:
#       Operator should set device online.
#       Issue "chccwdev -e <device number>".
#
# May 2007                      kmsgtest.2                 Kernel API(9)

A nice thing would be to include the online kernel message catalog in
the kernel rpm. One possibility for that would be to have one man page
per message. If an operator finds the message kmsgtest.2 in
var/log/messages and wants to know what the message means, he simply
issues "man kmsgtest.2" and gets the description.

To ensure that all messages are documented and there are no message
descriptions without corresponding messages, a checker tool is
provided. To enable message checking, in the toplevel Makefile the
following has to be added:

CHECK = scripts/kmsg_check.pl check

To enable message checking during kernel build, the "C" option has
to be used:

>> make modules C=1
  CHK     include/linux/version.h
  CHK     include/linux/utsrelease.h
  CHECK   drivers/kmsgtest/kmsgtest.c
drivers/kmsgtest/kmsgtest.c: Missing description for: kmsgtest.1
drivers/kmsgtest/kmsgtest.c: Description without message for: kmsgtest.3

Please note, that the patch for the kernel-doc and kmsg-doc tools
is just a prototype and is neither complete nor perfect.

Michael

Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com>
Signed-off-by: Michael Holzheu <holzheu@linux.vmnet.ibm.com>
---

 Makefile                    |    5
 drivers/Makefile            |    1
 drivers/kmsgtest/Makefile   |    5
 drivers/kmsgtest/kmsgtest.c |   59 +++++++++++
 include/linux/kmsg.h        |   32 ++++++
 scripts/kernel-doc          |  116 +++++++++++++++++++++-
 scripts/kmsg-doc            |  231 ++++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 445 insertions(+), 4 deletions(-)

diff -Naur linux-2.6.21/Makefile linux-2.6.21-kmsg/Makefile
--- linux-2.6.21/Makefile	2007-04-26 05:08:32.000000000 +0200
+++ linux-2.6.21-kmsg/Makefile	2007-06-05 15:17:51.000000000 +0200
@@ -293,9 +293,8 @@
 DEPMOD		= /sbin/depmod
 KALLSYMS	= scripts/kallsyms
 PERL		= perl
-CHECK		= sparse
-
-CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ -Wbitwise $(CF)
+CHECK		= scripts/kmsg-doc check
+CHECKFLAGS	=
 MODFLAGS	= -DMODULE
 CFLAGS_MODULE   = $(MODFLAGS)
 AFLAGS_MODULE   = $(MODFLAGS)
diff -Naur linux-2.6.21/drivers/Makefile linux-2.6.21-kmsg/drivers/Makefile
--- linux-2.6.21/drivers/Makefile	2007-04-26 05:08:32.000000000 +0200
+++ linux-2.6.21-kmsg/drivers/Makefile	2007-06-05 15:17:51.000000000 +0200
@@ -8,6 +8,7 @@
 obj-$(CONFIG_PCI)		+= pci/
 obj-$(CONFIG_PARISC)		+= parisc/
 obj-$(CONFIG_RAPIDIO)		+= rapidio/
+obj-m				+= kmsgtest/
 obj-y				+= video/
 obj-$(CONFIG_ACPI)		+= acpi/
 # PnP must come after ACPI since it will eventually need to check if acpi
diff -Naur linux-2.6.21/drivers/kmsgtest/Makefile linux-2.6.21-kmsg/drivers/kmsgtest/Makefile
--- linux-2.6.21/drivers/kmsgtest/Makefile	1970-01-01 01:00:00.000000000 +0100
+++ linux-2.6.21-kmsg/drivers/kmsgtest/Makefile	2007-06-05 15:17:51.000000000 +0200
@@ -0,0 +1,5 @@
+#
+# Makefile for kernel message test module
+#
+
+obj-m		+= kmsgtest.o
diff -Naur linux-2.6.21/drivers/kmsgtest/kmsgtest.c linux-2.6.21-kmsg/drivers/kmsgtest/kmsgtest.c
--- linux-2.6.21/drivers/kmsgtest/kmsgtest.c	1970-01-01 01:00:00.000000000 +0100
+++ linux-2.6.21-kmsg/drivers/kmsgtest/kmsgtest.c	2007-06-05 15:17:51.000000000 +0200
@@ -0,0 +1,59 @@
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/kmsg.h>
+
+static int devno = 0x4711;
+static int status = 1;
+
+#define KMSG_COMPONENT "kmsgtest"
+
+static int __init kmsgtest_init(void)
+{
+	printk(KMSG_INFO(1) "device %x has status %i\n", devno, status);
+	printk(KMSG_ERR(2) "device %x not online\n", devno);
+
+	return 0;
+}
+
+static void __exit kmsgtest_exit(void)
+{
+	printk("kmsgtest module exit\n");
+}
+
+module_init(kmsgtest_init);
+module_exit(kmsgtest_exit);
+
+/**
+ * message
+ * @0: Device number of device
+ * @1: Status of device
+ *
+ * Description:
+ * Information message about the status of our virtual msgtest device. The
+ * following values for the status parameter are available.
+ *
+ *   0 - Device is offline
+ *
+ *   1 - Device is online
+ *
+ *   2 - Device is broken
+ *
+ * User Response:
+ * If device is broken, replace it or fix it.
+ */
+
+KMSG_DOC(kmsgtest, 1, "device %x has status %i");
+
+/**
+ * message
+ * @0: Device number of device.
+ *
+ * Description:
+ * An operation has been performed on the msgtest device, but the device has
+ * not been set online. Therefore the operation failed.
+ *
+ * User Response:
+ * Operator should set device online. Issue "chccwdev -e <device number>".
+ */
+
+KMSG_DOC(kmsgtest, 2, "device %x not online");
diff -Naur linux-2.6.21/include/linux/kmsg.h linux-2.6.21-kmsg/include/linux/kmsg.h
--- linux-2.6.21/include/linux/kmsg.h	1970-01-01 01:00:00.000000000 +0100
+++ linux-2.6.21-kmsg/include/linux/kmsg.h	2007-06-05 15:17:51.000000000 +0200
@@ -0,0 +1,32 @@
+#ifndef _LINUX_KMSG_H
+#define _LINUX_KMSG_H
+
+#ifdef __KMSG_CHECKER
+
+#define KMSG_EMERG(num)   __KMSG_CHECK(EMERG, num)
+#define KMSG_ALERT(num)   __KMSG_CHECK(ALERT, num)
+#define KMSG_CRIT(num)    __KMSG_CHECK(CRIT, num)
+#define KMSG_ERR(num)     __KMSG_CHECK(ERR, num)
+#define KMSG_WARNING(num) __KMSG_CHECK(WARNING, num)
+#define KMSG_NOTICE(num)  __KMSG_CHECK(NOTICE, num)
+#define KMSG_INFO(num)    __KMSG_CHECK(INFO, num)
+#define KMSG_DEBUG(num)   __KMSG_CHECK(DEBUG, num)
+
+#define KMSG_DOC(comp, num, str) __KMSG_DOC(comp, num, str)
+
+#else
+
+#define KMSG_EMERG(num)   KERN_EMERG KMSG_COMPONENT "." #num ": "
+#define KMSG_ALERT(num)   KERN_ALERT KMSG_COMPONENT "." #num ": "
+#define KMSG_CRIT(num)    KERN_CRIT KMSG_COMPONENT "." #num ": "
+#define KMSG_ERR(num)     KERN_ERR KMSG_COMPONENT "." #num ": "
+#define KMSG_WARNING(num) KERN_WARNING KMSG_COMPONENT "." #num ": "
+#define KMSG_NOTICE(num)  KERN_NOTICE KMSG_COMPONENT "." #num ": "
+#define KMSG_INFO(num)    KERN_INFO KMSG_COMPONENT "." #num ": "
+#define KMSG_DEBUG(num)   KERN_DEBUG KMSG_COMPONENT "." #num ": "
+
+#define KMSG_DOC(comp, num, str)
+
+#endif /* __KMSG_CHECKER */
+
+#endif /* _LINUX_KMSG_H */
diff -Naur linux-2.6.21/scripts/kernel-doc linux-2.6.21-kmsg/scripts/kernel-doc
--- linux-2.6.21/scripts/kernel-doc	2007-04-26 05:08:32.000000000 +0200
+++ linux-2.6.21-kmsg/scripts/kernel-doc	2007-06-05 15:17:51.000000000 +0200
@@ -256,7 +256,7 @@
 my $in_doc_sect;
 
 #declaration types: can be
-# 'function', 'struct', 'union', 'enum', 'typedef'
+# 'function', 'struct', 'union', 'enum', 'typedef', 'message'
 my $decl_type;
 
 my $doc_special = "\@\%\$\&";
@@ -1163,6 +1163,55 @@
     output_section_text(@_);
 }
 
+##
+# output message in text
+sub output_message_text(%) {
+	my %args = %{$_[0]};
+	my ($parameter);
+
+	print $args{'id'}.": \"".$args{'message'}."\"\n\n";
+
+	print "Parameters:\n\n";
+	foreach $parameter (@{$args{'parameterlist'}}) {
+		($parameter =~ /^#/) && next;
+		my $parameter_name = $parameter;
+		$parameter_name =~ s/\[.*//;
+		($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
+		print "$parameter\n\t";
+		print $args{'parameterdescs'}{$parameter_name}."\n";
+	}
+	output_section_text(@_);
+}
+
+##
+# output message in man
+sub output_message_man(%) {
+	my %args = %{$_[0]};
+	my ($parameter, $section);
+
+	print ".TH \"$args{'module'}\" 9 \"".$args{'id'}."\" \"$man_date\" \"Linux Messages\" LINUX\n";
+
+	print ".SH MESSAGE\n";
+	print $args{'id'}.": "."\"".$args{'message'}."\"\n";
+
+	print ".SH Parameters\n";
+	foreach $parameter (@{$args{'parameterlist'}}) {
+		($parameter =~ /^#/) && next;
+
+		my $parameter_name = $parameter;
+		$parameter_name =~ s/\[.*//;
+
+		($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
+		print ".IP \"".$parameter."\" 12\n";
+		output_highlight($args{'parameterdescs'}{$parameter_name});
+	}
+	foreach $section (@{$args{'sectionlist'}}) {
+		print ".SH \"$section\"\n";
+		output_highlight($args{'sections'}{$section});
+		print "\n";
+	}
+}
+
 #output sections in text
 sub output_section_text(%) {
     my %args = %{$_[0]};
@@ -1607,6 +1656,69 @@
 		       });
 }
 
+sub create_parameterlist_msg($$) {
+	my $args = shift;
+	my $file = shift;
+	my $splitter = "%";
+	my $type;
+	my $param = 0;
+	my $first = 1;
+
+	# temporarily replace commas
+	while ($args =~ /(\([^\),]+),/) {
+		$args =~ s/(\([^\),]+),/$1#/g;
+	}
+
+	foreach my $arg (split($splitter, $args)) {
+		if ($first) {
+			$first = 0;
+			next;
+		}
+		$type = substr($arg,0,1);
+
+		# XXX introduce better type checking
+
+		push_parameter($param, $type, $file);
+		$param += 1;
+	}
+}
+
+##
+# takes a message prototype and the name of the current file being
+# processed and spits out all the details stored in the global
+# arrays/hashes.
+sub dump_message($$) {
+	my $x = shift;
+	my $file = shift;
+
+	if ($x =~/(KMSG_DOC)\(\s*(\w+)\s*\,\s*(\w+)\s*\,\s*\"(.*)\"\)/) {
+	$declaration_name = "$2.$3";
+	my $members = $4;
+	# strip comments:
+	$members =~ s/\/\*.*?\*\///gos;
+
+	create_parameterlist_msg($members, $file);
+
+	output_declaration($declaration_name,
+			   'message',
+			   {'message' => $members,
+			    'id' => $declaration_name,
+			    'module' => $modulename,
+			    'parameterlist' => \@parameterlist,
+			    'parameterdescs' => \%parameterdescs,
+			    'parametertypes' => \%parametertypes,
+			    'sectionlist' => \@sectionlist,
+			    'sections' => \%sections,
+			    'purpose' => $declaration_purpose,
+			    'type' => $decl_type
+			   });
+	}
+	else {
+		print STDERR "Error(${file}:$.): Cannot parse message!\n";
+		++$errors;
+	}
+}
+
 sub process_file($);
 
 # Read the file that maps relative names to absolute names for
@@ -1782,6 +1894,8 @@
 		    $decl_type = 'enum';
 		} elsif ($identifier =~ m/^typedef/) {
 		    $decl_type = 'typedef';
+		} elsif ($identifier =~ m/^message/) {
+		    $decl_type = 'message';
 		} else {
 		    $decl_type = 'function';
 		}
diff -Naur linux-2.6.21/scripts/kmsg-doc linux-2.6.21-kmsg/scripts/kmsg-doc
--- linux-2.6.21/scripts/kmsg-doc	1970-01-01 01:00:00.000000000 +0100
+++ linux-2.6.21-kmsg/scripts/kmsg-doc	2007-06-05 15:17:51.000000000 +0200
@@ -0,0 +1,231 @@
+#!/usr/bin/perl
+#
+# Tool to check kernel messages
+#
+# Can be used in toplevel Linux Makefile in the following way:
+#
+# CHECK = scripts/kmsg-doc check
+# CHECKFLAGS =
+#
+# Note: This is just a prototype and neither perfect nor complete!
+#
+# Copyright (C) IBM Corp. 2007
+# Author(s): Michael Holzheu <holzheu@de.ibm.com>
+#
+
+sub create_message($$$$$)
+{
+	my ($sev, $component, $number, $text, $params) = @_;
+
+	$text =~ s/\\n//; # remove trailing newline character
+	$message_id = "$component.$number";
+	$messages{$message_count}->{'ID'} = $message_id;
+	$messages{$message_count}->{'COMP'} = $component;
+	$messages{$message_count}->{'NR'} = $number;
+	$messages{$message_count}->{'MSG'} = $text;
+	$messages{$message_count}->{'SEV'} = $sev;
+
+	@parms = split(/[\s]*,[\s]*/,$params);
+	$parm_count = 0;
+	foreach $parm (@parms) {
+		if (!($parm eq "")) {
+			$messages{$message_count}->{'PARM_NAME'}->{$parm_count} = $parm;
+			$parm_count += 1;
+		}
+	}
+	$messages{$message_count}->{'PARM_COUNT'} = $parm_count;
+	$message_count += 1;
+}
+
+sub get_msgs($)
+{
+	my ($filename)=@_;
+
+	$message_count = 0;
+	open(FD, $filename);
+	my @lines=<FD>;
+	foreach $line (@lines) {
+		if ($line =~ /\s*printk\([\s]*__KMSG_CHECK\((.*)\,[\s]*(.*)\)[\s]*"(.*)"[\s]*(.*)[\s]*\);/) {
+			create_message($1, $component, $2, $3, $4);
+		}
+	}
+}
+
+sub get_descriptions($)
+{
+	my ($filename)=@_;
+	my $desc_start;
+
+	$description_count = 0;
+	$desc_start = 0;
+	open(FD, $filename);
+	my @lines=<FD>;
+	foreach $line (@lines) {
+		if ($line =~ /#define [\s]*KMSG_COMPONENT [\s]*"(.*)"/) {
+			$component = $1;
+		}
+		if ($line =~ /\s*\/\*\*$/) {
+			$msg_start = 1;
+			$parm_count = 0;
+			next;
+		}
+		if (($msg_start == 1) && ($line =~ / \* message/)) {
+			$desc_start = 1;
+			next;
+		}
+		if ($line =~ / \*\//) {
+			$desc_start = 0;
+			next;
+		}
+		if ($desc_start == 1) {
+			$descriptions{$description_count}->{'DESC'} .= "$line";
+			next;
+		}
+		if ($line =~
+			/\s*KMSG_DOC\(\s*(.*)\s*\,\s*(.*)\s*\,\s*\"(.*)\"\s*\);/) {
+			my $param_count = 0;
+			my $first = 1;
+			my $type;
+
+			$descriptions{$description_count}->{'ID'} = "$1\.$2";
+			$descriptions{$description_count}->{'COMP'} = "$1";
+			$descriptions{$description_count}->{'NR'} = "$2";
+			$descriptions{$description_count}->{'MSG'} = "$3";
+			foreach my $arg (split("%", $3)) {
+				if ($first) {
+					$first = 0;
+					next;
+				}
+				$type = substr($arg, 0, 1);
+				$descriptions{$description_count}->{'PARM_TYPE'}->{$param_count} = $type;
+				$param_count += 1;
+			}
+			$descriptions{$description_count}->{'PARM_COUNT'} = $param_count;
+			$description_count += 1;
+		}
+	}
+}
+
+sub print_messages()
+{
+	for ($i = 0; $i < $message_count; $i++) {
+		print "MESSAGE: $messages{$i}->{'ID'}\n";
+	}
+}
+
+sub print_descriptions($)
+{
+	my ($message_id)=@_;
+
+	for ($i = 0; $i < $description_count; $i++) {
+		if (($descriptions{$i}->{'ID'} eq $message_id) || $message_id eq "all") {
+			print "==============================================================================\n";
+			print "\[$descriptions{$i}->{'COMP'}\.$descriptions{$i}->{'NR'}\] $descriptions{$i}->{'MSG'}\n";
+			print "\n";
+			print "Parameters:\n";
+			for ($j = 0; $j < $descriptions{$i}->{'PARM_COUNT'}; $j++) {
+				print " $descriptions{$i}->{'PARM_TYPE'}->{$j}: $descriptions{$i}->{'PARM_DESC'}->{$j}\n";
+			}
+			print "\n";
+			print "Description:\n";
+			print "$descriptions{$i}->{'DESC'}\n";
+			print "==============================================================================\n";
+		}
+	}
+}
+
+sub check_messages($)
+{
+	my ($filename)=@_;
+
+	for ($i = 0; $i < $message_count; $i++) {
+		$found = 0;
+		for ($j = 0; $j < $description_count; $j++) {
+			if ($messages{$i}->{'ID'} eq $descriptions{$j}->{'ID'}) {
+				$found = 1;
+				last;
+			}
+		}
+		if (!$found) {
+			print STDERR "$filename: Missing description for: $messages{$i}->{'ID'}\n";
+		}
+	}
+	for ($i = 0; $i < $description_count; $i++) {
+		$found = 0;
+		for ($j = 0; $j < $message_count; $j++) {
+			if ($messages{$j}->{'ID'} eq $descriptions{$i}->{'ID'}) {
+				$found = 1;
+				last;
+			}
+		}
+		if (!$found) {
+			print STDERR "$filename: Description without message for: $descriptions{$i}->{'ID'}\n";
+		}
+	}
+}
+
+sub print_templates()
+{
+
+	for ($i = 0; $i < $message_count; $i++) {
+		$found = 0;
+		for ($j = 0; $j < $description_count; $j++) {
+			if ($messages{$i}->{'ID'} eq $descriptions{$j}->{'ID'}) {
+				$found = 1;
+			}
+		}
+		if (!$found) {
+			print "/**\n";
+			print " * message\n";
+			for ($k = 0; $k < $messages{$i}->{'PARM_COUNT'}; $k++) {
+				print " * \@$k: $messages{$i}->{'PARM_NAME'}->{$k}\n";
+			}
+			print " *\n";
+			print " * Description:\n";
+			print " *\n";
+			print " * User Response:\n";
+			print " */\n";
+			print "\n";
+			print "KMSG_DOC($messages{$i}->{'COMP'}, $messages{$i}->{'NR'}, \"$messages{$i}->{'MSG'}\");\n"
+		}
+	}
+}
+
+sub usage()
+{
+	print "USAGE: kmsg_tool print | check <file>\n";
+	exit 1;
+}
+
+$option = shift;
+
+if ($option eq "check") {
+	$gcc_options = "-E -D __KMSG_CHECKER ";
+	do {
+		$filename = $tmp;
+		$tmp = shift;
+		$tmp =~ s/\(/\\\(/;
+		$tmp =~ s/\)/\\\)/;
+		$gcc_options .= " $tmp";
+	} while (!($tmp eq ""));
+
+	$gcc_options =~ s/-Wbitwise//; # XXX hack to remove -Wbitwise CHECKFLAG
+	$gcc_options =~ s/-D__STDC__//; # XXX hack to remove -D__STDC__
+	$tmp_file = "$filename.msg";
+	system("gcc $gcc_options > $tmp_file");
+	get_descriptions($filename);
+	get_msgs($tmp_file);
+	check_messages($filename);
+	print_templates();
+	system("rm $tmp_file");
+} elsif ($option eq "print") {
+	$filename = shift;
+	do {
+		print STDERR "Processing: $filename\n";
+		get_descriptions($filename);
+		print_descriptions("all");
+		$filename = shift;
+	} while (!($filename eq ""));
+} else {
+	usage();
+}

Re: [RFC/PATCH] Documentation of kernel messages

[Dave Hansen]

On Wed, 2007-06-13 at 17:06 +0200, holzheu wrote:
> The operation of a Linux system sometimes requires to decode the
> meaning of a specific kernel message, e.g. an error message of a
> driver. Especially on our mainframe zSeries platform system
> administrators want to have descriptions for Linux kernel messages.
> They are used to that, because all other operating systems on that
> platform like z/OS, z/VM or z/VSE have message catalogs with detailed
> descriptions about the semantics of the messages.

I'm not sure we want to make Linux more like z/* in this regard. :)

The problem with your proposal is that every time a new message in the
kernel is created or modified, you need somebody to go update that
documentation.  It's going to get out-of-sync very fast if this isn't
done, and you either need to convince and teach each and every kernel
contributor to follow your lead, or have a team of highly trained code
monkeys to watch git-commits and resubmit documentation for every diff
that touches a printk. 

I think it's a great idea to go and update some of these obtuse kernel
messages with more understandable wording.  It might even be nice to
update the Documentation/ about what kinds of messages are good or bad
to have.  But, I just don't think this is viable to convince everybody
to do this.  Do you have a list of printks that are causing your
customers trouble?

Does every printk need one of these?  What do we do with the existing
printks?  Who will convert them over?  How do we enforce the rules if
new printks must have documentation written for them?  Who writes the
documentation if the printk author does not?

If we have a catalog of kernel messages, it obviously needs to be
versioned, but does that mean that we need to keep it in the kernel
tree?

-- Dave

Re: [RFC/PATCH] Documentation of kernel messages

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 4047 bytes --]

On Wed, 13 Jun 2007 17:06:57 +0200, holzheu said:
> They are used to that, because all other operating systems on that
> platform like z/OS, z/VM or z/VSE have message catalogs with detailed
> descriptions about the semantics of the messages.

25 years ago, I did OS/MVT and OS/VS1 for a living, so I know *all* about
the infamous "What does IEF507E mean again?"...

> In general we think, that also for Linux it is a good thing to have
> documentation for the most important kernel/driver messages. Even
> kernel hackers not always are aware of the meaning of kernel messages
> for components, which they don't know in detail. Most of the messages
> are self explaining but sometimes you get something like "Clocksource
> tsc unstable (delta = 7304132729 ns)" and you wonder if your system is
> going to explode.

This is probably best addressed by cleaning up the actual messages so they're
a bit more informative.

> New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be
> used in printk. These macros have as parameter the message number and
> are using a per c-file defined macro KMSG_COMPONENT.

Gaak. *NO*.

The *only* reason that the MVS and VM message catalogs worked at all is
because each component had a message repository that went across *all* the
source files - the instant you saw IEFnnns, you knew that IEF covered the
job scheduler, nnn was a *unique* number, and s was a Severe/Warning/Info
flag.  IGG was always data management, and so on.  This breaks horribly if
you have 2 C files that define subtly different KMSG_COMPONENT values (or
even worse, 2 or more duplicates).

[/usr/src/linux-2.6.22-rc4-mm2] find . -name '*.c' | wc -l
9959
[/usr/src/linux-2.6.22-rc4-mm2] find . -name '*.h' | wc -l
9933
[/usr/src/linux-2.6.22-rc4-mm2] find . -type d | wc -l
1736

You plan to maintain message uniqueness how?

[/usr/src/linux-2.6.22-rc4-mm2]1 find . -name '*.c' | sed -r 's?.*/([^/]*)?\1?' | sort | uniq -c | sort -nr | head
    105 setup.c
     90 irq.c
     66 time.c
     58 init.c
     50 inode.c
     39 io.c
     38 pci.c
     37 file.c
     32 signal.c
     32 ptrace.c

Looks like you're going to have to embed a lot of the path in that KMSG_COMPONENT
to make it unique - and you want to keep that message under 80 or so chars total.

> /**
>  * message
>  * @0: device number of device.
>  *
>  * Description:
>  * An operation has been performed on the msgtest device, but the
>  * device has not been set online. Therefore the operation failed

If you don't understand 'Device /dev/foo offline', this description
doesn't help any.  And that's true for *most* of the kernel messages
already - if you don't understand the message already, a paragraph
explanation isn't going to help much.  Consider the average OOPS
message, which contains stuff like 'EIP=0x..'.  Telling the user that
EIP means Execution Instruction Pointer isn't likely to help - if they
knew what the pointer *did*, they'd probably already know EIP.

>  *
>  * User Response:
>  * Operator should set device online.
>  * Issue "chccwdev -e <device number>".

And this is where the weakness of this scheme *really* hits.  I've actually run
into cases where an operator followed the listed "Operator Response" for a
"device offline", and issued a 'VARY 0C0,ONLINE'.  And then we got a flood of
I/O errors because the previous shift downed the device because it was having
issues.  The response the operator *should* have done is "assign a different
tape drive, like, oh maybe the operational ones at 0C1 through 0C4"...

And it's the same here - if you get a message that /dev/sdb1 has no media
present, there's a good chance that you typo'ed, and meant /dev/sda1 or /dev/sdc1
So following the directions for 'sdb1 offline' and putting in a blank DVD
because sdb is the DVD burner won't fix things if what you were trying to do is
mkfs something on another disk... ;)

And while we're at it, I'll point out that any attempt to "fix" the kernel
messages on this scale had *better* solve all the I18N problems while we're
there....




[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: [RFC/PATCH] Documentation of kernel messages

[Sam Ravnborg]

On Wed, Jun 13, 2007 at 09:37:29AM -0700, Dave Hansen wrote:
> On Wed, 2007-06-13 at 17:06 +0200, holzheu wrote:
> > The operation of a Linux system sometimes requires to decode the
> > meaning of a specific kernel message, e.g. an error message of a
> > driver. Especially on our mainframe zSeries platform system
> > administrators want to have descriptions for Linux kernel messages.
> > They are used to that, because all other operating systems on that
> > platform like z/OS, z/VM or z/VSE have message catalogs with detailed
> > descriptions about the semantics of the messages.
> 
> I'm not sure we want to make Linux more like z/* in this regard. :)
> 
> The problem with your proposal is that every time a new message in the
> kernel is created or modified, you need somebody to go update that
> documentation.  It's going to get out-of-sync very fast if this isn't
> done, and you either need to convince and teach each and every kernel
> contributor to follow your lead, or have a team of highly trained code
> monkeys to watch git-commits and resubmit documentation for every diff
> that touches a printk. 

This is no more and no less the same situation that we have with
the kernel-doc documented functions/data in the kernel.

I like the concept that the description is kept close to the actual
usage, the tool support and that in general looks like ordinary
kernel-doc documentation.

And if people do not dare to update the kernel-doc documentation of
a function then maybe they should not send patches in first place..

If we then really want to have the important printk's documented
is another story.

	Sam

Re: [RFC/PATCH] Documentation of kernel messages

[Alexey Dobriyan]

On Wed, Jun 13, 2007 at 05:06:57PM +0200, holzheu wrote:
> -CHECK		= sparse
> -
> -CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ -Wbitwise $(CF)
> +CHECK		= scripts/kmsg-doc check
> +CHECKFLAGS	=

Ick. Don't touch those checking kernel with sparse.

Re: [RFC/PATCH] Documentation of kernel messages

[holzheu]

Hi Dave,

On Wed, 2007-06-13 at 09:37 -0700, Dave Hansen wrote:

[snip]

> I'm not sure we want to make Linux more like z/* in this regard. :)
> 
> The problem with your proposal is that every time a new message in the
> kernel is created or modified, you need somebody to go update that
> documentation.

You only have to document printks, which are using the KMSG macros.
Since there are tons of self-explaining printks, most of them do not
have to use them.

If you change the meaning of a KMSG printk and you don't update the
documentation, it is the same thing like changing a kernel API function
and forgetting to update the kernel-doc comment.

> It's going to get out-of-sync very fast if this isn't
> done, and you either need to convince and teach each and every kernel
> contributor to follow your lead, or have a team of highly trained code
> monkeys to watch git-commits and resubmit documentation for every diff
> that touches a printk. 

It is a matter of discipline. If a device driver maintainer decides to
document some messages using the KMSG macros, he has to take care that
the documentation is up-to-date. Since the printks and the descriptions
are maintained together in the kernel code, I think, that's doable. 

And the checker tool can help to keep the descriptions up-to-date, since
it produces warnings, if you use C=1 during the kernel build.

> I think it's a great idea to go and update some of these obtuse kernel
> messages with more understandable wording.  It might even be nice to
> update the Documentation/ about what kinds of messages are good or bad
> to have.  But, I just don't think this is viable to convince everybody
> to do this.

Again, not everybody has to use KMSG printks.

> Do you have a list of printks that are causing your
> customers trouble?

There are quite a lot of messages in our s390 device drivers which
should be documented. Especially for character device drivers (e.g.
tape), where the user has direct interaction with the device.

One Example from the tape device driver: "Another host has reserved the
tape device"

We should document, how to find out the host, which reserved the tape
and what has to be done on the other host in order to release the tape
etc.

> Does every printk need one of these?  What do we do with the existing
> printks?

Again: Most printks don't need documentation. We can leave them as they
are.

> Who will convert them over?  How do we enforce the rules if
> new printks must have documentation written for them?  Who writes the
> documentation if the printk author does not?

The maintainer of the component (e.g. device driver) is responsible for
the KMSG printks. If he accepts a patch, which modifies KMSG printks, he
has to take care, that the patch also updates the KMSG_DOC part.

> If we have a catalog of kernel messages, it obviously needs to be
> versioned, but does that mean that we need to keep it in the kernel
> tree?

For each kernel a separate message catalog can be automatically created.
This message catalog fits exactly to one kernel and should be installed
together with the kernel.

Michael

Re: [RFC/PATCH] Documentation of kernel messages

[holzheu]

On Wed, 2007-06-13 at 21:16 +0400, Alexey Dobriyan wrote:
> On Wed, Jun 13, 2007 at 05:06:57PM +0200, holzheu wrote:
> > -CHECK		= sparse
> > -
> > -CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ -Wbitwise $(CF)
> > +CHECK		= scripts/kmsg-doc check
> > +CHECKFLAGS	=
> 
> Ick. Don't touch those checking kernel with sparse.

Of course we don't want to have that in the vanilla kernel. I just
wanted to show, how the checker tool has to be used.

Michael

Re: [RFC/PATCH] Documentation of kernel messages

[Greg KH]

On Wed, Jun 13, 2007 at 05:06:57PM +0200, holzheu wrote:
> Current prototype implementation:
> ================================= 
> 
> The structure of a kernel message is: <component>.<msg number>: <msg>
> 
> * component: Name of the kernel or driver component e.g. "pci", "ide", 
>   etc.
> * msg number: Within the component unique number of a kernel message.
> * msg: printk message
> 
> New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be
> used in printk. These macros have as parameter the message number and
> are using a per c-file defined macro KMSG_COMPONENT.
> 
> Example: Define message 2 in component "kmsgtest":
> 
> #define KMSG_COMPONENT "kmsgtest"
> 
> void f(void)
> {
>         printk(KMSG_ERR(1) "device %x not online\n", devno);
> }

Ick, why are you ignoring what we have already with dev_printk() and
friends?  We are just finally getting developers to use that, I think it
will be almost impossible to get people to change to something else,
especially one that isn't even as "correct" as what dev_printk() offers
you today, will be quite hard.

So, why not use what we already have and work off of it?

thanks,

greg k-h

Re: [RFC/PATCH] Documentation of kernel messages

[Rob Landley]

On Wednesday 13 June 2007 12:50:55 Valdis.Kletnieks@vt.edu wrote:
> > In general we think, that also for Linux it is a good thing to have
> > documentation for the most important kernel/driver messages. Even
> > kernel hackers not always are aware of the meaning of kernel messages
> > for components, which they don't know in detail. Most of the messages
> > are self explaining but sometimes you get something like "Clocksource
> > tsc unstable (delta = 7304132729 ns)" and you wonder if your system is
> > going to explode.

Isn't this what the severity levels are for?  Not just filtering messages out 
but telling you whether or not it's important?

> This is probably best addressed by cleaning up the actual messages so
> they're a bit more informative.

Seconded.

The point of a diagnostic message is to convey information.  Deep in the 
bowels of chipsets there are diagnostic messages that get spit out as 
hexcodes that some program needs to interpret, but that's why we have tools 
like http://kernel.org/pub/linux/kernel/people/davej/tools/parsemce.c and 
such.

If the diagnostic messages need a talmud, I see this is as more of a problem 
with said diagnostic messages, rather than excuse to add another layer.

> > New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be
> > used in printk. These macros have as parameter the message number and
> > are using a per c-file defined macro KMSG_COMPONENT.
>
> Gaak. *NO*.

If you're going to tweak printk, please make the darn thing take an integer 
first argument rather than a string first argument, so we can use clever 
macros to remove them at compile time rather than having them compiled in but 
not displaying.

This is one of those things buried down on my todo list.  I think it can be 
converted incrementally

Going back to the original patch here, there are exactly two examples of what 
this brave new infrastructure would be used for:

> +/**
> + * message
> + * @0: Device number of device
> + * @1: Status of device
> + *
> + * Description:
> + * Information message about the status of our virtual msgtest device. The
> + * following values for the status parameter are available.
> + *
> + *   0 - Device is offline
> + *
> + *   1 - Device is online
> + *
> + *   2 - Device is broken
> + *
> + * User Response:
> + * If device is broken, replace it or fix it.
> + */
> +
> +KMSG_DOC(kmsgtest, 1, "device %x has status %i");

This is just a thought: why not have the actual kprintf say "Device number %x 
is %s" where %s is "online", "offline", or "broken"?

I.E. have the kernel output a user readable message in the first place.

> +/**
> + * message
> + * @0: Device number of device.
> + *
> + * Description:
> + * An operation has been performed on the msgtest device, but the device
> has + * not been set online. Therefore the operation failed.
> + *
> + * User Response:
> + * Operator should set device online. Issue "chccwdev -e <device number>".
> + */
> +
> +KMSG_DOC(kmsgtest, 2, "device %x not online");

kprintf(LEVEL "device %x not online, try 'chccwdev -e %x'", devnum, devnum)

Rob

Re: [RFC/PATCH] Documentation of kernel messages

[holzheu]

Hi Valdis,

On Wed, 2007-06-13 at 12:50 -0400, Valdis.Kletnieks@vt.edu wrote: 
> On Wed, 13 Jun 2007 17:06:57 +0200, holzheu said:
> > They are used to that, because all other operating systems on that
> > platform like z/OS, z/VM or z/VSE have message catalogs with detailed
> > descriptions about the semantics of the messages.
> 
> 25 years ago, I did OS/MVT and OS/VS1 for a living, so I know *all* about
> the infamous "What does IEF507E mean again?"...

:-)

> > In general we think, that also for Linux it is a good thing to have
> > documentation for the most important kernel/driver messages. Even
> > kernel hackers not always are aware of the meaning of kernel messages
> > for components, which they don't know in detail. Most of the messages
> > are self explaining but sometimes you get something like "Clocksource
> > tsc unstable (delta = 7304132729 ns)" and you wonder if your system is
> > going to explode.
> 
> This is probably best addressed by cleaning up the actual messages so they're
> a bit more informative.

Of course that would be good, too.

But I think, that we sometimes have the dilemma, that we want to keep
the printks short, but also want to provide as much information as
possible.

If the information is to big for the printk itself, because you would
need 10 lines to explain what happened, wouldn't it be good to have a
place where to put that information?

> > New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be
> > used in printk. These macros have as parameter the message number and
> > are using a per c-file defined macro KMSG_COMPONENT.
> 
> Gaak. *NO*.
> 
> The *only* reason that the MVS and VM message catalogs worked at all is
> because each component had a message repository that went across *all* the
> source files - the instant you saw IEFnnns, you knew that IEF covered the
> job scheduler, nnn was a *unique* number, and s was a Severe/Warning/Info
> flag.  IGG was always data management, and so on.  This breaks horribly if
> you have 2 C files that define subtly different KMSG_COMPONENT values (or
> even worse, 2 or more duplicates).
> 
> [/usr/src/linux-2.6.22-rc4-mm2] find . -name '*.c' | wc -l
> 9959
> [/usr/src/linux-2.6.22-rc4-mm2] find . -name '*.h' | wc -l
> 9933
> [/usr/src/linux-2.6.22-rc4-mm2] find . -type d | wc -l
> 1736
> 
> You plan to maintain message uniqueness how?
> [/usr/src/linux-2.6.22-rc4-mm2]1 find . -name '*.c' | sed -r 's?.*/([^/]*)?\1?' | sort | uniq -c | sort -nr | head
>     105 setup.c
>      90 irq.c
>      66 time.c
>      58 init.c
>      50 inode.c
>      39 io.c
>      38 pci.c
>      37 file.c
>      32 signal.c
>      32 ptrace.c
> 
> Looks like you're going to have to embed a lot of the path in that KMSG_COMPONENT
> to make it unique - and you want to keep that message under 80 or so chars total.
> 

For each kernel component, like a device driver, we could have one
KMSG_COMPONENT (e.g. "acpi", "pci", etc). Within that component the
message ids have to be unique. A tool could check, if messages are
unique within the kernel sources.

We could use something like a Documentation/kmsg-components file with a
list of all component names using KMSG printks.

> > /**
> >  * message
> >  * @0: device number of device.
> >  *
> >  * Description:
> >  * An operation has been performed on the msgtest device, but the
> >  * device has not been set online. Therefore the operation failed
> 
> If you don't understand 'Device /dev/foo offline', this description
> doesn't help any.  And that's true for *most* of the kernel messages
> already - if you don't understand the message already, a paragraph
> explanation isn't going to help much.  Consider the average OOPS
> message, which contains stuff like 'EIP=0x..'.  Telling the user that
> EIP means Execution Instruction Pointer isn't likely to help - if they
> knew what the pointer *did*, they'd probably already know EIP.

I agree with you, that most of the kernel messages do not need further
documentation. But I am convinced, that there are plenty of printks,
where additional documentation would be helpful.

> >  *
> >  * User Response:
> >  * Operator should set device online.
> >  * Issue "chccwdev -e <device number>".
> 
> And this is where the weakness of this scheme *really* hits.  I've actually run
> into cases where an operator followed the listed "Operator Response" for a
> "device offline", and issued a 'VARY 0C0,ONLINE'.  And then we got a flood of
> I/O errors because the previous shift downed the device because it was having
> issues.  The response the operator *should* have done is "assign a different
> tape drive, like, oh maybe the operational ones at 0C1 through 0C4"...

I can understand your frustration here. But that's a general problem
with documentation. You never can foresee everything.

But should this mean, that we shouldn't document anything?

Michael

Re: [RFC/PATCH] Documentation of kernel messages

[Andrew Morton]

> On Wed, 13 Jun 2007 17:06:57 +0200 holzheu <holzheu@linux.vnet.ibm.com> wrote:
> Greetings,
> 
> The operation of a Linux system sometimes requires to decode the
> meaning of a specific kernel message, e.g. an error message of a
> driver. Especially on our mainframe zSeries platform system
> administrators want to have descriptions for Linux kernel messages.
> They are used to that, because all other operating systems on that
> platform like z/OS, z/VM or z/VSE have message catalogs with detailed
> descriptions about the semantics of the messages.
> 
> In general we think, that also for Linux it is a good thing to have
> documentation for the most important kernel/driver messages. Even
> kernel hackers not always are aware of the meaning of kernel messages
> for components, which they don't know in detail. Most of the messages
> are self explaining but sometimes you get something like "Clocksource
> tsc unstable (delta = 7304132729 ns)" and you wonder if your system is
> going to explode.
> 
> Unfortunately currently there is no general infrastructure in the Linux
> kernel for the documentation of messages. I worked on a proposal, how
> that could be implemented in an easy way using the already existing
> kernel-doc infrastructure and using printk. The proposal is as follows
> 
> 1. We use message identifiers in order to map messages to message
> descriptions. A message identifier consists out of a component name and
> within that component unique message number.
> 
> 2. Messages and message descriptions are maintained together in the
> kernel sources in order to keep them up to date. Messages catalog are
> generated automatically for exactly one kernel level.
> 
> 3. A special tool checks, if messages are not documented or if there
> are message descriptions without corresponding messages.
> 
> 4. We use the already existing kernel-doc tool to generate an online
> message catalog or e.g. a pdf for offline documentation.
> 
> Current prototype implementation:
> ================================= 
> 
> The structure of a kernel message is: <component>.<msg number>: <msg>
> 
> * component: Name of the kernel or driver component e.g. "pci", "ide", 
>   etc.
> * msg number: Within the component unique number of a kernel message.
> * msg: printk message
> 
> New macros KMSG_ERR(), KMSG_WARN(), etc. are defined, which have to be
> used in printk. These macros have as parameter the message number and
> are using a per c-file defined macro KMSG_COMPONENT.
> 
> Example: Define message 2 in component "kmsgtest":
> 
> #define KMSG_COMPONENT "kmsgtest"
> 
> void f(void)
> {
>         printk(KMSG_ERR(1) "device %x not online\n", devno);
> }
> 
> The output of that kernel message would be:
> "kmsgtest.1: device 4711 not online"
> 
> The messages have to be documented within the C source file
> in the following way:
> 
> /**
>  * message
>  * @0: device number of device.
>  *
>  * Description:
>  * An operation has been performed on the msgtest device, but the
>  * device has not been set online. Therefore the operation failed
>  *
>  * User Response:
>  * Operator should set device online.
>  * Issue "chccwdev -e <device number>".
>  *
>  */
> 
> KMSG_DOC(kmsgtest, 2,  "Device %x not online");
> 
> I created a patch for the kernel-doc tool so it can be used to generate
> a catalog of all kernel messages:
> 
> >> kernel-doc -man kmsgtest.c > kmsgtest.2
> >> man ./kmsgtest.2 
> 
> # Kernel API(9)               Linux Messages               Kernel API(9)
> # 
> # MESSAGE:
> #       kmsgtest.2: "device %x not online"
> #
> # Parameter:
> #       1           Device number of device.
> #
> # Description:
> #       An  operation  has been performed on the msgtest device, but
> #       the device has not been set online. Therefore the operation failed.
> #
> # User Response:
> #       Operator should set device online.
> #       Issue "chccwdev -e <device number>".
> #
> # May 2007                      kmsgtest.2                 Kernel API(9)
> 
> A nice thing would be to include the online kernel message catalog in
> the kernel rpm. One possibility for that would be to have one man page
> per message. If an operator finds the message kmsgtest.2 in
> var/log/messages and wants to know what the message means, he simply
> issues "man kmsgtest.2" and gets the description.
> 
> To ensure that all messages are documented and there are no message
> descriptions without corresponding messages, a checker tool is
> provided. To enable message checking, in the toplevel Makefile the
> following has to be added:
> 
> CHECK = scripts/kmsg_check.pl check
> 
> To enable message checking during kernel build, the "C" option has
> to be used:
> 
> >> make modules C=1
>   CHK     include/linux/version.h
>   CHK     include/linux/utsrelease.h
>   CHECK   drivers/kmsgtest/kmsgtest.c
> drivers/kmsgtest/kmsgtest.c: Missing description for: kmsgtest.1
> drivers/kmsgtest/kmsgtest.c: Description without message for: kmsgtest.3
> 
> Please note, that the patch for the kernel-doc and kmsg-doc tools
> is just a prototype and is neither complete nor perfect.
> 
> Michael
> 
> Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
> Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com>
> Signed-off-by: Michael Holzheu <holzheu@linux.vmnet.ibm.com>

Your proposal is similar to one I made to some Japanese developers
earlier this year.  I was more modest, proposing that we

- add an enhanced printk

	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

- An externally managed database will provide translations, diagnostics,
  remedial actions, etc, keyed off the msgid string

- Format and management of the msgid hasn't been clearly identified yet
  as far as I know.  But all we really need is that each ID be unique,
  kernel-wide.  We develop a simple tool which can check the whole tree
  for duplicated msgids.

- From a practical day-to-day POV what this means is that a person
  from the kernel-messages team will prepare a patch for your driver
  which adds the msgids and you the driver author should take care not
  to break the tagging.

  This is deliberately designed to be minimum-impact upon general 
  developers.  We want a system in place where driver/fs/etc developers
  can concentrate on what they do, and kernel-message developers can
  as independently as possibe take care of what _they_ do.

- A project has started up and there is a mailing list (cc'ed here) and
  meetings are happening at the LF collaboration summit this week.

  Please see https://lists.linux-foundation.org/mailman/listinfo/lf_kernel_messages
  and don't miss the next conference call ;)

  (argh.  The lf_kernel_messages archives are subscriber-only and it could be that
  only members can post to it.  Oh well.  Please subscribe, and review the archives)

> 
>  Makefile                    |    5
>  drivers/Makefile            |    1
>  drivers/kmsgtest/Makefile   |    5
>  drivers/kmsgtest/kmsgtest.c |   59 +++++++++++
>  include/linux/kmsg.h        |   32 ++++++
>  scripts/kernel-doc          |  116 +++++++++++++++++++++-
>  scripts/kmsg-doc            |  231 ++++++++++++++++++++++++++++++++++++++++++++
>  7 files changed, 445 insertions(+), 4 deletions(-)
> 
> diff -Naur linux-2.6.21/Makefile linux-2.6.21-kmsg/Makefile
> --- linux-2.6.21/Makefile	2007-04-26 05:08:32.000000000 +0200
> +++ linux-2.6.21-kmsg/Makefile	2007-06-05 15:17:51.000000000 +0200
> @@ -293,9 +293,8 @@
>  DEPMOD		= /sbin/depmod
>  KALLSYMS	= scripts/kallsyms
>  PERL		= perl
> -CHECK		= sparse
> -
> -CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ -Wbitwise $(CF)
> +CHECK		= scripts/kmsg-doc check
> +CHECKFLAGS	=
>  MODFLAGS	= -DMODULE
>  CFLAGS_MODULE   = $(MODFLAGS)
>  AFLAGS_MODULE   = $(MODFLAGS)
> diff -Naur linux-2.6.21/drivers/Makefile linux-2.6.21-kmsg/drivers/Makefile
> --- linux-2.6.21/drivers/Makefile	2007-04-26 05:08:32.000000000 +0200
> +++ linux-2.6.21-kmsg/drivers/Makefile	2007-06-05 15:17:51.000000000 +0200
> @@ -8,6 +8,7 @@
>  obj-$(CONFIG_PCI)		+= pci/
>  obj-$(CONFIG_PARISC)		+= parisc/
>  obj-$(CONFIG_RAPIDIO)		+= rapidio/
> +obj-m				+= kmsgtest/
>  obj-y				+= video/
>  obj-$(CONFIG_ACPI)		+= acpi/
>  # PnP must come after ACPI since it will eventually need to check if acpi
> diff -Naur linux-2.6.21/drivers/kmsgtest/Makefile linux-2.6.21-kmsg/drivers/kmsgtest/Makefile
> --- linux-2.6.21/drivers/kmsgtest/Makefile	1970-01-01 01:00:00.000000000 +0100
> +++ linux-2.6.21-kmsg/drivers/kmsgtest/Makefile	2007-06-05 15:17:51.000000000 +0200
> @@ -0,0 +1,5 @@
> +#
> +# Makefile for kernel message test module
> +#
> +
> +obj-m		+= kmsgtest.o
> diff -Naur linux-2.6.21/drivers/kmsgtest/kmsgtest.c linux-2.6.21-kmsg/drivers/kmsgtest/kmsgtest.c
> --- linux-2.6.21/drivers/kmsgtest/kmsgtest.c	1970-01-01 01:00:00.000000000 +0100
> +++ linux-2.6.21-kmsg/drivers/kmsgtest/kmsgtest.c	2007-06-05 15:17:51.000000000 +0200
> @@ -0,0 +1,59 @@
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/kmsg.h>
> +
> +static int devno = 0x4711;
> +static int status = 1;
> +
> +#define KMSG_COMPONENT "kmsgtest"
> +
> +static int __init kmsgtest_init(void)
> +{
> +	printk(KMSG_INFO(1) "device %x has status %i\n", devno, status);
> +	printk(KMSG_ERR(2) "device %x not online\n", devno);
> +
> +	return 0;
> +}
> +
> +static void __exit kmsgtest_exit(void)
> +{
> +	printk("kmsgtest module exit\n");
> +}
> +
> +module_init(kmsgtest_init);
> +module_exit(kmsgtest_exit);
> +
> +/**
> + * message
> + * @0: Device number of device
> + * @1: Status of device
> + *
> + * Description:
> + * Information message about the status of our virtual msgtest device. The
> + * following values for the status parameter are available.
> + *
> + *   0 - Device is offline
> + *
> + *   1 - Device is online
> + *
> + *   2 - Device is broken
> + *
> + * User Response:
> + * If device is broken, replace it or fix it.
> + */
> +
> +KMSG_DOC(kmsgtest, 1, "device %x has status %i");
> +
> +/**
> + * message
> + * @0: Device number of device.
> + *
> + * Description:
> + * An operation has been performed on the msgtest device, but the device has
> + * not been set online. Therefore the operation failed.
> + *
> + * User Response:
> + * Operator should set device online. Issue "chccwdev -e <device number>".
> + */
> +
> +KMSG_DOC(kmsgtest, 2, "device %x not online");
> diff -Naur linux-2.6.21/include/linux/kmsg.h linux-2.6.21-kmsg/include/linux/kmsg.h
> --- linux-2.6.21/include/linux/kmsg.h	1970-01-01 01:00:00.000000000 +0100
> +++ linux-2.6.21-kmsg/include/linux/kmsg.h	2007-06-05 15:17:51.000000000 +0200
> @@ -0,0 +1,32 @@
> +#ifndef _LINUX_KMSG_H
> +#define _LINUX_KMSG_H
> +
> +#ifdef __KMSG_CHECKER
> +
> +#define KMSG_EMERG(num)   __KMSG_CHECK(EMERG, num)
> +#define KMSG_ALERT(num)   __KMSG_CHECK(ALERT, num)
> +#define KMSG_CRIT(num)    __KMSG_CHECK(CRIT, num)
> +#define KMSG_ERR(num)     __KMSG_CHECK(ERR, num)
> +#define KMSG_WARNING(num) __KMSG_CHECK(WARNING, num)
> +#define KMSG_NOTICE(num)  __KMSG_CHECK(NOTICE, num)
> +#define KMSG_INFO(num)    __KMSG_CHECK(INFO, num)
> +#define KMSG_DEBUG(num)   __KMSG_CHECK(DEBUG, num)
> +
> +#define KMSG_DOC(comp, num, str) __KMSG_DOC(comp, num, str)
> +
> +#else
> +
> +#define KMSG_EMERG(num)   KERN_EMERG KMSG_COMPONENT "." #num ": "
> +#define KMSG_ALERT(num)   KERN_ALERT KMSG_COMPONENT "." #num ": "
> +#define KMSG_CRIT(num)    KERN_CRIT KMSG_COMPONENT "." #num ": "
> +#define KMSG_ERR(num)     KERN_ERR KMSG_COMPONENT "." #num ": "
> +#define KMSG_WARNING(num) KERN_WARNING KMSG_COMPONENT "." #num ": "
> +#define KMSG_NOTICE(num)  KERN_NOTICE KMSG_COMPONENT "." #num ": "
> +#define KMSG_INFO(num)    KERN_INFO KMSG_COMPONENT "." #num ": "
> +#define KMSG_DEBUG(num)   KERN_DEBUG KMSG_COMPONENT "." #num ": "
> +
> +#define KMSG_DOC(comp, num, str)
> +
> +#endif /* __KMSG_CHECKER */
> +
> +#endif /* _LINUX_KMSG_H */
> diff -Naur linux-2.6.21/scripts/kernel-doc linux-2.6.21-kmsg/scripts/kernel-doc
> --- linux-2.6.21/scripts/kernel-doc	2007-04-26 05:08:32.000000000 +0200
> +++ linux-2.6.21-kmsg/scripts/kernel-doc	2007-06-05 15:17:51.000000000 +0200
> @@ -256,7 +256,7 @@
>  my $in_doc_sect;
>  
>  #declaration types: can be
> -# 'function', 'struct', 'union', 'enum', 'typedef'
> +# 'function', 'struct', 'union', 'enum', 'typedef', 'message'
>  my $decl_type;
>  
>  my $doc_special = "\@\%\$\&";
> @@ -1163,6 +1163,55 @@
>      output_section_text(@_);
>  }
>  
> +##
> +# output message in text
> +sub output_message_text(%) {
> +	my %args = %{$_[0]};
> +	my ($parameter);
> +
> +	print $args{'id'}.": \"".$args{'message'}."\"\n\n";
> +
> +	print "Parameters:\n\n";
> +	foreach $parameter (@{$args{'parameterlist'}}) {
> +		($parameter =~ /^#/) && next;
> +		my $parameter_name = $parameter;
> +		$parameter_name =~ s/\[.*//;
> +		($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
> +		print "$parameter\n\t";
> +		print $args{'parameterdescs'}{$parameter_name}."\n";
> +	}
> +	output_section_text(@_);
> +}
> +
> +##
> +# output message in man
> +sub output_message_man(%) {
> +	my %args = %{$_[0]};
> +	my ($parameter, $section);
> +
> +	print ".TH \"$args{'module'}\" 9 \"".$args{'id'}."\" \"$man_date\" \"Linux Messages\" LINUX\n";
> +
> +	print ".SH MESSAGE\n";
> +	print $args{'id'}.": "."\"".$args{'message'}."\"\n";
> +
> +	print ".SH Parameters\n";
> +	foreach $parameter (@{$args{'parameterlist'}}) {
> +		($parameter =~ /^#/) && next;
> +
> +		my $parameter_name = $parameter;
> +		$parameter_name =~ s/\[.*//;
> +
> +		($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
> +		print ".IP \"".$parameter."\" 12\n";
> +		output_highlight($args{'parameterdescs'}{$parameter_name});
> +	}
> +	foreach $section (@{$args{'sectionlist'}}) {
> +		print ".SH \"$section\"\n";
> +		output_highlight($args{'sections'}{$section});
> +		print "\n";
> +	}
> +}
> +
>  #output sections in text
>  sub output_section_text(%) {
>      my %args = %{$_[0]};
> @@ -1607,6 +1656,69 @@
>  		       });
>  }
>  
> +sub create_parameterlist_msg($$) {
> +	my $args = shift;
> +	my $file = shift;
> +	my $splitter = "%";
> +	my $type;
> +	my $param = 0;
> +	my $first = 1;
> +
> +	# temporarily replace commas
> +	while ($args =~ /(\([^\),]+),/) {
> +		$args =~ s/(\([^\),]+),/$1#/g;
> +	}
> +
> +	foreach my $arg (split($splitter, $args)) {
> +		if ($first) {
> +			$first = 0;
> +			next;
> +		}
> +		$type = substr($arg,0,1);
> +
> +		# XXX introduce better type checking
> +
> +		push_parameter($param, $type, $file);
> +		$param += 1;
> +	}
> +}
> +
> +##
> +# takes a message prototype and the name of the current file being
> +# processed and spits out all the details stored in the global
> +# arrays/hashes.
> +sub dump_message($$) {
> +	my $x = shift;
> +	my $file = shift;
> +
> +	if ($x =~/(KMSG_DOC)\(\s*(\w+)\s*\,\s*(\w+)\s*\,\s*\"(.*)\"\)/) {
> +	$declaration_name = "$2.$3";
> +	my $members = $4;
> +	# strip comments:
> +	$members =~ s/\/\*.*?\*\///gos;
> +
> +	create_parameterlist_msg($members, $file);
> +
> +	output_declaration($declaration_name,
> +			   'message',
> +			   {'message' => $members,
> +			    'id' => $declaration_name,
> +			    'module' => $modulename,
> +			    'parameterlist' => \@parameterlist,
> +			    'parameterdescs' => \%parameterdescs,
> +			    'parametertypes' => \%parametertypes,
> +			    'sectionlist' => \@sectionlist,
> +			    'sections' => \%sections,
> +			    'purpose' => $declaration_purpose,
> +			    'type' => $decl_type
> +			   });
> +	}
> +	else {
> +		print STDERR "Error(${file}:$.): Cannot parse message!\n";
> +		++$errors;
> +	}
> +}
> +
>  sub process_file($);
>  
>  # Read the file that maps relative names to absolute names for
> @@ -1782,6 +1894,8 @@
>  		    $decl_type = 'enum';
>  		} elsif ($identifier =~ m/^typedef/) {
>  		    $decl_type = 'typedef';
> +		} elsif ($identifier =~ m/^message/) {
> +		    $decl_type = 'message';
>  		} else {
>  		    $decl_type = 'function';
>  		}
> diff -Naur linux-2.6.21/scripts/kmsg-doc linux-2.6.21-kmsg/scripts/kmsg-doc
> --- linux-2.6.21/scripts/kmsg-doc	1970-01-01 01:00:00.000000000 +0100
> +++ linux-2.6.21-kmsg/scripts/kmsg-doc	2007-06-05 15:17:51.000000000 +0200
> @@ -0,0 +1,231 @@
> +#!/usr/bin/perl
> +#
> +# Tool to check kernel messages
> +#
> +# Can be used in toplevel Linux Makefile in the following way:
> +#
> +# CHECK = scripts/kmsg-doc check
> +# CHECKFLAGS =
> +#
> +# Note: This is just a prototype and neither perfect nor complete!
> +#
> +# Copyright (C) IBM Corp. 2007
> +# Author(s): Michael Holzheu <holzheu@de.ibm.com>
> +#
> +
> +sub create_message($$$$$)
> +{
> +	my ($sev, $component, $number, $text, $params) = @_;
> +
> +	$text =~ s/\\n//; # remove trailing newline character
> +	$message_id = "$component.$number";
> +	$messages{$message_count}->{'ID'} = $message_id;
> +	$messages{$message_count}->{'COMP'} = $component;
> +	$messages{$message_count}->{'NR'} = $number;
> +	$messages{$message_count}->{'MSG'} = $text;
> +	$messages{$message_count}->{'SEV'} = $sev;
> +
> +	@parms = split(/[\s]*,[\s]*/,$params);
> +	$parm_count = 0;
> +	foreach $parm (@parms) {
> +		if (!($parm eq "")) {
> +			$messages{$message_count}->{'PARM_NAME'}->{$parm_count} = $parm;
> +			$parm_count += 1;
> +		}
> +	}
> +	$messages{$message_count}->{'PARM_COUNT'} = $parm_count;
> +	$message_count += 1;
> +}
> +
> +sub get_msgs($)
> +{
> +	my ($filename)=@_;
> +
> +	$message_count = 0;
> +	open(FD, $filename);
> +	my @lines=<FD>;
> +	foreach $line (@lines) {
> +		if ($line =~ /\s*printk\([\s]*__KMSG_CHECK\((.*)\,[\s]*(.*)\)[\s]*"(.*)"[\s]*(.*)[\s]*\);/) {
> +			create_message($1, $component, $2, $3, $4);
> +		}
> +	}
> +}
> +
> +sub get_descriptions($)
> +{
> +	my ($filename)=@_;
> +	my $desc_start;
> +
> +	$description_count = 0;
> +	$desc_start = 0;
> +	open(FD, $filename);
> +	my @lines=<FD>;
> +	foreach $line (@lines) {
> +		if ($line =~ /#define [\s]*KMSG_COMPONENT [\s]*"(.*)"/) {
> +			$component = $1;
> +		}
> +		if ($line =~ /\s*\/\*\*$/) {
> +			$msg_start = 1;
> +			$parm_count = 0;
> +			next;
> +		}
> +		if (($msg_start == 1) && ($line =~ / \* message/)) {
> +			$desc_start = 1;
> +			next;
> +		}
> +		if ($line =~ / \*\//) {
> +			$desc_start = 0;
> +			next;
> +		}
> +		if ($desc_start == 1) {
> +			$descriptions{$description_count}->{'DESC'} .= "$line";
> +			next;
> +		}
> +		if ($line =~
> +			/\s*KMSG_DOC\(\s*(.*)\s*\,\s*(.*)\s*\,\s*\"(.*)\"\s*\);/) {
> +			my $param_count = 0;
> +			my $first = 1;
> +			my $type;
> +
> +			$descriptions{$description_count}->{'ID'} = "$1\.$2";
> +			$descriptions{$description_count}->{'COMP'} = "$1";
> +			$descriptions{$description_count}->{'NR'} = "$2";
> +			$descriptions{$description_count}->{'MSG'} = "$3";
> +			foreach my $arg (split("%", $3)) {
> +				if ($first) {
> +					$first = 0;
> +					next;
> +				}
> +				$type = substr($arg, 0, 1);
> +				$descriptions{$description_count}->{'PARM_TYPE'}->{$param_count} = $type;
> +				$param_count += 1;
> +			}
> +			$descriptions{$description_count}->{'PARM_COUNT'} = $param_count;
> +			$description_count += 1;
> +		}
> +	}
> +}
> +
> +sub print_messages()
> +{
> +	for ($i = 0; $i < $message_count; $i++) {
> +		print "MESSAGE: $messages{$i}->{'ID'}\n";
> +	}
> +}
> +
> +sub print_descriptions($)
> +{
> +	my ($message_id)=@_;
> +
> +	for ($i = 0; $i < $description_count; $i++) {
> +		if (($descriptions{$i}->{'ID'} eq $message_id) || $message_id eq "all") {
> +			print "==============================================================================\n";
> +			print "\[$descriptions{$i}->{'COMP'}\.$descriptions{$i}->{'NR'}\] $descriptions{$i}->{'MSG'}\n";
> +			print "\n";
> +			print "Parameters:\n";
> +			for ($j = 0; $j < $descriptions{$i}->{'PARM_COUNT'}; $j++) {
> +				print " $descriptions{$i}->{'PARM_TYPE'}->{$j}: $descriptions{$i}->{'PARM_DESC'}->{$j}\n";
> +			}
> +			print "\n";
> +			print "Description:\n";
> +			print "$descriptions{$i}->{'DESC'}\n";
> +			print "==============================================================================\n";
> +		}
> +	}
> +}
> +
> +sub check_messages($)
> +{
> +	my ($filename)=@_;
> +
> +	for ($i = 0; $i < $message_count; $i++) {
> +		$found = 0;
> +		for ($j = 0; $j < $description_count; $j++) {
> +			if ($messages{$i}->{'ID'} eq $descriptions{$j}->{'ID'}) {
> +				$found = 1;
> +				last;
> +			}
> +		}
> +		if (!$found) {
> +			print STDERR "$filename: Missing description for: $messages{$i}->{'ID'}\n";
> +		}
> +	}
> +	for ($i = 0; $i < $description_count; $i++) {
> +		$found = 0;
> +		for ($j = 0; $j < $message_count; $j++) {
> +			if ($messages{$j}->{'ID'} eq $descriptions{$i}->{'ID'}) {
> +				$found = 1;
> +				last;
> +			}
> +		}
> +		if (!$found) {
> +			print STDERR "$filename: Description without message for: $descriptions{$i}->{'ID'}\n";
> +		}
> +	}
> +}
> +
> +sub print_templates()
> +{
> +
> +	for ($i = 0; $i < $message_count; $i++) {
> +		$found = 0;
> +		for ($j = 0; $j < $description_count; $j++) {
> +			if ($messages{$i}->{'ID'} eq $descriptions{$j}->{'ID'}) {
> +				$found = 1;
> +			}
> +		}
> +		if (!$found) {
> +			print "/**\n";
> +			print " * message\n";
> +			for ($k = 0; $k < $messages{$i}->{'PARM_COUNT'}; $k++) {
> +				print " * \@$k: $messages{$i}->{'PARM_NAME'}->{$k}\n";
> +			}
> +			print " *\n";
> +			print " * Description:\n";
> +			print " *\n";
> +			print " * User Response:\n";
> +			print " */\n";
> +			print "\n";
> +			print "KMSG_DOC($messages{$i}->{'COMP'}, $messages{$i}->{'NR'}, \"$messages{$i}->{'MSG'}\");\n"
> +		}
> +	}
> +}
> +
> +sub usage()
> +{
> +	print "USAGE: kmsg_tool print | check <file>\n";
> +	exit 1;
> +}
> +
> +$option = shift;
> +
> +if ($option eq "check") {
> +	$gcc_options = "-E -D __KMSG_CHECKER ";
> +	do {
> +		$filename = $tmp;
> +		$tmp = shift;
> +		$tmp =~ s/\(/\\\(/;
> +		$tmp =~ s/\)/\\\)/;
> +		$gcc_options .= " $tmp";
> +	} while (!($tmp eq ""));
> +
> +	$gcc_options =~ s/-Wbitwise//; # XXX hack to remove -Wbitwise CHECKFLAG
> +	$gcc_options =~ s/-D__STDC__//; # XXX hack to remove -D__STDC__
> +	$tmp_file = "$filename.msg";
> +	system("gcc $gcc_options > $tmp_file");
> +	get_descriptions($filename);
> +	get_msgs($tmp_file);
> +	check_messages($filename);
> +	print_templates();
> +	system("rm $tmp_file");
> +} elsif ($option eq "print") {
> +	$filename = shift;
> +	do {
> +		print STDERR "Processing: $filename\n";
> +		get_descriptions($filename);
> +		print_descriptions("all");
> +		$filename = shift;
> +	} while (!($filename eq ""));
> +} else {
> +	usage();
> +}
> 

Re: [RFC/PATCH] Documentation of kernel messages

[holzheu]

Hi Greg,

> Ick, why are you ignoring what we have already with dev_printk() and
> friends?  We are just finally getting developers to use that, I think it
> will be almost impossible to get people to change to something else,
> especially one that isn't even as "correct" as what dev_printk() offers
> you today, will be quite hard.
>
> So, why not use what we already have and work off of it?

dev_printk() and friends are great, since they already define something
like KMSG_COMPONENT: The driver name.

But they unfortunately do not solve our problem. We need an identifier
for a documented message in order to find the right description for a
message.

We could add new dev_printk macros for KMSG printks:

#define dev_printk_kmsg(kmsg, dev, format, arg...) \
        printk(kmsg "%s: " format , (dev)->bus_id , ## arg)
#define dev_err_kmsg(dev, nr, format, arg...) \
        dev_printk_kmsg(KMSG_ERR(nr) , dev , format , ## arg)

E.g. hub.c:
===========

#define KMSG_COMPONENT "hub"

dev_err_kmsg(&udev->dev, 1,
       "can't read configurations, error %d\n", err);

Which would result in:

hub.1 5-0:1.0: can't read configuration error 4711

Michael

Re: [RFC/PATCH] Documentation of kernel messages

[David Miller]

I think my general response to something like this, if it goes
in, would be to stop emitting useful kernel log messages
in the code I write because having to document it too on top
of that is just too much extra work to be worthwhile.

Re: [RFC/PATCH] Documentation of kernel messages

[holzheu]

On Wed, 2007-06-13 at 11:23 -0700, David Miller wrote:
> I think my general response to something like this, if it goes
> in, would be to stop emitting useful kernel log messages
> in the code I write because having to document it too on top
> of that is just too much extra work to be worthwhile.

Nobody will force you to document your messages. As I said, there is no
need to document self-explaining printks.

Michael

Re: [RFC/PATCH] Documentation of kernel messages

[Greg KH]

On Wed, Jun 13, 2007 at 08:18:00PM +0200, holzheu wrote:
> Hi Greg,
> 
> > Ick, why are you ignoring what we have already with dev_printk() and
> > friends?  We are just finally getting developers to use that, I think it
> > will be almost impossible to get people to change to something else,
> > especially one that isn't even as "correct" as what dev_printk() offers
> > you today, will be quite hard.
> >
> > So, why not use what we already have and work off of it?
> 
> dev_printk() and friends are great, since they already define something
> like KMSG_COMPONENT: The driver name.

They provide way more than that, they also provide the explicit device
that is being discussed, as well as other things depending on the
device.

So if you are going to do this, please use the already-in-place macros
to hook into, don't try to get the driver authors to pick up something
new and different, as it's going to be _very_ difficult, trust me...

thanks,

greg k-h

Re: [RFC/PATCH] Documentation of kernel messages

[Dave Hansen]

On Wed, 2007-06-13 at 11:32 -0700, Greg KH wrote:
> 
> > dev_printk() and friends are great, since they already define
> something
> > like KMSG_COMPONENT: The driver name.
> 
> They provide way more than that, they also provide the explicit device
> that is being discussed, as well as other things depending on the
> device. 

Instead of keeping a list of kernel message ids somewhere, can you base
this off the dev_printk() format argument?

You could keep a list of those somewhere, extracted from the kernel
source, and correlate documentation with the _format_ instead of an
arbitrarily assigned message number.

-- Dave

Re: [RFC/PATCH] Documentation of kernel messages

[Kok, Auke]

Greg KH wrote:
> On Wed, Jun 13, 2007 at 08:18:00PM +0200, holzheu wrote:
>> Hi Greg,
>>
>>> Ick, why are you ignoring what we have already with dev_printk() and
>>> friends?  We are just finally getting developers to use that, I think it
>>> will be almost impossible to get people to change to something else,
>>> especially one that isn't even as "correct" as what dev_printk() offers
>>> you today, will be quite hard.
>>>
>>> So, why not use what we already have and work off of it?
>> dev_printk() and friends are great, since they already define something
>> like KMSG_COMPONENT: The driver name.
> 
> They provide way more than that, they also provide the explicit device
> that is being discussed, as well as other things depending on the
> device.
> 
> So if you are going to do this, please use the already-in-place macros
> to hook into, don't try to get the driver authors to pick up something
> new and different, as it's going to be _very_ difficult, trust me...

absolutely... I'm currently trying to convince everyone to get a generic 
netdevice-centric printk macro (dev_printk does not print the interface name, 
which the netdevice can't store locally since it can change) in and even that is 
a pain :)

Auke

Re: [RFC/PATCH] Documentation of kernel messages

[Joe Perches]

On Wed, 2007-06-13 at 20:18 +0200, holzheu wrote:
> But they unfortunately do not solve our problem. We need an identifier
> for a documented message in order to find the right description for a
> message.

I believe it better to simply add __FILE__ & __LINE__ to the
macro rather than some other externally specified unique
identifier that adds developer overhead and easily gets stale.

If you go overboard, you could do something like:

#define dev_dbg(dev, format, arg...)		\
	dev_printk(KERN_DEBUG, dev, DBG_FMT format DBG_ARG, ## arg)

#define dev_err(dev, format, arg...)		\
	dev_printk(KERN_ERR, dev, PR_FMT format PR_ARG, ## arg)

etc...

and use config variables to control what additional info is printk'ed.

Go overboard without a life preserver and add __TIMESTAMP__ too.

/*
 */
#if defined DBG_FMT
#undef DBG_FMT
#endif
#if defined DBG_ARG
#undef DBG_ARG
#endif

#if    defined DBG_FILE &&  defined DBG_FUNCTION &&  defined DBG_LINE
#define DBG_FMT "(%s:%s:%u) "
#define DBG_ARG , __FILE__ , __FUNCTION__ , __LINE__ 
#elif  defined DBG_FILE &&  defined DBG_FUNCTION && !defined DBG_LINE
#define DBG_FMT "(%s:%s) "
#define DBG_ARG , __FILE__ , __FUNCTION__ 
#elif  defined DBG_FILE && !defined DBG_FUNCTION &&  defined DBG_LINE
#define DBG_FMT "(%s:%u) "
#define DBG_ARG , __FILE__ , __LINE__ 
#elif  defined DBG_FILE && !defined DBG_FUNCTION && !defined DBG_LINE
#define DBG_FMT "(%s) "
#define DBG_ARG , __FILE__ 
#elif !defined DBG_FILE &&  defined DBG_FUNCTION &&  defined DBG_LINE
#define DBG_FMT "(%s:%u) "
#define DBG_ARG , __FUNCTION__ , __LINE__ 
#elif !defined DBG_FILE &&  defined DBG_FUNCTION && !defined DBG_LINE
#define DBG_FMT "(%s) "
#define DBG_ARG , __FUNCTION__ 
#elif !defined DBG_FILE && !defined DBG_FUNCTION && defined DBG_LINE
#define DBG_FMT "(%u) "
#define DBG_ARG , __LINE__ 
#else
#define DBG_FMT
#define DBG_ARG
#endif

/*
 */

#if defined PR_FMT
#undef PR_FMT
#endif
#if defined PR_ARG
#undef PR_ARG
#endif

#if    defined PR_FILE &&  defined PR_FUNCTION &&  defined PR_LINE
#define PR_FMT "(%s:%s:%u) "
#define PR_ARG , __FILE__ , __FUNCTION__ , __LINE__ 
#elif  defined PR_FILE &&  defined PR_FUNCTION && !defined PR_LINE
#define PR_FMT "(%s:%s) "
#define PR_ARG , __FILE__ , __FUNCTION__ 
#elif  defined PR_FILE && !defined PR_FUNCTION &&  defined PR_LINE
#define PR_FMT "(%s:%u) "
#define PR_ARG , __FILE__ , __LINE__ 
#elif  defined PR_FILE && !defined PR_FUNCTION && !defined PR_LINE
#define PR_FMT "(%s) "
#define PR_ARG , __FILE__ 
#elif !defined PR_FILE &&  defined PR_FUNCTION &&  defined PR_LINE
#define PR_FMT "(%s:%u) "
#define PR_ARG , __FUNCTION__ , __LINE__ 
#elif !defined PR_FILE &&  defined PR_FUNCTION && !defined PR_LINE
#define PR_FMT "(%s) "
#define PR_ARG , __FUNCTION__ 
#elif !defined PR_FILE && !defined PR_FUNCTION && defined PR_LINE
#define PR_FMT "(%u) "
#define PR_ARG , __LINE__ 
#else
#define PR_FMT
#define PR_ARG
#endif

Re: [RFC/PATCH] Documentation of kernel messages

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

On Wed, 13 Jun 2007 12:04:56 PDT, Joe Perches said:

> I believe it better to simply add __FILE__ & __LINE__ to the
> macro rather than some other externally specified unique
> identifier that adds developer overhead and easily gets stale.

There's been plenty of times I've wished for that.  Now if we just found a way
to do something sane for drivers/net/wireless/mac80211/iwlwifi/base.c ;)

Of course, I'm probably atypical - for me, kernel messages come in two forms,
the kind that are immediately obvious at first reading, and the kind that
you end up having to look at the code and wonder WTF? it was doing inside
that never-should-happen-on-this-hardware while loop in the first place... :)


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: [RFC/PATCH] Documentation of kernel messages

[H. Peter Anvin]

Valdis.Kletnieks@vt.edu wrote:
> On Wed, 13 Jun 2007 12:04:56 PDT, Joe Perches said:
> 
>> I believe it better to simply add __FILE__ & __LINE__ to the
>> macro rather than some other externally specified unique
>> identifier that adds developer overhead and easily gets stale.
> 
> There's been plenty of times I've wished for that.  Now if we just found a way
> to do something sane for drivers/net/wireless/mac80211/iwlwifi/base.c ;)
> 

Why don't you just redefine the KERN_* macros which is used by almost
every printk in the system to contain said __FILE__ and __LINE__, then?

	-hpa

Re: [RFC/PATCH] Documentation of kernel messages

[Martin Schwidefsky]

On Wed, 2007-06-13 at 11:32 -0700, Greg KH wrote:
> > > So, why not use what we already have and work off of it?
> > 
> > dev_printk() and friends are great, since they already define
> something
> > like KMSG_COMPONENT: The driver name.
> 
> They provide way more than that, they also provide the explicit device
> that is being discussed, as well as other things depending on the
> device.

dev_printk() and friends provide additional information for a printk
that is related to a device. Not every printk is about a device, so I
think Michaels proposal is orthorgonal to dev_printk. We definitly
should have a dev_printk variant that uses the kmsg documentation tag
AND we should have a normal printk variant as well that uses the kmsg
tagging.

> So if you are going to do this, please use the already-in-place macros
> to hook into, don't try to get the driver authors to pick up something
> new and different, as it's going to be _very_ difficult, trust me...

There is no doubt that it will be difficult to get a larger part of the
developers use the kmsg scheme (e.g. see the reaction of Dave M.). We do
not have the illusion that we can replace every single message in the
kernel, nor do we think that it would make sense to do so. What we would
do for s390 is to check each message in drivers/s390 and think hard if
the message should be 1) removed, 2) replaced with a {dev_}printk_kmsg
or 3) left as it is. Fortunatly for us there are not too many drivers
for s390 ..

-- 
blue skies,
  Martin.

"Reality continues to ruin my life." - Calvin.

Re: [RFC/PATCH] Documentation of kernel messages

[Martin Schwidefsky]

On Wed, 2007-06-13 at 11:15 -0700, Andrew Morton wrote: 
> Your proposal is similar to one I made to some Japanese developers
> earlier this year.

Interesting. Our requirement comes from Japan as well.

>   I was more modest, proposing that we
> 
> - add an enhanced printk
> 
>         xxprintk(msgid, KERN_ERR "some text %d\n", some_number);

Some kind of id is always required to be able to identify the message.
The task is to make the tagging as simple as possible.

> - An externally managed database will provide translations, diagnostics,
>   remedial actions, etc, keyed off the msgid string

There I have my doubts if this can work. If you keep an external
database with the additional information about the messages the kernel
code and the database will get out of sync. Thats why we advocate the
kernel-doc style approach: the message catalogue will always be in sync
because it is delivered with the kernel.

> - Format and management of the msgid hasn't been clearly identified yet
>   as far as I know.  But all we really need is that each ID be unique,
>   kernel-wide.  We develop a simple tool which can check the whole tree
>   for duplicated msgids.

This is a paint point with Michaels approach as well. The KMSG_COMPONENT
defines are rather ugly. A clever way how to generate kernel unique IDs
would be nice. Intervention required..

> - From a practical day-to-day POV what this means is that a person
>   from the kernel-messages team will prepare a patch for your driver
>   which adds the msgids and you the driver author should take care not
>   to break the tagging.
> 
>   This is deliberately designed to be minimum-impact upon general 
>   developers.  We want a system in place where driver/fs/etc developers
>   can concentrate on what they do, and kernel-message developers can
>   as independently as possibe take care of what _they_ do.

With the KMSG approach all that needs to be done is to replace the
KERN_xxx with KMSG_xxx(<number) and to add a comment at the end of the
file.

> - A project has started up and there is a mailing list (cc'ed here) and
>   meetings are happening at the LF collaboration summit this week.
> 
>   Please see https://lists.linux-foundation.org/mailman/listinfo/lf_kernel_messages
>   and don't miss the next conference call ;)
> 
>   (argh.  The lf_kernel_messages archives are subscriber-only and it could be that
>   only members can post to it.  Oh well.  Please subscribe, and review the archives)

Very interesting. We did not know about this project. Thanks for the
hint.

-- 
blue skies,
  Martin.

"Reality continues to ruin my life." - Calvin.

Re: [RFC/PATCH] Documentation of kernel messages

[Krzysztof Halasa]

Hi,

holzheu <holzheu@linux.vnet.ibm.com> writes:

> If the information is to big for the printk itself, because you would
> need 10 lines to explain what happened, wouldn't it be good to have a
> place where to put that information?

I think if a message really has to be 10 lines long to be clear then
it should just be 10 lines long. The keyword is "really".

Example: oops.
-- 
Krzysztof Halasa

Re: [RFC/PATCH] Documentation of kernel messages

[Jan Kara]

  <snip>

> Your proposal is similar to one I made to some Japanese developers
> earlier this year.  I was more modest, proposing that we
> 
> - add an enhanced printk
> 
> 	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
  Maybe a stupid idea but why do we want to assign these numbers by hand?
I can imagine it could introduce collisions when merging tons of patches
with new messages... Wouldn't it be better to compute say, 8-byte hash
from the message and use it as it's identifier? We could do this
automagically at compile time. I know it also has it's problems - you
fix a spelling and the message gets a different id and you have to
update translation/documentation catalogue but maybe that could be
solved too...

								Honza

Re: [RFC/PATCH] Documentation of kernel messages

[holzheu]

On Thu, 2007-06-14 at 11:41 +0200, Jan Kara wrote:
>   <snip>
> 
> > Your proposal is similar to one I made to some Japanese developers
> > earlier this year.  I was more modest, proposing that we
> > 
> > - add an enhanced printk
> > 
> > 	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
>   Maybe a stupid idea but why do we want to assign these numbers by hand?
> I can imagine it could introduce collisions when merging tons of patches
> with new messages... Wouldn't it be better to compute say, 8-byte hash
> from the message and use it as it's identifier? We could do this
> automagically at compile time.

Of course automatically generated message numbers would be great and
something like:

hub.4a5bcd77: Detected some problem.

looks acceptable for me.

We could generate the hash using the format string of the printk. Since
we specify the format string also in KMSG_DOC, the hash for the
KMSG_DOC and the printk should match and we have the required link
between printk and description.

So technically that's probably doable.

Problems are:

* hashes are not unique
* We need an additional preprocessor step
* The might be people, who find 8 character hash values ugly in printks

The big advantage is, that we do not need to maintain message numbers.

> I know it also has it's problems - you
> fix a spelling and the message gets a different id and you have to
> update translation/documentation catalogue but maybe that could be
> solved too...

Since in our approach the message catalog is created automatically for
exactly one kernel and the message catalog belongs therefore to exactly
one kernel, I think the problem of changing error numbers is not too
big.

Michael

Re: [RFC/PATCH] Documentation of kernel messages

[holzheu]

On Thu, 2007-06-14 at 12:38 +0200, holzheu wrote:
> On Thu, 2007-06-14 at 11:41 +0200, Jan Kara wrote:
> >   <snip>
> > 
> > > Your proposal is similar to one I made to some Japanese developers
> > > earlier this year.  I was more modest, proposing that we
> > > 
> > > - add an enhanced printk
> > > 
> > > 	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
> >   Maybe a stupid idea but why do we want to assign these numbers by hand?
> > I can imagine it could introduce collisions when merging tons of patches
> > with new messages... Wouldn't it be better to compute say, 8-byte hash
> > from the message and use it as it's identifier? We could do this
> > automagically at compile time.
> 
> Of course automatically generated message numbers would be great and
> something like:
> 
> hub.4a5bcd77: Detected some problem.

Sorry, I first read: 8 characters not 8 bytes.

Indeed, "hub.d41d8cd98f00b204: Detected some problem." ... does not look
like very beautiful.

But maybe also 4 bytes would be enough, since the hash only has to be
unique within one component e.g. "hub".

Michael

Re: [RFC/PATCH] Documentation of kernel messages

[Jan Kara]

On Thu 14-06-07 13:47:41, holzheu wrote:
> On Thu, 2007-06-14 at 12:38 +0200, holzheu wrote:
> > On Thu, 2007-06-14 at 11:41 +0200, Jan Kara wrote:
> > >   <snip>
> > > 
> > > > Your proposal is similar to one I made to some Japanese developers
> > > > earlier this year.  I was more modest, proposing that we
> > > > 
> > > > - add an enhanced printk
> > > > 
> > > > 	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
> > >   Maybe a stupid idea but why do we want to assign these numbers by hand?
> > > I can imagine it could introduce collisions when merging tons of patches
> > > with new messages... Wouldn't it be better to compute say, 8-byte hash
> > > from the message and use it as it's identifier? We could do this
> > > automagically at compile time.
> > 
> > Of course automatically generated message numbers would be great and
> > something like:
> > 
> > hub.4a5bcd77: Detected some problem.
> 
> Sorry, I first read: 8 characters not 8 bytes.
> 
> Indeed, "hub.d41d8cd98f00b204: Detected some problem." ... does not look
> like very beautiful.
> 
> But maybe also 4 bytes would be enough, since the hash only has to be
> unique within one component e.g. "hub".
  It depends how large components you expect. For example for 10000
messages there is already 1% probability of collision so it means sooner or
later we are going to hit it... For 1000 messages the probability is
roughly 0.1% which is still not so small that I'd be comfortable with it.
On the other hand we could use something like gperf to generate perfect
hashing function and then we don't have to worry about colisions.

									Honza
-- 
Jan Kara <jack@suse.cz>
SuSE CR Labs

Re: [RFC/PATCH] Documentation of kernel messages

[holzheu]

On Thu, 2007-06-14 at 14:26 +0200, Jan Kara wrote:

<snip>

> > But maybe also 4 bytes would be enough, since the hash only has to be
> > unique within one component e.g. "hub".
>   It depends how large components you expect. For example for 10000
> messages there is already 1% probability of collision so it means sooner or
> later we are going to hit it... For 1000 messages the probability is
> roughly 0.1% which is still not so small that I'd be comfortable with it.

I wouldn't expect, that a "normal" component like a device driver will
have more than 100 documented messages.

Collisions are ugly, but will not hurt much. The operator will see for a
message two possible descriptions. Normally it should be possible to
figure out manually which description is the right one.

Michael

Re: [RFC/PATCH] Documentation of kernel messages

[Jan Engelhardt]

On Jun 13 2007 12:50, Valdis.Kletnieks@vt.edu wrote:
>> In general we think, that also for Linux it is a good thing to have
>> documentation for the most important kernel/driver messages. Even
>> kernel hackers not always are aware of the meaning of kernel messages
>> for components, which they don't know in detail. Most of the messages
>> are self explaining but sometimes you get something like "Clocksource
>> tsc unstable (delta = 7304132729 ns)" and you wonder if your system is
>> going to explode.

I suppose no. If it is going to explode, it will tell you :-)

/* fs/super.c */
printk("VFS: Busy inodes after unmount of %s. "
       "Self-destruct in 5 seconds.  Have a nice day...\n",
       sb->s_id);

>This is probably best addressed by cleaning up the actual messages so they're
>a bit more informative.


	Jan
-- 

Re: [RFC/PATCH] Documentation of kernel messages

[Jan Engelhardt]

On Jun 13 2007 12:04, Joe Perches wrote:
>On Wed, 2007-06-13 at 20:18 +0200, holzheu wrote:
>> But they unfortunately do not solve our problem. We need an identifier
>> for a documented message in order to find the right description for a
>> message.
>
>I believe it better to simply add __FILE__ & __LINE__ to the

Erk, no. That bloats the kernel to the max. Especially since kernel
files are compiled with a rather longish path.



	Jan
-- 

Re: [RFC/PATCH] Documentation of kernel messages

[Pavel Machek]

Hi!

> > Your proposal is similar to one I made to some Japanese developers
> > earlier this year.  I was more modest, proposing that we
> > 
> > - add an enhanced printk
> > 
> > 	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
>   Maybe a stupid idea but why do we want to assign these numbers by hand?
> I can imagine it could introduce collisions when merging tons of patches
> with new messages... Wouldn't it be better to compute say, 8-byte hash
> from the message and use it as it's identifier? We could do this

Why hashes? Just key it off full message, and treat it like normal
translation problem.

'lp%d: on fire' -> 'your printer failed basic tests, you should check
cabling'.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [RFC/PATCH] Documentation of kernel messages

[Gerrit Huizenga]

On Thu, 14 Jun 2007 12:38:53 +0200, holzheu wrote:
> On Thu, 2007-06-14 at 11:41 +0200, Jan Kara wrote:
> >   <snip>
> > 
> > > Your proposal is similar to one I made to some Japanese developers
> > > earlier this year.  I was more modest, proposing that we
> > > 
> > > - add an enhanced printk
> > > 
> > > 	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
> >   Maybe a stupid idea but why do we want to assign these numbers by hand?
> > I can imagine it could introduce collisions when merging tons of patches
> > with new messages... Wouldn't it be better to compute say, 8-byte hash
> > from the message and use it as it's identifier? We could do this
> > automagically at compile time.
> 
> Of course automatically generated message numbers would be great and
> something like:
> 
> hub.4a5bcd77: Detected some problem.
> 
> looks acceptable for me.
> 
> We could generate the hash using the format string of the printk. Since
> we specify the format string also in KMSG_DOC, the hash for the
> KMSG_DOC and the printk should match and we have the required link
> between printk and description.
> 
> So technically that's probably doable.
> 
> Problems are:
> 
> * hashes are not unique
> * We need an additional preprocessor step
> * The might be people, who find 8 character hash values ugly in printks
> 
> The big advantage is, that we do not need to maintain message numbers.
> 
> > I know it also has it's problems - you
> > fix a spelling and the message gets a different id and you have to
> > update translation/documentation catalogue but maybe that could be
> > solved too...
> 
> Since in our approach the message catalog is created automatically for
> exactly one kernel and the message catalog belongs therefore to exactly
> one kernel, I think the problem of changing error numbers is not too
> big.

We just had a meeting with the Japanese and several other participants
from the vendor and community side and came up with a potential proposal
that is similar to many things discussed here.  It has the benefit that
it seems implementable and low/no overhead on most kernel developers.

The basic proposal is to use a tool, run by the kernel Makefile to
extract kernel messages from either the kernel source or the .i files
(both have advantages, although I prefer the source to the .i file
since it 1) gets all messages and 2) is probably a little quicker with
less impact to the standard kernel make.

These messages would be stored in a file in the source tree, e.g.
usr/src/linux/Translations/English.  As each message is added to that
file, we calculate, say, an MD5 sum of the printk (dev_printk, sdev_printk,
etc.) string, and the text file ultimately contains:

MD5 Checksum of text; the printk text itself, the File name, the line number.

The checksum is run over just the printk.  We definitely would not include
the line number since the line number is too volatile.  Including the
file name in the hash *might* help disambiguate the hash a bit better in
the case of duplicates, but there was some debate that duplicates might
be better handled in other ways.

Andrew mentioned a mechanism for adding a subsystem tag or other tag
which helps disambiguate the message, either in the message file or in
the end user documentation (e.g. the Message Pedia/mPedia that the Japanese
have already created with ~350 messages, and a total of ~700 targetted
by the end of the year).

That tag could be appended to the beginning of the printk, to the end of
the printk, or even in a formatted comment at the end of the printk that
the tool could extract.

Then, the translations could be managed by anyone outside of the normal/
core kernel community, by simply creating a translation file, e.g.
usr/src/linux/Translations/Japanese, which contained the MD5 sum, the
translated message, the file name and line number (the last two redundent
perhaps but informational, and automatically generated if possible).

The files in the Translations directory could be uesd as the unique
keys for an external database (such as the Message Pedia, vendors or
distributions help pages, etc.) to help look up and explain root cause
of a problem.  The key property here is that the MD5 sum becomes the
key to all database entries to look up that key.

Further, yet another kernel config option could allow distros to output
the calculated MD5 sum to be printed, much like we do with timestamps
today.

End result is that these in-kernel message catalogs for translation are
updated automatically (mostly no kernel developer changes needed) and
the translations can be maintained by anyone who is interested.

On the topic of MD5 collisions, using a disambiguating tag would be a
simple addition for the few cases where that happens, the tool could
be educated to use that tag in the calculation of the MD5 sum, and we
have a 98% solution which impacts <1% of the kernel developers.

Folks present for this discussion included Ted T'so, James Bottomley,
several of the key Japenese folks interested in using this for debugging,
and reps from several vendors who would find this sort of info useful
for their folks supporting Linux in the field.

Comments?

gerrit

Re: [RFC/PATCH] Documentation of kernel messages

[Randy Dunlap]

Gerrit Huizenga wrote:
> On Thu, 14 Jun 2007 12:38:53 +0200, holzheu wrote:
>> On Thu, 2007-06-14 at 11:41 +0200, Jan Kara wrote:
>>>   <snip>
>>>
>>>> Your proposal is similar to one I made to some Japanese developers
>>>> earlier this year.  I was more modest, proposing that we
>>>>
>>>> - add an enhanced printk
>>>>
>>>> 	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
>>>   Maybe a stupid idea but why do we want to assign these numbers by hand?
>>> I can imagine it could introduce collisions when merging tons of patches
>>> with new messages... Wouldn't it be better to compute say, 8-byte hash
>>> from the message and use it as it's identifier? We could do this
>>> automagically at compile time.
>> Of course automatically generated message numbers would be great and
>> something like:
>>
>> hub.4a5bcd77: Detected some problem.
>>
>> looks acceptable for me.
>>
>> We could generate the hash using the format string of the printk. Since
>> we specify the format string also in KMSG_DOC, the hash for the
>> KMSG_DOC and the printk should match and we have the required link
>> between printk and description.
>>
>> So technically that's probably doable.
>>
>> Problems are:
>>
>> * hashes are not unique
>> * We need an additional preprocessor step
>> * The might be people, who find 8 character hash values ugly in printks
>>
>> The big advantage is, that we do not need to maintain message numbers.
>>
>>> I know it also has it's problems - you
>>> fix a spelling and the message gets a different id and you have to
>>> update translation/documentation catalogue but maybe that could be
>>> solved too...
>> Since in our approach the message catalog is created automatically for
>> exactly one kernel and the message catalog belongs therefore to exactly
>> one kernel, I think the problem of changing error numbers is not too
>> big.
> 
> We just had a meeting with the Japanese and several other participants
> from the vendor and community side and came up with a potential proposal
> that is similar to many things discussed here.  It has the benefit that
> it seems implementable and low/no overhead on most kernel developers.
> 
> The basic proposal is to use a tool, run by the kernel Makefile to
> extract kernel messages from either the kernel source or the .i files
> (both have advantages, although I prefer the source to the .i file
> since it 1) gets all messages and 2) is probably a little quicker with
> less impact to the standard kernel make.
> 
> These messages would be stored in a file in the source tree, e.g.
> usr/src/linux/Translations/English.  As each message is added to that
> file, we calculate, say, an MD5 sum of the printk (dev_printk, sdev_printk,
> etc.) string, and the text file ultimately contains:
> 
> MD5 Checksum of text; the printk text itself, the File name, the line number.
> 
> The checksum is run over just the printk.  We definitely would not include
> the line number since the line number is too volatile.  Including the
> file name in the hash *might* help disambiguate the hash a bit better in
> the case of duplicates, but there was some debate that duplicates might
> be better handled in other ways.
> 
> Andrew mentioned a mechanism for adding a subsystem tag or other tag
> which helps disambiguate the message, either in the message file or in
> the end user documentation (e.g. the Message Pedia/mPedia that the Japanese
> have already created with ~350 messages, and a total of ~700 targetted
> by the end of the year).
> 
> That tag could be appended to the beginning of the printk, to the end of
> the printk, or even in a formatted comment at the end of the printk that
> the tool could extract.
> 
> Then, the translations could be managed by anyone outside of the normal/
> core kernel community, by simply creating a translation file, e.g.
> usr/src/linux/Translations/Japanese, which contained the MD5 sum, the
> translated message, the file name and line number (the last two redundent
> perhaps but informational, and automatically generated if possible).
> 
> The files in the Translations directory could be uesd as the unique
> keys for an external database (such as the Message Pedia, vendors or
> distributions help pages, etc.) to help look up and explain root cause
> of a problem.  The key property here is that the MD5 sum becomes the
> key to all database entries to look up that key.
> 
> Further, yet another kernel config option could allow distros to output
> the calculated MD5 sum to be printed, much like we do with timestamps
> today.
> 
> End result is that these in-kernel message catalogs for translation are
> updated automatically (mostly no kernel developer changes needed) and
> the translations can be maintained by anyone who is interested.
> 
> On the topic of MD5 collisions, using a disambiguating tag would be a
> simple addition for the few cases where that happens, the tool could
> be educated to use that tag in the calculation of the MD5 sum, and we
> have a 98% solution which impacts <1% of the kernel developers.
> 
> Folks present for this discussion included Ted T'so, James Bottomley,
> several of the key Japenese folks interested in using this for debugging,
> and reps from several vendors who would find this sort of info useful
> for their folks supporting Linux in the field.
> 
> Comments?

For those of us who have not been in on these meetings, I think that
some serious justifications are needed.  The last paragraph began to
go in that direction, but it needs to be more detailed and convincing.
And "for debugging" doesn't cut it IMO.

-- 
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

Re: [RFC/PATCH] Documentation of kernel messages

[Gerrit Huizenga]

On Fri, 15 Jun 2007 11:51:51 PDT, Randy Dunlap wrote:
> 
> For those of us who have not been in on these meetings, I think that
> some serious justifications are needed.  The last paragraph began to
> go in that direction, but it needs to be more detailed and convincing.
> And "for debugging" doesn't cut it IMO.
> 
> -- 
> ~Randy

Fair point, Randy.

This came up this time because people in Japan supporting Linux customers
are not always familiar with Linux kernel internals and thus debugging
problems based on a terse, English language message is just painful.  Also,
any given message doesn't really provide any input about what *really*
failed, what should be done about it, how to correct this issue, or what
parts to pull out of your machine and replace with properly working parts.

So, the Message Pedia is basically a Japanese language wiki today where
people doing support can annotate the messages with real life experience on
what to *do* about the error.

Part of the problem is that the error messages when extracted don't
stand alone and don't easily allow tracing back to the real source.  When
the problem is handed from the second line support folks deeper into the
support or development organization, the message has to be found in the
source tree in some non-ambiguous fashion.

The other problem that comes up in other, non-English locales is that
debugging your Linux box when you are a non-native English speaker is
just plain impossible.  Providing a means for getting localized kernel
messages allows more end users to at least reasonably debug their own
machine.  Most every user level project today allows localization directly
but the kernel has never had any reasonable mechanism for localisation.

Now, the proposal here doesn't make the kernel "as good as" user level
application, but it would at least provide a searchable key to help
people get localized guidance on problems.

I'm sure there are other benefits that most can see based on just simple
debuggability.  But the Japanese goal is really to construct a database
of debugging info & context based on their experiences.

And, I'm sure that others can speak for themselves.  I'm mostly playing
scribe on this one - we abandoned our error and event subsystem work years
ago, even though this was one of the biggest weaknesses we saw in customer
support 3-5 years ago, simply because our changes were still viewed as
too invasive for mainline kernel.  These proposed changes help a lot, while
remaining almost completely invisible to most developers.

gerrit

Re: [RFC/PATCH] Documentation of kernel messages

[Greg KH]

On Fri, Jun 15, 2007 at 11:51:51AM -0700, Randy Dunlap wrote:
>  Gerrit Huizenga wrote:
> > On Thu, 14 Jun 2007 12:38:53 +0200, holzheu wrote:
> >> On Thu, 2007-06-14 at 11:41 +0200, Jan Kara wrote:
> >>>   <snip>
> >>>
> >>>> Your proposal is similar to one I made to some Japanese developers
> >>>> earlier this year.  I was more modest, proposing that we
> >>>>
> >>>> - add an enhanced printk
> >>>>
> >>>> 	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
> >>>   Maybe a stupid idea but why do we want to assign these numbers by hand?
> >>> I can imagine it could introduce collisions when merging tons of patches
> >>> with new messages... Wouldn't it be better to compute say, 8-byte hash
> >>> from the message and use it as it's identifier? We could do this
> >>> automagically at compile time.
> >> Of course automatically generated message numbers would be great and
> >> something like:
> >>
> >> hub.4a5bcd77: Detected some problem.
> >>
> >> looks acceptable for me.
> >>
> >> We could generate the hash using the format string of the printk. Since
> >> we specify the format string also in KMSG_DOC, the hash for the
> >> KMSG_DOC and the printk should match and we have the required link
> >> between printk and description.
> >>
> >> So technically that's probably doable.
> >>
> >> Problems are:
> >>
> >> * hashes are not unique
> >> * We need an additional preprocessor step
> >> * The might be people, who find 8 character hash values ugly in printks
> >>
> >> The big advantage is, that we do not need to maintain message numbers.
> >>
> >>> I know it also has it's problems - you
> >>> fix a spelling and the message gets a different id and you have to
> >>> update translation/documentation catalogue but maybe that could be
> >>> solved too...
> >> Since in our approach the message catalog is created automatically for
> >> exactly one kernel and the message catalog belongs therefore to exactly
> >> one kernel, I think the problem of changing error numbers is not too
> >> big.
> > We just had a meeting with the Japanese and several other participants
> > from the vendor and community side and came up with a potential proposal
> > that is similar to many things discussed here.  It has the benefit that
> > it seems implementable and low/no overhead on most kernel developers.
> > The basic proposal is to use a tool, run by the kernel Makefile to
> > extract kernel messages from either the kernel source or the .i files
> > (both have advantages, although I prefer the source to the .i file
> > since it 1) gets all messages and 2) is probably a little quicker with
> > less impact to the standard kernel make.
> > These messages would be stored in a file in the source tree, e.g.
> > usr/src/linux/Translations/English.  As each message is added to that
> > file, we calculate, say, an MD5 sum of the printk (dev_printk, sdev_printk,
> > etc.) string, and the text file ultimately contains:
> > MD5 Checksum of text; the printk text itself, the File name, the line 
> > number.
> > The checksum is run over just the printk.  We definitely would not include
> > the line number since the line number is too volatile.  Including the
> > file name in the hash *might* help disambiguate the hash a bit better in
> > the case of duplicates, but there was some debate that duplicates might
> > be better handled in other ways.
> > Andrew mentioned a mechanism for adding a subsystem tag or other tag
> > which helps disambiguate the message, either in the message file or in
> > the end user documentation (e.g. the Message Pedia/mPedia that the Japanese
> > have already created with ~350 messages, and a total of ~700 targetted
> > by the end of the year).
> > That tag could be appended to the beginning of the printk, to the end of
> > the printk, or even in a formatted comment at the end of the printk that
> > the tool could extract.
> > Then, the translations could be managed by anyone outside of the normal/
> > core kernel community, by simply creating a translation file, e.g.
> > usr/src/linux/Translations/Japanese, which contained the MD5 sum, the
> > translated message, the file name and line number (the last two redundent
> > perhaps but informational, and automatically generated if possible).
> > The files in the Translations directory could be uesd as the unique
> > keys for an external database (such as the Message Pedia, vendors or
> > distributions help pages, etc.) to help look up and explain root cause
> > of a problem.  The key property here is that the MD5 sum becomes the
> > key to all database entries to look up that key.
> > Further, yet another kernel config option could allow distros to output
> > the calculated MD5 sum to be printed, much like we do with timestamps
> > today.
> > End result is that these in-kernel message catalogs for translation are
> > updated automatically (mostly no kernel developer changes needed) and
> > the translations can be maintained by anyone who is interested.
> > On the topic of MD5 collisions, using a disambiguating tag would be a
> > simple addition for the few cases where that happens, the tool could
> > be educated to use that tag in the calculation of the MD5 sum, and we
> > have a 98% solution which impacts <1% of the kernel developers.
> > Folks present for this discussion included Ted T'so, James Bottomley,
> > several of the key Japenese folks interested in using this for debugging,
> > and reps from several vendors who would find this sort of info useful
> > for their folks supporting Linux in the field.
> > Comments?
> 
>  For those of us who have not been in on these meetings, I think that
>  some serious justifications are needed.  The last paragraph began to
>  go in that direction, but it needs to be more detailed and convincing.
>  And "for debugging" doesn't cut it IMO.

Full "justification" has never really been clear outside of the need to
try to duplicate what other Unixes have had for their support
departments in the past.  And yes, I've seen and sat through the
presentation by the Japanese group many different times, and talked to
the individual people who want this.

And you know, that's fine.  If anyone wants to go off and do such a
project like this, in a way that is almost entirely self-contained with
no affect on the development process or any kernel developer, I have no
objection to it at all.

And I think the proposal that Andrew made meets that requirement, so
let's wait and see what actual code comes out of this and judge it on
that merit at that time.

thanks,

greg k-h

Re: [RFC/PATCH] Documentation of kernel messages

[holzheu]

Hi Gerrit,

The common thing of your and our approach is, that we need an ID to
identify a message either by:

* Using hashes + component name (maybe)
* Or using hand-made message numbers + component name. Numbers have
  to be unique within the kernel component

I think the main difference of your approach is, that documentation will
be maintained outside the kernel. This has advantages, but also
disadvantages:

(+) Developers have less work to do :-)
(+) Kernel sources and patches have less lines of code, since message
    descriptions are not contained in the kernel sources.
(+) You can support multiple languages

(-) Hard to keep printks and descriptions up-to-date. If using hashes,
    each typo fix in a printk breaks the link to your documentation.
(-) Since the developer knows the meaning of a printk best, he probably
    would be the right person to write the description.
(-) For every Distribution or vanilla kernel you probably need separate
    external message databases.

I like the fact, that every printk gets an ID in your approach and you
do not need additional printk macros.

Maybe both approaches are not mutually exclusive. If developers would
like to document their messages within the kernel sources, they still
could use kernel-doc and our KMSG_DOC macro do do that.

I admit, that I am not sure about the the best solution here.

Michael

Re: [RFC/PATCH] Documentation of kernel messages

[Sam Ravnborg]

On Fri, Jun 15, 2007 at 11:42:15AM -0700, Gerrit Huizenga wrote:
> 
> Andrew mentioned a mechanism for adding a subsystem tag or other tag
> which helps disambiguate the message, either in the message file or in
> the end user documentation (e.g. the Message Pedia/mPedia that the Japanese
> have already created with ~350 messages, and a total of ~700 targetted
> by the end of the year).
> 
> That tag could be appended to the beginning of the printk, to the end of
> the printk, or even in a formatted comment at the end of the printk that
> the tool could extract.

The best way to maintain such a subsystem tag most be supported by
kbuild. So say we have the subsystem "scsi" then we can set this subsystem
tag in drivers/scsi/Makefile and let drivers/scsi/arm/Makefile inherit
this vaule doing nothing.

Addign the subsystem tag to the printk could than be done using the
C preprocessor and we are all fine.

If we go by the kernel-doc way to document kernel messages (the few that
actually needs additional explanation) then we need to teach kernel-doc
to collect the same subsystem tag but that should be doable.

	Sam

Re: [RFC/PATCH] Documentation of kernel messages

[Arjan van de Ven]

On Mon, 2007-06-18 at 14:55 +0200, holzheu wrote:
> Hi Gerrit,
> 
> The common thing of your and our approach is, that we need an ID to
> identify a message either by:


Maybe I am missing something big, but why is an ID needed?
The message IS the ID right? That's the only thing that is robust
against code moving about....

(And yes I am aware that very occasionally there might be a duplicate
message that REALLY is different, but I suspect that that is so
incredibly rare that it's just simpler to reword one of the two in such
case)

Re: [RFC/PATCH] Documentation of kernel messages

[Jan Kara]

On Mon 18-06-07 06:12:54, Arjan van de Ven wrote:
> On Mon, 2007-06-18 at 14:55 +0200, holzheu wrote:
> > Hi Gerrit,
> > 
> > The common thing of your and our approach is, that we need an ID to
> > identify a message either by:
> 
> 
> Maybe I am missing something big, but why is an ID needed?
> The message IS the ID right? That's the only thing that is robust
> against code moving about....
  I think the problem is messages contain device numbers, and all similar
kind of data. So what you'd really like is to use >format string< as ID.
But that is not easily accessible for the user, so you'd have to print it
with each message. And that's ... ugly IMHO.

> (And yes I am aware that very occasionally there might be a duplicate
> message that REALLY is different, but I suspect that that is so
> incredibly rare that it's just simpler to reword one of the two in such
> case)

									Honza
-- 
Jan Kara <jack@suse.cz>
SuSE CR Labs

Re: [RFC/PATCH] Documentation of kernel messages

[holzheu]

Hi Pavel,

On Fri, 2007-06-15 at 12:40 +0000, Pavel Machek wrote:
> Hi!
> 
> > > Your proposal is similar to one I made to some Japanese developers
> > > earlier this year.  I was more modest, proposing that we
> > > 
> > > - add an enhanced printk
> > > 
> > > 	xxprintk(msgid, KERN_ERR "some text %d\n", some_number);
> >   Maybe a stupid idea but why do we want to assign these numbers by hand?
> > I can imagine it could introduce collisions when merging tons of patches
> > with new messages... Wouldn't it be better to compute say, 8-byte hash
> > from the message and use it as it's identifier? We could do this
> 
> Why hashes? Just key it off full message, and treat it like normal
> translation problem.
> 
> 'lp%d: on fire' -> 'your printer failed basic tests, you should check
> cabling'.

Hmmm... What about extracting all format strings from the KMSG_DOC
macros and storing them in a file. E.g.:

  printk("lp%d: on fire");

  /**
   * message
   *
   * Description:
   * Your printer failed basic tests, you should check cabling
   ...

  KMSG_DOC(lp, "lp%d: on fire");

Some build mechanism creates a file with all format strings of the
KMSG_DOC macros:

E.g.: cat kmsgs.txt

# MSG:       "lp%d: on fire"
# COMPONENT: lp
# DESC:      Your printer failed basic tests, you should check cabling

Then we provide a tool e.g. "kmsg", which gets as input the actual
kernel message and searches the message database for matching format
strings:

>> kmsg "lp0: on fire"
>> Your printer ...

Of course, we can get duplicate hits, too.

The advantage of that approach is, that we don't need additional message
identifiers like hashes.

Michael
 

Re: [RFC/PATCH] Documentation of kernel messages

[holzheu]

On Mon, 2007-06-18 at 06:12 -0700, Arjan van de Ven wrote:
> On Mon, 2007-06-18 at 14:55 +0200, holzheu wrote:
> > Hi Gerrit,
> > 
> > The common thing of your and our approach is, that we need an ID to
> > identify a message either by:
> 
> 
> Maybe I am missing something big, but why is an ID needed?
> The message IS the ID right? That's the only thing that is robust
> against code moving about....

Yes. As already discussed with Pavel, it is one option to use the format
string of the message as message ID. The disadvantage compared to
message IDs like hashes is, that format strings might be even less
unique than hashes and it's probably less convenient for searching by
operators.

An operator has to issue either:

>> info lp.4711
or
>> info "lp0: on fire"

Michael

Re: [RFC/PATCH] Documentation of kernel messages

[Pavel Machek]

Hi!

> > 'lp%d: on fire' -> 'your printer failed basic tests, you should check
> > cabling'.
> 
> Hmmm... What about extracting all format strings from the KMSG_DOC
> macros and storing them in a file. E.g.:
> 
>   printk("lp%d: on fire");
> 
>   /**
>    * message
>    *
>    * Description:
>    * Your printer failed basic tests, you should check cabling
>    ...
> 
>   KMSG_DOC(lp, "lp%d: on fire");

Yes, that was the idea. But we have already tools and standard format
for this (gettext), we should just use that.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [RFC/PATCH] Documentation of kernel messages

[Randy Dunlap]

On Mon, 18 Jun 2007 15:11:57 +0200 Sam Ravnborg wrote:

> On Fri, Jun 15, 2007 at 11:42:15AM -0700, Gerrit Huizenga wrote:
> > 
> > Andrew mentioned a mechanism for adding a subsystem tag or other tag
> > which helps disambiguate the message, either in the message file or in
> > the end user documentation (e.g. the Message Pedia/mPedia that the Japanese
> > have already created with ~350 messages, and a total of ~700 targetted
> > by the end of the year).
> > 
> > That tag could be appended to the beginning of the printk, to the end of
> > the printk, or even in a formatted comment at the end of the printk that
> > the tool could extract.
> 
> The best way to maintain such a subsystem tag most be supported by
> kbuild. So say we have the subsystem "scsi" then we can set this subsystem
> tag in drivers/scsi/Makefile and let drivers/scsi/arm/Makefile inherit
> this vaule doing nothing.
> 
> Addign the subsystem tag to the printk could than be done using the
> C preprocessor and we are all fine.
> 
> If we go by the kernel-doc way to document kernel messages (the few that
> actually needs additional explanation) then we need to teach kernel-doc
> to collect the same subsystem tag but that should be doable.

Actually I would prefer that such a tool be a clone of
scripts/kernel-doc and use a (slightly?) modified tag instead of
the kernel-doc "/**" tag if this route is used at all.


---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

Re: [RFC/PATCH] Documentation of kernel messages

[Tim Bird]

Gerrit Huizenga wrote:
> Further, yet another kernel config option could allow distros to output
> the calculated MD5 sum to be printed, much like we do with timestamps
> today.

> Comments?

Would the compiled-in text then also become replaceable?
Or is the MD5 sum output expected to be in addition to
the regular English message?

If message replacement at compile-time is supported, this
could allow for creating "short" versions of the messages,
which could have a beneficial impact on kernel size.

Right now, it is possible to completely disable printks
and re-claim about 100K of memory.  However, in some
embedded configurations, even if you are space-constrained
it's desirable to retain some of the printks, for in-field
debugging.  Thus not very many embedded developers
disable printks completely, even though the option has
been supported for a while.  (That, and many aren't caught
up to the kernel version where it was introduced (2.6.10) :-)

But compressed messages (shortened text through abbreviations,
or just outputting the ID alone, etc.) could save SOME
of the space, in trade for less readability.  Heck, just
removing all vowels would probably save 10k, and not
hurt readability that much.

Finally, for testing, it's handy to also
have automatic translation generators.
At a former company I worked for, they had translators
that would output:
 * all messages shortened by 20%
 * all messages lengthened by 20%
 * every message converted to pig-latin

These were mostly used for testing if the strings broke
screen real-estate constraints (which don't apply to
kernel messages).  But the automatic translators
would sometimes catch messages with weird attributes.

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: [RFC/PATCH] Documentation of kernel messages

[Arjan van de Ven]

On Mon, 2007-06-18 at 15:53 +0200, holzheu wrote:
> On Mon, 2007-06-18 at 06:12 -0700, Arjan van de Ven wrote:
> > On Mon, 2007-06-18 at 14:55 +0200, holzheu wrote:
> > > Hi Gerrit,
> > > 
> > > The common thing of your and our approach is, that we need an ID to
> > > identify a message either by:
> > 
> > 
> > Maybe I am missing something big, but why is an ID needed?
> > The message IS the ID right? That's the only thing that is robust
> > against code moving about....
> 
> Yes. As already discussed with Pavel, it is one option to use the format
> string of the message as message ID. The disadvantage compared to
> message IDs like hashes is, that format strings might be even less
> unique than hashes 

if the hash comes from the string in the first place I have a hard time
believing that.

> and it's probably less convenient for searching by
> operators.

I'm not convinced of that...

> 
> An operator has to issue either:
> 
> >> info lp.4711
> or
> >> info "lp0: on fire"

well.... surely the messages are caught by some userspace program,
right? (like syslog).. that can do the lookup and make it all
conveniently lookup-able and cross-referencable etc etc....

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: [RFC/PATCH] Documentation of kernel messages

[Gerrit Huizenga]

On Mon, 18 Jun 2007 17:13:19 PDT, Tim Bird wrote:
> Gerrit Huizenga wrote:
> > Further, yet another kernel config option could allow distros to output
> > the calculated MD5 sum to be printed, much like we do with timestamps
> > today.
> 
> > Comments?
> 
> Would the compiled-in text then also become replaceable?
> Or is the MD5 sum output expected to be in addition to
> the regular English message?
 
 The MD5 sum would be calculated at print time, no proposal in
 here to replace the in-kernel text.  And, I'm not sure that replacing
 it with an MD5 sum would every be significantly shorter, ergo
 I don't think this helps your problem.

 The methods of post-processing to "shrink" the kernel text here
 seem too ugly to ponder.  And I just pondered a few that made my
 head hurt (sort the MD5 sums, re-insert the number of the MD5 sum
 as an integer instead of the original text, and, beyond being gross,
 what do you do with the formatting info about the args?).

> If message replacement at compile-time is supported, this
> could allow for creating "short" versions of the messages,
> which could have a beneficial impact on kernel size.
> 
> Right now, it is possible to completely disable printks
> and re-claim about 100K of memory.  However, in some
> embedded configurations, even if you are space-constrained
> it's desirable to retain some of the printks, for in-field
> debugging.  Thus not very many embedded developers
> disable printks completely, even though the option has
> been supported for a while.  (That, and many aren't caught
> up to the kernel version where it was introduced (2.6.10) :-)

 Which ones matter?  Odds are you could use the KERNEL_LOGLEVEL or
 such to mark those, then compile out the rest.

> But compressed messages (shortened text through abbreviations,
> or just outputting the ID alone, etc.) could save SOME
> of the space, in trade for less readability.  Heck, just
> removing all vowels would probably save 10k, and not
> hurt readability that much.
 
 Ick.  Are you serious?  How about running them through a valley
 girl filter and then converting to high school text messaging?

> Finally, for testing, it's handy to also
> have automatic translation generators.
> At a former company I worked for, they had translators
> that would output:
>  * all messages shortened by 20%
>  * all messages lengthened by 20%
>  * every message converted to pig-latin

 Double yucko.

> These were mostly used for testing if the strings broke
> screen real-estate constraints (which don't apply to
> kernel messages).  But the automatic translators
> would sometimes catch messages with weird attributes.
 
  I don't think people are worried about the correctness of
  the messages and message formats - much of that can actually
  be detected by simple tools.  The goal here was to at least
  allow people that support operating systems and for whom
  English (and English collation sequences) is not a primary
  language do some initial system diagosis.

  I think compiling out the messages of something below some LOGLEVEL
  is a lot more practical.

gerrit

Re: [Lf_kernel_messages] Re: [RFC/PATCH] Documentation of kernel messages

[Kunai, Takashi]

Hi Michael,

I initiated this discussion in The Linux Foundation Summit@Google Campus
last week from the viewpoint of requirements by Japanese vendors/users.

As for a file to store message descriptions, I perfectly agree with your
analysis and we also noticed some (+) and (-) you pointed out in the
meeting last week. Our conclusion was to have a file outside of the
kernel source codes to lessen developers burdens and to encourage better
documentations of descriptions (as Andrew mentioned in his initial message).

Takashi Kunai,
The Linux Foundation Japan

holzheu wrote:
> Hi Gerrit,
> 
> The common thing of your and our approach is, that we need an ID to
> identify a message either by:
> 
> * Using hashes + component name (maybe)
> * Or using hand-made message numbers + component name. Numbers have
>   to be unique within the kernel component
> 
> I think the main difference of your approach is, that documentation will
> be maintained outside the kernel. This has advantages, but also
> disadvantages:
> 
> (+) Developers have less work to do :-)
> (+) Kernel sources and patches have less lines of code, since message
>     descriptions are not contained in the kernel sources.
> (+) You can support multiple languages
> 
> (-) Hard to keep printks and descriptions up-to-date. If using hashes,
>     each typo fix in a printk breaks the link to your documentation.
> (-) Since the developer knows the meaning of a printk best, he probably
>     would be the right person to write the description.
> (-) For every Distribution or vanilla kernel you probably need separate
>     external message databases.
> 
> I like the fact, that every printk gets an ID in your approach and you
> do not need additional printk macros.
> 
> Maybe both approaches are not mutually exclusive. If developers would
> like to document their messages within the kernel sources, they still
> could use kernel-doc and our KMSG_DOC macro do do that.
> 
> I admit, that I am not sure about the the best solution here.
> 
> Michael
> 
> 
> _______________________________________________
> Lf_kernel_messages mailing list
> Lf_kernel_messages@lists.linux-foundation.org
> https://lists.linux-foundation.org/mailman/listinfo/lf_kernel_messages
> 
> 


-- 
Kunai, Takashi
The Linux Foundation Japan (New since '07/2/5)
Shibuya Mark City West 22nd Floor
1-12-1 Dogenzaka,Shibuya-ku,Tokyo,Japan
zip150-0043 tel+81-3-4360-5493

Re: [RFC/PATCH] Documentation of kernel messages

[holzheu]

On Mon, 2007-06-18 at 18:36 -0700, Arjan van de Ven wrote:
> On Mon, 2007-06-18 at 15:53 +0200, holzheu wrote:
> > On Mon, 2007-06-18 at 06:12 -0700, Arjan van de Ven wrote:
> > > On Mon, 2007-06-18 at 14:55 +0200, holzheu wrote:
> > > > Hi Gerrit,
> > > > 
> > > > The common thing of your and our approach is, that we need an ID to
> > > > identify a message either by:
> > > 
> > > 
> > > Maybe I am missing something big, but why is an ID needed?
> > > The message IS the ID right? That's the only thing that is robust
> > > against code moving about....
> > 
> > Yes. As already discussed with Pavel, it is one option to use the format
> > string of the message as message ID. The disadvantage compared to
> > message IDs like hashes is, that format strings might be even less
> > unique than hashes 
> 
> if the hash comes from the string in the first place I have a hard time
> believing that.
> 

Just think of all messages containing %s, which can expand to every
possible string.

Michael

Re: [RFC/PATCH] Documentation of kernel messages

[holzheu]

On Mon, 2007-06-18 at 18:36 -0700, Arjan van de Ven wrote:

<snip>

> well.... surely the messages are caught by some userspace program,
> right? (like syslog).. that can do the lookup and make it all
> conveniently lookup-able and cross-referencable etc etc....

Ok, I agree. Maybe that's really a good idea!

So what about the following proposal:

* Use printk format strings as message IDs
* Use kernel-doc (or similar mechanism) for message documentation
* Use Subsystem tag/component ID for each printk. This might be
  optional, but would make sense to get messages more unique.
* Implement a new tool (kmsg) to allow operators
  to access kernel message descriptions in a convenient way.

The kmsg tool will read the dmsg buffer together with the list of
documented printks, which has been created in the kernel build using
the format strings defined in the KMSG_DOC macros. It matches the known
format strings to the dmesg messages and informs the operator, for
which message he can get a description.

Something like:

>> kmsg
Linux version 2.6.9-42.0.3.ELsmp...
....
[<ID>] lp0 on fire

>> kmsg <ID>
Desciption: your printer failed basic tests, you should check cabling

The <ID> is generated by the kmsg tool and could be e.g. the hashes for
the matching format strings.

Michael

Re: [RFC/PATCH] Documentation of kernel messages

[Arjan van de Ven]

On Tue, 2007-06-19 at 10:51 +0200, holzheu wrote:
> On Mon, 2007-06-18 at 18:36 -0700, Arjan van de Ven wrote:
> > On Mon, 2007-06-18 at 15:53 +0200, holzheu wrote:
> > > On Mon, 2007-06-18 at 06:12 -0700, Arjan van de Ven wrote:
> > > > On Mon, 2007-06-18 at 14:55 +0200, holzheu wrote:
> > > > > Hi Gerrit,
> > > > > 
> > > > > The common thing of your and our approach is, that we need an ID to
> > > > > identify a message either by:
> > > > 
> > > > 
> > > > Maybe I am missing something big, but why is an ID needed?
> > > > The message IS the ID right? That's the only thing that is robust
> > > > against code moving about....
> > > 
> > > Yes. As already discussed with Pavel, it is one option to use the format
> > > string of the message as message ID. The disadvantage compared to
> > > message IDs like hashes is, that format strings might be even less
> > > unique than hashes 
> > 
> > if the hash comes from the string in the first place I have a hard time
> > believing that.
> > 
> 
> Just think of all messages containing %s, which can expand to every
> possible string.

of course you hash on the string before you do the format stuff....


-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Documentation of kernel messages (Summary)

[Michael Holzheu]

Hi all,

Any idea, how to proceed with this topic? Do you think that any of the
suggested solutions for documentation / translation of kernel messages
will have a chance to be included in the kernel?

I tried to summarize the outcome of this discussion...

There are two main issues:

* Translate messages
* Document messages

For both, we have to give messages identifiers in order to match them
to the corresponding documentation / translation. Three possible
solutions have been suggested:

1. Use message numbers maintained by driver owners
2. Automatically create hashes for printks
3. Use printk format string as message identifier

Regarding the question where to put documentation and translations we
have two options:

* In the kernel tree
* Somewhere outside the kernel tree

Another issue of the discussion was the usage of the component name for
printks.

Useful comments:
================
Andrew Morton:
Suggested an alternative approach using a new printk function with
message ID parameter. Documentation / Translation should be done
outside the kernel.

Gerrit Huizenga:
Suggested an alternative approach using hashes as message IDs.

Greg Kroah-Hartman:
He pointed out, that we need a solution which integrates dev_xxx()
macros.

Randy Dunlop:
He doesn't want to have message documentation included in the
kernel-doc tool. We should create new tool for kernel messages.

Pavel Machek:
Suggested to use gettext for message documentation / translation.

Useful links:
=============
There already exists a mailing list for the kernel messages topic:
https://lists.linux-foundation.org/mailman/listinfo/lf_kernel_messages

There was an LWN article summarizing some aspects of the discussion:
http://lwn.net/Articles/238948/

Pros and Cons of the different Message ID methods:
==================================================
Message numbers maintained by driver owners:
+ Unique IDs
+ Can be kept stable between different kernel versions
+ Efficient way of matching printk to Documentation or translation
- Difficult to maintain

printk hashes:
+ Easy to maintain
+ Efficient way of matching printk to Documentation or translation
- Additional preprocessor step needed
- No unique IDs
- Ugly in messages, since hashes can have many characters
- Volatile, since typo fixes change hash values

Format strings:
+ No change needed for printks
+ Matching of printk to corresponding documentation or translation is
  not as efficient as using hashes
- No unique IDs

Pros and Cons of external documentation (vs. internal documentation):
====================================================================
+ Developers have less work to do
+ Kernel sources and patches have less lines of code, since message
  descriptions are not contained in the kernel sources.
+ You can support multiple languages

- Hard to keep printks and descriptions up-to-date. If using hashes,
  each typo fix in a printk breaks the link to your documentation.
- Since the developer knows the meaning of a printk best, he probably
  would be the right person to write the description.
- For every Distribution or vanilla kernel you probably need separate
  external message databases.

Pros and Cons of using component names in printks:
==================================================
+ Makes it easier to identify source of message
+ Makes message more unique (especially, when using Hashes or format
  strings as message IDs)
- Changes of printks necessary

Implementation issues (depending on what solution we implement):
===============================================================
* Preprocessor tool for hashes
* Check tool for messages/message description
* Tool to extract kernel documentation from kernel tree
* Sysadmin tool to query message descriptions
* New kernel macros for Messages to support Component names / message
  numbers

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Monday 25 June 2007 09:48:41 Michael Holzheu wrote:
> Hi all,
>
> Any idea, how to proceed with this topic? Do you think that any of the
> suggested solutions for documentation / translation of kernel messages
> will have a chance to be included in the kernel?

Personally?  No to the second question, which renders the first "do it 
yourself outside of the tree".

Just a guess, and I don't speak for anyone else here, but I think most of us 
are waiting to see how long it takes you to lose interest.

> I tried to summarize the outcome of this discussion...
>
> There are two main issues:
>
> * Translate messages
> * Document messages

I'm somewhat sympathetic to translation (although I tend to think of it as a 
distro's job, and still wonder why IBM is pushing this directly rather than 
trying to bounce it off Red Hat or somebody).  But you could put a sed filter 
around dmesg to give you swahili output entirely in userspace.  Have you 
shown the simple approach to be insufficient yet?

Is "document messages" the same problem as "translate messages", or are these 
two different problems conflated together?

I find "document messages" to be a horrible idea conceptually, because I think 
the messages should _be_ documentation.  If the message is unclear it should 
be clarified.  "There's too much garbage on the floor, we should laminate it 
so we can't smell it anymore."  Er, no...

> Three possible solutions have been suggested:
>
> 1. Use message numbers maintained by driver owners
> 2. Automatically create hashes for printks
> 3. Use printk format string as message identifier

#2 has the advantage that you can do it without buy-in from the community, and 
without touching the code.  (Of course it doesn't necessarily help you if the 
printk string is "%s:%s".)

But I'm still convinced you could tackle over 90% of the problem from 
userspace, without touching the kernel.

> Pros and Cons of the different Message ID methods:
> ==================================================
> Message numbers maintained by driver owners:
> + Unique IDs
> + Can be kept stable between different kernel versions
> + Efficient way of matching printk to Documentation or translation
> - Difficult to maintain

Will at best only ever be done for some drivers, not all.  But that's probably 
inescapable no matter what your approach is.

> printk hashes:
> - Additional preprocessor step needed

Only for you guys.  Those of us building our own kernels and NOT translating 
them or producing "the big book of explanations of kernel messages for IBM 
customers who don't understand them" shouldn't have to run this preprocessor 
step.

> Pros and Cons of external documentation (vs. internal documentation):
> ====================================================================
> + Developers have less work to do
> + Kernel sources and patches have less lines of code, since message
>   descriptions are not contained in the kernel sources.
> + You can support multiple languages
>
> - Hard to keep printks and descriptions up-to-date. If using hashes,
>   each typo fix in a printk breaks the link to your documentation.
> - Since the developer knows the meaning of a printk best, he probably
>   would be the right person to write the description.

Imposing additional requirements on developers isn't going to work unless you 
reject patches from developers who don't follow your procedures and submit 
the appropriately filled-out forms with each patch for processing by the 
central bureaucracy.

Signed-off-by works because A) Linus enforces it, B) The overhead's very low, 
C) attribution is already highly good thing within our community values, D) 
it annoys SCO.

The javadoc function comments in the source don't have complete coverage and 
probably never will (although they're getting better).  The premise of doing 
something similar for messages is that a translating dmesg in userspace can't 
necessarily get complete coverage, so you need a solution modeled on 
something that doesn't have complete covereage years after its 
introduction...

> - For every Distribution or vanilla kernel you probably need separate
>   external message databases.

"If you're resigned to doing that anyway you don't actually need him to lie 
there all the time, do you?" - Ford Prefect

If you need an external message database what's wrong with doing it entirely 
in userspace via a sed-based translating dmesg?

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Sam Ravnborg]

> I find "document messages" to be a horrible idea conceptually, because I think 
> the messages should _be_ documentation.  If the message is unclear it should 
> be clarified.  "There's too much garbage on the floor, we should laminate it 
> so we can't smell it anymore."  Er, no...

"Document message" is for the cases where the message itself does not convey
enough info.
No need to add more text when the initial message is good enough.

	Sam

Re: Documentation of kernel messages (Summary)

[Michael Holzheu]

On Mon, 2007-06-25 at 11:44 -0400, Rob Landley wrote:
> On Monday 25 June 2007 09:48:41 Michael Holzheu wrote:
> > Hi all,
> >
> > Any idea, how to proceed with this topic? Do you think that any of the
> > suggested solutions for documentation / translation of kernel messages
> > will have a chance to be included in the kernel?
> 
> Personally?  No to the second question, which renders the first "do it 
> yourself outside of the tree".

If that is the opinion of the majority here, fine. If there is no hard
rule on how to define printk macros, one option for us would be to
define some new s390 specific printk macros for our device drivers.
Similar to hundreds of other driver specific printk macros in the
kernel.

> Just a guess, and I don't speak for anyone else here, but I think most of us 
> are waiting to see how long it takes you to lose interest.

:-) Work is not always fun. Sometimes it is just a duty.

Michael

Re: Documentation of kernel messages (Summary)

[Kunai, Takashi]

Hi Micheal,

This is a late response to your summary below.

First of all, there is a lot of interest in this issue among Japanese
vendor community. Recently, I have built a Japanese mailing list to
discuss this issue. I'm posting this comment resulting from such discussion.

(1) Your kernel development proposal will be greatly supported by
Japanese vendor community. At the same time, it needs support from the
kernel communities, as well.
(2) As for message ID's, you proposed 3 ways; (a)adding message numbers,
(b)printk hashes and (c)Format strings. (a) seems difficult to get
supports from kernel developers and (c) lacks uniqueness of each
message. Though (b) also lacks uniqueness, adding component id and/or
file name (files name could be hashed, as well), it could achieve some
practical level of uniqueness. To avoid ugly message format, it could be
possible to introduce a system parameter to suppress this hash ids.
Thus, our preference is (b).
(3) As for a documentation file, outside of kernel source tree would be
fine.
(4) When this kernel message scheme become feasible, it will be needed
to talk about the standardized contents of each description
(meanings/actions/etc).

Takashi Kunai
-- 
Kunai, Takashi
The Linux Foundation Japan (New since '07/2/5)
Shibuya Mark City West 22nd Floor
1-12-1 Dogenzaka,Shibuya-ku,Tokyo,Japan
zip150-0043 tel+81-3-4360-5493

Michael Holzheu wrote:
> On Mon, 2007-06-25 at 11:44 -0400, Rob Landley wrote:
>> On Monday 25 June 2007 09:48:41 Michael Holzheu wrote:
>>> Hi all,
>>>
>>> Any idea, how to proceed with this topic? Do you think that any of the
>>> suggested solutions for documentation / translation of kernel messages
>>> will have a chance to be included in the kernel?
>> Personally?  No to the second question, which renders the first "do it 
>> yourself outside of the tree".
> 
> If that is the opinion of the majority here, fine. If there is no hard
> rule on how to define printk macros, one option for us would be to
> define some new s390 specific printk macros for our device drivers.
> Similar to hundreds of other driver specific printk macros in the
> kernel.
> 
>> Just a guess, and I don't speak for anyone else here, but I think most of us 
>> are waiting to see how long it takes you to lose interest.
> 
> :-) Work is not always fun. Sometimes it is just a duty.
> 
> Michael
> 
> 
> 
> 

Re: Documentation of kernel messages (Summary)

[H. Peter Anvin]

Kunai, Takashi wrote:
> (1) Your kernel development proposal will be greatly supported by
> Japanese vendor community. At the same time, it needs support from the
> kernel communities, as well.

There is a very strong reason for the kernel community to NOT support
this: it makes it much harder to deal with bug reports.

Like it or not, English is a lingua franca[1] of the technology
industry.  When I deal with the rest of the kernel community, I use
English, instead of my native language (Swedish).

For things that are targetted toward end users, that's a different
matter, of course, but kernel messages are inherently directed toward
developers and technically sophisticated users.

	-hpa

Re: Documentation of kernel messages (Summary)

[Oliver Neukum]

Am Montag, 9. Juli 2007 schrieb Kunai, Takashi:
> (b)printk hashes and (c)Format strings. (a) seems difficult to get
> supports from kernel developers and (c) lacks uniqueness of each
> message. Though (b) also lacks uniqueness, adding component id and/or

We have generators for hash functions that guarantee unique results
for a set of inputs. They are even GPLed. As you are operating against
a known target, that should work.

	Regards
		Oliver

Re: Documentation of kernel messages (Summary)

[Rob Landley]

In regard to translating kernel messages:

On Monday 09 July 2007 01:36:31 H. Peter Anvin wrote:
> Kunai, Takashi wrote:
> > (1) Your kernel development proposal will be greatly supported by
> > Japanese vendor community. At the same time, it needs support from the
> > kernel communities, as well.
>
> There is a very strong reason for the kernel community to NOT support
> this: it makes it much harder to deal with bug reports.

Agreed.

There's a proposal in play for translations of documentation, and to 
recruit "language maintainers" (which is an idea I see that Matt Mackall came 
up with before I did).

A language maintainer is somebody who can accept a patch in language X, 
translate its description to post on the appropriate english-only list with a 
cc: to the appropriate maintainer(s), and translate questions and reviews 
back and forth for the original author until the patch can be merged.  (This 
means that documentation translations don't result in a stream of unmergeable 
patches.)

As for the documentation translations themselves, I note that Eric Raymond 
dissuaded me from actually hosting translations of kernel documentation on 
http://kernel.org/doc due to his experience with translations of his own 
writings.  If he hosts the translations on his website, they never get 
updated again.  But if he makes the contributor host them on their own 
website, then they sometimes get updated.

For my part, I can't _tell_ when a given translation is out of date because I 
can't read it, and I certainly can't update it.  So I agree with Eric and I'm 
linking to sites hosting kernel documentation in other languages rather than 
hosting them myself.  I've got a good link to a Japanese site, need to ping 
the contributors of the chinese documentation and see if they have a site for 
it...

I await hearing back from Satoru Ueda who might have a lead on a potential 
Japanese maintainer, probably after some event in Japan (a "Jamboree") in a 
week or two...

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Gerrit Huizenga]

On Mon, 09 Jul 2007 12:48:24 EDT, Rob Landley wrote:
> In regard to translating kernel messages:
> 
> On Monday 09 July 2007 01:36:31 H. Peter Anvin wrote:
> > Kunai, Takashi wrote:
> > > (1) Your kernel development proposal will be greatly supported by
> > > Japanese vendor community. At the same time, it needs support from the
> > > kernel communities, as well.
> >
> > There is a very strong reason for the kernel community to NOT support
> > this: it makes it much harder to deal with bug reports.
> 
> Agreed.

[...]

> 
> As for the documentation translations themselves, I note that Eric Raymond 
> dissuaded me from actually hosting translations of kernel documentation on 
> http://kernel.org/doc due to his experience with translations of his own 
> writings.  If he hosts the translations on his website, they never get 
> updated again.  But if he makes the contributor host them on their own 
> website, then they sometimes get updated.
> 
> For my part, I can't _tell_ when a given translation is out of date because I 
> can't read it, and I certainly can't update it.  So I agree with Eric and I'm 
> linking to sites hosting kernel documentation in other languages rather than 
> hosting them myself.  I've got a good link to a Japanese site, need to ping 
> the contributors of the chinese documentation and see if they have a site for 
> it...

Yeah, but it seems like having a translations directory in the kernel
avoids that problem - anyone can update, it is a single source, no digging
for sites that aren't tied to the kernel, available in the distros
directly, etc.

I'm not sure that the web site hosting argument is a good one.  Arch
maintainers and Language Maintainers have some similarities.  Also, being
able to clearly mark which version of a document was last translated
would help a bit with the fact that most of us aren't proficient in all
of the world's languages.  But, knowing who the translation maintainer is,
and when the translation was last updated allows both the original doc
maintainer and the translation document maintainer to see when a document
likely needs to be updated.  And that is probably good enough.

gerrit

Re: [Lf_kernel_messages] Re: Documentation of kernel messages (Summary)

[Theodore Tso]

On Mon, Jun 25, 2007 at 11:44:45AM -0400, Rob Landley wrote:
> Personally?  No to the second question, which renders the first "do it 
> yourself outside of the tree".
> 
> Just a guess, and I don't speak for anyone else here, but I think most of us 
> are waiting to see how long it takes you to lose interest.

Actually, at a brainstorming session at the recent Linux Foundation
collaboration summit, there were a number of kernel developers,
including James Bottomley, Greg K-H, Cristoph Hellwig (I think), and
others, who were actually generally symapthetic to the idea, if done
right.

The idea that was tossed about was essentially printk hashes (hashed
on message plus filename, with optional reporting of filename+line
number), with the messaging catalog being maintained externally.  Yes,
that will be a major pain for whoever wants to do the translation ---
but it puts the onus on the people who would be getting the value for
this to do the work.

The other idea that was tossed about was that for subsystems that are
using structured reporting (i.e., dev_printk, and sdev_printk in the
SCSI subsystem --- guess who suggested that :-), that there is an
opportunity to push a lot more information out to userspace for people
who want to use this facility to more easily determine when something
has bad happened to a particular disk device, for example.  This goes
a little a beyond the original stated desire for translation support,
but it was another idea to explore.

> I find "document messages" to be a horrible idea conceptually, because I think 
> the messages should _be_ documentation.  If the message is unclear it should 
> be clarified.  "There's too much garbage on the floor, we should laminate it 
> so we can't smell it anymore."  Er, no...

There are two separable desires being addressed here.  The first is
"cute" messages such as "lp0: on fire".  Yes, maybe those should be
fixed, but sometimes kernel developers *like* cute messages.  (Sniff,
I'm pretty sure there used to be a "eaten by a grue" printk, but it
doesn't seem to be there any more.  :-) Fixing these is largely a
political problem: "of course printer on fire makes sense".

The second desire is where people want entire paragraphs discsusing
possible causes of the messages and recommended responses to be
carried out by the data center's 3rd shift operations guy (aka "tape
monkey").  That's not something you *ever* want to put in the kernel,
and who ever compiles such a long document will probably sell it for
$$$ as significant value add.  Which is fine, because it is going to
cost a lot of $$$ to keep it up to date.  :-)

> > printk hashes:
> > - Additional preprocessor step needed
> 
> Only for you guys.  Those of us building our own kernels and NOT translating 
> them or producing "the big book of explanations of kernel messages for IBM 
> customers who don't understand them" shouldn't have to run this preprocessor 
> step.

Yep, I suspect it will only be used by distro kernels.  I probably
wouldn't turn it on myself.  :-)

> > - Hard to keep printks and descriptions up-to-date. If using hashes,
> >   each typo fix in a printk breaks the link to your documentation.
> > - Since the developer knows the meaning of a printk best, he probably
> >   would be the right person to write the description.
> 
> Imposing additional requirements on developers isn't going to work unless you 
> reject patches from developers who don't follow your procedures and submit 
> the appropriately filled-out forms with each patch for processing by the 
> central bureaucracy.

Yep, it's going to be up to the translators to keep the message
catalogs up to date.  The gettext folks have figured out ways of doing
fuzzy string matches, and so if you keep the external database of
filename, line #, string, hash, and translation(s), when you go to new
version of the kernel, it's not that hard to make the fuzzy
translations work.

	       	       	    	     - Ted

Re: Documentation of kernel messages (Summary)

[Adrian Bunk]

On Mon, Jul 09, 2007 at 12:48:24PM -0400, Rob Landley wrote:
>...
> [regarding translated documentation]
> 
> For my part, I can't _tell_ when a given translation is out of date because I 
> can't read it, and I certainly can't update it.  So I agree with Eric and I'm 
> linking to sites hosting kernel documentation in other languages rather than 
> hosting them myself.  I've got a good link to a Japanese site, need to ping 
> the contributors of the chinese documentation and see if they have a site for 
> it...
>...

Include the last git commit on this file in the translated document?

This would also make it easier for a translator to update a document 
because a diff on the original document since the last translation would 
be trivial.

> Rob

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [Lf_kernel_messages] Re: Documentation of kernel messages (Summary)

[Alan Cox]

O> Yep, it's going to be up to the translators to keep the message
> catalogs up to date.  The gettext folks have figured out ways of doing
> fuzzy string matches, and so if you keep the external database of
> filename, line #, string, hash, and translation(s), when you go to new
> version of the kernel, it's not that hard to make the fuzzy
> translations work.

The tools can extract the strings if marked _() which is easy work to do
and you usually only want to hit a subset of strings anyway [the ones
that occur usually].

Having implemented an encheferizer in a much older kernel its worth it
even for the humour value of a valspeak kernel (plus some people will
never be happy until it says things like

SYS-E-FOO: Incorrect operating system in use

Re: Documentation of kernel messages (Summary)

[Satyam Sharma]

On 7/9/07, Oliver Neukum <oliver@neukum.org> wrote:
> Am Montag, 9. Juli 2007 schrieb Kunai, Takashi:
> > (b)printk hashes and (c)Format strings. (a) seems difficult to get
> > supports from kernel developers and (c) lacks uniqueness of each
> > message. Though (b) also lacks uniqueness, adding component id and/or
>
> We have generators for hash functions that guarantee unique results
> for a set of inputs.

Right, perfect hash generators require the input set to be known
*a priori*.

> They are even GPLed. As you are operating against
> a known target, that should work.

But, I'm not sure they'd be operating against a known target -- I don't
really know what exactly would be hashed, but if it's kernel printk()
messages (the format string, obviously), then please remember that
new messages would get added all the time, and unless we're also
willing to change the hash function every time a printk() gets added
to the kernel (whew!) it's going to be a problem -- especially because
we don't really want to generate new hash functions for every kernel
version / or for every kernel build, because we'd obviously like the
same error message to hash to the same value across versions.

If we really care about uniqueness, the only solution I see is to use
a strong/cryptographic hash function (say SHA-1, which would never
change in the future) that generates the hashes for all the printk()
messages that are getting compiled-in at build-time ... but it would be
slow, of course.

Satyam

Re: Documentation of kernel messages (Summary)

[Oliver Neukum]

Am Dienstag, 10. Juli 2007 schrieb Satyam Sharma:
> But, I'm not sure they'd be operating against a known target -- I don't
> really know what exactly would be hashed, but if it's kernel printk()
> messages (the format string, obviously), then please remember that
> new messages would get added all the time, and unless we're also
> willing to change the hash function every time a printk() gets added
> to the kernel (whew!) it's going to be a problem -- especially because
> we don't really want to generate new hash functions for every kernel
> version / or for every kernel build, because we'd obviously like the
> same error message to hash to the same value across versions.

I suggest exactly that you build a new hash function for every build.
Nothing but your translating demon need ever see it. We build a new
System.map every time and it doesn't hurt.

	Regards
		Oliver

Re: Documentation of kernel messages (Summary)

[Dave Young]

Hi,
On 7/9/07, H. Peter Anvin <hpa@zytor.com> wrote:
> Kunai, Takashi wrote:
> > (1) Your kernel development proposal will be greatly supported by
> > Japanese vendor community. At the same time, it needs support from the
> > kernel communities, as well.
>
> There is a very strong reason for the kernel community to NOT support
> this: it makes it much harder to deal with bug reports.
>
> Like it or not, English is a lingua franca[1] of the technology
> industry.  When I deal with the rest of the kernel community, I use
> English, instead of my native language (Swedish).
>
> For things that are targetted toward end users, that's a different
> matter, of course, but kernel messages are inherently directed toward
> developers and technically sophisticated users.
>
I strongly agree with you.

Please keep kernel clean.

RE: Documentation of kernel messages (Summary)

[Li Yang-r58472]

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org
> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Rob Landley
> Sent: Tuesday, July 10, 2007 12:48 AM
> To: H. Peter Anvin
> Cc: Kunai, Takashi; holzheu@linux.vnet.ibm.com; Andrew Morton;
> linux-kernel@vger.kernel.org; lf_kernel_messages@linux-foundation.org;
> mtk-manpages@gmx.net; jack@suse.cz; randy.dunlap@oracle.com;
gregkh@suse.de;
> pavel@ucw.cz; tim.bird@am.sony.com; gh@us.ibm.com;
arjan@infradead.org;
> sam@ravnborg.org; jengelh@computergmbh.de; joe@perches.com;
> auke-jan.h.kok@intel.com; hansendc@us.ibm.com; davem@davemloft.net;
> Valdis.Kletnieks@vt.edu; kenistoj@us.ibm.com; schwidefsky@de.ibm.com;
> heiko.carstens@de.ibm.com; linux-doc@vger.kernel.org
> Subject: Re: Documentation of kernel messages (Summary)
> 
> In regard to translating kernel messages:
> 
> On Monday 09 July 2007 01:36:31 H. Peter Anvin wrote:
> > Kunai, Takashi wrote:
> > > (1) Your kernel development proposal will be greatly supported by
> > > Japanese vendor community. At the same time, it needs support from
the
> > > kernel communities, as well.
> >
> > There is a very strong reason for the kernel community to NOT
support
> > this: it makes it much harder to deal with bug reports.
> 
> Agreed.
> 
> There's a proposal in play for translations of documentation, and to
> recruit "language maintainers" (which is an idea I see that Matt
Mackall came
> up with before I did).
> 
> A language maintainer is somebody who can accept a patch in language
X,
> translate its description to post on the appropriate english-only list
with a
> cc: to the appropriate maintainer(s), and translate questions and
reviews
> back and forth for the original author until the patch can be merged.
(This
> means that documentation translations don't result in a stream of
unmergeable
> patches.)
> 
> As for the documentation translations themselves, I note that Eric
Raymond
> dissuaded me from actually hosting translations of kernel
documentation on
> http://kernel.org/doc due to his experience with translations of his
own
> writings.  If he hosts the translations on his website, they never get
> updated again.  But if he makes the contributor host them on their own
> website, then they sometimes get updated.

If the documents are already maintained together on one site, it won't
be very hard to synchronize with kernel.org.  Eric is right for the case
that separate documents on separate sites.

> 
> For my part, I can't _tell_ when a given translation is out of date
because I
> can't read it, and I certainly can't update it.  So I agree with Eric
and I'm
> linking to sites hosting kernel documentation in other languages
rather than
> hosting them myself.  I've got a good link to a Japanese site, need to
ping
> the contributors of the chinese documentation and see if they have a
site for
> it...

Please add a link to http://zh-kernel.org/docs for Chinese translated
kernel documentation.

- Leo

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Monday 09 July 2007 13:18:57 Gerrit Huizenga wrote:
> On Mon, 09 Jul 2007 12:48:24 EDT, Rob Landley wrote:
> > In regard to translating kernel messages:
> >
> > On Monday 09 July 2007 01:36:31 H. Peter Anvin wrote:
> > > Kunai, Takashi wrote:
> > > > (1) Your kernel development proposal will be greatly supported by
> > > > Japanese vendor community. At the same time, it needs support from
> > > > the kernel communities, as well.
> > >
> > > There is a very strong reason for the kernel community to NOT support
> > > this: it makes it much harder to deal with bug reports.
> >
> > Agreed.
>
> [...]
>
> > As for the documentation translations themselves, I note that Eric
> > Raymond dissuaded me from actually hosting translations of kernel
> > documentation on http://kernel.org/doc due to his experience with
> > translations of his own writings.  If he hosts the translations on his
> > website, they never get updated again.  But if he makes the contributor
> > host them on their own website, then they sometimes get updated.
> >
> > For my part, I can't _tell_ when a given translation is out of date
> > because I can't read it, and I certainly can't update it.  So I agree
> > with Eric and I'm linking to sites hosting kernel documentation in other
> > languages rather than hosting them myself.  I've got a good link to a
> > Japanese site, need to ping the contributors of the chinese documentation
> > and see if they have a site for it...
>
> Yeah, but it seems like having a translations directory in the kernel
> avoids that problem - anyone can update, it is a single source, no digging
> for sites that aren't tied to the kernel, available in the distros
> directly, etc.

No.  It doesn't help.

99% of the kernel directory is C.  That means any random passerby can review 
code.  Everyone who has the kernel tarball should be able to review code 
that's in there, plus when you compile it breaks.  So merging _code_ into the 
kernel helps keep it up to date.

Merging documentation into the kernel doesn't help keep it up to date, because 
documentation being out of date doesn't break the build.  It may get the 
documentation more review, but the existing state of Documentation/* argues 
against that.  It's a struggle to keep the english versions on the same 
continent as "up to date" or "complete", and most of the _good_ documentation 
is out in OLS papers and such (which I'm off indexing as we speak).

Now add in the non-english nature of the new Documentation/stale 
subdirectories you're proposing.  Almost nobody in the existing kernel 
community can even _read_ this documentation, let alone update it.  People 
who update Documentation/* _can't_ update the translations, and that will 
_never_change_.

Merging into the kernel is a great way to keep CODE up to date.  Don't think 
it's magic pixie dust for documentation.  It never has been before.

> I'm not sure that the web site hosting argument is a good one.  Arch
> maintainers and Language Maintainers have some similarities.

Ah, but It's not a language maintainer's job to update documentation.  It's 
their job to ACCEPT PATCHES.  Translate patches, translate questions back and 
forth.  This has NOTHING to do with documentation unless they're converting 
documentation accompanying the patch one way; into english.

> Also, being 
> able to clearly mark which version of a document was last translated
> would help a bit with the fact that most of us aren't proficient in all
> of the world's languages.

Keeping translations up to date is not something I can do, and thus not my 
problem.

> But, knowing who the translation maintainer is, 
> and when the translation was last updated allows both the original doc
> maintainer and the translation document maintainer to see when a document
> likely needs to be updated.  And that is probably good enough.

Thank you.  You've just scared away all potential language maintainers by 
making them think we want them to translate all the world's existing 
documentation.  Do you know how long it takes just to read through the 
existing Documentation directory?  (Have you done it?)

When I'm talking about language maintainers I am NOT looking for them to 
translate existing documentation or keep Documentation/* up to date for some 
other langauge.  That's a more than full-time job.  I've been looking around 
for weeks for someone to just accept Japanese patches, and everybody I've 
talked to says it's too much time commitment without any extra work piled on 
top of it in by armchair commentators who won't be _doing_ that work.

> gerrit

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Monday 09 July 2007 14:33:09 Adrian Bunk wrote:
> On Mon, Jul 09, 2007 at 12:48:24PM -0400, Rob Landley wrote:
> >...
> > [regarding translated documentation]
> >
> > For my part, I can't _tell_ when a given translation is out of date
> > because I can't read it, and I certainly can't update it.  So I agree
> > with Eric and I'm linking to sites hosting kernel documentation in other
> > languages rather than hosting them myself.  I've got a good link to a
> > Japanese site, need to ping the contributors of the chinese documentation
> > and see if they have a site for it...
> >...
>
> Include the last git commit on this file in the translated document?
>
> This would also make it easier for a translator to update a document
> because a diff on the original document since the last translation would
> be trivial.

Ok, back up.

Let's look at _english_ documentation.

If you go to http://kernel.org/doc/ols you should find, nicely split up, all 
the OLS papers from 2002-2007.  By my count, there are 337 of them, totaling 
61.8 megabytes when they're split up like that.  Being PDFs, they don't 
compress all that well.

Which subset of these should be copied into the Documentation directory?  How 
do you update them?  This is the documentation from just ONE SOURCE.

And PDF isn't close to the worst of it: Linuxconf.au has lots of video, as 
does google video.  (I don't even have _english_ transcripts for those 
videos, which would be nice.)  Do you want to merge all of this into the 
kernel tarball as well?

How does this help keep anything up to date?

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: [Lf_kernel_messages] Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Monday 09 July 2007 14:10:07 Theodore Tso wrote:
> The idea that was tossed about was essentially printk hashes (hashed
> on message plus filename, with optional reporting of filename+line
> number), with the messaging catalog being maintained externally.  Yes,
> that will be a major pain for whoever wants to do the translation ---
> but it puts the onus on the people who would be getting the value for
> this to do the work.

Sounds good to me.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Adrian Bunk]

On Tue, Jul 10, 2007 at 12:25:31PM -0400, Rob Landley wrote:
>...
> Let's look at _english_ documentation.
> 
> If you go to http://kernel.org/doc/ols you should find, nicely split up, all 
> the OLS papers from 2002-2007.  By my count, there are 337 of them, totaling 
> 61.8 megabytes when they're split up like that.  Being PDFs, they don't 
> compress all that well.
> 
> Which subset of these should be copied into the Documentation directory?

You don't have to copy everything.

And you anyway have to handle cases like e.g. most books where you are 
not permitted to copy them, but might want to point people to them.

> How 
> do you update them?  This is the documentation from just ONE SOURCE.

You aren't legally permitted to update any OLS paper unless you've 
gotten explicit permission by the author. And when he gives legal 
permission, he can as well give you the LaTeX source. I'd expect legal 
and technical update possibilities to usually be related this way.

And you shouldn't include non-free documentation into the kernel sources.

> And PDF isn't close to the worst of it: Linuxconf.au has lots of video, as 
> does google video.  (I don't even have _english_ transcripts for those 
> videos, which would be nice.)  Do you want to merge all of this into the 
> kernel tarball as well?

Not at all.

> How does this help keep anything up to date?

You can only update things you are legally and technically able to 
update. For the rest, you can only ship pointers like kernel-docs.txt.

> Rob

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Tuesday 10 July 2007 14:54:15 Adrian Bunk wrote:
> On Tue, Jul 10, 2007 at 12:25:31PM -0400, Rob Landley wrote:
> >...
> > Let's look at _english_ documentation.
> >
> > If you go to http://kernel.org/doc/ols you should find, nicely split up,
> > all the OLS papers from 2002-2007.  By my count, there are 337 of them,
> > totaling 61.8 megabytes when they're split up like that.  Being PDFs,
> > they don't compress all that well.
> >
> > Which subset of these should be copied into the Documentation directory?
>
> You don't have to copy everything.
>
> And you anyway have to handle cases like e.g. most books where you are
> not permitted to copy them, but might want to point people to them.

This is exactly what I'm saying.  I don't want to copy the translated 
documentation, I want to point people to it.  Out on the web.  Where it's 
maintained by the people who translated it, which history says they're 
noticeably less likely to do with a copy.

> > How
> > do you update them?  This is the documentation from just ONE SOURCE.
>
> You aren't legally permitted to update any OLS paper unless you've
> gotten explicit permission by the author.

I wasn't planning to.  I'm pointing out that it's hard.

> And you shouldn't include non-free documentation into the kernel sources.

A) http://www.linuxsymposium.org/2007/cfp.php under "Publication Rights":

  Further, as stated in the official templates, and on the Credits page from
  the Proceedings of this year and prior years: "Authors retain copyright to
  all submitted papers, but have granted unlimited redistribution rights to
  all as a condition of submission."

So it's free as in speech but not beer.

B) I'm using these as examples of things to NOT merge into the kernel sources.  
Thank you for making my point.  There's a LOT of this stuff out there.

> > How does this help keep anything up to date?
>
> You can only update things you are legally and technically able to
> update. For the rest, you can only ship pointers like kernel-docs.txt.

As an english speaker, I am not technically able to update non-english 
documentation.  Since the only common language required of linux kernel 
developers has ever been English, only the english documentation makes sense 
to merge into the kernel tarball.  Merging anything else guarantees that the 
vast majority of the contributors will never be able to review or update it, 
so merging it into the kernel tarball does not result in extra review or 
maintenance.

Extrapolating the review benefits of C code that breaks the build for 
everybody if you do it wrong to chinese documentation you can't even 
_read_...  Apples and oranges, it's a false comparison.  I don't see how 
merging translations of documentation is ever likely to be a win.

> > Rob
>
> cu
> Adrian

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Michael Holzheu]

Hi Rob,

On Tue, 2007-07-10 at 12:12 -0400, Rob Landley wrote:

[snip]

> > Yeah, but it seems like having a translations directory in the kernel
> > avoids that problem - anyone can update, it is a single source, no digging
> > for sites that aren't tied to the kernel, available in the distros
> > directly, etc.
> 
> No.  It doesn't help.
> 
> 99% of the kernel directory is C.  That means any random passerby can review 
> code.  Everyone who has the kernel tarball should be able to review code 
> that's in there, plus when you compile it breaks.  So merging _code_ into the 
> kernel helps keep it up to date.
> 
> Merging documentation into the kernel doesn't help keep it up to date, because 
> documentation being out of date doesn't break the build.  It may get the 
> documentation more review, but the existing state of Documentation/* argues 
> against that.  It's a struggle to keep the english versions on the same 
> continent as "up to date" or "complete", and most of the _good_ documentation 
> is out in OLS papers and such (which I'm off indexing as we speak).

With the checker tool, which we suggested in the initial proposal, it is
possible to verify

 * that every marked message has a description
 * that there are no descriptions without corresponding messages
 * that format strings of message and description match

So when compiling the kernel using C=1, you will at least see warnings,
when a message has changed or a message disappeared:

>> make modules C=1
  CHK     include/linux/version.h
  CHK     include/linux/utsrelease.h
  CHECK   drivers/kmsgtest/kmsgtest.c
drivers/kmsgtest/kmsgtest.c: Missing description for: kmsgtest.1
drivers/kmsgtest/kmsgtest.c: Description without message for: kmsgtest.3

Michael

Re: Documentation of kernel messages (Summary)

[Li Yang]

On Wed, 2007-07-11 at 00:12 +0800, Rob Landley wrote:
> On Monday 09 July 2007 13:18:57 Gerrit Huizenga wrote: 
> > On Mon, 09 Jul 2007 12:48:24 EDT, Rob Landley wrote: 
> > > In regard to translating kernel messages: 
> > > 
> > > On Monday 09 July 2007 01:36:31 H. Peter Anvin wrote: 
> > > > Kunai, Takashi wrote: 
> > > > > (1) Your kernel development proposal will be greatly supported
> by 
> > > > > Japanese vendor community. At the same time, it needs support
> from 
> > > > > the kernel communities, as well. 
> > > > 
> > > > There is a very strong reason for the kernel community to NOT
> support 
> > > > this: it makes it much harder to deal with bug reports. 
> > > 
> > > Agreed. 
> > 
> > [...] 
> > 
> > > As for the documentation translations themselves, I note that
> Eric 
> > > Raymond dissuaded me from actually hosting translations of kernel 
> > > documentation on http://kernel.org/doc due to his experience with 
> > > translations of his own writings.  If he hosts the translations on
> his 
> > > website, they never get updated again.  But if he makes the
> contributor 
> > > host them on their own website, then they sometimes get updated. 
> > > 
> > > For my part, I can't _tell_ when a given translation is out of
> date 
> > > because I can't read it, and I certainly can't update it.  So I
> agree 
> > > with Eric and I'm linking to sites hosting kernel documentation in
> other 
> > > languages rather than hosting them myself.  I've got a good link
> to a 
> > > Japanese site, need to ping the contributors of the chinese
> documentation 
> > > and see if they have a site for it... 
> > 
> > Yeah, but it seems like having a translations directory in the
> kernel 
> > avoids that problem - anyone can update, it is a single source, no
> digging 
> > for sites that aren't tied to the kernel, available in the distros 
> > directly, etc.
> 
> No.  It doesn't help.
> 
> 99% of the kernel directory is C.  That means any random passerby can
> review 
> code.  Everyone who has the kernel tarball should be able to review
> code 
> that's in there, plus when you compile it breaks.  So merging _code_
> into the 
> kernel helps keep it up to date.
> 
> Merging documentation into the kernel doesn't help keep it up to date,
> because 
> documentation being out of date doesn't break the build.  It may get
> the 
> documentation more review, but the existing state of Documentation/*
> argues 
> against that.  It's a struggle to keep the english versions on the
> same 
> continent as "up to date" or "complete", and most of the _good_
> documentation 
> is out in OLS papers and such (which I'm off indexing as we speak).
> 
> Now add in the non-english nature of the new Documentation/stale 
> subdirectories you're proposing.  Almost nobody in the existing
> kernel 
> community can even _read_ this documentation, let alone update it.
> People 
> who update Documentation/* _can't_ update the translations, and that
> will 
> _never_change_.

There are quite a lot kernel developers for each of the popular
language, AFAIK.  For non-popular languages, there shouldn't be
translation available in the first place.  It was really a surprise when
I post my Chinese translation on the LKML so many Chinese speakers have
commented on the translation and encouraged the translation work.  They
are not visible as they usually don't talk too much.  :) So I set up the
Chinese kernel development maillist(linux-kernel@zh-kernel.org), there
will be more and more experienced kernel developer who can read and
update the Chinese documents after they read the translated
documentation and become kernel hacker.

> 
> Merging into the kernel is a great way to keep CODE up to date.  Don't
> think 
> it's magic pixie dust for documentation.  It never has been before.

IMHO, having contribution merged into the kernel has the MAGIC to
attract people to work for recognition.  When more and more people
volunteer to work on it, the documentation will be up to date magically.

> 
> > I'm not sure that the web site hosting argument is a good one.
> Arch 
> > maintainers and Language Maintainers have some similarities.
> 
> Ah, but It's not a language maintainer's job to update documentation.
> It's 
> their job to ACCEPT PATCHES.  Translate patches, translate questions
> back and 
> forth.  This has NOTHING to do with documentation unless they're
> converting 
> documentation accompanying the patch one way; into english.

Language maintainers can integrate updates to the documentation just
like integrating any updates to the code.  Working on the documentation
is ,IMHO, a perfect task for novice kernel hacker to get familiar with
the process.

> 
> > Also, being 
> > able to clearly mark which version of a document was last
> translated 
> > would help a bit with the fact that most of us aren't proficient in
> all 
> > of the world's languages.
> 
> Keeping translations up to date is not something I can do, and thus
> not my 
> problem.
> 
> > But, knowing who the translation maintainer is, 
> > and when the translation was last updated allows both the original
> doc 
> > maintainer and the translation document maintainer to see when a
> document 
> > likely needs to be updated.  And that is probably good enough.
> 
> Thank you.  You've just scared away all potential language maintainers
> by 
> making them think we want them to translate all the world's existing 
> documentation.  Do you know how long it takes just to read through
> the 
> existing Documentation directory?  (Have you done it?)
> 
> When I'm talking about language maintainers I am NOT looking for them
> to 
> translate existing documentation or keep Documentation/* up to date
> for some 
> other langauge.  That's a more than full-time job.  I've been looking
> around 
> for weeks for someone to just accept Japanese patches, and everybody
> I've 
> talked to says it's too much time commitment without any extra work
> piled on 
> top of it in by armchair commentators who won't be _doing_ that work.

It won't be too hard if the work is shared by a community.  Like the one
I'm trying to establish.

- Leo

Advertisement time:
Chinese kernel development community http://zh-kernel.org    :)

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Wednesday 11 July 2007 10:26:30 am Li Yang wrote:
> There are quite a lot kernel developers for each of the popular
> language, AFAIK.  For non-popular languages, there shouldn't be
> translation available in the first place.

I don't distinguish between "popular" and "non-popular" languages.  If 
somebody's willing to do the translation work, it's popular.  If nobody's 
willing to do the work, then even a language 1/3 of the planet's population 
speaks isn't "popular" for kernel development.

I wouldn't discourage a translator into Klingon if they were willing to keep 
their translation up to date and/or it actually resulted in patches.

> It was really a surprise when 
> I post my Chinese translation on the LKML so many Chinese speakers have
> commented on the translation and encouraged the translation work.  They
> are not visible as they usually don't talk too much.  :) So I set up the
> Chinese kernel development maillist(linux-kernel@zh-kernel.org), there
> will be more and more experienced kernel developer who can read and
> update the Chinese documents after they read the translated
> documentation and become kernel hacker.

The chinese mailing list is highly cool, and my first instinct was to note it 
on kernel.org/doc, but it would be better if the chinese website I already 
link to notes it instead.  (That way I don't have to worry about how to 
spam-guard your email address. :)

This also highlights the need for language maintainers to help the patches go 
upstream to the english list.  (If we can avoid armchair commentators 
saddling them with the task of translating the entire Documentation directory 
and keeping such a translation up to date, we might actually get one too.  
Fielding patches and questions sounds like plenty to me...)

Could you ask on said list if anyone is likely to volunteer for the position?  
(JUST translating patches and questions, as part of pushing code upstream.)

> > Merging into the kernel is a great way to keep CODE up to date.  Don't
> > think
> > it's magic pixie dust for documentation.  It never has been before.
>
> IMHO, having contribution merged into the kernel has the MAGIC to
> attract people to work for recognition.  When more and more people
> volunteer to work on it, the documentation will be up to date magically.

Obvious counter-arguments include the floppy driver, the floppy tape driver, 
the tty layer, and most of the existing english Documentation directory...

I'll happily stay out of the way of people who actually want to merge 
translations of documentation into the kernel.  I reserve the right to say "I 
told you so" in about five years.

> > Ah, but It's not a language maintainer's job to update documentation.
> > It's
> > their job to ACCEPT PATCHES.  Translate patches, translate questions
> > back and
> > forth.  This has NOTHING to do with documentation unless they're
> > converting
> > documentation accompanying the patch one way; into english.
>
> Language maintainers can integrate updates to the documentation just
> like integrating any updates to the code.  Working on the documentation
> is ,IMHO, a perfect task for novice kernel hacker to get familiar with
> the process.

>From a language maintainer's perspective documentation patches wouldn't be any 
different than any other patches.  Translate the description and questions 
going back and forth.  The patch itself doesn't get translated when it's C 
and applies to scripts/kconfig/, why would it when it's UTF-8 and applies to 
Documentation/?

Of course this brings up the question "what kind of function and variable 
names do chinese people pick?"  (I honestly don't know, but I note that 
attempts to use names that don't fit in 7-bit ascii would probably be frowned 
upon in a big way...)

> It won't be too hard if the work is shared by a community.  Like the one
> I'm trying to establish.

A list works fine as a point of contact.  I note that in general, maintainers 
are individuals (who delegate like mad, of course), because otherwise agenda 
items languish with everyone thinking it's someone else's responsibility.

http://www.trumanlibrary.org/buckstop.htm

> - Leo
>
> Advertisement time:
> Chinese kernel development community http://zh-kernel.org    :)

Hmmm...  Now I'm wondering if I should link directly to the docs pages of the 
chinese and japanese sites, or to the top level community pages?

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Theodore Tso]

On Wed, Jul 11, 2007 at 02:13:02PM -0400, Rob Landley wrote:
> > IMHO, having contribution merged into the kernel has the MAGIC to
> > attract people to work for recognition.  When more and more people
> > volunteer to work on it, the documentation will be up to date magically.
> 
> Obvious counter-arguments include the floppy driver, the floppy tape driver, 
> the tty layer, and most of the existing english Documentation directory...
> 
> I'll happily stay out of the way of people who actually want to merge 
> translations of documentation into the kernel.  I reserve the right to say "I 
> told you so" in about five years.

One thing that would be a VERY good idea is to make sure that each
translated document is tagged with when it was last translated, and
against which kernel version (using either a GIT commit id or a
2.6.x.y revision, I don't care), and who is responsible for keeping
that particular document translated.  That way, it becomes a lot
easier to recognize when something is stale, and if has gotten stale,
we can just delete it.

If someone keeping some files translated into Klingon up to date for 3
months, and then disappears, we can just delete it, and it's no big
deal.  You can even say "I told you so" when you submit the patch to
delete the stale translation.  :-)

> A list works fine as a point of contact.  I note that in general, maintainers 
> are individuals (who delegate like mad, of course), because otherwise agenda 
> items languish with everyone thinking it's someone else's responsibility.

I agree; if we're going to have a language translated in
Documentation/<lang>, there ought to be one or two overall language
maintainer, plus a maintainer for each file that is being translated,
who is responsible for updating the translated file when the base
English Documentation file is updated.

					- Ted

Re: Documentation of kernel messages (Summary)

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 344 bytes --]

On Wed, 11 Jul 2007 14:13:02 EDT, Rob Landley said:
> I wouldn't discourage a translator into Klingon if they were willing to keep
> their translation up to date and/or it actually resulted in patches.

The guys at the Klingon Language Institute are going to have a fit - what's
the Klingon word for 'freezer' as used in our suspend code? :)



[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

RE: Documentation of kernel messages (Summary)

[Li Yang-r58472]

On Thursday, July 12, 2007 2:13 AM Rob Landley wrote:
> On Wednesday 11 July 2007 10:26:30 am Li Yang wrote:
> > There are quite a lot kernel developers for each of the popular
> > language, AFAIK.  For non-popular languages, there shouldn't be
> > translation available in the first place.
> 
> I don't distinguish between "popular" and "non-popular" languages.  If
> somebody's willing to do the translation work, it's popular.  If
nobody's
> willing to do the work, then even a language 1/3 of the planet's
population
> speaks isn't "popular" for kernel development.
> 
> I wouldn't discourage a translator into Klingon if they were willing
to keep
> their translation up to date and/or it actually resulted in patches.
> 
> > It was really a surprise when
> > I post my Chinese translation on the LKML so many Chinese speakers
have
> > commented on the translation and encouraged the translation work.
They
> > are not visible as they usually don't talk too much.  :) So I set up
the
> > Chinese kernel development maillist(linux-kernel@zh-kernel.org),
there
> > will be more and more experienced kernel developer who can read and
> > update the Chinese documents after they read the translated
> > documentation and become kernel hacker.
> 
> The chinese mailing list is highly cool, and my first instinct was to
note it
> on kernel.org/doc, but it would be better if the chinese website I
already
> link to notes it instead.  (That way I don't have to worry about how
to
> spam-guard your email address. :)

The benefit of a localized maillist is that patches can be reviewed and
commented in native language for common problems (like style, API).

> 
> This also highlights the need for language maintainers to help the
patches go
> upstream to the english list.  (If we can avoid armchair commentators
> saddling them with the task of translating the entire Documentation
directory
> and keeping such a translation up to date, we might actually get one
too.
> Fielding patches and questions sounds like plenty to me...)

I do think the documentation translation is very necessary even when
there is a language maintainer, especially for the policy documents as
HOWTO, codestyle , and etc.  The contributors should go through these
policies and check their code for compliance before going to the
language maintainer for help, or there will be too much for the language
maintainer to translate.  The language maintainer doesn't need to
translate all the documents himself, but he can help to coordinate the
translation effort and help to make it update to date.

> 
> Could you ask on said list if anyone is likely to volunteer for the
position?
> (JUST translating patches and questions, as part of pushing code
upstream.)

If we do need a contact person, I can do it.  However I don't think
there will be much translation work to do here.  As I stated before,
most Chinese programmers are more or less capable of read/write
technical English.  The difficult part is to let them know the benefit
of merging code in kernel and teach them how to do it.  That's why the
policy documents in native language will be very useful.

> 
> > > Merging into the kernel is a great way to keep CODE up to date.
Don't
> > > think
> > > it's magic pixie dust for documentation.  It never has been
before.
> >
> > IMHO, having contribution merged into the kernel has the MAGIC to
> > attract people to work for recognition.  When more and more people
> > volunteer to work on it, the documentation will be up to date
magically.
> 
> Obvious counter-arguments include the floppy driver, the floppy tape
driver,
> the tty layer, and most of the existing english Documentation
directory...
> 
> I'll happily stay out of the way of people who actually want to merge
> translations of documentation into the kernel.  I reserve the right to
say "I
> told you so" in about five years.
> 
> > > Ah, but It's not a language maintainer's job to update
documentation.
> > > It's
> > > their job to ACCEPT PATCHES.  Translate patches, translate
questions
> > > back and
> > > forth.  This has NOTHING to do with documentation unless they're
> > > converting
> > > documentation accompanying the patch one way; into english.
> >
> > Language maintainers can integrate updates to the documentation just
> > like integrating any updates to the code.  Working on the
documentation
> > is ,IMHO, a perfect task for novice kernel hacker to get familiar
with
> > the process.
> 
> From a language maintainer's perspective documentation patches
wouldn't be any
> different than any other patches.  Translate the description and
questions
> going back and forth.  The patch itself doesn't get translated when
it's C
> and applies to scripts/kconfig/, why would it when it's UTF-8 and
applies to
> Documentation/?
> 
> Of course this brings up the question "what kind of function and
variable
> names do chinese people pick?"  (I honestly don't know, but I note
that
> attempts to use names that don't fit in 7-bit ascii would probably be
frowned
> upon in a big way...)
> 
> > It won't be too hard if the work is shared by a community.  Like the
one
> > I'm trying to establish.
> 
> A list works fine as a point of contact.  I note that in general,
maintainers
> are individuals (who delegate like mad, of course), because otherwise
agenda
> items languish with everyone thinking it's someone else's
responsibility.

Got it.  I can do it as long as it doesn't consume too much time.  :)

> 
> http://www.trumanlibrary.org/buckstop.htm
> 
> > - Leo
> >
> > Advertisement time:
> > Chinese kernel development community http://zh-kernel.org    :)
> 
> Hmmm...  Now I'm wondering if I should link directly to the docs pages
of the
> chinese and japanese sites, or to the top level community pages?

The documents and maillist are also linked in the front page.  It's
better to link the top level page.

- Leo

[PATCH] Chinese Language Maintainer

[Rob Landley]

Proposed patch adding Li Yang to MAINTAINERS, and Documentation
describing what a language maintainer is.

Signed-off-by: Rob Landley <rob@landley.net>

--

On Thursday 12 July 2007 9:53:54 am Li Yang-r58472 wrote:

> > > I'm trying to establish.
> >
> > A list works fine as a point of contact.  I note that in general,
> maintainers
> > are individuals (who delegate like mad, of course), because otherwise
> agenda
> > items languish with everyone thinking it's someone else's
> responsibility.
>
> Got it.  I can do it as long as it doesn't consume too much time.  :)

Woot!  See attached patch and sign-off-by if you want the job. :)

The bit about "the maintainer forwards, not the list" is a suggestion to
prevent duplicates.  It doesn't mean you can't delegate, just that if it's
open season on the list translating stuff _and_ forwarding it, then
linux-kernel will get dupes.  (Plus other maintainers are more likely
to pay attention to somebody they've heard from before.)

Greg KH had entire an OLS presentation about how the developer ->
maintainer -> subsystem -> anderew+linus forwarding is more
an ideal than a reality, but at least it gives us a frame of reference
to diverge from:
http://lwn.net/Articles/240402/

diff -r edfd2d6f670d MAINTAINERS
--- a/MAINTAINERS	Tue Jul 10 17:51:13 2007 -0700
+++ b/MAINTAINERS	Thu Jul 12 11:51:19 2007 -0400
@@ -2146,6 +2146,13 @@ W:	http://auxdisplay.googlepages.com/
 W:	http://auxdisplay.googlepages.com/
 S:	Maintained
 
+LANGUAGE (CHINESE)
+P:	Li Yang
+M:	LeoLi@freescale.com
+L:	linux-kernel@zh-kernel.org
+W:	http://zh-kernel.org
+S:	Maintained
+
 LAPB module
 L:	linux-x25@vger.kernel.org
 S:	Orphan
--- /dev/null	2007-04-23 11:59:00.000000000 -0400
+++ linux-2.6/Documentation/language-maintainers.txt	2007-07-12 11:49:20.000000000 -0400
@@ -0,0 +1,20 @@
+The langauges in which the Linux kernel is developed are C and English.
+(Note that Linus Torvalds' native language is Swedish, yet he chose
+to hold technical discussions in English.)
+
+The job of a language maintainer is to provide a point of contact for
+non-english speaking developers who wish to merge their patches into the
+Linux kernel.  Each language needs a specific language maintainer, who
+accepts non-english patch submissions on behalf of the Linux kernel.
+
+A language maintainer accepts patches to the Linux kernel, written in C, from
+authors who do not also speak English.  The language maintainer translates the
+description of each patch into English, forwards the patches to linux-kernel
+and to the appropriate maintainers for inclusion in the Linux kernel, and
+translates questions and replies about such patches as part of the
+patch review process.
+
+Some language maintainers provide a mailing list as a point of contact, to
+distribute the translation work, but the maintainer is still the person who
+ultimately forwards the results (to prevent duplicates), and the one to contact
+if patches and questions don't get translated and forwarded in a timely fashion.

-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Tsugikazu Shibata]

On Thu, 12 Jul 2007 21:53:54 +0800, LeoLi wrote:
> On Thursday, July 12, 2007 2:13 AM Rob Landley wrote:
> > On Wednesday 11 July 2007 10:26:30 am Li Yang wrote:
> > > There are quite a lot kernel developers for each of the popular
> > > language, AFAIK.  For non-popular languages, there shouldn't be
> > > translation available in the first place.
> > 
> > I don't distinguish between "popular" and "non-popular" languages.  If
> > somebody's willing to do the translation work, it's popular.  If
> nobody's
> > willing to do the work, then even a language 1/3 of the planet's
> population
> > speaks isn't "popular" for kernel development.
> > 
> > I wouldn't discourage a translator into Klingon if they were willing
> to keep
> > their translation up to date and/or it actually resulted in patches.
> > 
> > > It was really a surprise when
> > > I post my Chinese translation on the LKML so many Chinese speakers
> have
> > > commented on the translation and encouraged the translation work.
> They
> > > are not visible as they usually don't talk too much.  :) So I set up
> the
> > > Chinese kernel development maillist(linux-kernel@zh-kernel.org),
> there
> > > will be more and more experienced kernel developer who can read and
> > > update the Chinese documents after they read the translated
> > > documentation and become kernel hacker.
> > 
> > The chinese mailing list is highly cool, and my first instinct was to
> note it
> > on kernel.org/doc, but it would be better if the chinese website I
> already
> > link to notes it instead.  (That way I don't have to worry about how
> to
> > spam-guard your email address. :)
> 
> The benefit of a localized maillist is that patches can be reviewed and
> commented in native language for common problems (like style, API).
> 
> > 
> > This also highlights the need for language maintainers to help the
> patches go
> > upstream to the english list.  (If we can avoid armchair commentators
> > saddling them with the task of translating the entire Documentation
> directory
> > and keeping such a translation up to date, we might actually get one
> too.
> > Fielding patches and questions sounds like plenty to me...)
> 
> I do think the documentation translation is very necessary even when
> there is a language maintainer, especially for the policy documents as
> HOWTO, codestyle , and etc.  The contributors should go through these
> policies and check their code for compliance before going to the
> language maintainer for help, or there will be too much for the language
> maintainer to translate.  The language maintainer doesn't need to
> translate all the documents himself, but he can help to coordinate the
> translation effort and help to make it update to date.

I would like to comment from Japanese situation.
I do also think the documentation translation is necessary even if
there are language maintainer are not there. HOWTO like very
fundamental documents will great help for early stage of developers.
The contributors/early stage of developer will be able to follow
coding style and so on if the documents are easy to read.

So, maintain this kind of document in local language are necessary
even if there are no language maintainer.

I believe the language maintainer should mostly take care about the
patch.

> > Could you ask on said list if anyone is likely to volunteer for the
> position?
> > (JUST translating patches and questions, as part of pushing code
> upstream.)
> 
> If we do need a contact person, I can do it.  However I don't think
> there will be much translation work to do here.  As I stated before,
> most Chinese programmers are more or less capable of read/write
> technical English.  The difficult part is to let them know the benefit
> of merging code in kernel and teach them how to do it.  That's why the
> policy documents in native language will be very useful.

I also think that most Japanese developers can read/write technical
English at some level.


> > > > Merging into the kernel is a great way to keep CODE up to date.
> Don't
> > > > think
> > > > it's magic pixie dust for documentation.  It never has been
> before.
> > >
> > > IMHO, having contribution merged into the kernel has the MAGIC to
> > > attract people to work for recognition.  When more and more people
> > > volunteer to work on it, the documentation will be up to date
> magically.
> > 
> > Obvious counter-arguments include the floppy driver, the floppy tape
> driver,
> > the tty layer, and most of the existing english Documentation
> directory...
> > 
> > I'll happily stay out of the way of people who actually want to merge
> > translations of documentation into the kernel.  I reserve the right to
> say "I
> > told you so" in about five years.
> > 
> > > > Ah, but It's not a language maintainer's job to update
> documentation.
> > > > It's
> > > > their job to ACCEPT PATCHES.  Translate patches, translate
> questions
> > > > back and
> > > > forth.  This has NOTHING to do with documentation unless they're
> > > > converting
> > > > documentation accompanying the patch one way; into english.
> > >
> > > Language maintainers can integrate updates to the documentation just
> > > like integrating any updates to the code.  Working on the
> documentation
> > > is ,IMHO, a perfect task for novice kernel hacker to get familiar
> with
> > > the process.
> > 
> > From a language maintainer's perspective documentation patches
> wouldn't be any
> > different than any other patches.  Translate the description and
> questions
> > going back and forth.  The patch itself doesn't get translated when
> it's C
> > and applies to scripts/kconfig/, why would it when it's UTF-8 and
> applies to
> > Documentation/?
> > 
> > Of course this brings up the question "what kind of function and
> variable
> > names do chinese people pick?"  (I honestly don't know, but I note
> that
> > attempts to use names that don't fit in 7-bit ascii would probably be
> frowned
> > upon in a big way...)
> > 
> > > It won't be too hard if the work is shared by a community.  Like the
> one
> > > I'm trying to establish.
> > 
> > A list works fine as a point of contact.  I note that in general,
> maintainers
> > are individuals (who delegate like mad, of course), because otherwise
> agenda
> > items languish with everyone thinking it's someone else's
> responsibility.
> 
> Got it.  I can do it as long as it doesn't consume too much time.  :)
> 
> > 
> > http://www.trumanlibrary.org/buckstop.htm
> > 
> > > - Leo
> > >
> > > Advertisement time:
> > > Chinese kernel development community http://zh-kernel.org    :)
> > 
> > Hmmm...  Now I'm wondering if I should link directly to the docs pages
> of the
> > chinese and japanese sites, or to the top level community pages?
> 
> The documents and maillist are also linked in the front page.  It's
> better to link the top level page.
> 
> - Leo

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Wednesday 11 July 2007 5:53:54 pm Valdis.Kletnieks@vt.edu wrote:
> On Wed, 11 Jul 2007 14:13:02 EDT, Rob Landley said:
> > I wouldn't discourage a translator into Klingon if they were willing to
> > keep their translation up to date and/or it actually resulted in patches.
>
> The guys at the Klingon Language Institute are going to have a fit - what's
> the Klingon word for 'freezer' as used in our suspend code? :)

Considering that Klingon didn't have "to be" before Star Trek 6 needed hamlet, 
I suspect they'll manage somehow.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Thursday 12 July 2007 9:53:54 am Li Yang-r58472 wrote:
> > Fielding patches and questions sounds like plenty to me...)
>
> I do think the documentation translation is very necessary even when
> there is a language maintainer, especially for the policy documents as
> HOWTO, codestyle , and etc.  The contributors should go through these
> policies and check their code for compliance before going to the
> language maintainer for help, or there will be too much for the language
> maintainer to translate.  The language maintainer doesn't need to
> translate all the documents himself, but he can help to coordinate the
> translation effort and help to make it update to date.

It would help if all the policy documents got grouped into a single 
Documentation/development directory so we could separate "policy documents in 
each language would be nice" from "that document about the amiga zorro bus 
really needs to be kept up-to-date in Navajo and that should be in the kernel 
tarball please".

Lemme see, which ones are we talking about?  The candidates are:
  applying-patches.txt
  BUG-HUNTING
  Changes
  CodingStyle
  debugging-modules.txt
  feature-removal-schedule.txt
  HOWTO
  kernel-docs.txt
  language-maintainers.txt
  ManagementStyle
  oops-tracing.txt
  SecurityBugs
  sparse.txt
  stable_api_nonsense.txt
  stable_kernel_rules.txt
  SubmitChecklist
  SubmittingDrivers
  SubmittingPatches
  volatile-considered-harmful.txt

That's everything I noticed in the top level directory that's a good candidate 
to be grouped into a "development" subdirectory.  Did I miss anything?

I note that Changes is a bit stale in places (16 bit assembly?), 
feature-removal-schedule.txt changes often but is good to know, 
kernel-docs.txt might be useless to translate considering it's mostly links 
to english documentation, language-maintainers.txt is assuming my patch from 
earlier today gets accepted...

I can submit a patch grouping all that stuff together into a subdirectory if 
it would help...

> If we do need a contact person, I can do it.  However I don't think
> there will be much translation work to do here.  As I stated before,
> most Chinese programmers are more or less capable of read/write
> technical English.  The difficult part is to let them know the benefit
> of merging code in kernel and teach them how to do it.  That's why the
> policy documents in native language will be very useful.

Does the above look like a good list?  There are more that need to be written, 
but that's what I saw in Documentation...

Rob

P.S.  Dear kmail/postfix developers: having a transient DNS lookup failure on 
one address in a long cc: list is _NOT_ a reason to have the message stay in 
the kmail outbox.  It should go into the postfix send queue and be retried 
from there.  Right now, if I tell it to resend, how do I know who it has and 
hasn't been successfully distributed to on the first attempt?  I've gotten 
dinged for trimming the cc: list before, but now I'm about to send out 
duplicates.  Wheee...
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: [PATCH] Chinese Language Maintainer

[Geert Uytterhoeven]

On Thu, 12 Jul 2007, Rob Landley wrote:
> --- /dev/null	2007-04-23 11:59:00.000000000 -0400
> +++ linux-2.6/Documentation/language-maintainers.txt	2007-07-12 11:49:20.000000000 -0400
> @@ -0,0 +1,20 @@
> +The langauges in which the Linux kernel is developed are C and English.
       ^^^^^^^^^
       languages

> +(Note that Linus Torvalds' native language is Swedish, yet he chose
> +to hold technical discussions in English.)
> +
> +The job of a language maintainer is to provide a point of contact for
> +non-english speaking developers who wish to merge their patches into the
> +Linux kernel.  Each language needs a specific language maintainer, who
> +accepts non-english patch submissions on behalf of the Linux kernel.
           ^^^^^^^^^^^
	   non-English
> +
> +A language maintainer accepts patches to the Linux kernel, written in C, from
> +authors who do not also speak English.  The language maintainer translates the

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: [PATCH] Chinese Language Maintainer

[Pavel Machek]

Hi!

> @@ -0,0 +1,20 @@
> +The langauges in which the Linux kernel is developed are C and English.
> +(Note that Linus Torvalds' native language is Swedish, yet he chose

choose?

> +A language maintainer accepts patches to the Linux kernel, written in C, from
> +authors who do not also speak English.  The language maintainer translates the
> +description of each patch into English, forwards the patches to linux-kernel
> +and to the appropriate maintainers for inclusion in the Linux kernel, and
> +translates questions and replies about such patches as part of the
> +patch review process.

Poor person... I'm not sure this can work for non-trivial patches.

							Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Documentation of kernel messages (Summary)

[Tsugikazu Shibata]

On Thu, 12 Jul 2007 13:35:54 -0400, rob wrote:
> On Thursday 12 July 2007 9:53:54 am Li Yang-r58472 wrote:
> > > Fielding patches and questions sounds like plenty to me...)
> >
> > I do think the documentation translation is very necessary even when
> > there is a language maintainer, especially for the policy documents as
> > HOWTO, codestyle , and etc.  The contributors should go through these
> > policies and check their code for compliance before going to the
> > language maintainer for help, or there will be too much for the language
> > maintainer to translate.  The language maintainer doesn't need to
> > translate all the documents himself, but he can help to coordinate the
> > translation effort and help to make it update to date.
> 
> It would help if all the policy documents got grouped into a single 
> Documentation/development directory so we could separate "policy documents in 
> each language would be nice" from "that document about the amiga zorro bus 
> really needs to be kept up-to-date in Navajo and that should be in the kernel 
> tarball please".
> 
> Lemme see, which ones are we talking about?  The candidates are:
>   applying-patches.txt
>   BUG-HUNTING
>   Changes
>   CodingStyle
>   debugging-modules.txt
This file seems mostly technical. may not policy related document.

>   feature-removal-schedule.txt
>   HOWTO
>   kernel-docs.txt
>   language-maintainers.txt
>   ManagementStyle
>   oops-tracing.txt
>   SecurityBugs
>   sparse.txt
>   stable_api_nonsense.txt
>   stable_kernel_rules.txt
>   SubmitChecklist
>   SubmittingDrivers
>   SubmittingPatches
>   volatile-considered-harmful.txt

How about adding; 
kernel-doc-nano-HOWTO.txt

These three slightly include some policy in the documents but purpose
are mostly technical. (So, it's just comment)
devices.txt
kernel-parameters.txt
unicode.txt

> That's everything I noticed in the top level directory that's a good candidate 
> to be grouped into a "development" subdirectory.  Did I miss anything?
> 
> I note that Changes is a bit stale in places (16 bit assembly?), 
> feature-removal-schedule.txt changes often but is good to know, 
> kernel-docs.txt might be useless to translate considering it's mostly links 
> to english documentation, language-maintainers.txt is assuming my patch from 
> earlier today gets accepted...
> 
> I can submit a patch grouping all that stuff together into a subdirectory if 
> it would help...
> 
> > If we do need a contact person, I can do it.  However I don't think
> > there will be much translation work to do here.  As I stated before,
> > most Chinese programmers are more or less capable of read/write
> > technical English.  The difficult part is to let them know the benefit
> > of merging code in kernel and teach them how to do it.  That's why the
> > policy documents in native language will be very useful.
> 
> Does the above look like a good list?  There are more that need to be written, 
> but that's what I saw in Documentation...
> 
> Rob
> 
> P.S.  Dear kmail/postfix developers: having a transient DNS lookup failure on 
> one address in a long cc: list is _NOT_ a reason to have the message stay in 
> the kmail outbox.  It should go into the postfix send queue and be retried 
> from there.  Right now, if I tell it to resend, how do I know who it has and 
> hasn't been successfully distributed to on the first attempt?  I've gotten 
> dinged for trimming the cc: list before, but now I'm about to send out 
> duplicates.  Wheee...

Re: [PATCH] Chinese Language Maintainer

[Geert Uytterhoeven]

On Thu, 12 Jul 2007, Pavel Machek wrote:
> > @@ -0,0 +1,20 @@
> > +The langauges in which the Linux kernel is developed are C and English.
> > +(Note that Linus Torvalds' native language is Swedish, yet he chose
> 
> choose?

>From `dict choose':
  Choose \Choose\, v. t. [imp. {Chose}; p. p. {Chosen}, {Chose}
     (Obs.); p. pr. & vb. n. {Choosing}.] [OE. chesen, cheosen,
     AS. ce['o]san; akin to OS. kiosan, D. kiezen, G. kiesen,
     Icel. kj[=o]sa, Goth. kiusan, L. gustare to taste, Gr. ?,
     Skr. jush to enjoy. [root]46. Cf. {Choice}, 2d {Gust}.]

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: Documentation of kernel messages (Summary)

[Alan Cox]

> One thing that would be a VERY good idea is to make sure that each
> translated document is tagged with when it was last translated, and
> against which kernel version (using either a GIT commit id or a

The English versions need a last updated too, that way we would know when
they are past their best before date (as most of them are)

Re: Documentation of kernel messages (Summary)

[Li Yang]

On Thu, 2007-07-12 at 13:35 -0400, Rob Landley wrote:
> On Thursday 12 July 2007 9:53:54 am Li Yang-r58472 wrote:
> > > Fielding patches and questions sounds like plenty to me...)
> >
> > I do think the documentation translation is very necessary even when
> > there is a language maintainer, especially for the policy documents as
> > HOWTO, codestyle , and etc.  The contributors should go through these
> > policies and check their code for compliance before going to the
> > language maintainer for help, or there will be too much for the language
> > maintainer to translate.  The language maintainer doesn't need to
> > translate all the documents himself, but he can help to coordinate the
> > translation effort and help to make it update to date.
> 
> It would help if all the policy documents got grouped into a single 
> Documentation/development directory so we could separate "policy documents in 
> each language would be nice" from "that document about the amiga zorro bus 
> really needs to be kept up-to-date in Navajo and that should be in the kernel 
> tarball please".
> 
> Lemme see, which ones are we talking about?  The candidates are:
>   applying-patches.txt
>   BUG-HUNTING
>   Changes
>   CodingStyle
>   debugging-modules.txt
>   feature-removal-schedule.txt
>   HOWTO
>   kernel-docs.txt
>   language-maintainers.txt
>   ManagementStyle
>   oops-tracing.txt
>   SecurityBugs
>   sparse.txt
>   stable_api_nonsense.txt
>   stable_kernel_rules.txt
>   SubmitChecklist
>   SubmittingDrivers
>   SubmittingPatches
>   volatile-considered-harmful.txt
> 
> That's everything I noticed in the top level directory that's a good candidate 
> to be grouped into a "development" subdirectory.  Did I miss anything?
> 
> I note that Changes is a bit stale in places (16 bit assembly?), 
> feature-removal-schedule.txt changes often but is good to know, 
> kernel-docs.txt might be useless to translate considering it's mostly links 
> to english documentation, language-maintainers.txt is assuming my patch from 
> earlier today gets accepted...
> 
> I can submit a patch grouping all that stuff together into a subdirectory if 
> it would help...
> 
> > If we do need a contact person, I can do it.  However I don't think
> > there will be much translation work to do here.  As I stated before,
> > most Chinese programmers are more or less capable of read/write
> > technical English.  The difficult part is to let them know the benefit
> > of merging code in kernel and teach them how to do it.  That's why the
> > policy documents in native language will be very useful.
> 
> Does the above look like a good list?  There are more that need to be written, 
> but that's what I saw in Documentation...

Yes, I do think this is a good list for translation.

Greg,
Do you have anything to add?

- Leo

Re: [PATCH] Chinese Language Maintainer

[Li Yang]

On Thu, 2007-07-12 at 12:05 -0400, Rob Landley wrote:
> Proposed patch adding Li Yang to MAINTAINERS, and Documentation
> describing what a language maintainer is.
> 
> Signed-off-by: Rob Landley <rob@landley.net>

Signed-off-by: Li Yang <leo@zh-kernel.org>
> 
> --
> 
> On Thursday 12 July 2007 9:53:54 am Li Yang-r58472 wrote:
> 
> > > > I'm trying to establish.
> > >
> > > A list works fine as a point of contact.  I note that in general,
> > maintainers
> > > are individuals (who delegate like mad, of course), because otherwise
> > agenda
> > > items languish with everyone thinking it's someone else's
> > responsibility.
> >
> > Got it.  I can do it as long as it doesn't consume too much time.  :)
> 
> Woot!  See attached patch and sign-off-by if you want the job. :)
> 
> The bit about "the maintainer forwards, not the list" is a suggestion to
> prevent duplicates.  It doesn't mean you can't delegate, just that if it's
> open season on the list translating stuff _and_ forwarding it, then
> linux-kernel will get dupes.  (Plus other maintainers are more likely
> to pay attention to somebody they've heard from before.)
> 
> Greg KH had entire an OLS presentation about how the developer ->
> maintainer -> subsystem -> anderew+linus forwarding is more
> an ideal than a reality, but at least it gives us a frame of reference
> to diverge from:
> http://lwn.net/Articles/240402/
> 
> diff -r edfd2d6f670d MAINTAINERS
> --- a/MAINTAINERS	Tue Jul 10 17:51:13 2007 -0700
> +++ b/MAINTAINERS	Thu Jul 12 11:51:19 2007 -0400
> @@ -2146,6 +2146,13 @@ W:	http://auxdisplay.googlepages.com/
>  W:	http://auxdisplay.googlepages.com/
>  S:	Maintained
>  
> +LANGUAGE (CHINESE)
> +P:	Li Yang
> +M:	LeoLi@freescale.com
> +L:	linux-kernel@zh-kernel.org
> +W:	http://zh-kernel.org
> +S:	Maintained
> +
>  LAPB module
>  L:	linux-x25@vger.kernel.org
>  S:	Orphan
> --- /dev/null	2007-04-23 11:59:00.000000000 -0400
> +++ linux-2.6/Documentation/language-maintainers.txt	2007-07-12 11:49:20.000000000 -0400
> @@ -0,0 +1,20 @@
> +The langauges in which the Linux kernel is developed are C and English.
> +(Note that Linus Torvalds' native language is Swedish, yet he chose
> +to hold technical discussions in English.)
> +
> +The job of a language maintainer is to provide a point of contact for
> +non-english speaking developers who wish to merge their patches into the
> +Linux kernel.  Each language needs a specific language maintainer, who
> +accepts non-english patch submissions on behalf of the Linux kernel.
> +
> +A language maintainer accepts patches to the Linux kernel, written in C, from
> +authors who do not also speak English.  The language maintainer translates the
> +description of each patch into English, forwards the patches to linux-kernel
> +and to the appropriate maintainers for inclusion in the Linux kernel, and
> +translates questions and replies about such patches as part of the
> +patch review process.

In addiction to this responsibility, I would like to add two more which,
in my opinion, are more important.  And these are what I'm trying to
do.  :)
First, promoting contribution to Linux kernel in local language.
Second, coordinate the translation effort of key kernel documents.

> +
> +Some language maintainers provide a mailing list as a point of contact, to
> +distribute the translation work, but the maintainer is still the person who
> +ultimately forwards the results (to prevent duplicates), and the one to contact
> +if patches and questions don't get translated and forwarded in a timely fashion.
> 

Re: Documentation of kernel messages (Summary)

[Li Yang]

On Fri, 2007-07-13 at 11:53 +0100, Alan Cox wrote:
> > One thing that would be a VERY good idea is to make sure that each
> > translated document is tagged with when it was last translated, and
> > against which kernel version (using either a GIT commit id or a
> 
> The English versions need a last updated too, that way we would know when
> they are past their best before date (as most of them are)

Good idea.  So a "last updated" date field will be required for both the
English and translated documents.  Date will be more straight forward
than using commit id, IMHO.

- Leo

Re: Documentation of kernel messages (Summary)

[Stefan Richter]

Alan Cox wrote:
> The English versions need a last updated too, that way we would know when
> they are past their best before date (as most of them are)

I've got the impression that whenever I saw a "last updated:" note in a
documentation, this note was out of date.
-- 
Stefan Richter
-=====-=-=== -=== -==-=
http://arcgraph.de/sr/

Re: Documentation of kernel messages (Summary)

[Theodore Tso]

On Fri, Jul 13, 2007 at 08:49:00PM +0800, Li Yang wrote:
> On Fri, 2007-07-13 at 11:53 +0100, Alan Cox wrote:
> > > One thing that would be a VERY good idea is to make sure that each
> > > translated document is tagged with when it was last translated, and
> > > against which kernel version (using either a GIT commit id or a
> > 
> > The English versions need a last updated too, that way we would know when
> > they are past their best before date (as most of them are)
> 
> Good idea.  So a "last updated" date field will be required for both the
> English and translated documents.  Date will be more straight forward
> than using commit id, IMHO.

Well, date and kernel version id, then.  Otherwise they may be
confusion if the translator is using 2.6.16.35 as opposed to
2.6.22-git3, etc.  I would hope that the translator would always be
using only the latest development branch, but I wouldn't want to
always bet on that.  One advantage of using git commit id is that
there's no question about what the source document version was for the
translation.  It also makes it easier for the translator since they
can just diff between the last git commit id, and HEAD, and then
update their translation accordingly.

							- Ted

Re: [PATCH] Chinese Language Maintainer

[Rob Landley]

On Thursday 12 July 2007 4:02:47 pm Pavel Machek wrote:
> Hi!
>
> > @@ -0,0 +1,20 @@
> > +The langauges in which the Linux kernel is developed are C and English.
> > +(Note that Linus Torvalds' native language is Swedish, yet he chose
>
> choose?

I was trying for past tense.

> > +A language maintainer accepts patches to the Linux kernel, written in C,
> > from +authors who do not also speak English.  The language maintainer
> > translates the +description of each patch into English, forwards the
> > patches to linux-kernel +and to the appropriate maintainers for inclusion
> > in the Linux kernel, and +translates questions and replies about such
> > patches as part of the +patch review process.
>
> Poor person... I'm not sure this can work for non-trivial patches.

We'll see.  Personally, I'm more worried about how people who don't speak 
english are supposed to choose function and variable names.  But translating 
_just_ the patch is not the end of it, and I wanted to make that clear.

I suppose a translator can always push back and ask for a _concise_ review, 
for the sake of translation.  But let's not invent problems before they 
actually occur...

> 							Pavel

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Thursday 12 July 2007 10:54:46 pm Tsugikazu Shibata wrote:
> On Thu, 12 Jul 2007 13:35:54 -0400, rob wrote:
> > On Thursday 12 July 2007 9:53:54 am Li Yang-r58472 wrote:
> > > > Fielding patches and questions sounds like plenty to me...)
> > >
> > > I do think the documentation translation is very necessary even when
> > > there is a language maintainer, especially for the policy documents as
> > > HOWTO, codestyle , and etc.  The contributors should go through these
> > > policies and check their code for compliance before going to the
> > > language maintainer for help, or there will be too much for the
> > > language maintainer to translate.  The language maintainer doesn't need
> > > to translate all the documents himself, but he can help to coordinate
> > > the translation effort and help to make it update to date.
> >
> > It would help if all the policy documents got grouped into a single
> > Documentation/development directory so we could separate "policy
> > documents in each language would be nice" from "that document about the
> > amiga zorro bus really needs to be kept up-to-date in Navajo and that
> > should be in the kernel tarball please".
> >
> > Lemme see, which ones are we talking about?  The candidates are:
> >   applying-patches.txt
> >   BUG-HUNTING
> >   Changes
> >   CodingStyle
> >   debugging-modules.txt
>
> This file seems mostly technical. may not policy related document.

Yeah, I guess so.  My mental filter was actually more like "how to do linux 
development", and it seemed both short and easy to translate, and also 
generally relevant to the majority of developers no matter what subsystem 
they're working on.

But I agree, it's not policy.

> >   feature-removal-schedule.txt
> >   HOWTO
> >   kernel-docs.txt
> >   language-maintainers.txt
> >   ManagementStyle
> >   oops-tracing.txt
> >   SecurityBugs
> >   sparse.txt

That one's not policy either, but it is general non-subsystem-specific 
development.  Probably "applying-patches.txt" above goes in the same bucket.

Whether or not to include that bucket in the new directory depends on whether 
the directory is labeled "development" or "policy" or something else...

> >   stable_api_nonsense.txt
> >   stable_kernel_rules.txt
> >   SubmitChecklist
> >   SubmittingDrivers
> >   SubmittingPatches
> >   volatile-considered-harmful.txt
>
> How about adding;
> kernel-doc-nano-HOWTO.txt

The problem is, the generated htmdocs are in english.  This file is about how 
to generate (and author) English documentation that won't be translated.  
What's the point of translating the instructions if the result won't be 
translated?

On the other hand, if the authors of patches _do_ put javadoc comments into 
the source code, the language maintainer should be aware of them and should 
have some easy way to translate them for the list...

> These three slightly include some policy in the documents but purpose
> are mostly technical. (So, it's just comment)
> devices.txt
> kernel-parameters.txt
> unicode.txt

Extracting the policy out into separate documents I could see, but I agree 
those are primarily technical.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: [PATCH] Chinese Language Maintainer

[Rob Landley]

On Friday 13 July 2007 8:43:03 am Li Yang wrote:
> On Thu, 2007-07-12 at 12:05 -0400, Rob Landley wrote:
> > Proposed patch adding Li Yang to MAINTAINERS, and Documentation
> > describing what a language maintainer is.
> >
> > Signed-off-by: Rob Landley <rob@landley.net>
>
> Signed-off-by: Li Yang <leo@zh-kernel.org>

Yay!

> > +A language maintainer accepts patches to the Linux kernel, written in C,
> > from +authors who do not also speak English.  The language maintainer
> > translates the +description of each patch into English, forwards the
> > patches to linux-kernel +and to the appropriate maintainers for inclusion
> > in the Linux kernel, and +translates questions and replies about such
> > patches as part of the +patch review process.
>
> In addiction to this responsibility, I would like to add two more which,
> in my opinion, are more important.  And these are what I'm trying to
> do.  :)
> First, promoting contribution to Linux kernel in local language.
> Second, coordinate the translation effort of key kernel documents.

Cool.  It's good to do that, but not the problem I'm worried about solving.

I was trying to describe the minimum requirements for being a language 
maintainer, I.E. what non-english users need in order to be able to merge 
their patches.  Because without someone to contribute patches to (I.E a 
language maintainer), documentation in non-english languages promotes the 
creation of patches that can't be merged.  That's the problem I'm trying to 
solve.

To me, finding language maintainers is the flip side of the coin of 
translating documentation.

Rob

-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Friday 13 July 2007 6:53:21 am Alan Cox wrote:
> > One thing that would be a VERY good idea is to make sure that each
> > translated document is tagged with when it was last translated, and
> > against which kernel version (using either a GIT commit id or a
>
> The English versions need a last updated too, that way we would know when
> they are past their best before date (as most of them are)

You can get this out of git annotate.  (Or in my case, hg annotate from 
http://kernel.org/hg/linux-2.6 .)

Cleaning up Documentation is on my todo list, but for this month I'm trying to 
integrate the existing Documentation files with things like the OLS papers, 
linux journal articles, lwn.net kernel articles, man-pages, and so on into 
one big index.  (Indexing it requires reading large chunks of it.  My brain 
hurts.)

If there's interest, I can push some patches to clean up Documentation by 
moving files into subdirectories, but Documentation's not well-suited to link 
out to the web.  (You need html for that, and it's text.)

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 571 bytes --]

On Fri, 13 Jul 2007 15:25:49 +0200, Stefan Richter said:
> Alan Cox wrote:
> > The English versions need a last updated too, that way we would know when
> > they are past their best before date (as most of them are)
> 
> I've got the impression that whenever I saw a "last updated:" note in a
> documentation, this note was out of date.

RCS and SCCS let you stick stuff like $DATE and $VERSION into the source, so
it would be automagically replaced at checkin/checkout time.  Not sure how
well that would work across a mesh of 'git pull' and 'git push' across trees. :)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Documentation of kernel messages (Summary)

[Tsugikazu Shibata]

On Fri, 13 Jul 2007 13:12:27 -0400, rob wrote:
> On Thursday 12 July 2007 10:54:46 pm Tsugikazu Shibata wrote:
> > On Thu, 12 Jul 2007 13:35:54 -0400, rob wrote:
> > > On Thursday 12 July 2007 9:53:54 am Li Yang-r58472 wrote:
> > > > > Fielding patches and questions sounds like plenty to me...)
> > > >
> > > > I do think the documentation translation is very necessary even when
> > > > there is a language maintainer, especially for the policy documents as
> > > > HOWTO, codestyle , and etc.  The contributors should go through these
> > > > policies and check their code for compliance before going to the
> > > > language maintainer for help, or there will be too much for the
> > > > language maintainer to translate.  The language maintainer doesn't need
> > > > to translate all the documents himself, but he can help to coordinate
> > > > the translation effort and help to make it update to date.
> > >
> > > It would help if all the policy documents got grouped into a single
> > > Documentation/development directory so we could separate "policy
> > > documents in each language would be nice" from "that document about the
> > > amiga zorro bus really needs to be kept up-to-date in Navajo and that
> > > should be in the kernel tarball please".
> > >
> > > Lemme see, which ones are we talking about?  The candidates are:
> > >   applying-patches.txt
> > >   BUG-HUNTING
> > >   Changes
> > >   CodingStyle
> > >   debugging-modules.txt
> >
> > This file seems mostly technical. may not policy related document.
> 
> Yeah, I guess so.  My mental filter was actually more like "how to do linux 
> development", and it seemed both short and easy to translate, and also 
> generally relevant to the majority of developers no matter what subsystem 
> they're working on.
> 
> But I agree, it's not policy.
> 
> > >   feature-removal-schedule.txt
> > >   HOWTO
> > >   kernel-docs.txt
> > >   language-maintainers.txt
> > >   ManagementStyle
> > >   oops-tracing.txt
> > >   SecurityBugs
> > >   sparse.txt
> 
> That one's not policy either, but it is general non-subsystem-specific 
> development.  Probably "applying-patches.txt" above goes in the same bucket.
> 
> Whether or not to include that bucket in the new directory depends on whether 
> the directory is labeled "development" or "policy" or something else...

Agree.

> > >   stable_api_nonsense.txt
> > >   stable_kernel_rules.txt
> > >   SubmitChecklist
> > >   SubmittingDrivers
> > >   SubmittingPatches
> > >   volatile-considered-harmful.txt
> >
> > How about adding;
> > kernel-doc-nano-HOWTO.txt
> 
> The problem is, the generated htmdocs are in english.  This file is about how 
> to generate (and author) English documentation that won't be translated.  
> What's the point of translating the instructions if the result won't be 
> translated?

People (who even in non-native) would better to know and read it such
htmldocs because there are good and important documents even in English.
I thought that translation of this file may help.

> On the other hand, if the authors of patches _do_ put javadoc comments into 
> the source code, the language maintainer should be aware of them and should 
> have some easy way to translate them for the list...

maybe.

> 
> > These three slightly include some policy in the documents but purpose
> > are mostly technical. (So, it's just comment)
> > devices.txt
> > kernel-parameters.txt
> > unicode.txt
> 
> Extracting the policy out into separate documents I could see, but I agree 
> those are primarily technical.

Further comment on this is that we may need to have another 00-INDEX
file, for this new directory.

> 
> Rob
> -- 
> "One of my most productive days was throwing away 1000 lines of code."
>   - Ken Thompson.
> 

Re: Documentation of kernel messages (Summary)

[Randy Dunlap]

On Fri, 13 Jul 2007 14:05:49 -0400 Rob Landley wrote:

> Cleaning up Documentation is on my todo list, but for this month I'm trying to 
> integrate the existing Documentation files with things like the OLS papers, 
> linux journal articles, lwn.net kernel articles, man-pages, and so on into 
> one big index.  (Indexing it requires reading large chunks of it.  My brain 
> hurts.)
> 
> If there's interest, I can push some patches to clean up Documentation by 
> moving files into subdirectories, but Documentation's not well-suited to link 
> out to the web.  (You need html for that, and it's text.)

I think that you should start moving Documentation/ files around
and adding subdirectories -- if you are pretty sure of where they
should go (i.e., they won't likely be moved again later on).

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Friday 13 July 2007 11:54:41 pm Randy Dunlap wrote:
> > If there's interest, I can push some patches to clean up Documentation by
> > moving files into subdirectories, but Documentation's not well-suited to
> > link out to the web.  (You need html for that, and it's text.)
>
> I think that you should start moving Documentation/ files around
> and adding subdirectories -- if you are pretty sure of where they
> should go (i.e., they won't likely be moved again later on).

You mean like these?
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Friday 13 July 2007 9:46:59 pm Tsugikazu Shibata wrote:
> > > How about adding;
> > > kernel-doc-nano-HOWTO.txt
> >
> > The problem is, the generated htmdocs are in english.  This file is about
> > how to generate (and author) English documentation that won't be
> > translated. What's the point of translating the instructions if the
> > result won't be translated?
>
> People (who even in non-native) would better to know and read it such
> htmldocs because there are good and important documents even in English.
> I thought that translation of this file may help.

The english version of htmldocs is generated from the source code.    If 
translating remotely current versions of htmldocs into other languages was 
feasible, then the english version wouldn't need to be in the source code in 
the first place.

Translating a document _about_ htmldocs boils down to "Dear non-english 
speaker: here's something you can't read, nor can you update it either except 
though your language maintainer."

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: [PATCH] Chinese Language Maintainer

[Li Yang]

On 7/14/07, Rob Landley <rob@landley.net> wrote:
> On Friday 13 July 2007 8:43:03 am Li Yang wrote:
> > On Thu, 2007-07-12 at 12:05 -0400, Rob Landley wrote:
> > > +A language maintainer accepts patches to the Linux kernel, written in C,
> > > from +authors who do not also speak English.  The language maintainer
> > > translates the +description of each patch into English, forwards the
> > > patches to linux-kernel +and to the appropriate maintainers for inclusion
> > > in the Linux kernel, and +translates questions and replies about such
> > > patches as part of the +patch review process.
> >
> > In addiction to this responsibility, I would like to add two more which,
> > in my opinion, are more important.  And these are what I'm trying to
> > do.  :)
> > First, promoting contribution to Linux kernel in local language.
> > Second, coordinate the translation effort of key kernel documents.
>
> Cool.  It's good to do that, but not the problem I'm worried about solving.
>
> I was trying to describe the minimum requirements for being a language
> maintainer, I.E. what non-english users need in order to be able to merge
> their patches.  Because without someone to contribute patches to (I.E a
> language maintainer), documentation in non-english languages promotes the
> creation of patches that can't be merged.  That's the problem I'm trying to
> solve.
>
> To me, finding language maintainers is the flip side of the coin of
> translating documentation.

I think you worried too much about this problem.  :)  Let me explain
the situation here in China more clearly.  Actually, English is
mandatory in most schools and universities.  Only very few people
learn other language as a second language.  Therefore software
developers who are almost educated should have the basic English
skill.  However, that doesn't mean that they can read English or
communicate with native English speaker very easily.  Consider your
second language learn in school for analogy.  Read in English will be
much slower and more likely to cause misunderstanding.  This will
reduce the likelihood greatly of English documentation being read.  If
we are promoting contribution to the Linux community, we should
maximum the possibility that these key documents being read.
Translation will serve this purpose very well.

So the possibility is very little that a translator is needed between
the Linux maintainer and a Chinese developer.  Although sometimes help
is needed when there is misunderstanding.

After a brief talk with the Japanese translator, I think the case is
similar for Japanese too.

Therefore, in my opinion, language maintainer should be more a helper
and promoter rather than a gatekeeper.  I will give a proposed process
later about how this helper mechanism works.

- Leo

Re: Documentation of kernel messages (Summary)

[Randy Dunlap]

On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:

> On Friday 13 July 2007 11:54:41 pm Randy Dunlap wrote:
> > > If there's interest, I can push some patches to clean up Documentation by
> > > moving files into subdirectories, but Documentation's not well-suited to
> > > link out to the web.  (You need html for that, and it's text.)
> >
> > I think that you should start moving Documentation/ files around
> > and adding subdirectories -- if you are pretty sure of where they
> > should go (i.e., they won't likely be moved again later on).
> 
> You mean like these?
> http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
> http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html

Yes.  Have any of these been merged?

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

Re: Documentation of kernel messages (Summary)

[Tsugikazu Shibata]

On Sat, 14 Jul 2007 22:12:35 -0400, rob wrote:
> On Friday 13 July 2007 9:46:59 pm Tsugikazu Shibata wrote:
> > > > How about adding;
> > > > kernel-doc-nano-HOWTO.txt
> > >
> > > The problem is, the generated htmdocs are in english.  This file is about
> > > how to generate (and author) English documentation that won't be
> > > translated. What's the point of translating the instructions if the
> > > result won't be translated?
> >
> > People (who even in non-native) would better to know and read it such
> > htmldocs because there are good and important documents even in English.
> > I thought that translation of this file may help.
> 
> The english version of htmldocs is generated from the source code.    If 
> translating remotely current versions of htmldocs into other languages was 
> feasible, then the english version wouldn't need to be in the source code in 
> the first place.
> 
> Translating a document _about_ htmldocs boils down to "Dear non-english 
> speaker: here's something you can't read, nor can you update it either except 
> though your language maintainer."

No, kernel-doc-nano-HOWTO includes explanation of tools to extract the
document, format of extractable document in source files and so on.
I thought this file would help non-native people, how to extract or add
in-kernel extractable documents.
I understand that these tools will generate English documents but
that's OK. Such large volume of documents are not easy to translate
into other languages.

> 
> Rob
> -- 
> "One of my most productive days was throwing away 1000 lines of code."
>   - Ken Thompson.
> 

Re: [PATCH] Chinese Language Maintainer

[H. Peter Anvin]

Li Yang wrote:
> 
> I think you worried too much about this problem.  :)  Let me explain
> the situation here in China more clearly.  Actually, English is
> mandatory in most schools and universities.  Only very few people
> learn other language as a second language.  Therefore software
> developers who are almost educated should have the basic English
> skill.  However, that doesn't mean that they can read English or
> communicate with native English speaker very easily.  Consider your
> second language learn in school for analogy.

Actually, I disagree.  English *is* the second language learned in
school for most European developers (except, obviously, the ones from
the British isles), and we don't have that problem.

> Read in English will be much slower and more likely to cause
> misunderstanding.  This will
> reduce the likelihood greatly of English documentation being read.  If
> we are promoting contribution to the Linux community, we should
> maximum the possibility that these key documents being read.
> Translation will serve this purpose very well.

What we have found in Europe, is that that it has limited value, and
that the closer to the core you are, the less value it is, because at
that stage you should be communicating more with other developers.
Putting yourself behind a wall of translation is unfortunately a
detriment in that way.

	-hpa

Re: [PATCH] Chinese Language Maintainer

[Alan Cox]

O> Actually, I disagree.  English *is* the second language learned in
> school for most European developers (except, obviously, the ones from
> the British isles), and we don't have that problem.

Not all those from the British Isles. A first language an English as
school language is not that uncommon for segments of the population, and
in Wales schooling may also be such that English is learned as a first
foreign language.

That aside, please remember that Europe as a whole is a small place on the
bigger world stage. The total volume of potential developers in the huge
and rapidly modernising nations like India and China is vast, and there
are large highly skilled technical nations that don't teach English to
everyone technical by default.

Key documents in other languages is a big help, especially those about
values, culture and standards because they are things that are not easy
to understand if your use of English is technically oriented.

Alan

Re: [PATCH] Chinese Language Maintainer

[Li Yang]

On 7/16/07, H. Peter Anvin <hpa@zytor.com> wrote:
> Li Yang wrote:
> >
> > I think you worried too much about this problem.  :)  Let me explain
> > the situation here in China more clearly.  Actually, English is
> > mandatory in most schools and universities.  Only very few people
> > learn other language as a second language.  Therefore software
> > developers who are almost educated should have the basic English
> > skill.  However, that doesn't mean that they can read English or
> > communicate with native English speaker very easily.  Consider your
> > second language learn in school for analogy.
>
> Actually, I disagree.  English *is* the second language learned in
> school for most European developers (except, obviously, the ones from
> the British isles), and we don't have that problem.

Then this could be the main reason for the disagreement we have on
this topic.  Many people here don't grasp English very well even after
the English classes.

- Leo

Re: [PATCH] Chinese Language Maintainer

[H. Peter Anvin]

Alan Cox wrote:
> O> Actually, I disagree.  English *is* the second language learned in
>> school for most European developers (except, obviously, the ones from
>> the British isles), and we don't have that problem.
> 
> Not all those from the British Isles. A first language an English as
> school language is not that uncommon for segments of the population, and
> in Wales schooling may also be such that English is learned as a first
> foreign language.
> 
> That aside, please remember that Europe as a whole is a small place on the
> bigger world stage. The total volume of potential developers in the huge
> and rapidly modernising nations like India and China is vast, and there
> are large highly skilled technical nations that don't teach English to
> everyone technical by default.
> 
> Key documents in other languages is a big help, especially those about
> values, culture and standards because they are things that are not easy
> to understand if your use of English is technically oriented.

No doubt.  My point was merely that the closer to the core of
development, the less translated documents help as the emphasis on
interaction *has* to increase.

In that sense, a translation of "Linux Kernel Internals" and the other
books written on the Linux kernel is more useful.  That doesn't mean, of
course, that there isn't documentation distributed with the kernel that
is intended for users and therefore more useful in translation.
However, I do feel that trying to keep up-to-date translations of design
documentation is at least to some degree a fool's errand, which ends up
having people rely on incomplete and outdated documentation, and cut
them off from the overall community.

	-hpa

Re: [PATCH] Chinese Language Maintainer

[Rik van Riel]

H. Peter Anvin wrote:
> Alan Cox wrote:

>> Key documents in other languages is a big help, especially those about
>> values, culture and standards because they are things that are not easy
>> to understand if your use of English is technically oriented.

> In that sense, a translation of "Linux Kernel Internals" and the other
> books written on the Linux kernel is more useful. 

Several groups of people are translating the "important bits"
(a different selection for each culture) of the kernelnewbies
documentation to different languages already:

http://kernelnewbies.org/RegionalNewbies

I can easily set up additional wikis for anybody who wants
to set up such a kernel documentation site in their language.

-- 
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

Re: [PATCH] Chinese Language Maintainer

[Rene Herman]

On 07/15/2007 08:12 PM, H. Peter Anvin wrote:

> Li Yang wrote:

>> I think you worried too much about this problem. :) Let me explain the
>> situation here in China more clearly. Actually, English is mandatory in
>> most schools and universities. Only very few people learn other
>> language as a second language. Therefore software developers who are
>> almost educated should have the basic English skill. However, that
>> doesn't mean that they can read English or communicate with native
>> English speaker very easily. Consider your second language learn in
>> school for analogy
> 
> Actually, I disagree.  English *is* the second language learned in school
> for most European developers (except, obviously, the ones from the
> British isles), and we don't have that problem.

There is somewhat of a difference though: English and both our native 
languages are very much related. English, Dutch (my native language) and 
Swedish (yours) are all Germanic languages -- English and Dutch both West 
Germanic and Swedish only slightly farther removed as North Germanic. Word 
for word translations not _too_ infrequently actually make okay or at least 
understandable Dutch...

Other than Germanic, the Slavic (Polish, Czech, Russian, ...) and Latin 
(Italian, Spanish, French, ...) language families are two other direct 
Indo-European descendants that are fairly well represented in the kernel 
community but I believe it's a largely unsurprising observation that members 
of both these families have a somewhat harder time adopting English. And 
Chinese is not even Indo-European...

However -- in Europe I notice culture might be even more important than 
school _or_ language family is and as such I believe the above argument 
isn't all that important anyway. English is the second language we learn in 
school here in the Netherlands but it's much more popular than say German, a 
language even closer to Dutch, due to us having heard English basically from 
the time we're old enough to hear music and watch TV. Most _Dutch_ bands 
sing in English and the ones that don't are for a part targetting the 
elderly and/or mentally handicapped...

German is close enough to Dutch (and English itself) to rule out most 
differences related to native language, but in Germany a significantly 
smaller percentage of people speaks English well enough to be comfortable 
with it than in the Netherlands simply due to them producing more of their 
own culture. It's a larger language zone to market to and they dub the 
remaining English-language content on TV. Over here in the Netherlands we 
sub-title.

France is another good example. While a bit farther removed from English 
family-wise, English has had lots of influences from French as well and in 
any case, learning English shouldn't be harder for a Frenchman than learning 
French is for a Dutchman (we are, or were when I was there, taught French in 
school as well) which is to say not very hard. Yet, mastery of English is 
extremely poor in France. Not as a coincedence, most _all_ of the French 
culture is in the French language including dubbed originally English songs 
for example.

Both the German and, slower, French examples get less true with every new 
generation, but both still hold...

Popular culture in the sense of music, tv and these days very much games is 
something you start to experience at a very young age, years younger than 
the Chinese will be taught English in school and age is extremely important 
in mastering a language -- the human brain is by far best at it at the time 
you start mastering your native language (in fact, this is what defines 
"native") and every single year after that makes it harder.

That is -- let's just solve our Chinese translation problem by overthrowing 
the Chinese government and forcing the sub-titled Harry Potter film-series 
down the throat of the population... |-/

Seriously, I only wanted to say that both language family and (import of) 
culture are very important and as such, concentrating on schooling alone 
might not be all that sensible. I did not want to say that I feel that all 
these translations make a great deal of sense. Some of us have an easier 
time learning English than others do, the Chinese probably don't have an 
easy time at all, but a single common language is still the thing to aim 
for. The subset and semblance of English spoken on this list seems like 
something that should serve well as that common language, especially given 
the _help_ English language education gives in it. Also, I hear China is in 
fact fairly rapidly opening up to Western (American) popular culture which 
might be an argument if you have a generation or two to spare...

Apologies for ranting...

Cheers,
Rene.

Re: Documentation of kernel messages (Summary)

[Rik van Riel]

Li Yang wrote:


> Yes, I do think this is a good list for translation.
> 
> Greg,
> Do you have anything to add?

There are a few documents not in the kernel tree that you
may want to translate too, if your goal is to increase the
number of kernel developers:

http://kernelnewbies.org/KernelGlossary

http://kernelnewbies.org/FAQ

http://kernelnewbies.org/UpstreamMerge

-- 
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is.  Each group
calls the other unpatriotic.

Re: [PATCH] Chinese Language Maintainer

[Tsugikazu Shibata]

On Sun, 15 Jul 2007 22:42:07 +0800, leo wrote:
> On 7/14/07, Rob Landley <rob@landley.net> wrote:
> > On Friday 13 July 2007 8:43:03 am Li Yang wrote:
> > > On Thu, 2007-07-12 at 12:05 -0400, Rob Landley wrote:
> > > > +A language maintainer accepts patches to the Linux kernel, written in C,
> > > > from +authors who do not also speak English.  The language maintainer
> > > > translates the +description of each patch into English, forwards the
> > > > patches to linux-kernel +and to the appropriate maintainers for inclusion
> > > > in the Linux kernel, and +translates questions and replies about such
> > > > patches as part of the +patch review process.
> > >
> > > In addiction to this responsibility, I would like to add two more which,
> > > in my opinion, are more important.  And these are what I'm trying to
> > > do.  :)
> > > First, promoting contribution to Linux kernel in local language.
> > > Second, coordinate the translation effort of key kernel documents.
> >
> > Cool.  It's good to do that, but not the problem I'm worried about solving.
> >
> > I was trying to describe the minimum requirements for being a language
> > maintainer, I.E. what non-english users need in order to be able to merge
> > their patches.  Because without someone to contribute patches to (I.E a
> > language maintainer), documentation in non-english languages promotes the
> > creation of patches that can't be merged.  That's the problem I'm trying to
> > solve.
> >
> > To me, finding language maintainers is the flip side of the coin of
> > translating documentation.
> 
> I think you worried too much about this problem.  :)  Let me explain
> the situation here in China more clearly.  Actually, English is
> mandatory in most schools and universities.  Only very few people
> learn other language as a second language.  Therefore software
> developers who are almost educated should have the basic English
> skill.  However, that doesn't mean that they can read English or
> communicate with native English speaker very easily.  Consider your
> second language learn in school for analogy.  Read in English will be
> much slower and more likely to cause misunderstanding.  This will
> reduce the likelihood greatly of English documentation being read.  If
> we are promoting contribution to the Linux community, we should
> maximum the possibility that these key documents being read.
> Translation will serve this purpose very well.
> 
> So the possibility is very little that a translator is needed between
> the Linux maintainer and a Chinese developer.  Although sometimes help
> is needed when there is misunderstanding.
> 
> After a brief talk with the Japanese translator, I think the case is
> similar for Japanese too.

Yes, In Japan, situation is mostly the same.
We are trying to increase number of Linux community developer with
Linux Foundation Japan or CELF people in Japan.
In our discussion, the problem is not only Language.

In case of some developer, once he step forward (he try to send patch
or comment on LKML), he got some comment and he can work with
community even if it's slow (because of he was non-native).
So, I thought if some key document are available in Japanese like
HOWTO, that will help such early stage of developers.

> Therefore, in my opinion, language maintainer should be more a helper
> and promoter rather than a gatekeeper.  I will give a proposed process
> later about how this helper mechanism works.

I will be able to help this as a stand point of Japanese situation.

> - Leo
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 
> 

Re: [PATCH] Chinese Language Maintainer

[H. Peter Anvin]

Tsugikazu Shibata wrote:
> 
> Yes, In Japan, situation is mostly the same.
> We are trying to increase number of Linux community developer with
> Linux Foundation Japan or CELF people in Japan.
> In our discussion, the problem is not only Language.
> 
> In case of some developer, once he step forward (he try to send patch
> or comment on LKML), he got some comment and he can work with
> community even if it's slow (because of he was non-native).
> So, I thought if some key document are available in Japanese like
> HOWTO, that will help such early stage of developers.
> 

I think you're absolutely right about that.  The stuff that will help
most of have translated is (kernel)newbie type documentation, the stuff
one wants when being introduced to the community.

	-hpa

Re: [PATCH] Chinese Language Maintainer

[Ganesan Rajagopal]

>>>>> Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> That aside, please remember that Europe as a whole is a small place on the
> bigger world stage. The total volume of potential developers in the huge
> and rapidly modernising nations like India and China is vast, and there
> are large highly skilled technical nations that don't teach English to
> everyone technical by default.

Speaking for India only, while the medium of instruction in primary and high
school is available in native languages, English is almost universally
preferred.  Moreover, technical education is completely in English. I don't
see much value in translating developer documentation into Indian
languages. User documentation is another story.

Ganesan

-- 
Ganesan Rajagopal

Re: [PATCH] Chinese Language Maintainer

[Dr. Keith G. Bowden]

What I would like to see is the Americans learning English as a FIRST
language.

Keith (England)

----- Original Message ----- 
From: "H. Peter Anvin" <hpa@zytor.com>
To: "Li Yang" <leo@zh-kernel.org>
Cc: "Rob Landley" <rob@landley.net>; "Gerrit Huizenga" <gh@us.ibm.com>;
"Kunai, Takashi" <kunai@linux-foundation.jp>; <holzheu@linux.vnet.ibm.com>;
"Andrew Morton" <akpm@linux-foundation.org>; <linux-kernel@vger.kernel.org>;
<lf_kernel_messages@linux-foundation.org>; <mtk-manpages@gmx.net>;
<jack@suse.cz>; <randy.dunlap@oracle.com>; <gregkh@suse.de>; <pavel@ucw.cz>;
<tim.bird@am.sony.com>; <arjan@infradead.org>; <sam@ravnborg.org>;
<jengelh@computergmbh.de>; <joe@perches.com>; <auke-jan.h.kok@intel.com>;
<hansendc@us.ibm.com>; <davem@davemloft.net>; <Valdis.Kletnieks@vt.edu>;
<kenistoj@us.ibm.com>; <schwidefsky@de.ibm.com>;
<heiko.carstens@de.ibm.com>; <linux-doc@vger.kernel.org>
Sent: Sunday, July 15, 2007 7:12 PM
Subject: Re: [PATCH] Chinese Language Maintainer


> Li Yang wrote:
> >
> > I think you worried too much about this problem.  :)  Let me explain
> > the situation here in China more clearly.  Actually, English is
> > mandatory in most schools and universities.  Only very few people
> > learn other language as a second language.  Therefore software
> > developers who are almost educated should have the basic English
> > skill.  However, that doesn't mean that they can read English or
> > communicate with native English speaker very easily.  Consider your
> > second language learn in school for analogy.
>
> Actually, I disagree.  English *is* the second language learned in
> school for most European developers (except, obviously, the ones from
> the British isles), and we don't have that problem.
>
> > Read in English will be much slower and more likely to cause
> > misunderstanding.  This will
> > reduce the likelihood greatly of English documentation being read.  If
> > we are promoting contribution to the Linux community, we should
> > maximum the possibility that these key documents being read.
> > Translation will serve this purpose very well.
>
> What we have found in Europe, is that that it has limited value, and
> that the closer to the core you are, the less value it is, because at
> that stage you should be communicating more with other developers.
> Putting yourself behind a wall of translation is unfortunately a
> detriment in that way.
>
> -hpa
> -
> To unsubscribe from this list: send the line "unsubscribe linux-doc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Sunday 15 July 2007 12:28:06 pm Randy Dunlap wrote:
> On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
> > On Friday 13 July 2007 11:54:41 pm Randy Dunlap wrote:
> > > > If there's interest, I can push some patches to clean up
> > > > Documentation by moving files into subdirectories, but
> > > > Documentation's not well-suited to link out to the web.  (You need
> > > > html for that, and it's text.)
> > >
> > > I think that you should start moving Documentation/ files around
> > > and adding subdirectories -- if you are pretty sure of where they
> > > should go (i.e., they won't likely be moved again later on).
> >
> > You mean like these?
> > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
> > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html
>
> Yes.  Have any of these been merged?

Not that I know of.  They seemed to meet with resounding indifference, so I 
went on to other things...

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: [PATCH] Chinese Language Maintainer

[Alan Cox]

> preferred.  Moreover, technical education is completely in English. I don't
> see much value in translating developer documentation into Indian
> languages. User documentation is another story.

I am not convinced that users do not become developers, nor that we
should have language barriers that make this unneccessarily hard.

Alan

Re: Documentation of kernel messages (Summary)

[Tim Bird]

Rob Landley wrote:
> If you go to http://kernel.org/doc/ols you should find, nicely split up, all 
> the OLS papers from 2002-2007.  

Oooh!  That's nice!  I didn't notice the "nicely split up" part earlier.
Any chance we can get the original docbook inputs that OLS uses
for paper submissions?  Have you asked Andrew or Craig about this?

Also, it would be nice to correlate the talk names with the individual
PDFs.  Do you want me to solicit inside CELF for a volunteer for this?
We'd probably produce a wiki page of links, based on your
list.txt file and the proceedings index.
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: Documentation of kernel messages (Summary)

[H. Peter Anvin]

Tim Bird wrote:
> 
> Oooh!  That's nice!  I didn't notice the "nicely split up" part earlier.
> Any chance we can get the original docbook inputs that OLS uses
> for paper submissions?  Have you asked Andrew or Craig about this?
> 

OLS uses LaTeX, not DocBook, for submissions AFAIK.

	-hpa

Re: [PATCH] Chinese Language Maintainer

[Ganesan Rajagopal]

>>>>> Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

>> preferred.  Moreover, technical education is completely in English. I don't
>> see much value in translating developer documentation into Indian
>> languages. User documentation is another story.

> I am not convinced that users do not become developers, nor that we
> should have language barriers that make this unneccessarily hard.

Again speaking only from the Indian context, we're somewhat unique when
compared to China and Japan ;-). It's extremely unlikely that a non-english
speaking user will become a developer. As I mentioned in my previous mail
technical education is almost completely in English. 

Most Indian IT work places communicate almost exclusively in English,
including verbal communication because of employees from all over India with
different native languages.  English is not a language barrier for
developers or even technical users from India.

Ganesan

-- 
Ganesan Rajagopal

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Sunday 15 July 2007 12:46:42 pm Tsugikazu Shibata wrote:
> On Sat, 14 Jul 2007 22:12:35 -0400, rob wrote:
> > On Friday 13 July 2007 9:46:59 pm Tsugikazu Shibata wrote:
> > > > > How about adding;
> > > > > kernel-doc-nano-HOWTO.txt
> > > >
> > > > The problem is, the generated htmdocs are in english.  This file is
> > > > about how to generate (and author) English documentation that won't
> > > > be translated. What's the point of translating the instructions if
> > > > the result won't be translated?
> > >
> > > People (who even in non-native) would better to know and read it such
> > > htmldocs because there are good and important documents even in
> > > English. I thought that translation of this file may help.
> >
> > The english version of htmldocs is generated from the source code.    If
> > translating remotely current versions of htmldocs into other languages
> > was feasible, then the english version wouldn't need to be in the source
> > code in the first place.
> >
> > Translating a document _about_ htmldocs boils down to "Dear non-english
> > speaker: here's something you can't read, nor can you update it either
> > except though your language maintainer."
>
> No, kernel-doc-nano-HOWTO includes explanation of tools to extract the
> document, format of extractable document in source files and so on.
> I thought this file would help non-native people, how to extract or add
> in-kernel extractable documents.

Which will be in english, so why can't the instructions on how to do it be in 
english too?

> I understand that these tools will generate English documents but
> that's OK.

Then the instructions how to do it in english should be ok too.  If they can't 
read the instructions, they won't be able to read the result.

> Such large volume of documents are not easy to translate 
> into other languages.

"Here's how to get a result that will be useless to you."

Why?  What's the point?  I'm not seeing it.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: [PATCH] Chinese Language Maintainer

[Li Yang]

Tsugikazu Shibata wrote:
> On Sun, 15 Jul 2007 22:42:07 +0800, leo wrote:
>> On 7/14/07, Rob Landley <rob@landley.net> wrote:
>>> On Friday 13 July 2007 8:43:03 am Li Yang wrote:
>>>> On Thu, 2007-07-12 at 12:05 -0400, Rob Landley wrote:
>>>>> +A language maintainer accepts patches to the Linux kernel, written in C,
>>>>> from +authors who do not also speak English.  The language maintainer
>>>>> translates the +description of each patch into English, forwards the
>>>>> patches to linux-kernel +and to the appropriate maintainers for inclusion
>>>>> in the Linux kernel, and +translates questions and replies about such
>>>>> patches as part of the +patch review process.
>>>> In addiction to this responsibility, I would like to add two more which,
>>>> in my opinion, are more important.  And these are what I'm trying to
>>>> do.  :)
>>>> First, promoting contribution to Linux kernel in local language.
>>>> Second, coordinate the translation effort of key kernel documents.
>>> Cool.  It's good to do that, but not the problem I'm worried about solving.
>>>
>>> I was trying to describe the minimum requirements for being a language
>>> maintainer, I.E. what non-english users need in order to be able to merge
>>> their patches.  Because without someone to contribute patches to (I.E a
>>> language maintainer), documentation in non-english languages promotes the
>>> creation of patches that can't be merged.  That's the problem I'm trying to
>>> solve.
>>>
>>> To me, finding language maintainers is the flip side of the coin of
>>> translating documentation.
>> I think you worried too much about this problem.  :)  Let me explain
>> the situation here in China more clearly.  Actually, English is
>> mandatory in most schools and universities.  Only very few people
>> learn other language as a second language.  Therefore software
>> developers who are almost educated should have the basic English
>> skill.  However, that doesn't mean that they can read English or
>> communicate with native English speaker very easily.  Consider your
>> second language learn in school for analogy.  Read in English will be
>> much slower and more likely to cause misunderstanding.  This will
>> reduce the likelihood greatly of English documentation being read.  If
>> we are promoting contribution to the Linux community, we should
>> maximum the possibility that these key documents being read.
>> Translation will serve this purpose very well.
>>
>> So the possibility is very little that a translator is needed between
>> the Linux maintainer and a Chinese developer.  Although sometimes help
>> is needed when there is misunderstanding.
>>
>> After a brief talk with the Japanese translator, I think the case is
>> similar for Japanese too.
> 
> Yes, In Japan, situation is mostly the same.
> We are trying to increase number of Linux community developer with
> Linux Foundation Japan or CELF people in Japan.
> In our discussion, the problem is not only Language.
> 
> In case of some developer, once he step forward (he try to send patch
> or comment on LKML), he got some comment and he can work with
> community even if it's slow (because of he was non-native).
> So, I thought if some key document are available in Japanese like
> HOWTO, that will help such early stage of developers.
> 
>> Therefore, in my opinion, language maintainer should be more a helper
>> and promoter rather than a gatekeeper.  I will give a proposed process
>> later about how this helper mechanism works.
> 
> I will be able to help this as a stand point of Japanese situation.

Here is the helper process I propose to help more people to participate. 
  Suggestions and comments are welcomed.

1) Developer who can't speak English or has a problem going through the 
submission process sends patches to the Language maintainer and cc the 
local mail list provided.
2) Language maintainer and experienced Linux developer on the local mail 
list can review the patch and give comments in native language.
3) When the language maintainer thinks the patch is good in general, he 
will pass the patch onto corresponding subsystem maintainer and mail 
list for further review, cc the author and local list.
4) The author communicates directly with the community.
5) If the author has a problem understanding the comments or expressing 
himself in English, he can ask for help in native language on the local 
list.  The language maintainer and the developers on the list are 
responsible to help out.
6) When the developer gets familiar with the submission process and can 
go through it with no problem, he can then submit patch directly without 
the helper process and probably join the mail list to help other new 
developer.

- Leo

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Monday 16 July 2007 8:31:52 pm Tim Bird wrote:
> Rob Landley wrote:
> > If you go to http://kernel.org/doc/ols you should find, nicely split up,
> > all the OLS papers from 2002-2007.
>
> Oooh!  That's nice!  I didn't notice the "nicely split up" part earlier.
> Any chance we can get the original docbook inputs that OLS uses
> for paper submissions?  Have you asked Andrew or Craig about this?

I've asked, but OLS was in the process of happening and they were kind of 
swamped...

> Also, it would be nice to correlate the talk names with the individual
> PDFs.

I'm working on that: http://kernel.org/doc/ols/2002

I keep getting distracted by new material coming in.  Need to shut it out for 
a bit and focus on processing the existing pile...

> Do you want me to solicit inside CELF for a volunteer for this?

Yes please.  I'm happy to have volunteers help out with any of this, if I can 
figure out how to sanely partition it.

I'm not just collating names with papers, I'm also reading the paper and 
trying to come up with a one page summary.  If you look at 
http://kernel.org/doc you'll see a (now stale) version of an index skeleton.  
I'd like to link to all these papers from one big index.  And also link the 
Documentation files into the same index.  And http://lwn.net/Kernel/Index/ 
and Linux Journal's archives and...

That's the hard part.  Indexing it.  Piling up a big heap is easy.

You'll notice that Documentation has its own 00-index file, which doesn't even 
refer to the make htmldocs output.  The lwn kernel material has an excellent 
index.  The Linux Journal articles have a chronological archive.  Every 
volume of OLS papers starts with a brief index.

And the problem is, every time somebody notices the problem they start a _new_ 
index that doesn't use any of the others.  (And they keep wanting to do it as 
a wiki, which dooms the project.  In my experience wikis aren't stable and 
only locally versioned.  They're not designed to be snapshotted as a coherent 
whole, which is the _point_ of an index.  If you deep-link into a wiki you 
get 404 errors after a while.  They're a good tool for the "piling up 
information" part, but a bad tool for editorial jobs like organizing and 
indexing existing information.  Wikis are designed to be decentralized, so 
the left hand doesn't have to know what the right hand is doing, but 
editorial functions is all about collating, coordinating, and correlating 
into a coherent whole.  Which is nontrivial.)

> We'd probably produce a wiki page of links, based on your
> list.txt file and the proceedings index.


I have a list.txt file?  (Rummages.)  Oh, that's actually extracted from Red 
Hat's 2007 OLS web page (see the link at top, "mirrored from here".)

I was using Red hat's broken-up pages (and pre-broken-up ones I found for 
2004) for a while, but I went back and I re-broke up the pages with my 
scripts (which you'll notice I link to; I want people other than me to be 
able to reproduce this.)  I did it because all the others I broke up have the 
two OLS boilerplate pages at the _end_ rather than the beginning, and I 
wanted to be consistent.

And that list.txt page doesn't have the paragraph per article summary (which I 
can sometimes take from the abstract, but even when there is an abstract it's 
often not what I need and I read the darn paper anyway to see what 
interesting stuff's buried in it).  I need that in order to index the 
articles, title isn't enough.  Some articles need to get linked from more 
than one place.  (James Bottomley's 2002 scsi paper has good "history of 
hotplug" material, for example.)

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Monday 16 July 2007 9:17:28 pm H. Peter Anvin wrote:
> Tim Bird wrote:
> > Oooh!  That's nice!  I didn't notice the "nicely split up" part earlier.
> > Any chance we can get the original docbook inputs that OLS uses
> > for paper submissions?  Have you asked Andrew or Craig about this?
>
> OLS uses LaTeX, not DocBook, for submissions AFAIK.

Yeah, but their retention of the original latex has been...  Iffy.  And their 
indexing was a bit rushed.  http://www.linuxsymposium.org/proceedings.php 
goes up to 2005.  If you go to 
http://www.linuxsymposium.org/2006/proceedings.php you'll see a mirror of the 
other page with 2006 added (why not update the top level one?  No idea.)  
Then 2007 is off on a Red Hat site (mis-numbered) and not even hosted on 
linuxsymposium.org.

If you look at http://kernel.org/doc/ols/2006/slides/ you'll notice I mirrored 
all the presentation slides (not the same as the papers).  Two of them had 
already gone down before I got there, but I managed to get a fresh copy of 
Steven Hill's uClibc NPTL slides when I bumped into him at OLS.  (My mirror 
has it, the original site it links to is 404.)  I still haven't tracked down 
Kristen Accardi's slides, and that's one that was theoretically mirrored on 
the OLS website itself, yet the link is 404...

I'm not complaining, by the way: busy people get the data out and then move 
on, it's up to people like me to go back and clean it up if you want any kind 
of data retention.  (And THEN you have to periodically figure out what's 
still relevant and what's only of historical interest.  Wheee...)

Right now, I'd be happy to just get the PDFs of 2001 and earlier.  There's a 
apparently one year they only have on paper, and one of the others is on a CD 
that Andrew thinks he knows where it is, but hadn't had time to dig up when 
last I poked him...

If latex shows up, I'm all for it.  But for now, I'm happy with the PDFs.  
(Hey Tim, you want something for a volunteer to do?  Index the 2006 papers, 
then match up the 2006 slides with the corresponding papers.  I can go 
through and add in summaries afterwards...  And note that the index.html 
should have _relative_ links, not absolute ones, because I want anyone to be 
able to tarball up the whole site and locally mirror it.  I need to do a big 
script that fetches the external things like the ols papers and breaks them 
up so you have reproducible local copies without blindly copying mine.  It's 
on my todo list...)

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: [PATCH] Chinese Language Maintainer

[Rob Landley]

On Tuesday 17 July 2007 12:24:07 pm Li Yang wrote:
> Here is the helper process I propose to help more people to participate.
>   Suggestions and comments are welcomed.
>
> 1) Developer who can't speak English or has a problem going through the
> submission process sends patches to the Language maintainer and cc the
> local mail list provided.
> 2) Language maintainer and experienced Linux developer on the local mail
> list can review the patch and give comments in native language.
> 3) When the language maintainer thinks the patch is good in general, he
> will pass the patch onto corresponding subsystem maintainer and mail
> list for further review, cc the author and local list.
> 4) The author communicates directly with the community.
> 5) If the author has a problem understanding the comments or expressing
> himself in English, he can ask for help in native language on the local
> list.  The language maintainer and the developers on the list are
> responsible to help out.
> 6) When the developer gets familiar with the submission process and can
> go through it with no problem, he can then submit patch directly without
> the helper process and probably join the mail list to help other new
> developer.

You know your area better than I do.  If you think that will work, I certainly 
have no objections...

> - Leo

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Tsugikazu Shibata]

On Tue, 17 Jul 2007 12:06:56 -0400, rob wrote:
> On Sunday 15 July 2007 12:46:42 pm Tsugikazu Shibata wrote:
> > On Sat, 14 Jul 2007 22:12:35 -0400, rob wrote:
> > > On Friday 13 July 2007 9:46:59 pm Tsugikazu Shibata wrote:
> > > > > > How about adding;
> > > > > > kernel-doc-nano-HOWTO.txt
> > > > >
> > > > > The problem is, the generated htmdocs are in english.  This file is
> > > > > about how to generate (and author) English documentation that won't
> > > > > be translated. What's the point of translating the instructions if
> > > > > the result won't be translated?
> > > >
> > > > People (who even in non-native) would better to know and read it such
> > > > htmldocs because there are good and important documents even in
> > > > English. I thought that translation of this file may help.
> > >
> > > The english version of htmldocs is generated from the source code.    If
> > > translating remotely current versions of htmldocs into other languages
> > > was feasible, then the english version wouldn't need to be in the source
> > > code in the first place.
> > >
> > > Translating a document _about_ htmldocs boils down to "Dear non-english
> > > speaker: here's something you can't read, nor can you update it either
> > > except though your language maintainer."
> >
> > No, kernel-doc-nano-HOWTO includes explanation of tools to extract the
> > document, format of extractable document in source files and so on.
> > I thought this file would help non-native people, how to extract or add
> > in-kernel extractable documents.
> 
> Which will be in english, so why can't the instructions on how to do it be in 
> english too?
> 
> > I understand that these tools will generate English documents but
> > that's OK.
> 
> Then the instructions how to do it in english should be ok too.  If they can't 
> read the instructions, they won't be able to read the result.

It seems my explanation was not enough.
We (like me, usually not using English) can read English but it takes
more time than you (and any natives). So, reading large document takes
huge amount of time. In case of htmldocs, it is actually large and
developer may give up before start.
I thought that if some part (like how to do it) are translated into
his/her familiar language, he/she may not give it up.
Yes, it may not be.

Another viewpoint is that if he/she need to modify some interface and
it may need to change some part of extractable document, this file
includes, "How to add extractable documentation to your source files".
It will be some help as well. 
Yes, He/she should write English and it may take long time but hopefully
the translation result would push his/her back. 


> > Such large volume of documents are not easy to translate 
> > into other languages.
> 
> "Here's how to get a result that will be useless to you."
> 
> Why?  What's the point?  I'm not seeing it.
> 
> Rob
> -- 
> "One of my most productive days was throwing away 1000 lines of code."
>   - Ken Thompson.
> 

Re: Documentation of kernel messages (Summary)

[Randy Dunlap]

On Mon, 16 Jul 2007 15:53:06 -0400 Rob Landley wrote:

> On Sunday 15 July 2007 12:28:06 pm Randy Dunlap wrote:
> > On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
> > > On Friday 13 July 2007 11:54:41 pm Randy Dunlap wrote:
> > > > > If there's interest, I can push some patches to clean up
> > > > > Documentation by moving files into subdirectories, but
> > > > > Documentation's not well-suited to link out to the web.  (You need
> > > > > html for that, and it's text.)
> > > >
> > > > I think that you should start moving Documentation/ files around
> > > > and adding subdirectories -- if you are pretty sure of where they
> > > > should go (i.e., they won't likely be moved again later on).
> > >
> > > You mean like these?
> > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
> > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html
> >
> > Yes.  Have any of these been merged?
> 
> Not that I know of.  They seemed to meet with resounding indifference, so I 
> went on to other things...

You need to be persistent.  Please re-submit this.

and since it's just file & dir moves and done via git, it should go to
Linus.  However, he may not want it right now since the primary
merge window for 2.6.23 has closed (although he has been merging
some other doc updates recently).

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Friday 03 August 2007 1:11:55 pm Randy Dunlap wrote:
> On Mon, 16 Jul 2007 15:53:06 -0400 Rob Landley wrote:
> > On Sunday 15 July 2007 12:28:06 pm Randy Dunlap wrote:
> > > On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
> > > > On Friday 13 July 2007 11:54:41 pm Randy Dunlap wrote:
> > > > > > If there's interest, I can push some patches to clean up
> > > > > > Documentation by moving files into subdirectories, but
> > > > > > Documentation's not well-suited to link out to the web.  (You
> > > > > > need html for that, and it's text.)
> > > > >
> > > > > I think that you should start moving Documentation/ files around
> > > > > and adding subdirectories -- if you are pretty sure of where they
> > > > > should go (i.e., they won't likely be moved again later on).
> > > >
> > > > You mean like these?
> > > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
> > > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html
> > >
> > > Yes.  Have any of these been merged?
> >
> > Not that I know of.  They seemed to meet with resounding indifference, so
> > I went on to other things...
>
> You need to be persistent.  Please re-submit this.

Greg KH thinks it's a good idea to add language directories to the top level 
of Documentation, and there are slightly more than two of those:
http://www.translatorscafe.com/cafe/Help.asp?HelpID=59

Since you think it's worth doing, I'll resubmit the work I've already done 
here, but I no longer think trying to shovel Documentation into some 
semblance of order against the will of people like Greg is a good use of 
time.

These days I'm trying to create an html index that links into Documentation in 
a coherent order (with categories and everything), and using automated tools 
to detect files that aren't linked to, or links that point to a file that 
isn't there anymore.  This is obviously still a work in progress, but I think 
it's a better approach.

> and since it's just file & dir moves and done via git, it should go to
> Linus.  However, he may not want it right now since the primary
> merge window for 2.6.23 has closed (although he has been merging
> some other doc updates recently).

If moving text files from one location to another in the Documentation 
directory breaks the build or causes bugs in the kernel, we have bigger 
problems. :)

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Greg KH]

On Fri, Aug 03, 2007 at 03:32:04PM -0400, Rob Landley wrote:
> On Friday 03 August 2007 1:11:55 pm Randy Dunlap wrote:
> > On Mon, 16 Jul 2007 15:53:06 -0400 Rob Landley wrote:
> > > On Sunday 15 July 2007 12:28:06 pm Randy Dunlap wrote:
> > > > On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
> > > > > On Friday 13 July 2007 11:54:41 pm Randy Dunlap wrote:
> > > > > > > If there's interest, I can push some patches to clean up
> > > > > > > Documentation by moving files into subdirectories, but
> > > > > > > Documentation's not well-suited to link out to the web.  (You
> > > > > > > need html for that, and it's text.)
> > > > > >
> > > > > > I think that you should start moving Documentation/ files around
> > > > > > and adding subdirectories -- if you are pretty sure of where they
> > > > > > should go (i.e., they won't likely be moved again later on).
> > > > >
> > > > > You mean like these?
> > > > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
> > > > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html
> > > >
> > > > Yes.  Have any of these been merged?
> > >
> > > Not that I know of.  They seemed to meet with resounding indifference, so
> > > I went on to other things...
> >
> > You need to be persistent.  Please re-submit this.
> 
> Greg KH thinks it's a good idea to add language directories to the top level 
> of Documentation, and there are slightly more than two of those:
> http://www.translatorscafe.com/cafe/Help.asp?HelpID=59
> 
> Since you think it's worth doing, I'll resubmit the work I've already done 
> here, but I no longer think trying to shovel Documentation into some 
> semblance of order against the will of people like Greg is a good use of 
> time.

My "will" is only that some of these documents be translated _and_ put
in the main kernel source tree.  I really have no opinion on where they
need to go within that directory.  If you have a better place for them,
please let me know.

thanks,

greg k-h

Re: Documentation of kernel messages (Summary)

[Greg KH]

On Fri, Aug 03, 2007 at 03:32:04PM -0400, Rob Landley wrote:
> 
> These days I'm trying to create an html index that links into Documentation in 
> a coherent order (with categories and everything), and using automated tools 
> to detect files that aren't linked to, or links that point to a file that 
> isn't there anymore.  This is obviously still a work in progress, but I think 
> it's a better approach.

Better than cleaning up what we have in the kernel source tree?  Why not
work on that first, then the "automated" type stuff would be much easier
to do later, right?

thanks,

greg k-h

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Friday 03 August 2007 10:52:06 pm Greg KH wrote:
> On Fri, Aug 03, 2007 at 03:32:04PM -0400, Rob Landley wrote:
> > These days I'm trying to create an html index that links into
> > Documentation in a coherent order (with categories and everything), and
> > using automated tools to detect files that aren't linked to, or links
> > that point to a file that isn't there anymore.  This is obviously still a
> > work in progress, but I think it's a better approach.
>
> Better than cleaning up what we have in the kernel source tree?

Yes, I just said that.

> Why not work on that first, then the "automated" type stuff would be much
> easier to do later, right?

How does an automated 404 checker that identifies files nothing's linking to 
get easier if the source files are in a different order?  They could be 
alphabetical in one big directory and that would work the same.

Moving Documentation around is pointless churn that does nothing to prevent 
you from adding language directories to the top level on a whim, as if there 
were only two instead of (as I pointed out last message), several dozen.  
(While Randy Dunlap's poking me to resubmit my patch to group the 
architecture documentation into an architecture subdirectory, you're adding 
language directories to the top level instead of in their own subdirectory.)

Documentation doesn't even cross-link to the output of make htmldocs (which 
has its own structure imposed on it due to being extracted from the kernel 
sources).  The kernel tarball has _two_ documentation sources that don't 
significantly cross-reference each other, and Rusty just submitted "make 
Preparation!" for lguest that's totally unrelated to either of them (and 
starts from a README file buried in the source code).  None of this links to 
the menuconfig help entries, and the only reference in Documentation/ 
to "make help" is in Documentation/kbuild/makefiles.txt which explains that 
its purpose is to list the available architecture targets (something it does 
not, in my experience, actually do).

The idea that the kernel Documentation directory is the master repository of 
kernel documentation is an unworkable fantasy.  The Documentation directory 
cannot index for all the kernel documentation resources out on the web, 
because it's in text not html.  Documentation was created on the assumption 
that it would contain all interesting resources, as text files, but that 
doesn't match reality.  Documentation is merely one resource among many, and 
to link _out_ you need HTML.

Some of the resources out there are organized chronologically, such as the 
Linux Journal archives http://www.linuxjournal.com/xstatic/magazine/archives, 
the Ottawa Linux Symposium papers (http://kernel.org/doc/ols), or the 
kernel-traffic archives 
http://www.kernel-traffic.org/kernel-traffic/archives.html.  Some have their 
own indexes, such as the Linux Weekly News kernel articles 
http://lwn.net/Kernel/Index/ and the Linux Device Drivers book 
http://lwn.net/Kernel/LDD3/.  Some are random bits picked out of developer 
blogs, found with google, reasonably coherent articles on wikipedia.  Some 
are huge self-contained lumps on specific topics such as Mel Gorman's mm 
documentation or lots of the embedded stuff.

So no matter what reorganization I do to Documentation, its structure (or lack 
thereof) is incidental to coherently indexing most of the kernel 
documentation that's out there.  (Right now the best index is "google" but 
that's only useful if you know what questions to ask.  But getting up-to-date 
versions of Documentation and the output of make htmldocs on the web lets 
google find it.  (Last year, back when I was still working on BusyBox, I did 
a google trawl for ext2 documentation to replace the horrible mke2fs busybox 
used to have, and the first three pages of hits did _NOT_ include a copy of 
Documentation/filesystems/ext2.txt.  Ironically, if you google for "ext2 
documentation" today the sixth hit is the unfinished ext2 documentation I'd 
just started to write before I found Documentation/filesystems/ext2.txt.)

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Friday 03 August 2007 10:50:38 pm Greg KH wrote:
> On Fri, Aug 03, 2007 at 03:32:04PM -0400, Rob Landley wrote:
> > On Friday 03 August 2007 1:11:55 pm Randy Dunlap wrote:
> > > On Mon, 16 Jul 2007 15:53:06 -0400 Rob Landley wrote:
> > > > On Sunday 15 July 2007 12:28:06 pm Randy Dunlap wrote:
> > > > > On Sat, 14 Jul 2007 21:56:15 -0400 Rob Landley wrote:
> > > > > > On Friday 13 July 2007 11:54:41 pm Randy Dunlap wrote:
> > > > > > > > If there's interest, I can push some patches to clean up
> > > > > > > > Documentation by moving files into subdirectories, but
> > > > > > > > Documentation's not well-suited to link out to the web.  (You
> > > > > > > > need html for that, and it's text.)
> > > > > > >
> > > > > > > I think that you should start moving Documentation/ files
> > > > > > > around and adding subdirectories -- if you are pretty sure of
> > > > > > > where they should go (i.e., they won't likely be moved again
> > > > > > > later on).
> > > > > >
> > > > > > You mean like these?
> > > > > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2925.html
> > > > > > http://www.uwsg.iu.edu/hypermail/linux/kernel/0705.3/2924.html
> > > > >
> > > > > Yes.  Have any of these been merged?
> > > >
> > > > Not that I know of.  They seemed to meet with resounding
> > > > indifference, so I went on to other things...
> > >
> > > You need to be persistent.  Please re-submit this.
> >
> > Greg KH thinks it's a good idea to add language directories to the top
> > level of Documentation, and there are slightly more than two of those:
> > http://www.translatorscafe.com/cafe/Help.asp?HelpID=59
> >
> > Since you think it's worth doing, I'll resubmit the work I've already
> > done here, but I no longer think trying to shovel Documentation into some
> > semblance of order against the will of people like Greg is a good use of
> > time.
>
> My "will" is only that some of these documents be translated _and_ put
> in the main kernel source tree.  I really have no opinion on where they
> need to go within that directory.  If you have a better place for them,
> please let me know.

A common subdirectory.  I'd probably put it in "Documentation/translated" with 
a README at the top level of that explaining what it's for.

I can push a patch to move them, but reorganizing Documentation doesn't turn 
it into an index that can link out resources outside itself.  If reorganizing 
it to be less of a compost heap is a separate goal in and of itself, I can do 
that...

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Documentation of kernel messages (Summary)

[Stefan Richter]

Rob Landley wrote:
> Documentation is merely one resource among many, and 
> to link _out_ you need HTML.

Do you suggest HTML files in linux/Documentation?  I think we can follow
URIs and other references in plaintext files just fine.

(People who want clickable links in plaintext files can use plaintext
viewers which implement this, or utilities or browser plugins which
follow URLs in the clipboard.)
-- 
Stefan Richter
-=====-=-=== =--- --=--
http://arcgraph.de/sr/

Re: Documentation of kernel messages (Summary)

[Rob Landley]

On Saturday 04 August 2007 3:04:33 pm Stefan Richter wrote:
> Rob Landley wrote:
> > Documentation is merely one resource among many, and
> > to link _out_ you need HTML.
>
> Do you suggest HTML files in linux/Documentation?

I've thought about it, but right now Documentation is text.  Converting all 
the existing text files to html is a fairly intrusive change, and having 
multiple file formats in there seems even more untidy.  (What next, pdf?).

I'm aware there are some docbook files in there, plus an ancient graphic file, 
plus the bloody stains of various chicken sacrificies, but the point would be 
how best to clean it up...

> I think we can follow 
> URIs and other references in plaintext files just fine.

Go for it.  Have fun.  I'll be over here.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.