LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* [PATCH 00/14] Add support to generate ABI documentation at admin-guide
@ 2019-06-14  2:04 Mauro Carvalho Chehab
  2019-06-14  2:04 ` [PATCH 01/14] ABI: fix some syntax issues at the ABI database Mauro Carvalho Chehab
                   ` (14 more replies)
  0 siblings, 15 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, linuxppc-dev, Andrew Donnellan,
	Jonathan Cameron, Lars-Peter Clausen, Anton Vorontsov, linux-pm,
	Colin Cross, linux-iio, Hartmut Knaack, Sebastian Reichel,
	Frederic Barrat, Andreas Klinger, Tony Luck,
	Peter Meerwald-Stadler, Stefan Achatz, Kees Cook

Greg,

As promised, I'm resending the patch series with adds the Kernel ABI to
Documentation/admin-guide.

Those patches are basically the version 3 patchset I sent back in 2017,
rebased on the top of linux-next (next-20190613), and with some fixes
in order for it to work.

- The 4 initial patches to fix some ABI descriptions that are violating 
  the syntax described at Documentation/ABI/README;

- The next 6 patches are the ones originally written in 2017 with a
  script with parses the ABI files;

- The 11th patch is a new one: it relaxes a little bit the parser in 
  order to parse file headers that contains colons on it;

- The 12th patch adds the new script to the documentation build
  system, together with a new python Sphinx extension with calls it;

- The 13th patch fixes the python script when running with newer
  Sphinx versions (1.7 and upper);

- The final patch fixes an UTF-8 trouble. I noticed it only with Sphinx
  1.4, but it could affect other versions too. So, I ended by changing
  the UTF-8 encoding logit to work version-independent, just like
  what happens with kerneldoc.py extension.

Mauro Carvalho Chehab (14):
  ABI: fix some syntax issues at the ABI database
  ABI: sysfs-driver-hid: the "What" field doesn't parse fine
  ABI: sysfs-class-uwb_rc: remove a duplicated incomplete entry
  ABI: better identificate tables
  scripts: add an script to parse the ABI files
  scripts/get_abi.pl: parse files with text at beginning
  scripts/get_abi.pl: avoid use literal blocks when not needed
  scripts/get_abi.pl: split label naming from xref logic
  scripts/get_abi.pl: add support for searching for ABI symbols
  scripts/get_abi.pl: represent what in tables
  scripts/get_abi.pl: fix parse issues with some files
  doc-rst: add ABI documentation to the admin-guide book
  sphinx/kernel_abi.py: make it compatible with Sphinx 1.7+
  docs: sphinx/kernel_abi.py: fix UTF-8 support

 .../ABI/obsolete/sysfs-driver-hid-roccat-pyra |   2 +-
 Documentation/ABI/testing/pstore              |   2 +-
 .../sysfs-bus-event_source-devices-format     |   2 +-
 .../ABI/testing/sysfs-bus-i2c-devices-hm6352  |   6 +-
 .../ABI/testing/sysfs-bus-iio-distance-srf08  |   4 +-
 .../testing/sysfs-bus-iio-proximity-as3935    |   4 +-
 .../ABI/testing/sysfs-bus-pci-devices-cciss   |  22 +-
 .../testing/sysfs-bus-usb-devices-usbsevseg   |  12 +-
 .../sysfs-class-backlight-driver-lm3533       |   6 +-
 Documentation/ABI/testing/sysfs-class-cxl     |   6 +-
 Documentation/ABI/testing/sysfs-class-devfreq |   2 +-
 .../ABI/testing/sysfs-class-led-driver-lm3533 |   8 +-
 .../ABI/testing/sysfs-class-leds-gt683r       |   4 +-
 .../ABI/testing/sysfs-class-powercap          |   2 +-
 Documentation/ABI/testing/sysfs-class-uwb_rc  |   6 -
 Documentation/ABI/testing/sysfs-driver-hid    |  12 +-
 .../ABI/testing/sysfs-driver-hid-roccat-kone  |   2 +-
 Documentation/ABI/testing/sysfs-kernel-fscaps |   2 +-
 .../ABI/testing/sysfs-kernel-vmcoreinfo       |   2 +-
 Documentation/admin-guide/abi-obsolete.rst    |  10 +
 Documentation/admin-guide/abi-removed.rst     |   4 +
 Documentation/admin-guide/abi-stable.rst      |  13 +
 Documentation/admin-guide/abi-testing.rst     |  19 +
 Documentation/admin-guide/abi.rst             |  11 +
 Documentation/admin-guide/index.rst           |   1 +
 Documentation/conf.py                         |   2 +-
 Documentation/sphinx/kernel_abi.py            | 172 +++++++
 scripts/get_abi.pl                            | 450 ++++++++++++++++++
 28 files changed, 731 insertions(+), 57 deletions(-)
 create mode 100644 Documentation/admin-guide/abi-obsolete.rst
 create mode 100644 Documentation/admin-guide/abi-removed.rst
 create mode 100644 Documentation/admin-guide/abi-stable.rst
 create mode 100644 Documentation/admin-guide/abi-testing.rst
 create mode 100644 Documentation/admin-guide/abi.rst
 create mode 100644 Documentation/sphinx/kernel_abi.py
 create mode 100755 scripts/get_abi.pl

-- 
2.21.0



^ permalink raw reply	[flat|nested] 63+ messages in thread

* [PATCH 01/14] ABI: fix some syntax issues at the ABI database
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
@ 2019-06-14  2:04 ` Mauro Carvalho Chehab
  2019-06-14  7:20   ` Andrew Donnellan
  2019-06-14  8:30   ` Rafael J. Wysocki
  2019-06-14  2:04 ` [PATCH 02/14] ABI: sysfs-driver-hid: the "What" field doesn't parse fine Mauro Carvalho Chehab
                   ` (13 subsequent siblings)
  14 siblings, 2 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Andreas Klinger, Jonathan Cameron,
	Hartmut Knaack, Lars-Peter Clausen, Peter Meerwald-Stadler,
	Frederic Barrat, Andrew Donnellan, Sebastian Reichel, Kees Cook,
	Anton Vorontsov, Colin Cross, Tony Luck, linux-iio, linuxppc-dev,
	linux-pm, Mauro Carvalho Chehab

From: Mauro Carvalho Chehab <mchehab@s-opensource.com>

On those three files, the ABI representation described at
README are violated.

- at sysfs-bus-iio-proximity-as3935:
	a ':' character is missing after "What"

- at sysfs-class-devfreq:
	there's a typo at Description

- at sysfs-class-cxl, it is using the ":" character at a
	file preamble, causing it to be misinterpreted as a
	tag.

- On the other files, instead of "What", they use "Where".

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 Documentation/ABI/testing/pstore              |  2 +-
 .../sysfs-bus-event_source-devices-format     |  2 +-
 .../ABI/testing/sysfs-bus-i2c-devices-hm6352  |  6 ++---
 .../ABI/testing/sysfs-bus-iio-distance-srf08  |  4 ++--
 .../testing/sysfs-bus-iio-proximity-as3935    |  4 ++--
 .../ABI/testing/sysfs-bus-pci-devices-cciss   | 22 +++++++++----------
 .../testing/sysfs-bus-usb-devices-usbsevseg   | 12 +++++-----
 Documentation/ABI/testing/sysfs-class-cxl     |  6 ++---
 Documentation/ABI/testing/sysfs-class-devfreq |  2 +-
 .../ABI/testing/sysfs-class-powercap          |  2 +-
 Documentation/ABI/testing/sysfs-kernel-fscaps |  2 +-
 .../ABI/testing/sysfs-kernel-vmcoreinfo       |  2 +-
 12 files changed, 33 insertions(+), 33 deletions(-)

diff --git a/Documentation/ABI/testing/pstore b/Documentation/ABI/testing/pstore
index 5fca9f5e10a3..8d6e48f4e8ef 100644
--- a/Documentation/ABI/testing/pstore
+++ b/Documentation/ABI/testing/pstore
@@ -1,4 +1,4 @@
-Where:		/sys/fs/pstore/... (or /dev/pstore/...)
+What:		/sys/fs/pstore/... (or /dev/pstore/...)
 Date:		March 2011
 Kernel Version: 2.6.39
 Contact:	tony.luck@intel.com
diff --git a/Documentation/ABI/testing/sysfs-bus-event_source-devices-format b/Documentation/ABI/testing/sysfs-bus-event_source-devices-format
index 77f47ff5ee02..b6f8748e0200 100644
--- a/Documentation/ABI/testing/sysfs-bus-event_source-devices-format
+++ b/Documentation/ABI/testing/sysfs-bus-event_source-devices-format
@@ -1,4 +1,4 @@
-Where:		/sys/bus/event_source/devices/<dev>/format
+What:		/sys/bus/event_source/devices/<dev>/format
 Date:		January 2012
 Kernel Version: 3.3
 Contact:	Jiri Olsa <jolsa@redhat.com>
diff --git a/Documentation/ABI/testing/sysfs-bus-i2c-devices-hm6352 b/Documentation/ABI/testing/sysfs-bus-i2c-devices-hm6352
index feb2e4a87075..29bd447e50a0 100644
--- a/Documentation/ABI/testing/sysfs-bus-i2c-devices-hm6352
+++ b/Documentation/ABI/testing/sysfs-bus-i2c-devices-hm6352
@@ -1,18 +1,18 @@
-Where:		/sys/bus/i2c/devices/.../heading0_input
+What:		/sys/bus/i2c/devices/.../heading0_input
 Date:		April 2010
 Kernel Version: 2.6.36?
 Contact:	alan.cox@intel.com
 Description:	Reports the current heading from the compass as a floating
 		point value in degrees.
 
-Where:		/sys/bus/i2c/devices/.../power_state
+What:		/sys/bus/i2c/devices/.../power_state
 Date:		April 2010
 Kernel Version: 2.6.36?
 Contact:	alan.cox@intel.com
 Description:	Sets the power state of the device. 0 sets the device into
 		sleep mode, 1 wakes it up.
 
-Where:		/sys/bus/i2c/devices/.../calibration
+What:		/sys/bus/i2c/devices/.../calibration
 Date:		April 2010
 Kernel Version: 2.6.36?
 Contact:	alan.cox@intel.com
diff --git a/Documentation/ABI/testing/sysfs-bus-iio-distance-srf08 b/Documentation/ABI/testing/sysfs-bus-iio-distance-srf08
index 0a1ca1487fa9..a133fd8d081a 100644
--- a/Documentation/ABI/testing/sysfs-bus-iio-distance-srf08
+++ b/Documentation/ABI/testing/sysfs-bus-iio-distance-srf08
@@ -1,4 +1,4 @@
-What		/sys/bus/iio/devices/iio:deviceX/sensor_sensitivity
+What:		/sys/bus/iio/devices/iio:deviceX/sensor_sensitivity
 Date:		January 2017
 KernelVersion:	4.11
 Contact:	linux-iio@vger.kernel.org
@@ -6,7 +6,7 @@ Description:
 		Show or set the gain boost of the amp, from 0-31 range.
 		default 31
 
-What		/sys/bus/iio/devices/iio:deviceX/sensor_max_range
+What:		/sys/bus/iio/devices/iio:deviceX/sensor_max_range
 Date:		January 2017
 KernelVersion:	4.11
 Contact:	linux-iio@vger.kernel.org
diff --git a/Documentation/ABI/testing/sysfs-bus-iio-proximity-as3935 b/Documentation/ABI/testing/sysfs-bus-iio-proximity-as3935
index 9a17ab5036a4..c59d95346341 100644
--- a/Documentation/ABI/testing/sysfs-bus-iio-proximity-as3935
+++ b/Documentation/ABI/testing/sysfs-bus-iio-proximity-as3935
@@ -1,4 +1,4 @@
-What		/sys/bus/iio/devices/iio:deviceX/in_proximity_input
+What:		/sys/bus/iio/devices/iio:deviceX/in_proximity_input
 Date:		March 2014
 KernelVersion:	3.15
 Contact:	Matt Ranostay <matt.ranostay@konsulko.com>
@@ -6,7 +6,7 @@ Description:
 		Get the current distance in meters of storm (1km steps)
 		1000-40000 = distance in meters
 
-What		/sys/bus/iio/devices/iio:deviceX/sensor_sensitivity
+What:		/sys/bus/iio/devices/iio:deviceX/sensor_sensitivity
 Date:		March 2014
 KernelVersion:	3.15
 Contact:	Matt Ranostay <matt.ranostay@konsulko.com>
diff --git a/Documentation/ABI/testing/sysfs-bus-pci-devices-cciss b/Documentation/ABI/testing/sysfs-bus-pci-devices-cciss
index 53d99edd1d75..eb449169c30b 100644
--- a/Documentation/ABI/testing/sysfs-bus-pci-devices-cciss
+++ b/Documentation/ABI/testing/sysfs-bus-pci-devices-cciss
@@ -1,66 +1,66 @@
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/model
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/model
 Date:		March 2009
 Kernel Version: 2.6.30
 Contact:	iss_storagedev@hp.com
 Description:	Displays the SCSI INQUIRY page 0 model for logical drive
 		Y of controller X.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/rev
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/rev
 Date:		March 2009
 Kernel Version: 2.6.30
 Contact:	iss_storagedev@hp.com
 Description:	Displays the SCSI INQUIRY page 0 revision for logical
 		drive Y of controller X.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/unique_id
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/unique_id
 Date:		March 2009
 Kernel Version: 2.6.30
 Contact:	iss_storagedev@hp.com
 Description:	Displays the SCSI INQUIRY page 83 serial number for logical
 		drive Y of controller X.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/vendor
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/vendor
 Date:		March 2009
 Kernel Version: 2.6.30
 Contact:	iss_storagedev@hp.com
 Description:	Displays the SCSI INQUIRY page 0 vendor for logical drive
 		Y of controller X.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/block:cciss!cXdY
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/block:cciss!cXdY
 Date:		March 2009
 Kernel Version: 2.6.30
 Contact:	iss_storagedev@hp.com
 Description:	A symbolic link to /sys/block/cciss!cXdY
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/rescan
+What:		/sys/bus/pci/devices/<dev>/ccissX/rescan
 Date:		August 2009
 Kernel Version:	2.6.31
 Contact:	iss_storagedev@hp.com
 Description:	Kicks of a rescan of the controller to discover logical
 		drive topology changes.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/lunid
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/lunid
 Date:		August 2009
 Kernel Version: 2.6.31
 Contact:	iss_storagedev@hp.com
 Description:	Displays the 8-byte LUN ID used to address logical
 		drive Y of controller X.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/raid_level
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/raid_level
 Date:		August 2009
 Kernel Version: 2.6.31
 Contact:	iss_storagedev@hp.com
 Description:	Displays the RAID level of logical drive Y of
 		controller X.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/usage_count
+What:		/sys/bus/pci/devices/<dev>/ccissX/cXdY/usage_count
 Date:		August 2009
 Kernel Version: 2.6.31
 Contact:	iss_storagedev@hp.com
 Description:	Displays the usage count (number of opens) of logical drive Y
 		of controller X.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/resettable
+What:		/sys/bus/pci/devices/<dev>/ccissX/resettable
 Date:		February 2011
 Kernel Version:	2.6.38
 Contact:	iss_storagedev@hp.com
@@ -71,7 +71,7 @@ Description:	Value of 1 indicates the controller can honor the reset_devices
 		a dump device, as kdump requires resetting the device in order
 		to work reliably.
 
-Where:		/sys/bus/pci/devices/<dev>/ccissX/transport_mode
+What:		/sys/bus/pci/devices/<dev>/ccissX/transport_mode
 Date:		July 2011
 Kernel Version:	3.0
 Contact:	iss_storagedev@hp.com
diff --git a/Documentation/ABI/testing/sysfs-bus-usb-devices-usbsevseg b/Documentation/ABI/testing/sysfs-bus-usb-devices-usbsevseg
index 70d00dfa443d..f6199b314196 100644
--- a/Documentation/ABI/testing/sysfs-bus-usb-devices-usbsevseg
+++ b/Documentation/ABI/testing/sysfs-bus-usb-devices-usbsevseg
@@ -1,12 +1,12 @@
-Where:		/sys/bus/usb/.../powered
+What:		/sys/bus/usb/.../powered
 Date:		August 2008
 Kernel Version:	2.6.26
 Contact:	Harrison Metzger <harrisonmetz@gmail.com>
 Description:	Controls whether the device's display will powered.
 		A value of 0 is off and a non-zero value is on.
 
-Where:		/sys/bus/usb/.../mode_msb
-Where:		/sys/bus/usb/.../mode_lsb
+What:		/sys/bus/usb/.../mode_msb
+What:		/sys/bus/usb/.../mode_lsb
 Date:		August 2008
 Kernel Version:	2.6.26
 Contact:	Harrison Metzger <harrisonmetz@gmail.com>
@@ -16,7 +16,7 @@ Description:	Controls the devices display mode.
 		for an 8 character display the values are
 			MSB 0x08; LSB 0xFF.
 
-Where:		/sys/bus/usb/.../textmode
+What:		/sys/bus/usb/.../textmode
 Date:		August 2008
 Kernel Version:	2.6.26
 Contact:	Harrison Metzger <harrisonmetz@gmail.com>
@@ -25,13 +25,13 @@ Description:	Controls the way the device interprets its text buffer.
 		hex:	each character is between 0-15
 		ascii:	each character is between '0'-'9' and 'A'-'F'.
 
-Where:		/sys/bus/usb/.../text
+What:		/sys/bus/usb/.../text
 Date:		August 2008
 Kernel Version:	2.6.26
 Contact:	Harrison Metzger <harrisonmetz@gmail.com>
 Description:	The text (or data) for the device to display
 
-Where:		/sys/bus/usb/.../decimals
+What:		/sys/bus/usb/.../decimals
 Date:		August 2008
 Kernel Version:	2.6.26
 Contact:	Harrison Metzger <harrisonmetz@gmail.com>
diff --git a/Documentation/ABI/testing/sysfs-class-cxl b/Documentation/ABI/testing/sysfs-class-cxl
index bbbabffc682a..fc7c6f7c21b3 100644
--- a/Documentation/ABI/testing/sysfs-class-cxl
+++ b/Documentation/ABI/testing/sysfs-class-cxl
@@ -1,6 +1,6 @@
-Note: Attributes that are shared between devices are stored in the directory
-pointed to by the symlink device/.
-Example: The real path of the attribute /sys/class/cxl/afu0.0s/irqs_max is
+Please notice that attributes that are shared between devices are stored in
+the directory pointed to by the symlink device/.
+For example, the real path of the attribute /sys/class/cxl/afu0.0s/irqs_max is
 /sys/class/cxl/afu0.0s/device/irqs_max, i.e. /sys/class/cxl/afu0.0/irqs_max.
 
 
diff --git a/Documentation/ABI/testing/sysfs-class-devfreq b/Documentation/ABI/testing/sysfs-class-devfreq
index ee39acacf6f8..01196e19afca 100644
--- a/Documentation/ABI/testing/sysfs-class-devfreq
+++ b/Documentation/ABI/testing/sysfs-class-devfreq
@@ -47,7 +47,7 @@ Description:
 What:		/sys/class/devfreq/.../trans_stat
 Date:		October 2012
 Contact:	MyungJoo Ham <myungjoo.ham@samsung.com>
-Descrtiption:
+Description:
 		This ABI shows the statistics of devfreq behavior on a
 		specific device. It shows the time spent in each state and
 		the number of transitions between states.
diff --git a/Documentation/ABI/testing/sysfs-class-powercap b/Documentation/ABI/testing/sysfs-class-powercap
index db3b3ff70d84..f333a0ccc29b 100644
--- a/Documentation/ABI/testing/sysfs-class-powercap
+++ b/Documentation/ABI/testing/sysfs-class-powercap
@@ -147,6 +147,6 @@ What:		/sys/class/powercap/.../<power zone>/enabled
 Date:		September 2013
 KernelVersion:	3.13
 Contact:	linux-pm@vger.kernel.org
-Description
+Description:
 		This allows to enable/disable power capping at power zone level.
 		This applies to current power zone and its children.
diff --git a/Documentation/ABI/testing/sysfs-kernel-fscaps b/Documentation/ABI/testing/sysfs-kernel-fscaps
index 50a3033b5e15..bcff34665192 100644
--- a/Documentation/ABI/testing/sysfs-kernel-fscaps
+++ b/Documentation/ABI/testing/sysfs-kernel-fscaps
@@ -2,7 +2,7 @@ What:		/sys/kernel/fscaps
 Date:		February 2011
 KernelVersion:	2.6.38
 Contact:	Ludwig Nussel <ludwig.nussel@suse.de>
-Description
+Description:
 		Shows whether file system capabilities are honored
 		when executing a binary
 
diff --git a/Documentation/ABI/testing/sysfs-kernel-vmcoreinfo b/Documentation/ABI/testing/sysfs-kernel-vmcoreinfo
index 7bd81168e063..1f1087a5f075 100644
--- a/Documentation/ABI/testing/sysfs-kernel-vmcoreinfo
+++ b/Documentation/ABI/testing/sysfs-kernel-vmcoreinfo
@@ -4,7 +4,7 @@ KernelVersion:	2.6.24
 Contact:	Ken'ichi Ohmichi <oomichi@mxs.nes.nec.co.jp>
 		Kexec Mailing List <kexec@lists.infradead.org>
 		Vivek Goyal <vgoyal@redhat.com>
-Description
+Description:
 		Shows physical address and size of vmcoreinfo ELF note.
 		First value contains physical address of note in hex and
 		second value contains the size of note in hex. This ELF
-- 
2.21.0


^ permalink raw reply	[flat|nested] 63+ messages in thread

* [PATCH 02/14] ABI: sysfs-driver-hid: the "What" field doesn't parse fine
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
  2019-06-14  2:04 ` [PATCH 01/14] ABI: fix some syntax issues at the ABI database Mauro Carvalho Chehab
@ 2019-06-14  2:04 ` Mauro Carvalho Chehab
  2019-06-14  2:04 ` [PATCH 03/14] ABI: sysfs-class-uwb_rc: remove a duplicated incomplete entry Mauro Carvalho Chehab
                   ` (12 subsequent siblings)
  14 siblings, 0 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Mauro Carvalho Chehab

From: Mauro Carvalho Chehab <mchehab@s-opensource.com>

The What: field on this ABI description use a different
syntax than expected, causing it to not be properly parsed
by script.

Fix it.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 Documentation/ABI/testing/sysfs-driver-hid | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-hid b/Documentation/ABI/testing/sysfs-driver-hid
index 48942cacb0bf..a59533410871 100644
--- a/Documentation/ABI/testing/sysfs-driver-hid
+++ b/Documentation/ABI/testing/sysfs-driver-hid
@@ -1,6 +1,6 @@
-What:		For USB devices	: /sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/report_descriptor
-		For BT devices	: /sys/class/bluetooth/hci<addr>/<hid-bus>:<vendor-id>:<product-id>.<num>/report_descriptor
-		Symlink		: /sys/class/hidraw/hidraw<num>/device/report_descriptor
+What:		/sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/report_descriptor
+What:		/sys/class/bluetooth/hci<addr>/<hid-bus>:<vendor-id>:<product-id>.<num>/report_descriptor
+What:		/sys/class/hidraw/hidraw<num>/device/report_descriptor
 Date:		Jan 2011
 KernelVersion:	2.0.39
 Contact:	Alan Ott <alan@signal11.us>
@@ -9,9 +9,9 @@ Description:	When read, this file returns the device's raw binary HID
 		This file cannot be written.
 Users:		HIDAPI library (http://www.signal11.us/oss/hidapi)
 
-What:		For USB devices	: /sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/country
-		For BT devices	: /sys/class/bluetooth/hci<addr>/<hid-bus>:<vendor-id>:<product-id>.<num>/country
-		Symlink		: /sys/class/hidraw/hidraw<num>/device/country
+What:		/sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/country
+What:		/sys/class/bluetooth/hci<addr>/<hid-bus>:<vendor-id>:<product-id>.<num>/country
+What:		/sys/class/hidraw/hidraw<num>/device/country
 Date:		February 2015
 KernelVersion:	3.19
 Contact:	Olivier Gay <ogay@logitech.com>
-- 
2.21.0


^ permalink raw reply	[flat|nested] 63+ messages in thread

* [PATCH 03/14] ABI: sysfs-class-uwb_rc: remove a duplicated incomplete entry
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
  2019-06-14  2:04 ` [PATCH 01/14] ABI: fix some syntax issues at the ABI database Mauro Carvalho Chehab
  2019-06-14  2:04 ` [PATCH 02/14] ABI: sysfs-driver-hid: the "What" field doesn't parse fine Mauro Carvalho Chehab
@ 2019-06-14  2:04 ` Mauro Carvalho Chehab
  2019-06-14  2:04 ` [PATCH 04/14] ABI: better identificate tables Mauro Carvalho Chehab
                   ` (11 subsequent siblings)
  14 siblings, 0 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Mauro Carvalho Chehab

From: Mauro Carvalho Chehab <mchehab@s-opensource.com>

There are two entries for /sys/class/uwb_rc/uwbN/<EUI-48>/BPST.
The second one has a missing description.

Get rid of it.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 Documentation/ABI/testing/sysfs-class-uwb_rc | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-class-uwb_rc b/Documentation/ABI/testing/sysfs-class-uwb_rc
index 85f4875d16ac..a0578751c1e3 100644
--- a/Documentation/ABI/testing/sysfs-class-uwb_rc
+++ b/Documentation/ABI/testing/sysfs-class-uwb_rc
@@ -125,12 +125,6 @@ Description:
                 The EUI-48 of this device in colon separated hex
                 octets.
 
-What:           /sys/class/uwb_rc/uwbN/<EUI-48>/BPST
-Date:           July 2008
-KernelVersion:  2.6.27
-Contact:        linux-usb@vger.kernel.org
-Description:
-
 What:           /sys/class/uwb_rc/uwbN/<EUI-48>/IEs
 Date:           July 2008
 KernelVersion:  2.6.27
-- 
2.21.0


^ permalink raw reply	[flat|nested] 63+ messages in thread

* [PATCH 04/14] ABI: better identificate tables
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
                   ` (2 preceding siblings ...)
  2019-06-14  2:04 ` [PATCH 03/14] ABI: sysfs-class-uwb_rc: remove a duplicated incomplete entry Mauro Carvalho Chehab
@ 2019-06-14  2:04 ` Mauro Carvalho Chehab
  2019-06-19 12:51   ` Johan Hovold
  2019-06-14  2:04 ` [PATCH 05/14] scripts: add an script to parse the ABI files Mauro Carvalho Chehab
                   ` (10 subsequent siblings)
  14 siblings, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Stefan Achatz, Mauro Carvalho Chehab

From: Mauro Carvalho Chehab <mchehab@s-opensource.com>

When parsing via script, it is important to know if the script
should consider a description as a literal block that should
be displayed as-is, or if the description can be considered
as a normal text.

Change descriptions to ensure that the preceding line of a table
ends with a colon. That makes easy to identify the need of a
literal block.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra   | 2 +-
 .../ABI/testing/sysfs-class-backlight-driver-lm3533       | 6 +++---
 Documentation/ABI/testing/sysfs-class-led-driver-lm3533   | 8 ++++----
 Documentation/ABI/testing/sysfs-class-leds-gt683r         | 4 ++--
 Documentation/ABI/testing/sysfs-driver-hid-roccat-kone    | 2 +-
 5 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra b/Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra
index 16020b31ae64..5d41ebadf15e 100644
--- a/Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra
+++ b/Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra
@@ -5,7 +5,7 @@ Description:	It is possible to switch the cpi setting of the mouse with the
 		press of a button.
 		When read, this file returns the raw number of the actual cpi
 		setting reported by the mouse. This number has to be further
-		processed to receive the real dpi value.
+		processed to receive the real dpi value:
 
 		VALUE DPI
 		1     400
diff --git a/Documentation/ABI/testing/sysfs-class-backlight-driver-lm3533 b/Documentation/ABI/testing/sysfs-class-backlight-driver-lm3533
index 77cf7ac949af..c0e0a9ae7b3d 100644
--- a/Documentation/ABI/testing/sysfs-class-backlight-driver-lm3533
+++ b/Documentation/ABI/testing/sysfs-class-backlight-driver-lm3533
@@ -4,7 +4,7 @@ KernelVersion:	3.5
 Contact:	Johan Hovold <jhovold@gmail.com>
 Description:
 		Get the ALS output channel used as input in
-		ALS-current-control mode (0, 1), where
+		ALS-current-control mode (0, 1), where:
 
 		0 - out_current0 (backlight 0)
 		1 - out_current1 (backlight 1)
@@ -28,7 +28,7 @@ Date:		April 2012
 KernelVersion:	3.5
 Contact:	Johan Hovold <jhovold@gmail.com>
 Description:
-		Set the brightness-mapping mode (0, 1), where
+		Set the brightness-mapping mode (0, 1), where:
 
 		0 - exponential mode
 		1 - linear mode
@@ -38,7 +38,7 @@ Date:		April 2012
 KernelVersion:	3.5
 Contact:	Johan Hovold <jhovold@gmail.com>
 Description:
-		Set the PWM-input control mask (5 bits), where
+		Set the PWM-input control mask (5 bits), where:
 
 		bit 5 - PWM-input enabled in Zone 4
 		bit 4 - PWM-input enabled in Zone 3
diff --git a/Documentation/ABI/testing/sysfs-class-led-driver-lm3533 b/Documentation/ABI/testing/sysfs-class-led-driver-lm3533
index 620ebb3b9baa..e4c89b261546 100644
--- a/Documentation/ABI/testing/sysfs-class-led-driver-lm3533
+++ b/Documentation/ABI/testing/sysfs-class-led-driver-lm3533
@@ -4,7 +4,7 @@ KernelVersion:	3.5
 Contact:	Johan Hovold <jhovold@gmail.com>
 Description:
 		Set the ALS output channel to use as input in
-		ALS-current-control mode (1, 2), where
+		ALS-current-control mode (1, 2), where:
 
 		1 - out_current1
 		2 - out_current2
@@ -22,7 +22,7 @@ Date:		April 2012
 KernelVersion:	3.5
 Contact:	Johan Hovold <jhovold@gmail.com>
 Description:
-		Set the pattern generator fall and rise times (0..7), where
+		Set the pattern generator fall and rise times (0..7), where:
 
 		0 - 2048 us
 		1 - 262 ms
@@ -45,7 +45,7 @@ Date:		April 2012
 KernelVersion:	3.5
 Contact:	Johan Hovold <jhovold@gmail.com>
 Description:
-		Set the brightness-mapping mode (0, 1), where
+		Set the brightness-mapping mode (0, 1), where:
 
 		0 - exponential mode
 		1 - linear mode
@@ -55,7 +55,7 @@ Date:		April 2012
 KernelVersion:	3.5
 Contact:	Johan Hovold <jhovold@gmail.com>
 Description:
-		Set the PWM-input control mask (5 bits), where
+		Set the PWM-input control mask (5 bits), where:
 
 		bit 5 - PWM-input enabled in Zone 4
 		bit 4 - PWM-input enabled in Zone 3
diff --git a/Documentation/ABI/testing/sysfs-class-leds-gt683r b/Documentation/ABI/testing/sysfs-class-leds-gt683r
index e4fae6026e79..6adab27f646e 100644
--- a/Documentation/ABI/testing/sysfs-class-leds-gt683r
+++ b/Documentation/ABI/testing/sysfs-class-leds-gt683r
@@ -5,7 +5,7 @@ Contact:	Janne Kanniainen <janne.kanniainen@gmail.com>
 Description:
 		Set the mode of LEDs. You should notice that changing the mode
 		of one LED will update the mode of its two sibling devices as
-		well.
+		well. Possible values are:
 
 		0 - normal
 		1 - audio
@@ -13,4 +13,4 @@ Description:
 
 		Normal: LEDs are fully on when enabled
 		Audio:  LEDs brightness depends on sound level
-		Breathing: LEDs brightness varies at human breathing rate
\ No newline at end of file
+		Breathing: LEDs brightness varies at human breathing rate
diff --git a/Documentation/ABI/testing/sysfs-driver-hid-roccat-kone b/Documentation/ABI/testing/sysfs-driver-hid-roccat-kone
index 3ca3971109bf..8f7982c70d72 100644
--- a/Documentation/ABI/testing/sysfs-driver-hid-roccat-kone
+++ b/Documentation/ABI/testing/sysfs-driver-hid-roccat-kone
@@ -5,7 +5,7 @@ Description:	It is possible to switch the dpi setting of the mouse with the
 		press of a button.
 		When read, this file returns the raw number of the actual dpi
 		setting reported by the mouse. This number has to be further
-		processed to receive the real dpi value.
+		processed to receive the real dpi value:
 
 		VALUE DPI
 		1     800
-- 
2.21.0


^ permalink raw reply	[flat|nested] 63+ messages in thread

* [PATCH 05/14] scripts: add an script to parse the ABI files
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
                   ` (3 preceding siblings ...)
  2019-06-14  2:04 ` [PATCH 04/14] ABI: better identificate tables Mauro Carvalho Chehab
@ 2019-06-14  2:04 ` Mauro Carvalho Chehab
  2019-06-14 13:31   ` Jani Nikula
  2019-06-14  2:04 ` [PATCH 06/14] scripts/get_abi.pl: parse files with text at beginning Mauro Carvalho Chehab
                   ` (9 subsequent siblings)
  14 siblings, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Mauro Carvalho Chehab

From: Mauro Carvalho Chehab <mchehab@s-opensource.com>

Add a script to parse the Documentation/ABI files and produce
an output with all entries inside an ABI (sub)directory.

Right now, it outputs its contents on ReST format. It shouldn't
be hard to make it produce other kind of outputs, since the ABI
file parser is implemented in separate than the output generator.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 scripts/get_abi.pl | 212 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 212 insertions(+)
 create mode 100755 scripts/get_abi.pl

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
new file mode 100755
index 000000000000..f7c9944a833c
--- /dev/null
+++ b/scripts/get_abi.pl
@@ -0,0 +1,212 @@
+#!/usr/bin/perl
+
+use strict;
+use Pod::Usage;
+use Getopt::Long;
+use File::Find;
+use Fcntl ':mode';
+
+my $help;
+my $man;
+my $debug;
+
+GetOptions(
+	"debug|d+" => \$debug,
+	'help|?' => \$help,
+	man => \$man
+) or pod2usage(2);
+
+pod2usage(1) if $help;
+pod2usage(-exitstatus => 0, -verbose => 2) if $man;
+
+pod2usage(2) if (scalar @ARGV != 1);
+
+my ($prefix) = @ARGV;
+
+require Data::Dumper if ($debug);
+
+my %data;
+
+#
+# Displays an error message, printing file name and line
+#
+sub parse_error($$$$) {
+	my ($file, $ln, $msg, $data) = @_;
+
+	print STDERR "file $file#$ln: $msg at\n\t$data";
+}
+
+#
+# Parse an ABI file, storing its contents at %data
+#
+sub parse_abi {
+	my $file = $File::Find::name;
+
+	my $mode = (stat($file))[2];
+	return if ($mode & S_IFDIR);
+	return if ($file =~ m,/README,);
+
+	my $name = $file;
+	$name =~ s,.*/,,;
+
+	my $type = $file;
+	$type =~ s,.*/(.*)/.*,$1,;
+
+	my $what;
+	my $new_what;
+	my $tag;
+	my $ln;
+
+	print STDERR "Opening $file\n" if ($debug > 1);
+	open IN, $file;
+	while(<IN>) {
+		$ln++;
+		if (m/^(\S+):\s*(.*)/i) {
+			my $new_tag = lc($1);
+			my $content = $2;
+
+			if (!($new_tag =~ m/(what|date|kernelversion|contact|description|users)/)) {
+				if ($tag eq "description") {
+					$data{$what}->{$tag} .= "\n$content";;
+					$data{$what}->{$tag} =~ s/\n+$//;
+					next;
+				} else {
+					parse_error($file, $ln, "tag '$tag' is invalid", $_);
+				}
+			}
+
+			if ($new_tag =~ m/what/) {
+				if ($tag =~ m/what/) {
+					$what .= ", " . $content;
+				} else {
+					$what = $content;
+					$new_what = 1;
+				}
+				$tag = $new_tag;
+				next;
+			}
+
+			$tag = $new_tag;
+
+			if ($new_what) {
+				$new_what = 0;
+
+				$data{$what}->{type} = $type;
+				$data{$what}->{file} = $name;
+				print STDERR "\twhat: $what\n" if ($debug > 1);
+			}
+
+			if (!$what) {
+				parse_error($file, $ln, "'What:' should come first:", $_);
+				next;
+			}
+			$data{$what}->{$tag} = $content;
+			next;
+		}
+
+		# Silently ignore any headers before the database
+		next if (!$tag);
+
+		if (m/^\s*(.*)/) {
+			$data{$what}->{$tag} .= "\n$1";
+			$data{$what}->{$tag} =~ s/\n+$//;
+			next;
+		}
+
+		# Everything else is error
+		parse_error($file, $ln, "Unexpected line:", $_);
+	}
+	close IN;
+}
+
+# Outputs the output on ReST format
+sub output_rest {
+	foreach my $what (sort keys %data) {
+		my $type = $data{$what}->{type};
+		my $file = $data{$what}->{file};
+
+		my $w = $what;
+		$w =~ s/([\(\)\_\-\*\=\^\~\\])/\\$1/g;
+
+		print "$w\n\n";
+		print "- defined on file $file (type: $type)\n\n::\n\n";
+
+		my $desc = $data{$what}->{description};
+		$desc =~ s/^\s+//;
+
+		# Remove title markups from the description, as they won't work
+		$desc =~ s/\n[\-\*\=\^\~]+\n/\n/g;
+
+		# put everything inside a code block
+		$desc =~ s/\n/\n /g;
+
+
+		if (!($desc =~ /^\s*$/)) {
+			print " $desc\n\n";
+		} else {
+			print " DESCRIPTION MISSING\n\n";
+		}
+	}
+}
+
+#
+# Parses all ABI files located at $prefix dir
+#
+find({wanted =>\&parse_abi, no_chdir => 1}, $prefix);
+
+print STDERR Data::Dumper->Dump([\%data], [qw(*data)]) if ($debug);
+
+#
+# Outputs an ReST file with the ABI contents
+#
+output_rest
+
+
+__END__
+
+=head1 NAME
+
+abi_book.pl - parse the Linux ABI files and produce a ReST book.
+
+=head1 SYNOPSIS
+
+B<abi_book.pl> [--debug] <ABI_DIR>]
+
+=head1 OPTIONS
+
+=over 8
+
+=item B<--debug>
+
+Put the script in verbose mode, useful for debugging. Can be called multiple
+times, to increase verbosity.
+
+=item B<--help>
+
+Prints a brief help message and exits.
+
+=item B<--man>
+
+Prints the manual page and exits.
+
+=back
+
+=head1 DESCRIPTION
+
+Parse the Linux ABI files from ABI DIR (usually located at Documentation/ABI)
+and produce a ReST book containing the Linux ABI.
+
+=head1 BUGS
+
+Report bugs to Mauro Carvalho Chehab <mchehab@s-opensource.com>
+
+=head1 COPYRIGHT
+
+Copyright (c) 2016 by Mauro Carvalho Chehab <mchehab@s-opensource.com>.
+
+License GPLv2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>.
+
+This is free software: you are free to change and redistribute it.
+There is NO WARRANTY, to the extent permitted by law.
+
+=cut
-- 
2.21.0


^ permalink raw reply	[flat|nested] 63+ messages in thread

* [PATCH 06/14] scripts/get_abi.pl: parse files with text at beginning
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
                   ` (4 preceding siblings ...)
  2019-06-14  2:04 ` [PATCH 05/14] scripts: add an script to parse the ABI files Mauro Carvalho Chehab
@ 2019-06-14  2:04 ` Mauro Carvalho Chehab
  2019-06-14  2:04 ` [PATCH 07/14] scripts/get_abi.pl: avoid use literal blocks when not needed Mauro Carvalho Chehab
                   ` (8 subsequent siblings)
  14 siblings, 0 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Mauro Carvalho Chehab

From: Mauro Carvalho Chehab <mchehab@s-opensource.com>

It sounds usefult o parse files with has some text at the
beginning. Add support for it.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 scripts/get_abi.pl | 59 ++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 54 insertions(+), 5 deletions(-)

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
index f7c9944a833c..ac0f057fa3ca 100755
--- a/scripts/get_abi.pl
+++ b/scripts/get_abi.pl
@@ -55,7 +55,10 @@ sub parse_abi {
 	my $what;
 	my $new_what;
 	my $tag;
+	my $label;
 	my $ln;
+	my $has_file;
+	my $xrefs;
 
 	print STDERR "Opening $file\n" if ($debug > 1);
 	open IN, $file;
@@ -67,7 +70,7 @@ sub parse_abi {
 
 			if (!($new_tag =~ m/(what|date|kernelversion|contact|description|users)/)) {
 				if ($tag eq "description") {
-					$data{$what}->{$tag} .= "\n$content";;
+					$data{$what}->{$tag} .= "\n$content";
 					$data{$what}->{$tag} =~ s/\n+$//;
 					next;
 				} else {
@@ -83,6 +86,25 @@ sub parse_abi {
 					$new_what = 1;
 				}
 				$tag = $new_tag;
+
+				if ($has_file) {
+					$label = "abi_" . $content . " ";
+					$label =~ tr/A-Z/a-z/;
+
+					# Convert special chars to "_"
+					$label =~s/[\x00-\x2f]+/_/g;
+					$label =~s/[\x3a-\x40]+/_/g;
+					$label =~s/[\x7b-\xff]+/_/g;
+					$label =~ s,_+,_,g;
+					$label =~ s,_$,,;
+
+					$data{$what}->{label} .= $label;
+
+					# Escape special chars from content
+					$content =~s/([\x00-\x1f\x21-\x2f\x3a-\x40\x7b-\xff])/\\$1/g;
+
+					$xrefs .= "- :ref:`$content <$label>`\n\n";
+				}
 				next;
 			}
 
@@ -104,8 +126,18 @@ sub parse_abi {
 			next;
 		}
 
-		# Silently ignore any headers before the database
-		next if (!$tag);
+		# Store any contents before the database
+		if (!$tag) {
+			next if (/^\n/);
+
+			my $my_what = "File $name";
+			$data{$my_what}->{what} = "File $name";
+			$data{$my_what}->{type} = "File";
+			$data{$my_what}->{file} = $name;
+			$data{$my_what}->{description} .= $_;
+			$has_file = 1;
+			next;
+		}
 
 		if (m/^\s*(.*)/) {
 			$data{$what}->{$tag} .= "\n$1";
@@ -117,6 +149,11 @@ sub parse_abi {
 		parse_error($file, $ln, "Unexpected line:", $_);
 	}
 	close IN;
+
+	if ($has_file) {
+		my $my_what = "File $name";
+                $data{$my_what}->{xrefs} = $xrefs;
+	}
 }
 
 # Outputs the output on ReST format
@@ -128,8 +165,17 @@ sub output_rest {
 		my $w = $what;
 		$w =~ s/([\(\)\_\-\*\=\^\~\\])/\\$1/g;
 
+		if ($data{$what}->{label}) {
+			my @labels = split(/\s/, $data{$what}->{label});
+			foreach my $label (@labels) {
+				printf ".. _%s:\n\n", $label;
+			}
+		}
+
 		print "$w\n\n";
-		print "- defined on file $file (type: $type)\n\n::\n\n";
+
+		print "- defined on file $file (type: $type)\n\n" if ($type ne "File");
+		print "::\n\n";
 
 		my $desc = $data{$what}->{description};
 		$desc =~ s/^\s+//;
@@ -144,8 +190,11 @@ sub output_rest {
 		if (!($desc =~ /^\s*$/)) {
 			print " $desc\n\n";
 		} else {
-			print " DESCRIPTION MISSING\n\n";
+			print " DESCRIPTION MISSING for $what\n\n";
 		}
+
+		printf "Has the following ABI:\n\n%s", $data{$what}->{xrefs} if ($data{$what}->{xrefs});
+
 	}
 }
 
-- 
2.21.0


^ permalink raw reply	[flat|nested] 63+ messages in thread

* [PATCH 07/14] scripts/get_abi.pl: avoid use literal blocks when not needed
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
                   ` (5 preceding siblings ...)
  2019-06-14  2:04 ` [PATCH 06/14] scripts/get_abi.pl: parse files with text at beginning Mauro Carvalho Chehab
@ 2019-06-14  2:04 ` Mauro Carvalho Chehab
  2019-06-14  2:04 ` [PATCH 08/14] scripts/get_abi.pl: split label naming from xref logic Mauro Carvalho Chehab
                   ` (7 subsequent siblings)
  14 siblings, 0 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Mauro Carvalho Chehab

From: Mauro Carvalho Chehab <mchehab@s-opensource.com>

The usage of literal blocks make the document very complex,
causing the browser to take a long time to load.

On most ABI descriptions, they're a plain text, and don't
require a literal block.

So, add a logic there with identifies when a literal block
is needed.

As, on literal blocks, we need to respect the original
document space, the most complex part of this patch is
to preserve the original spacing where needed.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 scripts/get_abi.pl | 104 +++++++++++++++++++++++++++++++++------------
 1 file changed, 78 insertions(+), 26 deletions(-)

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
index ac0f057fa3ca..ba8a7466f896 100755
--- a/scripts/get_abi.pl
+++ b/scripts/get_abi.pl
@@ -59,29 +59,34 @@ sub parse_abi {
 	my $ln;
 	my $has_file;
 	my $xrefs;
+	my $space;
 
 	print STDERR "Opening $file\n" if ($debug > 1);
 	open IN, $file;
 	while(<IN>) {
 		$ln++;
-		if (m/^(\S+):\s*(.*)/i) {
+		if (m/^(\S+)(:\s*)(.*)/i) {
 			my $new_tag = lc($1);
-			my $content = $2;
+			my $sep = $2;
+			my $content = $3;
 
 			if (!($new_tag =~ m/(what|date|kernelversion|contact|description|users)/)) {
 				if ($tag eq "description") {
-					$data{$what}->{$tag} .= "\n$content";
-					$data{$what}->{$tag} =~ s/\n+$//;
-					next;
+					# New "tag" is actually part of
+					# description. Don't consider it a tag
+					$new_tag = "";
 				} else {
 					parse_error($file, $ln, "tag '$tag' is invalid", $_);
 				}
 			}
 
 			if ($new_tag =~ m/what/) {
+				$space = "";
 				if ($tag =~ m/what/) {
 					$what .= ", " . $content;
 				} else {
+					parse_error($file, $ln, "What '$what' doesn't have a description", "") if ($what && !$data{$what}->{description});
+
 					$what = $content;
 					$new_what = 1;
 				}
@@ -108,25 +113,38 @@ sub parse_abi {
 				next;
 			}
 
-			$tag = $new_tag;
+			if ($new_tag) {
+				$tag = $new_tag;
 
-			if ($new_what) {
-				$new_what = 0;
+				if ($new_what) {
+					$new_what = 0;
 
-				$data{$what}->{type} = $type;
-				$data{$what}->{file} = $name;
-				print STDERR "\twhat: $what\n" if ($debug > 1);
-			}
+					$data{$what}->{type} = $type;
+					$data{$what}->{file} = $name;
+					print STDERR "\twhat: $what\n" if ($debug > 1);
+				}
 
-			if (!$what) {
-				parse_error($file, $ln, "'What:' should come first:", $_);
+				if (!$what) {
+					parse_error($file, $ln, "'What:' should come first:", $_);
+					next;
+				}
+				if ($tag eq "description") {
+					next if ($content =~ m/^\s*$/);
+					if ($content =~ m/^(\s*)(.*)/) {
+						my $new_content = $2;
+						$space = $new_tag . $sep . $1;
+						while ($space =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {}
+						$space =~ s/./ /g;
+						$data{$what}->{$tag} .= "$new_content\n";
+					}
+				} else {
+					$data{$what}->{$tag} = $content;
+				}
 				next;
 			}
-			$data{$what}->{$tag} = $content;
-			next;
 		}
 
-		# Store any contents before the database
+		# Store any contents before tags at the database
 		if (!$tag) {
 			next if (/^\n/);
 
@@ -139,6 +157,32 @@ sub parse_abi {
 			next;
 		}
 
+		if ($tag eq "description") {
+			if (!$data{$what}->{description}) {
+				next if (m/^\s*\n/);
+				if (m/^(\s*)(.*)/) {
+					$space = $1;
+					while ($space =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {}
+					$data{$what}->{$tag} .= "$2\n";
+				}
+			} else {
+				my $content = $_;
+				if (m/^\s*\n/) {
+					$data{$what}->{$tag} .= $content;
+					next;
+				}
+
+				while ($content =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {}
+				$space = "" if (!($content =~ s/^($space)//));
+
+				# Compress spaces with tabs
+				$content =~ s<^ {8}> <\t>;
+				$content =~ s<^ {1,7}\t> <\t>;
+				$content =~ s< {1,7}\t> <\t>;
+				$data{$what}->{$tag} .= $content;
+			}
+			next;
+		}
 		if (m/^\s*(.*)/) {
 			$data{$what}->{$tag} .= "\n$1";
 			$data{$what}->{$tag} =~ s/\n+$//;
@@ -165,6 +209,9 @@ sub output_rest {
 		my $w = $what;
 		$w =~ s/([\(\)\_\-\*\=\^\~\\])/\\$1/g;
 
+		my $bar = $w;
+		$bar =~ s/./-/g;
+
 		if ($data{$what}->{label}) {
 			my @labels = split(/\s/, $data{$what}->{label});
 			foreach my $label (@labels) {
@@ -172,10 +219,9 @@ sub output_rest {
 			}
 		}
 
-		print "$w\n\n";
+		print "$w\n$bar\n\n";
 
 		print "- defined on file $file (type: $type)\n\n" if ($type ne "File");
-		print "::\n\n";
 
 		my $desc = $data{$what}->{description};
 		$desc =~ s/^\s+//;
@@ -183,18 +229,24 @@ sub output_rest {
 		# Remove title markups from the description, as they won't work
 		$desc =~ s/\n[\-\*\=\^\~]+\n/\n/g;
 
-		# put everything inside a code block
-		$desc =~ s/\n/\n /g;
-
-
 		if (!($desc =~ /^\s*$/)) {
-			print " $desc\n\n";
+	                if ($desc =~ m/\:\n/ || $desc =~ m/\n[\t ]+/  || $desc =~ m/[\x00-\x08\x0b-\x1f\x7b-\xff]/) {
+				# put everything inside a code block
+				$desc =~ s/\n/\n /g;
+
+				print "::\n\n";
+				print " $desc\n\n";
+			} else {
+				# Escape any special chars from description
+				$desc =~s/([\x00-\x08\x0b-\x1f\x21-\x2a\x2d\x2f\x3c-\x40\x5c\x5e-\x60\x7b-\xff])/\\$1/g;
+
+				print "$desc\n\n";
+			}
 		} else {
-			print " DESCRIPTION MISSING for $what\n\n";
+			print "DESCRIPTION MISSING for $what\n\n";
 		}
 
 		printf "Has the following ABI:\n\n%s", $data{$what}->{xrefs} if ($data{$what}->{xrefs});
-
 	}
 }
 
-- 
2.21.0


^ permalink raw reply	[flat|nested] 63+ messages in thread

* [PATCH 08/14] scripts/get_abi.pl: split label naming from xref logic
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
                   ` (6 preceding siblings ...)
  2019-06-14  2:04 ` [PATCH 07/14] scripts/get_abi.pl: avoid use literal blocks when not needed Mauro Carvalho Chehab
@ 2019-06-14  2:04 ` Mauro Carvalho Chehab
  2019-06-14  2:04 ` [PATCH 09/14] scripts/get_abi.pl: add support for searching for ABI symbols Mauro Carvalho Chehab
                   ` (6 subsequent siblings)
  14 siblings, 0 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Mauro Carvalho Chehab

From: Mauro Carvalho Chehab <mchehab@s-opensource.com>

Instead of using a ReST compilant label while parsing,
move the label to ReST output. That makes the parsing logic
more generic, allowing it to provide other types of output.

As a side effect, now all files used to generate the output
will be output. We can later add command line arguments to
filter.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 scripts/get_abi.pl | 94 ++++++++++++++++++++++++++--------------------
 1 file changed, 53 insertions(+), 41 deletions(-)

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
index ba8a7466f896..d437e148b1c0 100755
--- a/scripts/get_abi.pl
+++ b/scripts/get_abi.pl
@@ -49,17 +49,23 @@ sub parse_abi {
 	my $name = $file;
 	$name =~ s,.*/,,;
 
+	my $nametag = "File $name";
+	$data{$nametag}->{what} = "File $name";
+	$data{$nametag}->{type} = "File";
+	$data{$nametag}->{file} = $name;
+	$data{$nametag}->{is_file} = 1;
+
 	my $type = $file;
 	$type =~ s,.*/(.*)/.*,$1,;
 
 	my $what;
 	my $new_what;
 	my $tag;
-	my $label;
 	my $ln;
-	my $has_file;
 	my $xrefs;
 	my $space;
+	my @labels;
+	my $label;
 
 	print STDERR "Opening $file\n" if ($debug > 1);
 	open IN, $file;
@@ -88,28 +94,13 @@ sub parse_abi {
 					parse_error($file, $ln, "What '$what' doesn't have a description", "") if ($what && !$data{$what}->{description});
 
 					$what = $content;
+					$label = $content;
 					$new_what = 1;
 				}
+				push @labels, [($content, $label)];
 				$tag = $new_tag;
 
-				if ($has_file) {
-					$label = "abi_" . $content . " ";
-					$label =~ tr/A-Z/a-z/;
-
-					# Convert special chars to "_"
-					$label =~s/[\x00-\x2f]+/_/g;
-					$label =~s/[\x3a-\x40]+/_/g;
-					$label =~s/[\x7b-\xff]+/_/g;
-					$label =~ s,_+,_,g;
-					$label =~ s,_$,,;
-
-					$data{$what}->{label} .= $label;
-
-					# Escape special chars from content
-					$content =~s/([\x00-\x1f\x21-\x2f\x3a-\x40\x7b-\xff])/\\$1/g;
-
-					$xrefs .= "- :ref:`$content <$label>`\n\n";
-				}
+				push @{$data{$nametag}->{xrefs}}, [($content, $label)] if ($data{$nametag}->{what});
 				next;
 			}
 
@@ -117,6 +108,9 @@ sub parse_abi {
 				$tag = $new_tag;
 
 				if ($new_what) {
+					@{$data{$what}->{label}} = @labels if ($data{$nametag}->{what});
+					@labels = ();
+					$label = "";
 					$new_what = 0;
 
 					$data{$what}->{type} = $type;
@@ -145,15 +139,8 @@ sub parse_abi {
 		}
 
 		# Store any contents before tags at the database
-		if (!$tag) {
-			next if (/^\n/);
-
-			my $my_what = "File $name";
-			$data{$my_what}->{what} = "File $name";
-			$data{$my_what}->{type} = "File";
-			$data{$my_what}->{file} = $name;
-			$data{$my_what}->{description} .= $_;
-			$has_file = 1;
+		if (!$tag && $data{$nametag}->{what}) {
+			$data{$nametag}->{description} .= $_;
 			next;
 		}
 
@@ -192,12 +179,8 @@ sub parse_abi {
 		# Everything else is error
 		parse_error($file, $ln, "Unexpected line:", $_);
 	}
+	$data{$nametag}->{description} =~ s/^\n+//;
 	close IN;
-
-	if ($has_file) {
-		my $my_what = "File $name";
-                $data{$my_what}->{xrefs} = $xrefs;
-	}
 }
 
 # Outputs the output on ReST format
@@ -212,11 +195,22 @@ sub output_rest {
 		my $bar = $w;
 		$bar =~ s/./-/g;
 
-		if ($data{$what}->{label}) {
-			my @labels = split(/\s/, $data{$what}->{label});
-			foreach my $label (@labels) {
-				printf ".. _%s:\n\n", $label;
-			}
+		foreach my $p (@{$data{$what}->{label}}) {
+			my ($content, $label) = @{$p};
+			$label = "abi_" . $label . " ";
+			$label =~ tr/A-Z/a-z/;
+
+			# Convert special chars to "_"
+			$label =~s/([\x00-\x2f\x3a-\x40\x5b-\x60\x7b-\xff])/_/g;
+			$label =~ s,_+,_,g;
+			$label =~ s,_$,,;
+
+			$data{$what}->{label} .= $label;
+
+			printf ".. _%s:\n\n", $label;
+
+			# only one label is enough
+			last;
 		}
 
 		print "$w\n$bar\n\n";
@@ -243,10 +237,28 @@ sub output_rest {
 				print "$desc\n\n";
 			}
 		} else {
-			print "DESCRIPTION MISSING for $what\n\n";
+			print "DESCRIPTION MISSING for $what\n\n" if (!$data{$what}->{is_file});
 		}
 
-		printf "Has the following ABI:\n\n%s", $data{$what}->{xrefs} if ($data{$what}->{xrefs});
+		if ($data{$what}->{xrefs}) {
+			printf "Has the following ABI:\n\n";
+
+			foreach my $p(@{$data{$what}->{xrefs}}) {
+				my ($content, $label) = @{$p};
+				$label = "abi_" . $label . " ";
+				$label =~ tr/A-Z/a-z/;
+
+				# Convert special chars to "_"
+				$label =~s/([\x00-\x2f\x3a-\x40\x5b-\x60\x7b-\xff])/_/g;
+				$label =~ s,_+,_,g;
+				$label =~ s,_$,,;
+
+				# Escape special chars from content
+				$content =~s/([\x00-\x1f\x21-\x2f\x3a-\x40\x7b-\xff])/\\$1/g;
+
+				print "- :ref:`$content <$label>`\n\n";
+			}
+		}
 	}
 }
 
-- 
2.21.0


^ permalink raw reply	[flat|nested] 63+ messages in thread

* [PATCH 09/14] scripts/get_abi.pl: add support for searching for ABI symbols
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
                   ` (7 preceding siblings ...)
  2019-06-14  2:04 ` [PATCH 08/14] scripts/get_abi.pl: split label naming from xref logic Mauro Carvalho Chehab
@ 2019-06-14  2:04 ` Mauro Carvalho Chehab
  2019-06-14  2:04 ` [PATCH 10/14] scripts/get_abi.pl: represent what in tables Mauro Carvalho Chehab
                   ` (5 subsequent siblings)
  14 siblings, 0 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Mauro Carvalho Chehab

From: Mauro Carvalho Chehab <mchehab@s-opensource.com>

Change its syntax to allow switching between ReST output mode
and a new search mode, with allows to seek for ABI symbols
using regex.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 scripts/get_abi.pl | 112 +++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 103 insertions(+), 9 deletions(-)

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
index d437e148b1c0..c202292af1a2 100755
--- a/scripts/get_abi.pl
+++ b/scripts/get_abi.pl
@@ -9,9 +9,11 @@ use Fcntl ':mode';
 my $help;
 my $man;
 my $debug;
+my $prefix="Documentation/ABI";
 
 GetOptions(
 	"debug|d+" => \$debug,
+	"dir=s" => \$prefix,
 	'help|?' => \$help,
 	man => \$man
 ) or pod2usage(2);
@@ -19,9 +21,12 @@ GetOptions(
 pod2usage(1) if $help;
 pod2usage(-exitstatus => 0, -verbose => 2) if $man;
 
-pod2usage(2) if (scalar @ARGV != 1);
+pod2usage(2) if (scalar @ARGV < 1 || @ARGV > 2);
 
-my ($prefix) = @ARGV;
+my ($cmd, $arg) = @ARGV;
+
+pod2usage(2) if ($cmd ne "search" && $cmd ne "rest");
+pod2usage(2) if ($cmd eq "search" && !$arg);
 
 require Data::Dumper if ($debug);
 
@@ -53,6 +58,7 @@ sub parse_abi {
 	$data{$nametag}->{what} = "File $name";
 	$data{$nametag}->{type} = "File";
 	$data{$nametag}->{file} = $name;
+	$data{$nametag}->{filepath} = $file;
 	$data{$nametag}->{is_file} = 1;
 
 	my $type = $file;
@@ -115,6 +121,7 @@ sub parse_abi {
 
 					$data{$what}->{type} = $type;
 					$data{$what}->{file} = $name;
+					$data{$what}->{filepath} = $file;
 					print STDERR "\twhat: $what\n" if ($debug > 1);
 				}
 
@@ -183,7 +190,9 @@ sub parse_abi {
 	close IN;
 }
 
-# Outputs the output on ReST format
+#
+# Outputs the book on ReST format
+#
 sub output_rest {
 	foreach my $what (sort keys %data) {
 		my $type = $data{$what}->{type};
@@ -262,6 +271,45 @@ sub output_rest {
 	}
 }
 
+#
+# Searches for ABI symbols
+#
+sub search_symbols {
+	foreach my $what (sort keys %data) {
+		next if (!($what =~ m/($arg)/));
+
+		my $type = $data{$what}->{type};
+		next if ($type eq "File");
+
+		my $file = $data{$what}->{filepath};
+
+		my $bar = $what;
+		$bar =~ s/./-/g;
+
+		print "\n$what\n$bar\n\n";
+
+		my $kernelversion = $data{$what}->{kernelversion};
+		my $contact = $data{$what}->{contact};
+		my $users = $data{$what}->{users};
+		my $date = $data{$what}->{date};
+		my $desc = $data{$what}->{description};
+		$kernelversion =~ s/^\s+//;
+		$contact =~ s/^\s+//;
+		$users =~ s/^\s+//;
+		$users =~ s/\n//g;
+		$date =~ s/^\s+//;
+		$desc =~ s/^\s+//;
+
+		printf "Kernel version:\t\t%s\n", $kernelversion if ($kernelversion);
+		printf "Date:\t\t\t%s\n", $date if ($date);
+		printf "Contact:\t\t%s\n", $contact if ($contact);
+		printf "Users:\t\t\t%s\n", $users if ($users);
+		print "Defined on file:\t$file\n\n";
+		print "Description:\n\n$desc";
+	}
+}
+
+
 #
 # Parses all ABI files located at $prefix dir
 #
@@ -270,9 +318,13 @@ find({wanted =>\&parse_abi, no_chdir => 1}, $prefix);
 print STDERR Data::Dumper->Dump([\%data], [qw(*data)]) if ($debug);
 
 #
-# Outputs an ReST file with the ABI contents
+# Handles the command
 #
-output_rest
+if ($cmd eq "rest") {
+	output_rest;
+} else {
+	search_symbols;
+}
 
 
 __END__
@@ -283,12 +335,27 @@ abi_book.pl - parse the Linux ABI files and produce a ReST book.
 
 =head1 SYNOPSIS
 
-B<abi_book.pl> [--debug] <ABI_DIR>]
+B<abi_book.pl> [--debug] <COMAND> [<ARGUMENT>]
+
+Where <COMMAND> can be:
+
+=over 8
+
+B<search> [SEARCH_REGEX] - search for [SEARCH_REGEX] inside ABI
+
+B<rest>   - output the ABI in ReST markup language
+
+=back
 
 =head1 OPTIONS
 
 =over 8
 
+=item B<--dir>
+
+Changes the location of the ABI search. By default, it uses
+the Documentation/ABI directory.
+
 =item B<--debug>
 
 Put the script in verbose mode, useful for debugging. Can be called multiple
@@ -306,8 +373,35 @@ Prints the manual page and exits.
 
 =head1 DESCRIPTION
 
-Parse the Linux ABI files from ABI DIR (usually located at Documentation/ABI)
-and produce a ReST book containing the Linux ABI.
+Parse the Linux ABI files from ABI DIR (usually located at Documentation/ABI),
+allowing to search for ABI symbols or to produce a ReST book containing
+the Linux ABI documentation.
+
+=head1 EXAMPLES
+
+Search for all stable symbols with the word "usb":
+
+=over 8
+
+$ scripts/get_abi.pl search usb --dir Documentation/ABI/stable
+
+=back
+
+Search for all symbols that match the regex expression "usb.*cap":
+
+=over 8
+
+$ scripts/get_abi.pl search usb.*cap
+
+=back
+
+Output all obsoleted symbols in ReST format
+
+=over 8
+
+$ scripts/get_abi.pl rest --dir Documentation/ABI/obsolete
+
+=back
 
 =head1 BUGS
 
@@ -315,7 +409,7 @@ Report bugs to Mauro Carvalho Chehab <mchehab@s-opensource.com>
 
 =head1 COPYRIGHT
 
-Copyright (c) 2016 by Mauro Carvalho Chehab <mchehab@s-opensource.com>.
+Copyright (c) 2016-2017 by Mauro Carvalho Chehab <mchehab@s-opensource.com>.
 
 License GPLv2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>.
 
-- 
2.21.0


^ permalink raw reply	[flat|nested] 63+ messages in thread

* [PATCH 10/14] scripts/get_abi.pl: represent what in tables
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
                   ` (8 preceding siblings ...)
  2019-06-14  2:04 ` [PATCH 09/14] scripts/get_abi.pl: add support for searching for ABI symbols Mauro Carvalho Chehab
@ 2019-06-14  2:04 ` Mauro Carvalho Chehab
  2019-06-14  2:04 ` [PATCH 11/14] scripts/get_abi.pl: fix parse issues with some files Mauro Carvalho Chehab
                   ` (4 subsequent siblings)
  14 siblings, 0 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Mauro Carvalho Chehab

From: Mauro Carvalho Chehab <mchehab@s-opensource.com>

Several entries at the ABI have multiple What: with the same
description.

Instead of showing those symbols as sections, let's show them
as tables. That makes easier to read on the final output,
and avoid too much recursion at Sphinx parsing.

We need to put file references at the end, as we don't want
non-file tables to be mangled with other entries.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 scripts/get_abi.pl | 41 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 36 insertions(+), 5 deletions(-)

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
index c202292af1a2..a2861fcec745 100755
--- a/scripts/get_abi.pl
+++ b/scripts/get_abi.pl
@@ -193,16 +193,19 @@ sub parse_abi {
 #
 # Outputs the book on ReST format
 #
+
 sub output_rest {
-	foreach my $what (sort keys %data) {
+	foreach my $what (sort {
+				($data{$a}->{type} eq "File") cmp ($data{$b}->{type} eq "File") ||
+				$a cmp $b
+			       } keys %data) {
 		my $type = $data{$what}->{type};
 		my $file = $data{$what}->{file};
+		my $filepath = $data{$what}->{filepath};
 
 		my $w = $what;
 		$w =~ s/([\(\)\_\-\*\=\^\~\\])/\\$1/g;
 
-		my $bar = $w;
-		$bar =~ s/./-/g;
 
 		foreach my $p (@{$data{$what}->{label}}) {
 			my ($content, $label) = @{$p};
@@ -222,9 +225,37 @@ sub output_rest {
 			last;
 		}
 
-		print "$w\n$bar\n\n";
 
-		print "- defined on file $file (type: $type)\n\n" if ($type ne "File");
+		$filepath =~ s,.*/(.*/.*),\1,;;
+		$filepath =~ s,[/\-],_,g;;
+		my $fileref = "abi_file_".$filepath;
+
+		if ($type eq "File") {
+			my $bar = $w;
+			$bar =~ s/./-/g;
+
+			print ".. _$fileref:\n\n";
+			print "$w\n$bar\n\n";
+		} else {
+			my @names = split /\s*,\s*/,$w;
+
+			my $len = 0;
+
+			foreach my $name (@names) {
+				$len = length($name) if (length($name) > $len);
+			}
+
+			print "What:\n\n";
+
+			print "+-" . "-" x $len . "-+\n";
+			foreach my $name (@names) {
+				printf "| %s", $name . " " x ($len - length($name)) . " |\n";
+				print "+-" . "-" x $len . "-+\n";
+			}
+			print "\n";
+		}
+
+		print "Defined on file :ref:`$file <$fileref>`\n\n" if ($type ne "File");
 
 		my $desc = $data{$what}->{description};
 		$desc =~ s/^\s+//;
-- 
2.21.0


^ permalink raw reply	[flat|nested] 63+ messages in thread

* [PATCH 11/14] scripts/get_abi.pl: fix parse issues with some files
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
                   ` (9 preceding siblings ...)
  2019-06-14  2:04 ` [PATCH 10/14] scripts/get_abi.pl: represent what in tables Mauro Carvalho Chehab
@ 2019-06-14  2:04 ` Mauro Carvalho Chehab
  2019-06-14  2:04 ` [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book Mauro Carvalho Chehab
                   ` (3 subsequent siblings)
  14 siblings, 0 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

A few files are failing to parse:

	Documentation/ABI/testing/sysfs-bus-pci-devices-aer_stats
	Documentation/ABI/testing/sysfs-class-pktcdvd
	Documentation/ABI/testing/sysfs-bus-nfit

On all three files, the problem is that there is a ":" character
at the initial file description.

Improve the parse in order to handle those special cases.

Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 scripts/get_abi.pl | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
index a2861fcec745..c3ccfdfc3a5d 100755
--- a/scripts/get_abi.pl
+++ b/scripts/get_abi.pl
@@ -87,7 +87,7 @@ sub parse_abi {
 					# New "tag" is actually part of
 					# description. Don't consider it a tag
 					$new_tag = "";
-				} else {
+				} elsif ($tag ne "") {
 					parse_error($file, $ln, "tag '$tag' is invalid", $_);
 				}
 			}
@@ -110,7 +110,7 @@ sub parse_abi {
 				next;
 			}
 
-			if ($new_tag) {
+			if ($tag ne "" && $new_tag) {
 				$tag = $new_tag;
 
 				if ($new_what) {
-- 
2.21.0


^ permalink raw reply	[flat|nested] 63+ messages in thread

* [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
                   ` (10 preceding siblings ...)
  2019-06-14  2:04 ` [PATCH 11/14] scripts/get_abi.pl: fix parse issues with some files Mauro Carvalho Chehab
@ 2019-06-14  2:04 ` Mauro Carvalho Chehab
  2019-06-14 13:42   ` Jani Nikula
  2019-06-14  2:04 ` [PATCH 13/14] sphinx/kernel_abi.py: make it compatible with Sphinx 1.7+ Mauro Carvalho Chehab
                   ` (2 subsequent siblings)
  14 siblings, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Mauro Carvalho Chehab

From: Mauro Carvalho Chehab <mchehab@s-opensource.com>

As we don't want a generic Sphinx extension to execute commands,
change the one proposed to Markus to call the abi_book.pl
script.

Use a script to parse the Documentation/ABI directory and output
it at the admin-guide.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 Documentation/admin-guide/abi-obsolete.rst |  10 ++
 Documentation/admin-guide/abi-removed.rst  |   4 +
 Documentation/admin-guide/abi-stable.rst   |  13 ++
 Documentation/admin-guide/abi-testing.rst  |  19 +++
 Documentation/admin-guide/abi.rst          |  11 ++
 Documentation/admin-guide/index.rst        |   1 +
 Documentation/conf.py                      |   2 +-
 Documentation/sphinx/kernel_abi.py         | 155 +++++++++++++++++++++
 8 files changed, 214 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/admin-guide/abi-obsolete.rst
 create mode 100644 Documentation/admin-guide/abi-removed.rst
 create mode 100644 Documentation/admin-guide/abi-stable.rst
 create mode 100644 Documentation/admin-guide/abi-testing.rst
 create mode 100644 Documentation/admin-guide/abi.rst
 create mode 100644 Documentation/sphinx/kernel_abi.py

diff --git a/Documentation/admin-guide/abi-obsolete.rst b/Documentation/admin-guide/abi-obsolete.rst
new file mode 100644
index 000000000000..cda9168445a5
--- /dev/null
+++ b/Documentation/admin-guide/abi-obsolete.rst
@@ -0,0 +1,10 @@
+ABI obsolete symbols
+====================
+
+Documents interfaces that are still remaining in the kernel, but are
+marked to be removed at some later point in time.
+
+The description of the interface will document the reason why it is
+obsolete and when it can be expected to be removed.
+
+.. kernel-abi:: $srctree/Documentation/ABI/obsolete
diff --git a/Documentation/admin-guide/abi-removed.rst b/Documentation/admin-guide/abi-removed.rst
new file mode 100644
index 000000000000..497978fc9632
--- /dev/null
+++ b/Documentation/admin-guide/abi-removed.rst
@@ -0,0 +1,4 @@
+ABI removed symbols
+===================
+
+.. kernel-abi:: $srctree/Documentation/ABI/removed
diff --git a/Documentation/admin-guide/abi-stable.rst b/Documentation/admin-guide/abi-stable.rst
new file mode 100644
index 000000000000..7495d7a35048
--- /dev/null
+++ b/Documentation/admin-guide/abi-stable.rst
@@ -0,0 +1,13 @@
+ABI stable symbols
+==================
+
+Documents the interfaces that the developer has defined to be stable.
+
+Userspace programs are free to use these interfaces with no
+restrictions, and backward compatibility for them will be guaranteed
+for at least 2 years.
+
+Most interfaces (like syscalls) are expected to never change and always
+be available.
+
+.. kernel-abi:: $srctree/Documentation/ABI/stable
diff --git a/Documentation/admin-guide/abi-testing.rst b/Documentation/admin-guide/abi-testing.rst
new file mode 100644
index 000000000000..5c886fc50b9e
--- /dev/null
+++ b/Documentation/admin-guide/abi-testing.rst
@@ -0,0 +1,19 @@
+ABI testing symbols
+===================
+
+Documents interfaces that are felt to be stable,
+as the main development of this interface has been completed.
+
+The interface can be changed to add new features, but the
+current interface will not break by doing this, unless grave
+errors or security problems are found in them.
+
+Userspace programs can start to rely on these interfaces, but they must
+be aware of changes that can occur before these interfaces move to
+be marked stable.
+
+Programs that use these interfaces are strongly encouraged to add their
+name to the description of these interfaces, so that the kernel
+developers can easily notify them if any changes occur.
+
+.. kernel-abi:: $srctree/Documentation/ABI/testing
diff --git a/Documentation/admin-guide/abi.rst b/Documentation/admin-guide/abi.rst
new file mode 100644
index 000000000000..3b9645c77469
--- /dev/null
+++ b/Documentation/admin-guide/abi.rst
@@ -0,0 +1,11 @@
+=====================
+Linux ABI description
+=====================
+
+.. toctree::
+   :maxdepth: 1
+
+   abi-stable
+   abi-testing
+   abi-obsolete
+   abi-removed
diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst
index 8001917ee012..20c3020fd73c 100644
--- a/Documentation/admin-guide/index.rst
+++ b/Documentation/admin-guide/index.rst
@@ -16,6 +16,7 @@ etc.
    README
    kernel-parameters
    devices
+   abi
 
 This section describes CPU vulnerabilities and their mitigations.
 
diff --git a/Documentation/conf.py b/Documentation/conf.py
index 7ace3f8852bd..598256fb5c98 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -34,7 +34,7 @@ needs_sphinx = '1.3'
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure', 'sphinx.ext.ifconfig']
+extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure', 'sphinx.ext.ifconfig', 'kernel_abi']
 
 # The name of the math extension changed on Sphinx 1.4
 if (major == 1 and minor > 3) or (major > 1):
diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
new file mode 100644
index 000000000000..32ce90775d96
--- /dev/null
+++ b/Documentation/sphinx/kernel_abi.py
@@ -0,0 +1,155 @@
+# -*- coding: utf-8; mode: python -*-
+u"""
+    kernel-abi
+    ~~~~~~~~~~
+
+    Implementation of the ``kernel-abi`` reST-directive.
+
+    :copyright:  Copyright (C) 2016  Markus Heiser
+    :copyright:  Copyright (C) 2016  Mauro Carvalho Chehab
+    :license:    GPL Version 2, June 1991 see Linux/COPYING for details.
+
+    The ``kernel-abi`` (:py:class:`KernelCmd`) directive calls the
+    scripts/get_abi.pl script to parse the Kernel ABI files.
+
+    Overview of directive's argument and options.
+
+    .. code-block:: rst
+
+        .. kernel-abi:: <ABI directory location>
+            :debug:
+
+    The argument ``<ABI directory location>`` is required. It contains the
+    location of the ABI files to be parsed.
+
+    ``debug``
+      Inserts a code-block with the *raw* reST. Sometimes it is helpful to see
+      what reST is generated.
+
+"""
+
+import sys
+import os
+from os import path
+import subprocess
+
+from sphinx.ext.autodoc import AutodocReporter
+
+from docutils import nodes
+from docutils.parsers.rst import Directive, directives
+from docutils.statemachine import ViewList
+from docutils.utils.error_reporting import ErrorString
+
+
+__version__  = '1.0'
+
+# We can't assume that six is installed
+PY3 = sys.version_info[0] == 3
+PY2 = sys.version_info[0] == 2
+if PY3:
+    # pylint: disable=C0103, W0622
+    unicode     = str
+    basestring  = str
+
+def setup(app):
+
+    app.add_directive("kernel-abi", KernelCmd)
+    return dict(
+        version = __version__
+        , parallel_read_safe = True
+        , parallel_write_safe = True
+    )
+
+class KernelCmd(Directive):
+
+    u"""KernelABI (``kernel-abi``) directive"""
+
+    required_arguments = 1
+    optional_arguments = 0
+    has_content = False
+    final_argument_whitespace = True
+
+    option_spec = {
+        "debug"     : directives.flag
+    }
+
+    def warn(self, message, **replace):
+        replace["fname"]   = self.state.document.current_source
+        replace["line_no"] = replace.get("line_no", self.lineno)
+        message = ("%(fname)s:%(line_no)s: [kernel-abi WARN] : " + message) % replace
+        self.state.document.settings.env.app.warn(message, prefix="")
+
+    def run(self):
+
+        doc = self.state.document
+        if not doc.settings.file_insertion_enabled:
+            raise self.warning("docutils: file insertion disabled")
+
+        env = doc.settings.env
+        cwd = path.dirname(doc.current_source)
+        cmd = "get_abi.pl rest --dir "
+        cmd += self.arguments[0]
+
+        srctree = path.abspath(os.environ["srctree"])
+
+        fname = cmd
+
+        # extend PATH with $(srctree)/scripts
+        path_env = os.pathsep.join([
+            srctree + os.sep + "scripts",
+            os.environ["PATH"]
+        ])
+        shell_env = os.environ.copy()
+        shell_env["PATH"]    = path_env
+        shell_env["srctree"] = srctree
+
+        lines = self.runCmd(cmd, shell=True, cwd=cwd, env=shell_env)
+        nodeList = self.nestedParse(lines, fname)
+        return nodeList
+
+    def runCmd(self, cmd, **kwargs):
+        u"""Run command ``cmd`` and return it's stdout as unicode."""
+
+        try:
+            proc = subprocess.Popen(
+                cmd
+                , stdout = subprocess.PIPE
+                , stderr = subprocess.PIPE
+                , universal_newlines = True
+                , **kwargs
+            )
+            out, err = proc.communicate()
+            if err:
+                self.warn(err)
+            if proc.returncode != 0:
+                raise self.severe(
+                    u"command '%s' failed with return code %d"
+                    % (cmd, proc.returncode)
+                )
+        except OSError as exc:
+            raise self.severe(u"problems with '%s' directive: %s."
+                              % (self.name, ErrorString(exc)))
+        return unicode(out)
+
+    def nestedParse(self, lines, fname):
+        content = ViewList()
+        node    = nodes.section()
+
+        if "debug" in self.options:
+            code_block = "\n\n.. code-block:: rst\n    :linenos:\n"
+            for l in lines.split("\n"):
+                code_block += "\n    " + l
+            lines = code_block + "\n\n"
+
+        for c, l in enumerate(lines.split("\n")):
+            content.append(l, fname, c)
+
+        buf  = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter
+        self.state.memo.title_styles  = []
+        self.state.memo.section_level = 0
+        self.state.memo.reporter      = AutodocReporter(content, self.state.memo.reporter)
+        try:
+            self.state.nested_parse(content, 0, node, match_titles=1)
+        finally:
+            self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf
+        return node.children
-- 
2.21.0


^ permalink raw reply	[flat|nested] 63+ messages in thread

* [PATCH 13/14] sphinx/kernel_abi.py: make it compatible with Sphinx 1.7+
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
                   ` (11 preceding siblings ...)
  2019-06-14  2:04 ` [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book Mauro Carvalho Chehab
@ 2019-06-14  2:04 ` Mauro Carvalho Chehab
  2019-06-14  2:04 ` [PATCH 14/14] docs: sphinx/kernel_abi.py: fix UTF-8 support Mauro Carvalho Chehab
  2019-06-14 16:20 ` [PATCH 00/14] Add support to generate ABI documentation at admin-guide Greg Kroah-Hartman
  14 siblings, 0 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

The same way kerneldoc.py needed changes to work with newer
Sphinx, this script needs the same changes.

Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 Documentation/sphinx/kernel_abi.py | 31 ++++++++++++++++++++++--------
 1 file changed, 23 insertions(+), 8 deletions(-)

diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index 32ce90775d96..7fa7806532dc 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -33,7 +33,16 @@ import os
 from os import path
 import subprocess
 
-from sphinx.ext.autodoc import AutodocReporter
+#
+# AutodocReporter is only good up to Sphinx 1.7
+#
+import sphinx
+
+Use_SSI = sphinx.__version__[:3] >= '1.7'
+if Use_SSI:
+    from sphinx.util.docutils import switch_source_input
+else:
+    from sphinx.ext.autodoc import AutodocReporter
 
 from docutils import nodes
 from docutils.parsers.rst import Directive, directives
@@ -145,11 +154,17 @@ class KernelCmd(Directive):
             content.append(l, fname, c)
 
         buf  = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter
-        self.state.memo.title_styles  = []
-        self.state.memo.section_level = 0
-        self.state.memo.reporter      = AutodocReporter(content, self.state.memo.reporter)
-        try:
-            self.state.nested_parse(content, 0, node, match_titles=1)
-        finally:
-            self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf
+
+        if Use_SSI:
+            with switch_source_input(self.state, content):
+                self.state.nested_parse(content, 0, node, match_titles=1)
+        else:
+            self.state.memo.title_styles  = []
+            self.state.memo.section_level = 0
+            self.state.memo.reporter      = AutodocReporter(content, self.state.memo.reporter)
+            try:
+                self.state.nested_parse(content, 0, node, match_titles=1)
+            finally:
+                self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf
+
         return node.children
-- 
2.21.0


^ permalink raw reply	[flat|nested] 63+ messages in thread

* [PATCH 14/14] docs: sphinx/kernel_abi.py: fix UTF-8 support
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
                   ` (12 preceding siblings ...)
  2019-06-14  2:04 ` [PATCH 13/14] sphinx/kernel_abi.py: make it compatible with Sphinx 1.7+ Mauro Carvalho Chehab
@ 2019-06-14  2:04 ` Mauro Carvalho Chehab
  2019-06-14 16:18   ` Greg Kroah-Hartman
  2019-06-14 16:20 ` [PATCH 00/14] Add support to generate ABI documentation at admin-guide Greg Kroah-Hartman
  14 siblings, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14  2:04 UTC (permalink / raw)
  To: Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

The parser breaks with UTF-8 characters with Sphinx 1.4.

Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---
 Documentation/sphinx/kernel_abi.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index 7fa7806532dc..460cee48a245 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -1,4 +1,5 @@
-# -*- coding: utf-8; mode: python -*-
+# coding=utf-8
+#
 u"""
     kernel-abi
     ~~~~~~~~~~
@@ -28,6 +29,7 @@ u"""
 
 """
 
+import codecs
 import sys
 import os
 from os import path
@@ -124,12 +126,12 @@ class KernelCmd(Directive):
                 cmd
                 , stdout = subprocess.PIPE
                 , stderr = subprocess.PIPE
-                , universal_newlines = True
                 , **kwargs
             )
             out, err = proc.communicate()
-            if err:
-                self.warn(err)
+
+            out, err = codecs.decode(out, 'utf-8'), codecs.decode(err, 'utf-8')
+
             if proc.returncode != 0:
                 raise self.severe(
                     u"command '%s' failed with return code %d"
-- 
2.21.0


^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 01/14] ABI: fix some syntax issues at the ABI database
  2019-06-14  2:04 ` [PATCH 01/14] ABI: fix some syntax issues at the ABI database Mauro Carvalho Chehab
@ 2019-06-14  7:20   ` Andrew Donnellan
  2019-06-14 16:16     ` Greg Kroah-Hartman
  2019-06-14  8:30   ` Rafael J. Wysocki
  1 sibling, 1 reply; 63+ messages in thread
From: Andrew Donnellan @ 2019-06-14  7:20 UTC (permalink / raw)
  To: Mauro Carvalho Chehab, Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Andreas Klinger, Jonathan Cameron,
	Hartmut Knaack, Lars-Peter Clausen, Peter Meerwald-Stadler,
	Frederic Barrat, Sebastian Reichel, Kees Cook, Anton Vorontsov,
	Colin Cross, Tony Luck, linux-iio, linuxppc-dev, linux-pm

On 14/6/19 12:04 pm, Mauro Carvalho Chehab wrote:
> diff --git a/Documentation/ABI/testing/sysfs-class-cxl b/Documentation/ABI/testing/sysfs-class-cxl
> index bbbabffc682a..fc7c6f7c21b3 100644
> --- a/Documentation/ABI/testing/sysfs-class-cxl
> +++ b/Documentation/ABI/testing/sysfs-class-cxl
> @@ -1,6 +1,6 @@
> -Note: Attributes that are shared between devices are stored in the directory
> -pointed to by the symlink device/.
> -Example: The real path of the attribute /sys/class/cxl/afu0.0s/irqs_max is
> +Please notice that attributes that are shared between devices are stored in

Would prefer "Please note" over "Please notice".

Acked-by: Andrew Donnellan <ajd@linux.ibm.com>  # cxl


-- 
Andrew Donnellan              OzLabs, ADL Canberra
ajd@linux.ibm.com             IBM Australia Limited


^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 01/14] ABI: fix some syntax issues at the ABI database
  2019-06-14  2:04 ` [PATCH 01/14] ABI: fix some syntax issues at the ABI database Mauro Carvalho Chehab
  2019-06-14  7:20   ` Andrew Donnellan
@ 2019-06-14  8:30   ` Rafael J. Wysocki
  1 sibling, 0 replies; 63+ messages in thread
From: Rafael J. Wysocki @ 2019-06-14  8:30 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Linux Doc Mailing List, Greg Kroah-Hartman,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab,
	Linux Kernel Mailing List, Jonathan Corbet, Andreas Klinger,
	Jonathan Cameron, Hartmut Knaack, Lars-Peter Clausen,
	Peter Meerwald-Stadler, Frederic Barrat, Andrew Donnellan,
	Sebastian Reichel, Kees Cook, Anton Vorontsov, Colin Cross,
	Tony Luck, linux-iio, linuxppc-dev, Linux PM

On Fri, Jun 14, 2019 at 4:04 AM Mauro Carvalho Chehab
<mchehab+samsung@kernel.org> wrote:
>
> From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
>
> On those three files, the ABI representation described at
> README are violated.
>
> - at sysfs-bus-iio-proximity-as3935:
>         a ':' character is missing after "What"
>
> - at sysfs-class-devfreq:
>         there's a typo at Description
>
> - at sysfs-class-cxl, it is using the ":" character at a
>         file preamble, causing it to be misinterpreted as a
>         tag.
>
> - On the other files, instead of "What", they use "Where".
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>

Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

> ---
>  Documentation/ABI/testing/pstore              |  2 +-
>  .../sysfs-bus-event_source-devices-format     |  2 +-
>  .../ABI/testing/sysfs-bus-i2c-devices-hm6352  |  6 ++---
>  .../ABI/testing/sysfs-bus-iio-distance-srf08  |  4 ++--
>  .../testing/sysfs-bus-iio-proximity-as3935    |  4 ++--
>  .../ABI/testing/sysfs-bus-pci-devices-cciss   | 22 +++++++++----------
>  .../testing/sysfs-bus-usb-devices-usbsevseg   | 12 +++++-----
>  Documentation/ABI/testing/sysfs-class-cxl     |  6 ++---
>  Documentation/ABI/testing/sysfs-class-devfreq |  2 +-
>  .../ABI/testing/sysfs-class-powercap          |  2 +-
>  Documentation/ABI/testing/sysfs-kernel-fscaps |  2 +-
>  .../ABI/testing/sysfs-kernel-vmcoreinfo       |  2 +-
>  12 files changed, 33 insertions(+), 33 deletions(-)
>
> diff --git a/Documentation/ABI/testing/pstore b/Documentation/ABI/testing/pstore
> index 5fca9f5e10a3..8d6e48f4e8ef 100644
> --- a/Documentation/ABI/testing/pstore
> +++ b/Documentation/ABI/testing/pstore
> @@ -1,4 +1,4 @@
> -Where:         /sys/fs/pstore/... (or /dev/pstore/...)
> +What:          /sys/fs/pstore/... (or /dev/pstore/...)
>  Date:          March 2011
>  Kernel Version: 2.6.39
>  Contact:       tony.luck@intel.com
> diff --git a/Documentation/ABI/testing/sysfs-bus-event_source-devices-format b/Documentation/ABI/testing/sysfs-bus-event_source-devices-format
> index 77f47ff5ee02..b6f8748e0200 100644
> --- a/Documentation/ABI/testing/sysfs-bus-event_source-devices-format
> +++ b/Documentation/ABI/testing/sysfs-bus-event_source-devices-format
> @@ -1,4 +1,4 @@
> -Where:         /sys/bus/event_source/devices/<dev>/format
> +What:          /sys/bus/event_source/devices/<dev>/format
>  Date:          January 2012
>  Kernel Version: 3.3
>  Contact:       Jiri Olsa <jolsa@redhat.com>
> diff --git a/Documentation/ABI/testing/sysfs-bus-i2c-devices-hm6352 b/Documentation/ABI/testing/sysfs-bus-i2c-devices-hm6352
> index feb2e4a87075..29bd447e50a0 100644
> --- a/Documentation/ABI/testing/sysfs-bus-i2c-devices-hm6352
> +++ b/Documentation/ABI/testing/sysfs-bus-i2c-devices-hm6352
> @@ -1,18 +1,18 @@
> -Where:         /sys/bus/i2c/devices/.../heading0_input
> +What:          /sys/bus/i2c/devices/.../heading0_input
>  Date:          April 2010
>  Kernel Version: 2.6.36?
>  Contact:       alan.cox@intel.com
>  Description:   Reports the current heading from the compass as a floating
>                 point value in degrees.
>
> -Where:         /sys/bus/i2c/devices/.../power_state
> +What:          /sys/bus/i2c/devices/.../power_state
>  Date:          April 2010
>  Kernel Version: 2.6.36?
>  Contact:       alan.cox@intel.com
>  Description:   Sets the power state of the device. 0 sets the device into
>                 sleep mode, 1 wakes it up.
>
> -Where:         /sys/bus/i2c/devices/.../calibration
> +What:          /sys/bus/i2c/devices/.../calibration
>  Date:          April 2010
>  Kernel Version: 2.6.36?
>  Contact:       alan.cox@intel.com
> diff --git a/Documentation/ABI/testing/sysfs-bus-iio-distance-srf08 b/Documentation/ABI/testing/sysfs-bus-iio-distance-srf08
> index 0a1ca1487fa9..a133fd8d081a 100644
> --- a/Documentation/ABI/testing/sysfs-bus-iio-distance-srf08
> +++ b/Documentation/ABI/testing/sysfs-bus-iio-distance-srf08
> @@ -1,4 +1,4 @@
> -What           /sys/bus/iio/devices/iio:deviceX/sensor_sensitivity
> +What:          /sys/bus/iio/devices/iio:deviceX/sensor_sensitivity
>  Date:          January 2017
>  KernelVersion: 4.11
>  Contact:       linux-iio@vger.kernel.org
> @@ -6,7 +6,7 @@ Description:
>                 Show or set the gain boost of the amp, from 0-31 range.
>                 default 31
>
> -What           /sys/bus/iio/devices/iio:deviceX/sensor_max_range
> +What:          /sys/bus/iio/devices/iio:deviceX/sensor_max_range
>  Date:          January 2017
>  KernelVersion: 4.11
>  Contact:       linux-iio@vger.kernel.org
> diff --git a/Documentation/ABI/testing/sysfs-bus-iio-proximity-as3935 b/Documentation/ABI/testing/sysfs-bus-iio-proximity-as3935
> index 9a17ab5036a4..c59d95346341 100644
> --- a/Documentation/ABI/testing/sysfs-bus-iio-proximity-as3935
> +++ b/Documentation/ABI/testing/sysfs-bus-iio-proximity-as3935
> @@ -1,4 +1,4 @@
> -What           /sys/bus/iio/devices/iio:deviceX/in_proximity_input
> +What:          /sys/bus/iio/devices/iio:deviceX/in_proximity_input
>  Date:          March 2014
>  KernelVersion: 3.15
>  Contact:       Matt Ranostay <matt.ranostay@konsulko.com>
> @@ -6,7 +6,7 @@ Description:
>                 Get the current distance in meters of storm (1km steps)
>                 1000-40000 = distance in meters
>
> -What           /sys/bus/iio/devices/iio:deviceX/sensor_sensitivity
> +What:          /sys/bus/iio/devices/iio:deviceX/sensor_sensitivity
>  Date:          March 2014
>  KernelVersion: 3.15
>  Contact:       Matt Ranostay <matt.ranostay@konsulko.com>
> diff --git a/Documentation/ABI/testing/sysfs-bus-pci-devices-cciss b/Documentation/ABI/testing/sysfs-bus-pci-devices-cciss
> index 53d99edd1d75..eb449169c30b 100644
> --- a/Documentation/ABI/testing/sysfs-bus-pci-devices-cciss
> +++ b/Documentation/ABI/testing/sysfs-bus-pci-devices-cciss
> @@ -1,66 +1,66 @@
> -Where:         /sys/bus/pci/devices/<dev>/ccissX/cXdY/model
> +What:          /sys/bus/pci/devices/<dev>/ccissX/cXdY/model
>  Date:          March 2009
>  Kernel Version: 2.6.30
>  Contact:       iss_storagedev@hp.com
>  Description:   Displays the SCSI INQUIRY page 0 model for logical drive
>                 Y of controller X.
>
> -Where:         /sys/bus/pci/devices/<dev>/ccissX/cXdY/rev
> +What:          /sys/bus/pci/devices/<dev>/ccissX/cXdY/rev
>  Date:          March 2009
>  Kernel Version: 2.6.30
>  Contact:       iss_storagedev@hp.com
>  Description:   Displays the SCSI INQUIRY page 0 revision for logical
>                 drive Y of controller X.
>
> -Where:         /sys/bus/pci/devices/<dev>/ccissX/cXdY/unique_id
> +What:          /sys/bus/pci/devices/<dev>/ccissX/cXdY/unique_id
>  Date:          March 2009
>  Kernel Version: 2.6.30
>  Contact:       iss_storagedev@hp.com
>  Description:   Displays the SCSI INQUIRY page 83 serial number for logical
>                 drive Y of controller X.
>
> -Where:         /sys/bus/pci/devices/<dev>/ccissX/cXdY/vendor
> +What:          /sys/bus/pci/devices/<dev>/ccissX/cXdY/vendor
>  Date:          March 2009
>  Kernel Version: 2.6.30
>  Contact:       iss_storagedev@hp.com
>  Description:   Displays the SCSI INQUIRY page 0 vendor for logical drive
>                 Y of controller X.
>
> -Where:         /sys/bus/pci/devices/<dev>/ccissX/cXdY/block:cciss!cXdY
> +What:          /sys/bus/pci/devices/<dev>/ccissX/cXdY/block:cciss!cXdY
>  Date:          March 2009
>  Kernel Version: 2.6.30
>  Contact:       iss_storagedev@hp.com
>  Description:   A symbolic link to /sys/block/cciss!cXdY
>
> -Where:         /sys/bus/pci/devices/<dev>/ccissX/rescan
> +What:          /sys/bus/pci/devices/<dev>/ccissX/rescan
>  Date:          August 2009
>  Kernel Version:        2.6.31
>  Contact:       iss_storagedev@hp.com
>  Description:   Kicks of a rescan of the controller to discover logical
>                 drive topology changes.
>
> -Where:         /sys/bus/pci/devices/<dev>/ccissX/cXdY/lunid
> +What:          /sys/bus/pci/devices/<dev>/ccissX/cXdY/lunid
>  Date:          August 2009
>  Kernel Version: 2.6.31
>  Contact:       iss_storagedev@hp.com
>  Description:   Displays the 8-byte LUN ID used to address logical
>                 drive Y of controller X.
>
> -Where:         /sys/bus/pci/devices/<dev>/ccissX/cXdY/raid_level
> +What:          /sys/bus/pci/devices/<dev>/ccissX/cXdY/raid_level
>  Date:          August 2009
>  Kernel Version: 2.6.31
>  Contact:       iss_storagedev@hp.com
>  Description:   Displays the RAID level of logical drive Y of
>                 controller X.
>
> -Where:         /sys/bus/pci/devices/<dev>/ccissX/cXdY/usage_count
> +What:          /sys/bus/pci/devices/<dev>/ccissX/cXdY/usage_count
>  Date:          August 2009
>  Kernel Version: 2.6.31
>  Contact:       iss_storagedev@hp.com
>  Description:   Displays the usage count (number of opens) of logical drive Y
>                 of controller X.
>
> -Where:         /sys/bus/pci/devices/<dev>/ccissX/resettable
> +What:          /sys/bus/pci/devices/<dev>/ccissX/resettable
>  Date:          February 2011
>  Kernel Version:        2.6.38
>  Contact:       iss_storagedev@hp.com
> @@ -71,7 +71,7 @@ Description:  Value of 1 indicates the controller can honor the reset_devices
>                 a dump device, as kdump requires resetting the device in order
>                 to work reliably.
>
> -Where:         /sys/bus/pci/devices/<dev>/ccissX/transport_mode
> +What:          /sys/bus/pci/devices/<dev>/ccissX/transport_mode
>  Date:          July 2011
>  Kernel Version:        3.0
>  Contact:       iss_storagedev@hp.com
> diff --git a/Documentation/ABI/testing/sysfs-bus-usb-devices-usbsevseg b/Documentation/ABI/testing/sysfs-bus-usb-devices-usbsevseg
> index 70d00dfa443d..f6199b314196 100644
> --- a/Documentation/ABI/testing/sysfs-bus-usb-devices-usbsevseg
> +++ b/Documentation/ABI/testing/sysfs-bus-usb-devices-usbsevseg
> @@ -1,12 +1,12 @@
> -Where:         /sys/bus/usb/.../powered
> +What:          /sys/bus/usb/.../powered
>  Date:          August 2008
>  Kernel Version:        2.6.26
>  Contact:       Harrison Metzger <harrisonmetz@gmail.com>
>  Description:   Controls whether the device's display will powered.
>                 A value of 0 is off and a non-zero value is on.
>
> -Where:         /sys/bus/usb/.../mode_msb
> -Where:         /sys/bus/usb/.../mode_lsb
> +What:          /sys/bus/usb/.../mode_msb
> +What:          /sys/bus/usb/.../mode_lsb
>  Date:          August 2008
>  Kernel Version:        2.6.26
>  Contact:       Harrison Metzger <harrisonmetz@gmail.com>
> @@ -16,7 +16,7 @@ Description:  Controls the devices display mode.
>                 for an 8 character display the values are
>                         MSB 0x08; LSB 0xFF.
>
> -Where:         /sys/bus/usb/.../textmode
> +What:          /sys/bus/usb/.../textmode
>  Date:          August 2008
>  Kernel Version:        2.6.26
>  Contact:       Harrison Metzger <harrisonmetz@gmail.com>
> @@ -25,13 +25,13 @@ Description:        Controls the way the device interprets its text buffer.
>                 hex:    each character is between 0-15
>                 ascii:  each character is between '0'-'9' and 'A'-'F'.
>
> -Where:         /sys/bus/usb/.../text
> +What:          /sys/bus/usb/.../text
>  Date:          August 2008
>  Kernel Version:        2.6.26
>  Contact:       Harrison Metzger <harrisonmetz@gmail.com>
>  Description:   The text (or data) for the device to display
>
> -Where:         /sys/bus/usb/.../decimals
> +What:          /sys/bus/usb/.../decimals
>  Date:          August 2008
>  Kernel Version:        2.6.26
>  Contact:       Harrison Metzger <harrisonmetz@gmail.com>
> diff --git a/Documentation/ABI/testing/sysfs-class-cxl b/Documentation/ABI/testing/sysfs-class-cxl
> index bbbabffc682a..fc7c6f7c21b3 100644
> --- a/Documentation/ABI/testing/sysfs-class-cxl
> +++ b/Documentation/ABI/testing/sysfs-class-cxl
> @@ -1,6 +1,6 @@
> -Note: Attributes that are shared between devices are stored in the directory
> -pointed to by the symlink device/.
> -Example: The real path of the attribute /sys/class/cxl/afu0.0s/irqs_max is
> +Please notice that attributes that are shared between devices are stored in
> +the directory pointed to by the symlink device/.
> +For example, the real path of the attribute /sys/class/cxl/afu0.0s/irqs_max is
>  /sys/class/cxl/afu0.0s/device/irqs_max, i.e. /sys/class/cxl/afu0.0/irqs_max.
>
>
> diff --git a/Documentation/ABI/testing/sysfs-class-devfreq b/Documentation/ABI/testing/sysfs-class-devfreq
> index ee39acacf6f8..01196e19afca 100644
> --- a/Documentation/ABI/testing/sysfs-class-devfreq
> +++ b/Documentation/ABI/testing/sysfs-class-devfreq
> @@ -47,7 +47,7 @@ Description:
>  What:          /sys/class/devfreq/.../trans_stat
>  Date:          October 2012
>  Contact:       MyungJoo Ham <myungjoo.ham@samsung.com>
> -Descrtiption:
> +Description:
>                 This ABI shows the statistics of devfreq behavior on a
>                 specific device. It shows the time spent in each state and
>                 the number of transitions between states.
> diff --git a/Documentation/ABI/testing/sysfs-class-powercap b/Documentation/ABI/testing/sysfs-class-powercap
> index db3b3ff70d84..f333a0ccc29b 100644
> --- a/Documentation/ABI/testing/sysfs-class-powercap
> +++ b/Documentation/ABI/testing/sysfs-class-powercap
> @@ -147,6 +147,6 @@ What:               /sys/class/powercap/.../<power zone>/enabled
>  Date:          September 2013
>  KernelVersion: 3.13
>  Contact:       linux-pm@vger.kernel.org
> -Description
> +Description:
>                 This allows to enable/disable power capping at power zone level.
>                 This applies to current power zone and its children.
> diff --git a/Documentation/ABI/testing/sysfs-kernel-fscaps b/Documentation/ABI/testing/sysfs-kernel-fscaps
> index 50a3033b5e15..bcff34665192 100644
> --- a/Documentation/ABI/testing/sysfs-kernel-fscaps
> +++ b/Documentation/ABI/testing/sysfs-kernel-fscaps
> @@ -2,7 +2,7 @@ What:           /sys/kernel/fscaps
>  Date:          February 2011
>  KernelVersion: 2.6.38
>  Contact:       Ludwig Nussel <ludwig.nussel@suse.de>
> -Description
> +Description:
>                 Shows whether file system capabilities are honored
>                 when executing a binary
>
> diff --git a/Documentation/ABI/testing/sysfs-kernel-vmcoreinfo b/Documentation/ABI/testing/sysfs-kernel-vmcoreinfo
> index 7bd81168e063..1f1087a5f075 100644
> --- a/Documentation/ABI/testing/sysfs-kernel-vmcoreinfo
> +++ b/Documentation/ABI/testing/sysfs-kernel-vmcoreinfo
> @@ -4,7 +4,7 @@ KernelVersion:  2.6.24
>  Contact:       Ken'ichi Ohmichi <oomichi@mxs.nes.nec.co.jp>
>                 Kexec Mailing List <kexec@lists.infradead.org>
>                 Vivek Goyal <vgoyal@redhat.com>
> -Description
> +Description:
>                 Shows physical address and size of vmcoreinfo ELF note.
>                 First value contains physical address of note in hex and
>                 second value contains the size of note in hex. This ELF
> --
> 2.21.0
>

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 05/14] scripts: add an script to parse the ABI files
  2019-06-14  2:04 ` [PATCH 05/14] scripts: add an script to parse the ABI files Mauro Carvalho Chehab
@ 2019-06-14 13:31   ` Jani Nikula
  2019-06-14 13:39     ` Greg Kroah-Hartman
  0 siblings, 1 reply; 63+ messages in thread
From: Jani Nikula @ 2019-06-14 13:31 UTC (permalink / raw)
  To: Mauro Carvalho Chehab, Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Mauro Carvalho Chehab

On Thu, 13 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
>
> Add a script to parse the Documentation/ABI files and produce
> an output with all entries inside an ABI (sub)directory.
>
> Right now, it outputs its contents on ReST format. It shouldn't
> be hard to make it produce other kind of outputs, since the ABI
> file parser is implemented in separate than the output generator.

Hum, or just convert the ABI files to rst directly.

BR,
Jani.



>
> Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> ---
>  scripts/get_abi.pl | 212 +++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 212 insertions(+)
>  create mode 100755 scripts/get_abi.pl
>
> diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
> new file mode 100755
> index 000000000000..f7c9944a833c
> --- /dev/null
> +++ b/scripts/get_abi.pl
> @@ -0,0 +1,212 @@
> +#!/usr/bin/perl
> +
> +use strict;
> +use Pod::Usage;
> +use Getopt::Long;
> +use File::Find;
> +use Fcntl ':mode';
> +
> +my $help;
> +my $man;
> +my $debug;
> +
> +GetOptions(
> +	"debug|d+" => \$debug,
> +	'help|?' => \$help,
> +	man => \$man
> +) or pod2usage(2);
> +
> +pod2usage(1) if $help;
> +pod2usage(-exitstatus => 0, -verbose => 2) if $man;
> +
> +pod2usage(2) if (scalar @ARGV != 1);
> +
> +my ($prefix) = @ARGV;
> +
> +require Data::Dumper if ($debug);
> +
> +my %data;
> +
> +#
> +# Displays an error message, printing file name and line
> +#
> +sub parse_error($$$$) {
> +	my ($file, $ln, $msg, $data) = @_;
> +
> +	print STDERR "file $file#$ln: $msg at\n\t$data";
> +}
> +
> +#
> +# Parse an ABI file, storing its contents at %data
> +#
> +sub parse_abi {
> +	my $file = $File::Find::name;
> +
> +	my $mode = (stat($file))[2];
> +	return if ($mode & S_IFDIR);
> +	return if ($file =~ m,/README,);
> +
> +	my $name = $file;
> +	$name =~ s,.*/,,;
> +
> +	my $type = $file;
> +	$type =~ s,.*/(.*)/.*,$1,;
> +
> +	my $what;
> +	my $new_what;
> +	my $tag;
> +	my $ln;
> +
> +	print STDERR "Opening $file\n" if ($debug > 1);
> +	open IN, $file;
> +	while(<IN>) {
> +		$ln++;
> +		if (m/^(\S+):\s*(.*)/i) {
> +			my $new_tag = lc($1);
> +			my $content = $2;
> +
> +			if (!($new_tag =~ m/(what|date|kernelversion|contact|description|users)/)) {
> +				if ($tag eq "description") {
> +					$data{$what}->{$tag} .= "\n$content";;
> +					$data{$what}->{$tag} =~ s/\n+$//;
> +					next;
> +				} else {
> +					parse_error($file, $ln, "tag '$tag' is invalid", $_);
> +				}
> +			}
> +
> +			if ($new_tag =~ m/what/) {
> +				if ($tag =~ m/what/) {
> +					$what .= ", " . $content;
> +				} else {
> +					$what = $content;
> +					$new_what = 1;
> +				}
> +				$tag = $new_tag;
> +				next;
> +			}
> +
> +			$tag = $new_tag;
> +
> +			if ($new_what) {
> +				$new_what = 0;
> +
> +				$data{$what}->{type} = $type;
> +				$data{$what}->{file} = $name;
> +				print STDERR "\twhat: $what\n" if ($debug > 1);
> +			}
> +
> +			if (!$what) {
> +				parse_error($file, $ln, "'What:' should come first:", $_);
> +				next;
> +			}
> +			$data{$what}->{$tag} = $content;
> +			next;
> +		}
> +
> +		# Silently ignore any headers before the database
> +		next if (!$tag);
> +
> +		if (m/^\s*(.*)/) {
> +			$data{$what}->{$tag} .= "\n$1";
> +			$data{$what}->{$tag} =~ s/\n+$//;
> +			next;
> +		}
> +
> +		# Everything else is error
> +		parse_error($file, $ln, "Unexpected line:", $_);
> +	}
> +	close IN;
> +}
> +
> +# Outputs the output on ReST format
> +sub output_rest {
> +	foreach my $what (sort keys %data) {
> +		my $type = $data{$what}->{type};
> +		my $file = $data{$what}->{file};
> +
> +		my $w = $what;
> +		$w =~ s/([\(\)\_\-\*\=\^\~\\])/\\$1/g;
> +
> +		print "$w\n\n";
> +		print "- defined on file $file (type: $type)\n\n::\n\n";
> +
> +		my $desc = $data{$what}->{description};
> +		$desc =~ s/^\s+//;
> +
> +		# Remove title markups from the description, as they won't work
> +		$desc =~ s/\n[\-\*\=\^\~]+\n/\n/g;
> +
> +		# put everything inside a code block
> +		$desc =~ s/\n/\n /g;
> +
> +
> +		if (!($desc =~ /^\s*$/)) {
> +			print " $desc\n\n";
> +		} else {
> +			print " DESCRIPTION MISSING\n\n";
> +		}
> +	}
> +}
> +
> +#
> +# Parses all ABI files located at $prefix dir
> +#
> +find({wanted =>\&parse_abi, no_chdir => 1}, $prefix);
> +
> +print STDERR Data::Dumper->Dump([\%data], [qw(*data)]) if ($debug);
> +
> +#
> +# Outputs an ReST file with the ABI contents
> +#
> +output_rest
> +
> +
> +__END__
> +
> +=head1 NAME
> +
> +abi_book.pl - parse the Linux ABI files and produce a ReST book.
> +
> +=head1 SYNOPSIS
> +
> +B<abi_book.pl> [--debug] <ABI_DIR>]
> +
> +=head1 OPTIONS
> +
> +=over 8
> +
> +=item B<--debug>
> +
> +Put the script in verbose mode, useful for debugging. Can be called multiple
> +times, to increase verbosity.
> +
> +=item B<--help>
> +
> +Prints a brief help message and exits.
> +
> +=item B<--man>
> +
> +Prints the manual page and exits.
> +
> +=back
> +
> +=head1 DESCRIPTION
> +
> +Parse the Linux ABI files from ABI DIR (usually located at Documentation/ABI)
> +and produce a ReST book containing the Linux ABI.
> +
> +=head1 BUGS
> +
> +Report bugs to Mauro Carvalho Chehab <mchehab@s-opensource.com>
> +
> +=head1 COPYRIGHT
> +
> +Copyright (c) 2016 by Mauro Carvalho Chehab <mchehab@s-opensource.com>.
> +
> +License GPLv2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>.
> +
> +This is free software: you are free to change and redistribute it.
> +There is NO WARRANTY, to the extent permitted by law.
> +
> +=cut

-- 
Jani Nikula, Intel Open Source Graphics Center

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 05/14] scripts: add an script to parse the ABI files
  2019-06-14 13:31   ` Jani Nikula
@ 2019-06-14 13:39     ` Greg Kroah-Hartman
  2019-06-14 13:58       ` Mauro Carvalho Chehab
  2019-06-14 14:00       ` Jani Nikula
  0 siblings, 2 replies; 63+ messages in thread
From: Greg Kroah-Hartman @ 2019-06-14 13:39 UTC (permalink / raw)
  To: Jani Nikula
  Cc: Mauro Carvalho Chehab, Linux Doc Mailing List,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

On Fri, Jun 14, 2019 at 04:31:56PM +0300, Jani Nikula wrote:
> On Thu, 13 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> > From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> >
> > Add a script to parse the Documentation/ABI files and produce
> > an output with all entries inside an ABI (sub)directory.
> >
> > Right now, it outputs its contents on ReST format. It shouldn't
> > be hard to make it produce other kind of outputs, since the ABI
> > file parser is implemented in separate than the output generator.
> 
> Hum, or just convert the ABI files to rst directly.

And what would that look like?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-14  2:04 ` [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book Mauro Carvalho Chehab
@ 2019-06-14 13:42   ` Jani Nikula
  2019-06-14 14:06     ` Greg Kroah-Hartman
  2019-06-14 14:10     ` Markus Heiser
  0 siblings, 2 replies; 63+ messages in thread
From: Jani Nikula @ 2019-06-14 13:42 UTC (permalink / raw)
  To: Mauro Carvalho Chehab, Linux Doc Mailing List, Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Mauro Carvalho Chehab

On Thu, 13 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
>
> As we don't want a generic Sphinx extension to execute commands,
> change the one proposed to Markus to call the abi_book.pl
> script.
>
> Use a script to parse the Documentation/ABI directory and output
> it at the admin-guide.

We had a legacy kernel-doc perl script so we used that instead of
rewriting it in python. Just to keep it bug-for-bug compatible with the
past. That was the only reason.

I see absolutely zero reason to add a new perl monstrosity with a python
extension to call it. All of this could be better done in python,
directly.

Please don't complicate the documentation build. I know you know we all
worked hard to take apart the old DocBook Rube Goldberg machine to
replace it with Sphinx. Please don't turn the Sphinx build to another
complicated mess.

My strong preferences are, in this order:

1) Convert the ABI documentation to reStructuredText

2) Have the python extension read the ABI files directly, without an
   extra pipeline.


BR,
Jani.


>
> Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> ---
>  Documentation/admin-guide/abi-obsolete.rst |  10 ++
>  Documentation/admin-guide/abi-removed.rst  |   4 +
>  Documentation/admin-guide/abi-stable.rst   |  13 ++
>  Documentation/admin-guide/abi-testing.rst  |  19 +++
>  Documentation/admin-guide/abi.rst          |  11 ++
>  Documentation/admin-guide/index.rst        |   1 +
>  Documentation/conf.py                      |   2 +-
>  Documentation/sphinx/kernel_abi.py         | 155 +++++++++++++++++++++
>  8 files changed, 214 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/admin-guide/abi-obsolete.rst
>  create mode 100644 Documentation/admin-guide/abi-removed.rst
>  create mode 100644 Documentation/admin-guide/abi-stable.rst
>  create mode 100644 Documentation/admin-guide/abi-testing.rst
>  create mode 100644 Documentation/admin-guide/abi.rst
>  create mode 100644 Documentation/sphinx/kernel_abi.py
>
> diff --git a/Documentation/admin-guide/abi-obsolete.rst b/Documentation/admin-guide/abi-obsolete.rst
> new file mode 100644
> index 000000000000..cda9168445a5
> --- /dev/null
> +++ b/Documentation/admin-guide/abi-obsolete.rst
> @@ -0,0 +1,10 @@
> +ABI obsolete symbols
> +====================
> +
> +Documents interfaces that are still remaining in the kernel, but are
> +marked to be removed at some later point in time.
> +
> +The description of the interface will document the reason why it is
> +obsolete and when it can be expected to be removed.
> +
> +.. kernel-abi:: $srctree/Documentation/ABI/obsolete
> diff --git a/Documentation/admin-guide/abi-removed.rst b/Documentation/admin-guide/abi-removed.rst
> new file mode 100644
> index 000000000000..497978fc9632
> --- /dev/null
> +++ b/Documentation/admin-guide/abi-removed.rst
> @@ -0,0 +1,4 @@
> +ABI removed symbols
> +===================
> +
> +.. kernel-abi:: $srctree/Documentation/ABI/removed
> diff --git a/Documentation/admin-guide/abi-stable.rst b/Documentation/admin-guide/abi-stable.rst
> new file mode 100644
> index 000000000000..7495d7a35048
> --- /dev/null
> +++ b/Documentation/admin-guide/abi-stable.rst
> @@ -0,0 +1,13 @@
> +ABI stable symbols
> +==================
> +
> +Documents the interfaces that the developer has defined to be stable.
> +
> +Userspace programs are free to use these interfaces with no
> +restrictions, and backward compatibility for them will be guaranteed
> +for at least 2 years.
> +
> +Most interfaces (like syscalls) are expected to never change and always
> +be available.
> +
> +.. kernel-abi:: $srctree/Documentation/ABI/stable
> diff --git a/Documentation/admin-guide/abi-testing.rst b/Documentation/admin-guide/abi-testing.rst
> new file mode 100644
> index 000000000000..5c886fc50b9e
> --- /dev/null
> +++ b/Documentation/admin-guide/abi-testing.rst
> @@ -0,0 +1,19 @@
> +ABI testing symbols
> +===================
> +
> +Documents interfaces that are felt to be stable,
> +as the main development of this interface has been completed.
> +
> +The interface can be changed to add new features, but the
> +current interface will not break by doing this, unless grave
> +errors or security problems are found in them.
> +
> +Userspace programs can start to rely on these interfaces, but they must
> +be aware of changes that can occur before these interfaces move to
> +be marked stable.
> +
> +Programs that use these interfaces are strongly encouraged to add their
> +name to the description of these interfaces, so that the kernel
> +developers can easily notify them if any changes occur.
> +
> +.. kernel-abi:: $srctree/Documentation/ABI/testing
> diff --git a/Documentation/admin-guide/abi.rst b/Documentation/admin-guide/abi.rst
> new file mode 100644
> index 000000000000..3b9645c77469
> --- /dev/null
> +++ b/Documentation/admin-guide/abi.rst
> @@ -0,0 +1,11 @@
> +=====================
> +Linux ABI description
> +=====================
> +
> +.. toctree::
> +   :maxdepth: 1
> +
> +   abi-stable
> +   abi-testing
> +   abi-obsolete
> +   abi-removed
> diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst
> index 8001917ee012..20c3020fd73c 100644
> --- a/Documentation/admin-guide/index.rst
> +++ b/Documentation/admin-guide/index.rst
> @@ -16,6 +16,7 @@ etc.
>     README
>     kernel-parameters
>     devices
> +   abi
>  
>  This section describes CPU vulnerabilities and their mitigations.
>  
> diff --git a/Documentation/conf.py b/Documentation/conf.py
> index 7ace3f8852bd..598256fb5c98 100644
> --- a/Documentation/conf.py
> +++ b/Documentation/conf.py
> @@ -34,7 +34,7 @@ needs_sphinx = '1.3'
>  # Add any Sphinx extension module names here, as strings. They can be
>  # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
>  # ones.
> -extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure', 'sphinx.ext.ifconfig']
> +extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure', 'sphinx.ext.ifconfig', 'kernel_abi']
>  
>  # The name of the math extension changed on Sphinx 1.4
>  if (major == 1 and minor > 3) or (major > 1):
> diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
> new file mode 100644
> index 000000000000..32ce90775d96
> --- /dev/null
> +++ b/Documentation/sphinx/kernel_abi.py
> @@ -0,0 +1,155 @@
> +# -*- coding: utf-8; mode: python -*-
> +u"""
> +    kernel-abi
> +    ~~~~~~~~~~
> +
> +    Implementation of the ``kernel-abi`` reST-directive.
> +
> +    :copyright:  Copyright (C) 2016  Markus Heiser
> +    :copyright:  Copyright (C) 2016  Mauro Carvalho Chehab
> +    :license:    GPL Version 2, June 1991 see Linux/COPYING for details.
> +
> +    The ``kernel-abi`` (:py:class:`KernelCmd`) directive calls the
> +    scripts/get_abi.pl script to parse the Kernel ABI files.
> +
> +    Overview of directive's argument and options.
> +
> +    .. code-block:: rst
> +
> +        .. kernel-abi:: <ABI directory location>
> +            :debug:
> +
> +    The argument ``<ABI directory location>`` is required. It contains the
> +    location of the ABI files to be parsed.
> +
> +    ``debug``
> +      Inserts a code-block with the *raw* reST. Sometimes it is helpful to see
> +      what reST is generated.
> +
> +"""
> +
> +import sys
> +import os
> +from os import path
> +import subprocess
> +
> +from sphinx.ext.autodoc import AutodocReporter
> +
> +from docutils import nodes
> +from docutils.parsers.rst import Directive, directives
> +from docutils.statemachine import ViewList
> +from docutils.utils.error_reporting import ErrorString
> +
> +
> +__version__  = '1.0'
> +
> +# We can't assume that six is installed
> +PY3 = sys.version_info[0] == 3
> +PY2 = sys.version_info[0] == 2
> +if PY3:
> +    # pylint: disable=C0103, W0622
> +    unicode     = str
> +    basestring  = str
> +
> +def setup(app):
> +
> +    app.add_directive("kernel-abi", KernelCmd)
> +    return dict(
> +        version = __version__
> +        , parallel_read_safe = True
> +        , parallel_write_safe = True
> +    )
> +
> +class KernelCmd(Directive):
> +
> +    u"""KernelABI (``kernel-abi``) directive"""
> +
> +    required_arguments = 1
> +    optional_arguments = 0
> +    has_content = False
> +    final_argument_whitespace = True
> +
> +    option_spec = {
> +        "debug"     : directives.flag
> +    }
> +
> +    def warn(self, message, **replace):
> +        replace["fname"]   = self.state.document.current_source
> +        replace["line_no"] = replace.get("line_no", self.lineno)
> +        message = ("%(fname)s:%(line_no)s: [kernel-abi WARN] : " + message) % replace
> +        self.state.document.settings.env.app.warn(message, prefix="")
> +
> +    def run(self):
> +
> +        doc = self.state.document
> +        if not doc.settings.file_insertion_enabled:
> +            raise self.warning("docutils: file insertion disabled")
> +
> +        env = doc.settings.env
> +        cwd = path.dirname(doc.current_source)
> +        cmd = "get_abi.pl rest --dir "
> +        cmd += self.arguments[0]
> +
> +        srctree = path.abspath(os.environ["srctree"])
> +
> +        fname = cmd
> +
> +        # extend PATH with $(srctree)/scripts
> +        path_env = os.pathsep.join([
> +            srctree + os.sep + "scripts",
> +            os.environ["PATH"]
> +        ])
> +        shell_env = os.environ.copy()
> +        shell_env["PATH"]    = path_env
> +        shell_env["srctree"] = srctree
> +
> +        lines = self.runCmd(cmd, shell=True, cwd=cwd, env=shell_env)
> +        nodeList = self.nestedParse(lines, fname)
> +        return nodeList
> +
> +    def runCmd(self, cmd, **kwargs):
> +        u"""Run command ``cmd`` and return it's stdout as unicode."""
> +
> +        try:
> +            proc = subprocess.Popen(
> +                cmd
> +                , stdout = subprocess.PIPE
> +                , stderr = subprocess.PIPE
> +                , universal_newlines = True
> +                , **kwargs
> +            )
> +            out, err = proc.communicate()
> +            if err:
> +                self.warn(err)
> +            if proc.returncode != 0:
> +                raise self.severe(
> +                    u"command '%s' failed with return code %d"
> +                    % (cmd, proc.returncode)
> +                )
> +        except OSError as exc:
> +            raise self.severe(u"problems with '%s' directive: %s."
> +                              % (self.name, ErrorString(exc)))
> +        return unicode(out)
> +
> +    def nestedParse(self, lines, fname):
> +        content = ViewList()
> +        node    = nodes.section()
> +
> +        if "debug" in self.options:
> +            code_block = "\n\n.. code-block:: rst\n    :linenos:\n"
> +            for l in lines.split("\n"):
> +                code_block += "\n    " + l
> +            lines = code_block + "\n\n"
> +
> +        for c, l in enumerate(lines.split("\n")):
> +            content.append(l, fname, c)
> +
> +        buf  = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter
> +        self.state.memo.title_styles  = []
> +        self.state.memo.section_level = 0
> +        self.state.memo.reporter      = AutodocReporter(content, self.state.memo.reporter)
> +        try:
> +            self.state.nested_parse(content, 0, node, match_titles=1)
> +        finally:
> +            self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf
> +        return node.children

-- 
Jani Nikula, Intel Open Source Graphics Center

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 05/14] scripts: add an script to parse the ABI files
  2019-06-14 13:39     ` Greg Kroah-Hartman
@ 2019-06-14 13:58       ` Mauro Carvalho Chehab
  2019-06-14 14:00       ` Jani Nikula
  1 sibling, 0 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14 13:58 UTC (permalink / raw)
  To: Greg Kroah-Hartman
  Cc: Jani Nikula, Linux Doc Mailing List, Mauro Carvalho Chehab,
	Mauro Carvalho Chehab, linux-kernel, Jonathan Corbet

Em Fri, 14 Jun 2019 15:39:33 +0200
Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:

> On Fri, Jun 14, 2019 at 04:31:56PM +0300, Jani Nikula wrote:
> > On Thu, 13 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:  
> > > From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> > >
> > > Add a script to parse the Documentation/ABI files and produce
> > > an output with all entries inside an ABI (sub)directory.
> > >
> > > Right now, it outputs its contents on ReST format. It shouldn't
> > > be hard to make it produce other kind of outputs, since the ABI
> > > file parser is implemented in separate than the output generator.  
> > 
> > Hum, or just convert the ABI files to rst directly.  

Converting ABI files to rst could be easily done using a modified
version of my script (plus the produced files will likely need some
manual review).

There is a drawback though: we'll lose the capability of being able
to parse ABI files via an script, as people would be freed to use a
different syntax.

We could minimize it by using things like:

:What:
:Kernel Version:
...

> And what would that look like?

That's the big question :-)

If you prefer something like that, I can write the patches.

Thanks,
Mauro

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 05/14] scripts: add an script to parse the ABI files
  2019-06-14 13:39     ` Greg Kroah-Hartman
  2019-06-14 13:58       ` Mauro Carvalho Chehab
@ 2019-06-14 14:00       ` Jani Nikula
  2019-06-14 14:14         ` Jonathan Corbet
  2019-06-14 16:24         ` Greg Kroah-Hartman
  1 sibling, 2 replies; 63+ messages in thread
From: Jani Nikula @ 2019-06-14 14:00 UTC (permalink / raw)
  To: Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Linux Doc Mailing List,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

On Fri, 14 Jun 2019, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> On Fri, Jun 14, 2019 at 04:31:56PM +0300, Jani Nikula wrote:
>> On Thu, 13 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
>> > From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
>> >
>> > Add a script to parse the Documentation/ABI files and produce
>> > an output with all entries inside an ABI (sub)directory.
>> >
>> > Right now, it outputs its contents on ReST format. It shouldn't
>> > be hard to make it produce other kind of outputs, since the ABI
>> > file parser is implemented in separate than the output generator.
>> 
>> Hum, or just convert the ABI files to rst directly.
>
> And what would that look like?

That pretty much depends on the requirements we want to set on both the
ABI source files and the generated output. Obviously the requirements
can be conflicting; might be hard to produce fancy output if the input
is very limited.

At the bare minimum, you could convert the files to contain
reStructuredText field lists [1]. Add a colon at the start of the field
name, and make sure field bodies (values) are not empty.

Conversion of a file selected at random; I've only added ":" and "N/A".

diff --git a/Documentation/ABI/stable/sysfs-devices-system-cpu b/Documentation/ABI/stable/sysfs-devices-system-cpu
index 33c133e2a631..34c218b344fb 100644
--- a/Documentation/ABI/stable/sysfs-devices-system-cpu
+++ b/Documentation/ABI/stable/sysfs-devices-system-cpu
@@ -1,19 +1,20 @@
-What: 		/sys/devices/system/cpu/dscr_default
-Date:		13-May-2014
-KernelVersion:	v3.15.0
-Contact:
-Description:	Writes are equivalent to writing to
+:What: 		/sys/devices/system/cpu/dscr_default
+:Date:		13-May-2014
+:KernelVersion:	v3.15.0
+:Contact:	N/A
+:Description:	Writes are equivalent to writing to
 		/sys/devices/system/cpu/cpuN/dscr on all CPUs.
 		Reads return the last written value or 0.
 		This value is not a global default: it is a way to set
 		all per-CPU defaults at the same time.
-Values:		64 bit unsigned integer (bit field)
+:Values:	64 bit unsigned integer (bit field)
 
-What: 		/sys/devices/system/cpu/cpu[0-9]+/dscr
-Date:		13-May-2014
-KernelVersion:	v3.15.0
-Contact:
-Description:	Default value for the Data Stream Control Register (DSCR) on
+
+:What: 		/sys/devices/system/cpu/cpu[0-9]+/dscr
+:Date:		13-May-2014
+:KernelVersion:	v3.15.0
+:Contact:	N/A
+:Description:	Default value for the Data Stream Control Register (DSCR) on
 		a CPU.
 		This default value is used when the kernel is executing and
 		for any process that has not set the DSCR itself.
@@ -22,4 +23,4 @@ Description:	Default value for the Data Stream Control Register (DSCR) on
 		on any CPU where it executes (overriding the value described
 		here).
 		If set by a process it will be inherited by child processes.
-Values:		64 bit unsigned integer (bit field)
+:Values:		64 bit unsigned integer (bit field)

---

Of course, you'd still need to add higher level files to include the ABI
files.

At the other end, you could add structure and syntax to your heart's
content, and make the output fancier too.

BR,
Jani.

[1] http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#field-lists


-- 
Jani Nikula, Intel Open Source Graphics Center

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-14 13:42   ` Jani Nikula
@ 2019-06-14 14:06     ` Greg Kroah-Hartman
  2019-06-14 15:27       ` Mauro Carvalho Chehab
  2019-06-14 14:10     ` Markus Heiser
  1 sibling, 1 reply; 63+ messages in thread
From: Greg Kroah-Hartman @ 2019-06-14 14:06 UTC (permalink / raw)
  To: Jani Nikula
  Cc: Mauro Carvalho Chehab, Linux Doc Mailing List,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

On Fri, Jun 14, 2019 at 04:42:20PM +0300, Jani Nikula wrote:
> On Thu, 13 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> > From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> >
> > As we don't want a generic Sphinx extension to execute commands,
> > change the one proposed to Markus to call the abi_book.pl
> > script.
> >
> > Use a script to parse the Documentation/ABI directory and output
> > it at the admin-guide.
> 
> We had a legacy kernel-doc perl script so we used that instead of
> rewriting it in python. Just to keep it bug-for-bug compatible with the
> past. That was the only reason.
> 
> I see absolutely zero reason to add a new perl monstrosity with a python
> extension to call it. All of this could be better done in python,
> directly.
> 
> Please don't complicate the documentation build. I know you know we all
> worked hard to take apart the old DocBook Rube Goldberg machine to
> replace it with Sphinx. Please don't turn the Sphinx build to another
> complicated mess.
> 
> My strong preferences are, in this order:
> 
> 1) Convert the ABI documentation to reStructuredText

What would that exactly look like?  What would it require for new
developers for when they write new entries?  Why not rely on a helper
script, that allows us to validate things better?

> 2) Have the python extension read the ABI files directly, without an
>    extra pipeline.

He who writes the script, get's to dictate the language of the script :)

Personally, this looks sane to me, I'm going to apply the ABI fixups to
my tree at least, and then see how the script works out.  The script can
always be replaced with a different one in a different language at a
later point in time of people think it really mattes.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-14 13:42   ` Jani Nikula
  2019-06-14 14:06     ` Greg Kroah-Hartman
@ 2019-06-14 14:10     ` Markus Heiser
  2019-06-14 14:15       ` Jonathan Corbet
  1 sibling, 1 reply; 63+ messages in thread
From: Markus Heiser @ 2019-06-14 14:10 UTC (permalink / raw)
  To: Jani Nikula, Mauro Carvalho Chehab, Linux Doc Mailing List,
	Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet


Am 14.06.19 um 15:42 schrieb Jani Nikula:
> On Thu, 13 Jun 2019, Mauro Carvalho Chehab<mchehab+samsung@kernel.org>  wrote:
>> From: Mauro Carvalho Chehab<mchehab@s-opensource.com>
>>
>> As we don't want a generic Sphinx extension to execute commands,
>> change the one proposed to Markus to call the abi_book.pl
>> script.
>>
>> Use a script to parse the Documentation/ABI directory and output
>> it at the admin-guide.
> We had a legacy kernel-doc perl script so we used that instead of
> rewriting it in python. Just to keep it bug-for-bug compatible with the
> past. That was the only reason.
> 
> I see absolutely zero reason to add a new perl monstrosity with a python
> extension to call it. All of this could be better done in python,
> directly.
> 
> Please don't complicate the documentation build. I know you know we all
> worked hard to take apart the old DocBook Rube Goldberg machine to
> replace it with Sphinx. Please don't turn the Sphinx build to another
> complicated mess.
> 
> My strong preferences are, in this order:
> 
> 1) Convert the ABI documentation to reStructuredText
> 
> 2) Have the python extension read the ABI files directly, without an
>     extra pipeline.
> 

I agree with Jani. No matter how the decision ends, since I can't help here, I'd 
rather not show up in the copyright.

   -- Markus --

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 05/14] scripts: add an script to parse the ABI files
  2019-06-14 14:00       ` Jani Nikula
@ 2019-06-14 14:14         ` Jonathan Corbet
  2019-06-14 16:24         ` Greg Kroah-Hartman
  1 sibling, 0 replies; 63+ messages in thread
From: Jonathan Corbet @ 2019-06-14 14:14 UTC (permalink / raw)
  To: Jani Nikula
  Cc: Greg Kroah-Hartman, Mauro Carvalho Chehab,
	Linux Doc Mailing List, Mauro Carvalho Chehab,
	Mauro Carvalho Chehab, linux-kernel

On Fri, 14 Jun 2019 17:00:55 +0300
Jani Nikula <jani.nikula@linux.intel.com> wrote:

> On Fri, 14 Jun 2019, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> > On Fri, Jun 14, 2019 at 04:31:56PM +0300, Jani Nikula wrote:  
> >> On Thu, 13 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:  
> >> > From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> >> >
> >> > Add a script to parse the Documentation/ABI files and produce
> >> > an output with all entries inside an ABI (sub)directory.
> >> >
> >> > Right now, it outputs its contents on ReST format. It shouldn't
> >> > be hard to make it produce other kind of outputs, since the ABI
> >> > file parser is implemented in separate than the output generator.  
> >> 
> >> Hum, or just convert the ABI files to rst directly.  
> >
> > And what would that look like?  
> 
> That pretty much depends on the requirements we want to set on both the
> ABI source files and the generated output. Obviously the requirements
> can be conflicting; might be hard to produce fancy output if the input
> is very limited.

The real question, I guess, is: is there anything else that depends on the
current format of those files?  That could make reformatting them harder.

Otherwise it seems like Jani's fieldlist idea might have some potential.
We could consider a *really* simple preprocessing step if we really wanted
("prepend a colon on relevant lines and join with the following line if
need be") to keep the current format nearly unchanged.

jon

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-14 14:10     ` Markus Heiser
@ 2019-06-14 14:15       ` Jonathan Corbet
  2019-06-16 16:04         ` Markus Heiser
  0 siblings, 1 reply; 63+ messages in thread
From: Jonathan Corbet @ 2019-06-14 14:15 UTC (permalink / raw)
  To: Markus Heiser
  Cc: Jani Nikula, Mauro Carvalho Chehab, Linux Doc Mailing List,
	Greg Kroah-Hartman, Mauro Carvalho Chehab, Mauro Carvalho Chehab,
	linux-kernel

On Fri, 14 Jun 2019 16:10:31 +0200
Markus Heiser <markus.heiser@darmarit.de> wrote:

> I agree with Jani. No matter how the decision ends, since I can't help here, I'd 
> rather not show up in the copyright.

Is there something specific you are asking us to do here?

Thanks,

jon

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-14 14:06     ` Greg Kroah-Hartman
@ 2019-06-14 15:27       ` Mauro Carvalho Chehab
  2019-06-17 12:36         ` Jani Nikula
  0 siblings, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14 15:27 UTC (permalink / raw)
  To: Greg Kroah-Hartman
  Cc: Jani Nikula, Linux Doc Mailing List, Mauro Carvalho Chehab,
	Mauro Carvalho Chehab, linux-kernel, Jonathan Corbet

Em Fri, 14 Jun 2019 16:06:03 +0200
Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:

> On Fri, Jun 14, 2019 at 04:42:20PM +0300, Jani Nikula wrote:
> > On Thu, 13 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:  
> > > From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> > >
> > > As we don't want a generic Sphinx extension to execute commands,
> > > change the one proposed to Markus to call the abi_book.pl
> > > script.
> > >
> > > Use a script to parse the Documentation/ABI directory and output
> > > it at the admin-guide.  
> > 
> > We had a legacy kernel-doc perl script so we used that instead of
> > rewriting it in python. Just to keep it bug-for-bug compatible with the
> > past. That was the only reason.
> > 
> > I see absolutely zero reason to add a new perl monstrosity with a python
> > extension to call it. All of this could be better done in python,
> > directly.
> > 
> > Please don't complicate the documentation build. I know you know we all
> > worked hard to take apart the old DocBook Rube Goldberg machine to
> > replace it with Sphinx. Please don't turn the Sphinx build to another
> > complicated mess.
> > 
> > My strong preferences are, in this order:
> > 
> > 1) Convert the ABI documentation to reStructuredText  
> 
> What would that exactly look like?  What would it require for new
> developers for when they write new entries?  Why not rely on a helper
> script, that allows us to validate things better?

Funny enough, this e-mail arrived here after Greg's reply, and my
reply over his one :-)

-

With regards to the script, my idea is to have it run on a new
"validate" mode, when the Kernel is built with COMPILE_TEST:

	https://git.linuxtv.org/mchehab/experimental.git/log/?h=abi_patches_v4

NB: the last patch is not yet... somehow, the building system is not
passing CONFIG_WARN_ABI_ERRORS to Documentation/Makefile. I'm
debugging it.

Personally, I would prefer to keep it the way it is, with two
additions:

1) I would add a SPDX header at the fist line of each file there;

2) It would make sense to have a new field - or indicator - to let
add ReST markups at the description. 

The advantage of using a parseable ABI file is that it is possible
to parse it, for example, to search for a symbol:

	$ ./scripts/get_abi.pl voltage_max

	/sys/class/power_supply/<supply_name>/voltage_max
	-------------------------------------------------

	Date:			January 2008
	Contact:		linux-pm@vger.kernel.org
	Defined on file:	Documentation/ABI/testing/sysfs-class-power

	Description:

	Reports the maximum VBUS voltage the supply can support.

	Access: Read
	Valid values: Represented in microvolts

	...

> 
> > 2) Have the python extension read the ABI files directly, without an
> >    extra pipeline.  
> 
> He who writes the script, get's to dictate the language of the script :)

No idea about how much time it would take if written in python,
but this perl script is really fast:

	$ time ./scripts/get_abi.pl search voltage_max >/dev/null
	real	0m0,139s
	user	0m0,132s
	sys	0m0,006s

That's the time it takes here (SSD disks) to read all files under
Documentation/ABI, parse them and seek for a string.

That's about half of the time a python script takes to just import the
the sphinx modules and print its version, running at the same machine:

	$ time sphinx-build --version >/dev/null

	real	0m0,224s
	user	0m0,199s
	sys	0m0,024s

> Personally, this looks sane to me, I'm going to apply the ABI fixups to
> my tree at least, and then see how the script works out.  The script can
> always be replaced with a different one in a different language at a
> later point in time of people think it really mattes.

Thanks,
Mauro

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 01/14] ABI: fix some syntax issues at the ABI database
  2019-06-14  7:20   ` Andrew Donnellan
@ 2019-06-14 16:16     ` Greg Kroah-Hartman
  0 siblings, 0 replies; 63+ messages in thread
From: Greg Kroah-Hartman @ 2019-06-14 16:16 UTC (permalink / raw)
  To: Andrew Donnellan
  Cc: Mauro Carvalho Chehab, Linux Doc Mailing List,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Andreas Klinger, Jonathan Cameron,
	Hartmut Knaack, Lars-Peter Clausen, Peter Meerwald-Stadler,
	Frederic Barrat, Sebastian Reichel, Kees Cook, Anton Vorontsov,
	Colin Cross, Tony Luck, linux-iio, linuxppc-dev, linux-pm

On Fri, Jun 14, 2019 at 05:20:29PM +1000, Andrew Donnellan wrote:
> On 14/6/19 12:04 pm, Mauro Carvalho Chehab wrote:
> > diff --git a/Documentation/ABI/testing/sysfs-class-cxl b/Documentation/ABI/testing/sysfs-class-cxl
> > index bbbabffc682a..fc7c6f7c21b3 100644
> > --- a/Documentation/ABI/testing/sysfs-class-cxl
> > +++ b/Documentation/ABI/testing/sysfs-class-cxl
> > @@ -1,6 +1,6 @@
> > -Note: Attributes that are shared between devices are stored in the directory
> > -pointed to by the symlink device/.
> > -Example: The real path of the attribute /sys/class/cxl/afu0.0s/irqs_max is
> > +Please notice that attributes that are shared between devices are stored in
> 
> Would prefer "Please note" over "Please notice".

Now changed, but that's a minor grammer thing, both are correct. :)

> Acked-by: Andrew Donnellan <ajd@linux.ibm.com>  # cxl

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 14/14] docs: sphinx/kernel_abi.py: fix UTF-8 support
  2019-06-14  2:04 ` [PATCH 14/14] docs: sphinx/kernel_abi.py: fix UTF-8 support Mauro Carvalho Chehab
@ 2019-06-14 16:18   ` Greg Kroah-Hartman
  2019-06-14 16:25     ` Mauro Carvalho Chehab
  0 siblings, 1 reply; 63+ messages in thread
From: Greg Kroah-Hartman @ 2019-06-14 16:18 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Linux Doc Mailing List, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

On Thu, Jun 13, 2019 at 11:04:20PM -0300, Mauro Carvalho Chehab wrote:
> The parser breaks with UTF-8 characters with Sphinx 1.4.
> 
> Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> ---
>  Documentation/sphinx/kernel_abi.py | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
> index 7fa7806532dc..460cee48a245 100644
> --- a/Documentation/sphinx/kernel_abi.py
> +++ b/Documentation/sphinx/kernel_abi.py
> @@ -1,4 +1,5 @@
> -# -*- coding: utf-8; mode: python -*-
> +# coding=utf-8
> +#

Is this an emacs vs. vim fight?

Why change this?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 00/14] Add support to generate ABI documentation at admin-guide
  2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
                   ` (13 preceding siblings ...)
  2019-06-14  2:04 ` [PATCH 14/14] docs: sphinx/kernel_abi.py: fix UTF-8 support Mauro Carvalho Chehab
@ 2019-06-14 16:20 ` Greg Kroah-Hartman
  14 siblings, 0 replies; 63+ messages in thread
From: Greg Kroah-Hartman @ 2019-06-14 16:20 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Linux Doc Mailing List, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, linuxppc-dev, Andrew Donnellan,
	Jonathan Cameron, Lars-Peter Clausen, Anton Vorontsov, linux-pm,
	Colin Cross, linux-iio, Hartmut Knaack, Sebastian Reichel,
	Frederic Barrat, Andreas Klinger, Tony Luck,
	Peter Meerwald-Stadler, Stefan Achatz, Kees Cook

On Thu, Jun 13, 2019 at 11:04:06PM -0300, Mauro Carvalho Chehab wrote:
> Greg,
> 
> As promised, I'm resending the patch series with adds the Kernel ABI to
> Documentation/admin-guide.
> 
> Those patches are basically the version 3 patchset I sent back in 2017,
> rebased on the top of linux-next (next-20190613), and with some fixes
> in order for it to work.
> 
> - The 4 initial patches to fix some ABI descriptions that are violating 
>   the syntax described at Documentation/ABI/README;

These 4 are now applied to my driver-core tree, thanks.

greg k-h

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 05/14] scripts: add an script to parse the ABI files
  2019-06-14 14:00       ` Jani Nikula
  2019-06-14 14:14         ` Jonathan Corbet
@ 2019-06-14 16:24         ` Greg Kroah-Hartman
  1 sibling, 0 replies; 63+ messages in thread
From: Greg Kroah-Hartman @ 2019-06-14 16:24 UTC (permalink / raw)
  To: Jani Nikula
  Cc: Mauro Carvalho Chehab, Linux Doc Mailing List,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

On Fri, Jun 14, 2019 at 05:00:55PM +0300, Jani Nikula wrote:
> On Fri, 14 Jun 2019, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> > On Fri, Jun 14, 2019 at 04:31:56PM +0300, Jani Nikula wrote:
> >> On Thu, 13 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> >> > From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> >> >
> >> > Add a script to parse the Documentation/ABI files and produce
> >> > an output with all entries inside an ABI (sub)directory.
> >> >
> >> > Right now, it outputs its contents on ReST format. It shouldn't
> >> > be hard to make it produce other kind of outputs, since the ABI
> >> > file parser is implemented in separate than the output generator.
> >> 
> >> Hum, or just convert the ABI files to rst directly.
> >
> > And what would that look like?
> 
> That pretty much depends on the requirements we want to set on both the
> ABI source files and the generated output. Obviously the requirements
> can be conflicting; might be hard to produce fancy output if the input
> is very limited.
> 
> At the bare minimum, you could convert the files to contain
> reStructuredText field lists [1]. Add a colon at the start of the field
> name, and make sure field bodies (values) are not empty.
> 
> Conversion of a file selected at random; I've only added ":" and "N/A".

N/A should be allowed to just drop the line entirely, right?

And what does this end up looking like?

I also hate "flag days" where all of a chunk of stuff needs to be
converted into another style.  Also it doesn't deal with merges from the
100+ different trees that all end up adding stuff to this directory over
time (slowly though, unfortunately)

So ideally, I'd like to keep the original format if at all possible.
Having the tool here allows people to do nice things like search for a
specific file easily, or a device type, which is something that I know I
have wanted, and others have asked for in the past as well.

It also might allow us to find out where we are missing documentation, a
long-term goal of mine.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 14/14] docs: sphinx/kernel_abi.py: fix UTF-8 support
  2019-06-14 16:18   ` Greg Kroah-Hartman
@ 2019-06-14 16:25     ` Mauro Carvalho Chehab
  2019-06-15  6:16       ` Greg Kroah-Hartman
  2019-06-16 15:43       ` Markus Heiser
  0 siblings, 2 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-14 16:25 UTC (permalink / raw)
  To: Greg Kroah-Hartman
  Cc: Linux Doc Mailing List, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

Em Fri, 14 Jun 2019 18:18:37 +0200
Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:

> On Thu, Jun 13, 2019 at 11:04:20PM -0300, Mauro Carvalho Chehab wrote:
> > The parser breaks with UTF-8 characters with Sphinx 1.4.
> > 
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> > ---
> >  Documentation/sphinx/kernel_abi.py | 10 ++++++----
> >  1 file changed, 6 insertions(+), 4 deletions(-)
> > 
> > diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
> > index 7fa7806532dc..460cee48a245 100644
> > --- a/Documentation/sphinx/kernel_abi.py
> > +++ b/Documentation/sphinx/kernel_abi.py
> > @@ -1,4 +1,5 @@
> > -# -*- coding: utf-8; mode: python -*-
> > +# coding=utf-8
> > +#  
> 
> Is this an emacs vs. vim fight?

No. This is a python-specific thing:

	https://www.python.org/dev/peps/pep-0263/

> 
> Why change this?

Just to keep the "header" part of the script closer to kerneldoc.py. 

You may keep the previous syntax if you want, as both ways are
equally recognized, as python actually checks for anything that
matches this regex at the first or second line:

	^[ \t\f]*#.*?coding[:=][ \t]*([-_.a-zA-Z0-9]+)

Thanks,
Mauro

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 14/14] docs: sphinx/kernel_abi.py: fix UTF-8 support
  2019-06-14 16:25     ` Mauro Carvalho Chehab
@ 2019-06-15  6:16       ` Greg Kroah-Hartman
  2019-06-16 15:43       ` Markus Heiser
  1 sibling, 0 replies; 63+ messages in thread
From: Greg Kroah-Hartman @ 2019-06-15  6:16 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Linux Doc Mailing List, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

On Fri, Jun 14, 2019 at 01:25:30PM -0300, Mauro Carvalho Chehab wrote:
> Em Fri, 14 Jun 2019 18:18:37 +0200
> Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:
> 
> > On Thu, Jun 13, 2019 at 11:04:20PM -0300, Mauro Carvalho Chehab wrote:
> > > The parser breaks with UTF-8 characters with Sphinx 1.4.
> > > 
> > > Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> > > ---
> > >  Documentation/sphinx/kernel_abi.py | 10 ++++++----
> > >  1 file changed, 6 insertions(+), 4 deletions(-)
> > > 
> > > diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
> > > index 7fa7806532dc..460cee48a245 100644
> > > --- a/Documentation/sphinx/kernel_abi.py
> > > +++ b/Documentation/sphinx/kernel_abi.py
> > > @@ -1,4 +1,5 @@
> > > -# -*- coding: utf-8; mode: python -*-
> > > +# coding=utf-8
> > > +#  
> > 
> > Is this an emacs vs. vim fight?
> 
> No. This is a python-specific thing:
> 
> 	https://www.python.org/dev/peps/pep-0263/

Ah, thanks, didn't know that.

greg k-h

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 14/14] docs: sphinx/kernel_abi.py: fix UTF-8 support
  2019-06-14 16:25     ` Mauro Carvalho Chehab
  2019-06-15  6:16       ` Greg Kroah-Hartman
@ 2019-06-16 15:43       ` Markus Heiser
  2019-06-17  9:16         ` Mauro Carvalho Chehab
  1 sibling, 1 reply; 63+ messages in thread
From: Markus Heiser @ 2019-06-16 15:43 UTC (permalink / raw)
  To: Mauro Carvalho Chehab, Greg Kroah-Hartman
  Cc: Linux Doc Mailing List, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet


Am 14.06.19 um 18:25 schrieb Mauro Carvalho Chehab:
> Em Fri, 14 Jun 2019 18:18:37 +0200
> Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:
> 
>> On Thu, Jun 13, 2019 at 11:04:20PM -0300, Mauro Carvalho Chehab wrote:
>>> The parser breaks with UTF-8 characters with Sphinx 1.4.
>>>
>>> Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
>>> ---
>>>   Documentation/sphinx/kernel_abi.py | 10 ++++++----
>>>   1 file changed, 6 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
>>> index 7fa7806532dc..460cee48a245 100644
>>> --- a/Documentation/sphinx/kernel_abi.py
>>> +++ b/Documentation/sphinx/kernel_abi.py
>>> @@ -1,4 +1,5 @@
>>> -# -*- coding: utf-8; mode: python -*-
>>> +# coding=utf-8
>>> +#
>>
>> Is this an emacs vs. vim fight?
> 
> No. This is a python-specific thing:
> 
> 	https://www.python.org/dev/peps/pep-0263/

No need to change, the emacs notation is also OK, see your link

   """or (using formats recognized by popular editors):"""

   https://www.python.org/dev/peps/pep-0263/#defining-the-encoding

I prefer emacs notation, this is also evaluated by many other editors / tools.

-- Markus --

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-14 14:15       ` Jonathan Corbet
@ 2019-06-16 16:04         ` Markus Heiser
  2019-06-17  9:11           ` Mauro Carvalho Chehab
  0 siblings, 1 reply; 63+ messages in thread
From: Markus Heiser @ 2019-06-16 16:04 UTC (permalink / raw)
  To: Jonathan Corbet
  Cc: Jani Nikula, Mauro Carvalho Chehab, Linux Doc Mailing List,
	Greg Kroah-Hartman, Mauro Carvalho Chehab, Mauro Carvalho Chehab,
	linux-kernel

Am 14.06.19 um 16:15 schrieb Jonathan Corbet:
> On Fri, 14 Jun 2019 16:10:31 +0200
> Markus Heiser <markus.heiser@darmarit.de> wrote:
> 
>> I agree with Jani. No matter how the decision ends, since I can't help here, I'd
>> rather not show up in the copyright.
> 
> Is there something specific you are asking us to do here?
> 


I have lost the overview, but there was a patch Mauro added a
kernel_abi.py.  There was my name (Markus Heiser) listed with a
copyright notation.

I guess Mauro picked up some old RFC or an other old patch of
mine from 2016 and made some C&P .. whatever .. ATM I do not have
time to give any support on parsing ABI and I'am not interested
in holding copyrights on a C&P of a old source  ;)

-- Markus --


^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-16 16:04         ` Markus Heiser
@ 2019-06-17  9:11           ` Mauro Carvalho Chehab
  2019-06-17 16:31             ` Markus Heiser
  0 siblings, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-17  9:11 UTC (permalink / raw)
  To: Markus Heiser
  Cc: Jonathan Corbet, Jani Nikula, Linux Doc Mailing List,
	Greg Kroah-Hartman, Mauro Carvalho Chehab, Mauro Carvalho Chehab,
	linux-kernel

Em Sun, 16 Jun 2019 18:04:01 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Am 14.06.19 um 16:15 schrieb Jonathan Corbet:
> > On Fri, 14 Jun 2019 16:10:31 +0200
> > Markus Heiser <markus.heiser@darmarit.de> wrote:
> >   
> >> I agree with Jani. No matter how the decision ends, since I can't help here, I'd
> >> rather not show up in the copyright.  
> > 
> > Is there something specific you are asking us to do here?
> >   
> 
> 
> I have lost the overview, but there was a patch Mauro added a
> kernel_abi.py.  There was my name (Markus Heiser) listed with a
> copyright notation.
> 
> I guess Mauro picked up some old RFC or an other old patch of
> mine from 2016 and made some C&P .. whatever .. ATM I do not have
> time to give any support on parsing ABI and I'am not interested
> in holding copyrights on a C&P of a old source  ;)

Well, the code was basically written by you :-)

It was written to be a script capable of running a generic
script. On that time, my contribution to it was basically
to hardcode it to run "get_abi.pl".

This came from an old branch where the last change was back in 2017. 
It was resurrected due to a discussion at KS ML.

There, the discussion was related to what's left to be converted
to ReST.

While I can't simply remove your copyright, would you be happy
with something like that?


Thanks,
Mauro

diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index 2d5d582207f7..ef91b1e1ff4b 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -7,7 +7,8 @@ u"""
     Implementation of the ``kernel-abi`` reST-directive.
 
     :copyright:  Copyright (C) 2016  Markus Heiser
-    :copyright:  Copyright (C) 2016  Mauro Carvalho Chehab
+    :copyright:  Copyright (C) 2016-2019  Mauro Carvalho Chehab
+    :maintained-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
     :license:    GPL Version 2, June 1991 see Linux/COPYING for details.
 
     The ``kernel-abi`` (:py:class:`KernelCmd`) directive calls the



^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 14/14] docs: sphinx/kernel_abi.py: fix UTF-8 support
  2019-06-16 15:43       ` Markus Heiser
@ 2019-06-17  9:16         ` Mauro Carvalho Chehab
  2019-06-17 13:56           ` Jonathan Corbet
  0 siblings, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-17  9:16 UTC (permalink / raw)
  To: Markus Heiser
  Cc: Greg Kroah-Hartman, Linux Doc Mailing List,
	Mauro Carvalho Chehab, linux-kernel, Jonathan Corbet

Em Sun, 16 Jun 2019 17:43:50 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Am 14.06.19 um 18:25 schrieb Mauro Carvalho Chehab:
> > Em Fri, 14 Jun 2019 18:18:37 +0200
> > Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:
> >   
> >> On Thu, Jun 13, 2019 at 11:04:20PM -0300, Mauro Carvalho Chehab wrote:  
> >>> The parser breaks with UTF-8 characters with Sphinx 1.4.
> >>>
> >>> Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> >>> ---
> >>>   Documentation/sphinx/kernel_abi.py | 10 ++++++----
> >>>   1 file changed, 6 insertions(+), 4 deletions(-)
> >>>
> >>> diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
> >>> index 7fa7806532dc..460cee48a245 100644
> >>> --- a/Documentation/sphinx/kernel_abi.py
> >>> +++ b/Documentation/sphinx/kernel_abi.py
> >>> @@ -1,4 +1,5 @@
> >>> -# -*- coding: utf-8; mode: python -*-
> >>> +# coding=utf-8
> >>> +#  
> >>
> >> Is this an emacs vs. vim fight?  
> > 
> > No. This is a python-specific thing:
> > 
> > 	https://www.python.org/dev/peps/pep-0263/  
> 
> No need to change, the emacs notation is also OK, see your link
> 
>    """or (using formats recognized by popular editors):"""
> 
>    https://www.python.org/dev/peps/pep-0263/#defining-the-encoding
> 
> I prefer emacs notation, this is also evaluated by many other editors / tools.

The usage of emacs notation is something that we don't like at the
Linux Kernel. With ~4K developers per release, if we add tags to
every single editor people use, it would be really messy, as one
developer would be adding a tag and the next one replacing it by its
some other favorite editor's tag.

There's even an item at Documentation/process/coding-style.rst
due to that (item 9).

Thanks,
Mauro

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-14 15:27       ` Mauro Carvalho Chehab
@ 2019-06-17 12:36         ` Jani Nikula
  2019-06-17 12:54           ` Greg Kroah-Hartman
  2019-06-17 13:51           ` Mauro Carvalho Chehab
  0 siblings, 2 replies; 63+ messages in thread
From: Jani Nikula @ 2019-06-17 12:36 UTC (permalink / raw)
  To: Mauro Carvalho Chehab, Greg Kroah-Hartman
  Cc: Linux Doc Mailing List, Mauro Carvalho Chehab,
	Mauro Carvalho Chehab, linux-kernel, Jonathan Corbet

On Fri, 14 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> Em Fri, 14 Jun 2019 16:06:03 +0200
> Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:
>
>> On Fri, Jun 14, 2019 at 04:42:20PM +0300, Jani Nikula wrote:
>> > 2) Have the python extension read the ABI files directly, without an
>> >    extra pipeline.  
>> 
>> He who writes the script, get's to dictate the language of the script :)

The point is, it's an extension to a python based tool, written in perl,
using pipes for communication, and losing any advantages of integrating
with the tool it's extending.

I doubt you'd want to see system() to be used to subsequently extend the
perl tool.

I think it's just sad to see the documentation system slowly drift
further away from the ideals we had, and towards the old ways we worked
so hard to fix.

> No idea about how much time it would take if written in python,
> but this perl script is really fast:
>
> 	$ time ./scripts/get_abi.pl search voltage_max >/dev/null
> 	real	0m0,139s
> 	user	0m0,132s
> 	sys	0m0,006s
>
> That's the time it takes here (SSD disks) to read all files under
> Documentation/ABI, parse them and seek for a string.
>
> That's about half of the time a python script takes to just import the
> the sphinx modules and print its version, running at the same machine:
>
> 	$ time sphinx-build --version >/dev/null
>
> 	real	0m0,224s
> 	user	0m0,199s
> 	sys	0m0,024s

Please at least use fair and sensible comparisons. If you want to make
the extension usable standalone on the command-line, bypassing Sphinx,
you can do that. No need to factor in Sphinx to your comparisons.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Graphics Center

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-17 12:36         ` Jani Nikula
@ 2019-06-17 12:54           ` Greg Kroah-Hartman
  2019-06-17 13:48             ` Jonathan Corbet
  2019-06-17 13:50             ` Jani Nikula
  2019-06-17 13:51           ` Mauro Carvalho Chehab
  1 sibling, 2 replies; 63+ messages in thread
From: Greg Kroah-Hartman @ 2019-06-17 12:54 UTC (permalink / raw)
  To: Jani Nikula
  Cc: Mauro Carvalho Chehab, Linux Doc Mailing List,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

On Mon, Jun 17, 2019 at 03:36:17PM +0300, Jani Nikula wrote:
> On Fri, 14 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> > Em Fri, 14 Jun 2019 16:06:03 +0200
> > Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:
> >
> >> On Fri, Jun 14, 2019 at 04:42:20PM +0300, Jani Nikula wrote:
> >> > 2) Have the python extension read the ABI files directly, without an
> >> >    extra pipeline.  
> >> 
> >> He who writes the script, get's to dictate the language of the script :)
> 
> The point is, it's an extension to a python based tool, written in perl,
> using pipes for communication, and losing any advantages of integrating
> with the tool it's extending.
> 
> I doubt you'd want to see system() to be used to subsequently extend the
> perl tool.
> 
> I think it's just sad to see the documentation system slowly drift
> further away from the ideals we had, and towards the old ways we worked
> so hard to fix.

What are those ideals?

I thought the goal was to be able to write documentation in a as much
as a normal text file as possible and have automation turn those files
into "pretty" documentation that we can all use.

And I think that fits with the way this patch set goes, right?  We are
not on a quest for purity of scripts to generate the documentation at
the expense of having to force hundreds, or thousands, of developers to
change their ways, or to force a "flag day" conversion of existing
documentation resulting in a huge merge mess.

So, we are stuck with the current structure that I totally made up for
Documentation/ABI/.  Turns out it is almost parsable, as Mauro's tool
shows.  His tool also validates the existing text, which is great, and
has caused fixes for it.

If someone wants to write that tool in some other language, like python,
wonderful, I have no objection, but as it is, this is a useful tool
already, allowing us to validate, and search, existing documentation
entries that we have never been able to do before.  It also provides an
output that can be turned into pretty html/pdf/whatever files by other
tools in the pipeline, a totally bonus benefit.

So what is going backwards here?

Maybe the processing pipeline isn't as nice as you would like, but
remember to view this from a normal developer's point of view, not a
documentation pipeline developer's point of view please.

So, in short, my requirements are:
	- keep Documentation/ABI/ file formats as close as possible to
	  what we have today, preventing any flag-day issues or merge
	  problems
	- be able to query and validate Documentation/ABI/
	- be able to turn Documentation/ABI into pretty documentation.

If you object to the mechanics of the last requirement here, I don't
object either, provide something else that works better.  But don't
throw away the whole thing just because you don't like how things are
hooked up here.

I'm going to go apply most of the rest of these patches to my
driver-core tree, stopping at the "hook it up to the kernel
documentation" point.  Is that ok?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-17 12:54           ` Greg Kroah-Hartman
@ 2019-06-17 13:48             ` Jonathan Corbet
  2019-06-17 13:50             ` Jani Nikula
  1 sibling, 0 replies; 63+ messages in thread
From: Jonathan Corbet @ 2019-06-17 13:48 UTC (permalink / raw)
  To: Greg Kroah-Hartman
  Cc: Jani Nikula, Mauro Carvalho Chehab, Linux Doc Mailing List,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel

On Mon, 17 Jun 2019 14:54:38 +0200
Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:

> > I think it's just sad to see the documentation system slowly drift
> > further away from the ideals we had, and towards the old ways we worked
> > so hard to fix.  
> 
> What are those ideals?
> 
> I thought the goal was to be able to write documentation in a as much
> as a normal text file as possible and have automation turn those files
> into "pretty" documentation that we can all use.

That was indeed one of the goals.  Another was to replace the incredible
pile of fragile duct tape that the docs build system had become with
something more robust, understandable, and maintainable.  We did that, to
an extent at least, and life is better.

Jani worries that we have been regressing toward duct-tape mode, and I
suspect he may be right.  I'm certainly as guilty as anybody of tossing
stuff in because it's expedient right now.  It is right to ask whether we
should continue in that direction.

Can we slow down just a bit on the ABI files?  It may be that Mauro's
solution is the best one, but I would really like to think a bit about
how all this stuff fits together, and life isn't really even giving me
time to tie my shoes these days.  I don't think that this is screamingly
urgent right now.

Thanks,

jon

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-17 12:54           ` Greg Kroah-Hartman
  2019-06-17 13:48             ` Jonathan Corbet
@ 2019-06-17 13:50             ` Jani Nikula
  1 sibling, 0 replies; 63+ messages in thread
From: Jani Nikula @ 2019-06-17 13:50 UTC (permalink / raw)
  To: Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Linux Doc Mailing List,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

On Mon, 17 Jun 2019, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> On Mon, Jun 17, 2019 at 03:36:17PM +0300, Jani Nikula wrote:
>> On Fri, 14 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
>> > Em Fri, 14 Jun 2019 16:06:03 +0200
>> > Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:
>> >
>> >> On Fri, Jun 14, 2019 at 04:42:20PM +0300, Jani Nikula wrote:
>> >> > 2) Have the python extension read the ABI files directly, without an
>> >> >    extra pipeline.  
>> >> 
>> >> He who writes the script, get's to dictate the language of the script :)
>> 
>> The point is, it's an extension to a python based tool, written in perl,
>> using pipes for communication, and losing any advantages of integrating
>> with the tool it's extending.
>> 
>> I doubt you'd want to see system() to be used to subsequently extend the
>> perl tool.
>> 
>> I think it's just sad to see the documentation system slowly drift
>> further away from the ideals we had, and towards the old ways we worked
>> so hard to fix.
>
> What are those ideals?

For example, have a single coherent system, instead of a fragile pipe
with each stage written in a different language, each having its own
idiosynchracies, each step losing something in translation.

Have a system that a normal developer can actually look at and
understand. It didn't use to be that way.

> I thought the goal was to be able to write documentation in a as much
> as a normal text file as possible and have automation turn those files
> into "pretty" documentation that we can all use.
>
> And I think that fits with the way this patch set goes, right?  We are
> not on a quest for purity of scripts to generate the documentation at
> the expense of having to force hundreds, or thousands, of developers to
> change their ways, or to force a "flag day" conversion of existing
> documentation resulting in a huge merge mess.

Fair enough, let's dismiss the thought of changing the ABI files. But I
never meant that would somehow be for the "purity of scripts", or that
those two would somehow be at odds here.

> So, we are stuck with the current structure that I totally made up for
> Documentation/ABI/.  Turns out it is almost parsable, as Mauro's tool
> shows.  His tool also validates the existing text, which is great, and
> has caused fixes for it.
>
> If someone wants to write that tool in some other language, like python,
> wonderful, I have no objection, but as it is, this is a useful tool
> already, allowing us to validate, and search, existing documentation
> entries that we have never been able to do before.  It also provides an
> output that can be turned into pretty html/pdf/whatever files by other
> tools in the pipeline, a totally bonus benefit.
>
> So what is going backwards here?
>
> Maybe the processing pipeline isn't as nice as you would like, but
> remember to view this from a normal developer's point of view, not a
> documentation pipeline developer's point of view please.
>
> So, in short, my requirements are:
> 	- keep Documentation/ABI/ file formats as close as possible to
> 	  what we have today, preventing any flag-day issues or merge
> 	  problems
> 	- be able to query and validate Documentation/ABI/
> 	- be able to turn Documentation/ABI into pretty documentation.
>
> If you object to the mechanics of the last requirement here, I don't
> object either, provide something else that works better.  But don't
> throw away the whole thing just because you don't like how things are
> hooked up here.
>
> I'm going to go apply most of the rest of these patches to my
> driver-core tree, stopping at the "hook it up to the kernel
> documentation" point.  Is that ok?

I'll leave it all up to Jon's discretion; I trust he'll understand my
concerns. I have no authority beyond the opinion I've voiced here
anyway.


BR,
Jani.

-- 
Jani Nikula, Intel Open Source Graphics Center

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-17 12:36         ` Jani Nikula
  2019-06-17 12:54           ` Greg Kroah-Hartman
@ 2019-06-17 13:51           ` Mauro Carvalho Chehab
  2019-06-18  8:47             ` Jani Nikula
  1 sibling, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-17 13:51 UTC (permalink / raw)
  To: Jani Nikula
  Cc: Greg Kroah-Hartman, Linux Doc Mailing List,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

Em Mon, 17 Jun 2019 15:36:17 +0300
Jani Nikula <jani.nikula@linux.intel.com> escreveu:

> On Fri, 14 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> > Em Fri, 14 Jun 2019 16:06:03 +0200
> > Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:
> >  
> >> On Fri, Jun 14, 2019 at 04:42:20PM +0300, Jani Nikula wrote:  
> >> > 2) Have the python extension read the ABI files directly, without an
> >> >    extra pipeline.    
> >> 
> >> He who writes the script, get's to dictate the language of the script :)  
> 
> The point is, it's an extension to a python based tool, written in perl,
> using pipes for communication, and losing any advantages of integrating
> with the tool it's extending.
> 
> I doubt you'd want to see system() to be used to subsequently extend the
> perl tool.
> 
> I think it's just sad to see the documentation system slowly drift
> further away from the ideals we had, and towards the old ways we worked
> so hard to fix.

Actually, it is a perl script that can be used standalone (just like
get_maintainers.pl and kernel-doc) with have some features including
producing a ReST output. We could easily get rid of the python extension,
if we add this to the Makefile (adjusted to work with O= option):

	./scripts/get_api.pl rest > Documentation/output/admin-guide/abi.rst

> 
> > No idea about how much time it would take if written in python,
> > but this perl script is really fast:
> >
> > 	$ time ./scripts/get_abi.pl search voltage_max >/dev/null
> > 	real	0m0,139s
> > 	user	0m0,132s
> > 	sys	0m0,006s
> >
> > That's the time it takes here (SSD disks) to read all files under
> > Documentation/ABI, parse them and seek for a string.
> >
> > That's about half of the time a python script takes to just import the
> > the sphinx modules and print its version, running at the same machine:
> >
> > 	$ time sphinx-build --version >/dev/null
> >
> > 	real	0m0,224s
> > 	user	0m0,199s
> > 	sys	0m0,024s  
> 
> Please at least use fair and sensible comparisons. If you want to make
> the extension usable standalone on the command-line, bypassing Sphinx,
> you can do that. No need to factor in Sphinx to your comparisons.

Yeah, I guess it should be possible to do that. How a python script
can identify if it was called by Sphinx, or if it was called directly?

Thanks,
Mauro

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 14/14] docs: sphinx/kernel_abi.py: fix UTF-8 support
  2019-06-17  9:16         ` Mauro Carvalho Chehab
@ 2019-06-17 13:56           ` Jonathan Corbet
  2019-06-17 15:55             ` Mauro Carvalho Chehab
  0 siblings, 1 reply; 63+ messages in thread
From: Jonathan Corbet @ 2019-06-17 13:56 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Markus Heiser, Greg Kroah-Hartman, Linux Doc Mailing List,
	Mauro Carvalho Chehab, linux-kernel

On Mon, 17 Jun 2019 06:16:59 -0300
Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:

> > No need to change, the emacs notation is also OK, see your link
> > 
> >    """or (using formats recognized by popular editors):"""
> > 
> >    https://www.python.org/dev/peps/pep-0263/#defining-the-encoding
> > 
> > I prefer emacs notation, this is also evaluated by many other editors / tools.  
> 
> The usage of emacs notation is something that we don't like at the
> Linux Kernel. With ~4K developers per release, if we add tags to
> every single editor people use, it would be really messy, as one
> developer would be adding a tag and the next one replacing it by its
> some other favorite editor's tag.

So "we" like a language-specific notation instead?  That seems a little
strange to me.  Lots of things understand the Emacs notation, it doesn't
seem like something that needs to be actively avoided here.

Thanks,

jon

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 14/14] docs: sphinx/kernel_abi.py: fix UTF-8 support
  2019-06-17 13:56           ` Jonathan Corbet
@ 2019-06-17 15:55             ` Mauro Carvalho Chehab
  0 siblings, 0 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-17 15:55 UTC (permalink / raw)
  To: Jonathan Corbet
  Cc: Markus Heiser, Greg Kroah-Hartman, Linux Doc Mailing List,
	Mauro Carvalho Chehab, linux-kernel

Em Mon, 17 Jun 2019 07:56:08 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> On Mon, 17 Jun 2019 06:16:59 -0300
> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> 
> > > No need to change, the emacs notation is also OK, see your link
> > > 
> > >    """or (using formats recognized by popular editors):"""
> > > 
> > >    https://www.python.org/dev/peps/pep-0263/#defining-the-encoding
> > > 
> > > I prefer emacs notation, this is also evaluated by many other editors / tools.    
> > 
> > The usage of emacs notation is something that we don't like at the
> > Linux Kernel. With ~4K developers per release, if we add tags to
> > every single editor people use, it would be really messy, as one
> > developer would be adding a tag and the next one replacing it by its
> > some other favorite editor's tag.  
> 
> So "we" like a language-specific notation instead?  That seems a little
> strange to me.  Lots of things understand the Emacs notation, it doesn't
> seem like something that needs to be actively avoided here.

From my side, I don't have any strong preference. Just saying that
people usually complain when e-macs or vim specific tags appear at the
Kernel. That's why I would prefer an editor-agnostic macro.

It won't make any difference for me, anyway, as the editors I use
don't recognize it.

Whatever you want is OK to me, provided that we use the same notation on
all Sphinx extensions... right now there's a mix of notations.

Thanks,
Mauro

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-17  9:11           ` Mauro Carvalho Chehab
@ 2019-06-17 16:31             ` Markus Heiser
  0 siblings, 0 replies; 63+ messages in thread
From: Markus Heiser @ 2019-06-17 16:31 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Jonathan Corbet, Jani Nikula, Linux Doc Mailing List,
	Greg Kroah-Hartman, Mauro Carvalho Chehab, Mauro Carvalho Chehab,
	linux-kernel


Am 17.06.19 um 11:11 schrieb Mauro Carvalho Chehab:
> Em Sun, 16 Jun 2019 18:04:01 +0200
> Markus Heiser <markus.heiser@darmarit.de> escreveu:
> 
>> Am 14.06.19 um 16:15 schrieb Jonathan Corbet:
>>> On Fri, 14 Jun 2019 16:10:31 +0200
>>> Markus Heiser <markus.heiser@darmarit.de> wrote:
>>>    
>>>> I agree with Jani. No matter how the decision ends, since I can't help here, I'd
>>>> rather not show up in the copyright.
>>>
>>> Is there something specific you are asking us to do here?
>>>    
>>
>>
>> I have lost the overview, but there was a patch Mauro added a
>> kernel_abi.py.  There was my name (Markus Heiser) listed with a
>> copyright notation.
>>
>> I guess Mauro picked up some old RFC or an other old patch of
>> mine from 2016 and made some C&P .. whatever .. ATM I do not have
>> time to give any support on parsing ABI and I'am not interested
>> in holding copyrights on a C&P of a old source  ;)
> 
> Well, the code was basically written by you :-)
> 
> It was written to be a script capable of running a generic
> script. On that time, my contribution to it was basically
> to hardcode it to run "get_abi.pl".

Thanks for clarifying.

> 
> This came from an old branch where the last change was back in 2017.
> It was resurrected due to a discussion at KS ML.
> 
> There, the discussion was related to what's left to be converted
> to ReST.
> 
> While I can't simply remove your copyright, would you be happy
> with something like that?

Yes, but basically I share Jani's and Jon's doubts about this solution.

-- Markus --

> 
> 
> Thanks,
> Mauro
> 
> diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
> index 2d5d582207f7..ef91b1e1ff4b 100644
> --- a/Documentation/sphinx/kernel_abi.py
> +++ b/Documentation/sphinx/kernel_abi.py
> @@ -7,7 +7,8 @@ u"""
>       Implementation of the ``kernel-abi`` reST-directive.
>   
>       :copyright:  Copyright (C) 2016  Markus Heiser
> -    :copyright:  Copyright (C) 2016  Mauro Carvalho Chehab
> +    :copyright:  Copyright (C) 2016-2019  Mauro Carvalho Chehab
> +    :maintained-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
>       :license:    GPL Version 2, June 1991 see Linux/COPYING for details.
>   
>       The ``kernel-abi`` (:py:class:`KernelCmd`) directive calls the
> 
> 



^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-17 13:51           ` Mauro Carvalho Chehab
@ 2019-06-18  8:47             ` Jani Nikula
  2019-06-19 16:37               ` Mauro Carvalho Chehab
  0 siblings, 1 reply; 63+ messages in thread
From: Jani Nikula @ 2019-06-18  8:47 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Greg Kroah-Hartman, Linux Doc Mailing List,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

On Mon, 17 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> Yeah, I guess it should be possible to do that. How a python script
> can identify if it was called by Sphinx, or if it was called directly?

if __name__ == '__main__':
	# run on the command-line, not imported

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Graphics Center

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 04/14] ABI: better identificate tables
  2019-06-14  2:04 ` [PATCH 04/14] ABI: better identificate tables Mauro Carvalho Chehab
@ 2019-06-19 12:51   ` Johan Hovold
  2019-06-19 13:56     ` Mauro Carvalho Chehab
  0 siblings, 1 reply; 63+ messages in thread
From: Johan Hovold @ 2019-06-19 12:51 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Linux Doc Mailing List, Greg Kroah-Hartman,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Stefan Achatz

On Thu, Jun 13, 2019 at 11:04:10PM -0300, Mauro Carvalho Chehab wrote:
> From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> 
> When parsing via script, it is important to know if the script
> should consider a description as a literal block that should
> be displayed as-is, or if the description can be considered
> as a normal text.
> 
> Change descriptions to ensure that the preceding line of a table
> ends with a colon. That makes easy to identify the need of a
> literal block.

In the cover letter you say that the first four patches of this series,
including this one, "fix some ABI descriptions that are violating the
syntax described at Documentation/ABI/README". This seems a bit harsh,
given that it's you that is now *introducing* a new syntax requirement
to assist your script.

Specifically, this new requirement isn't documented anywhere AFAICT, so
how will anyone adding new ABI descriptions learn about it?

> Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> ---
>  Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra   | 2 +-
>  .../ABI/testing/sysfs-class-backlight-driver-lm3533       | 6 +++---
>  Documentation/ABI/testing/sysfs-class-led-driver-lm3533   | 8 ++++----
>  Documentation/ABI/testing/sysfs-class-leds-gt683r         | 4 ++--
>  Documentation/ABI/testing/sysfs-driver-hid-roccat-kone    | 2 +-
>  5 files changed, 11 insertions(+), 11 deletions(-)
> 
> diff --git a/Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra b/Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra
> index 16020b31ae64..5d41ebadf15e 100644
> --- a/Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra
> +++ b/Documentation/ABI/obsolete/sysfs-driver-hid-roccat-pyra
> @@ -5,7 +5,7 @@ Description:	It is possible to switch the cpi setting of the mouse with the
>  		press of a button.
>  		When read, this file returns the raw number of the actual cpi
>  		setting reported by the mouse. This number has to be further
> -		processed to receive the real dpi value.
> +		processed to receive the real dpi value:
>  
>  		VALUE DPI
>  		1     400

Johan

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 04/14] ABI: better identificate tables
  2019-06-19 12:51   ` Johan Hovold
@ 2019-06-19 13:56     ` Mauro Carvalho Chehab
  2019-06-19 15:02       ` Greg Kroah-Hartman
  0 siblings, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-19 13:56 UTC (permalink / raw)
  To: Johan Hovold, Greg Kroah-Hartman
  Cc: Linux Doc Mailing List, Mauro Carvalho Chehab,
	Mauro Carvalho Chehab, linux-kernel, Jonathan Corbet,
	Stefan Achatz

Hi Johan,

Em Wed, 19 Jun 2019 14:51:35 +0200
Johan Hovold <johan@kernel.org> escreveu:

> On Thu, Jun 13, 2019 at 11:04:10PM -0300, Mauro Carvalho Chehab wrote:
> > From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> > 
> > When parsing via script, it is important to know if the script
> > should consider a description as a literal block that should
> > be displayed as-is, or if the description can be considered
> > as a normal text.
> > 
> > Change descriptions to ensure that the preceding line of a table
> > ends with a colon. That makes easy to identify the need of a
> > literal block.  
> 
> In the cover letter you say that the first four patches of this series,
> including this one, "fix some ABI descriptions that are violating the
> syntax described at Documentation/ABI/README". This seems a bit harsh,
> given that it's you that is now *introducing* a new syntax requirement
> to assist your script.

Yeah, what's there at the cover letter doesn't apply to this specific
patch. The thing is that I wrote this series a lot of time ago (2016/17).

I revived those per a request at KS ML, as we still need to expose the
ABI content on some book that will be used by userspace people.

So, I just rebased it on the top of curent Kernel, add a cover letter
with the things I remembered and re-sent.

In the specific case of this patch, the ":" there actually makes sense
for someone that it is reading it as a text file, and it is an easy
hack to make it parse better.

> Specifically, this new requirement isn't documented anywhere AFAICT, so
> how will anyone adding new ABI descriptions learn about it?

Yeah, either that or provide an alternative to "Description" tag, to be
used with more complex ABI descriptions.

One of the things that occurred to me, back on 2017, is that we should
have a way to to specify that an specific ABI description would have
a rich format. Something like:

What:		/sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/pyra/roccatpyra<minor>/actual_cpi
Date:		August 2010
Contact:	Stefan Achatz <erazor_de@users.sourceforge.net>
RST-Description:
		It is possible to switch the cpi setting of the mouse with the
		press of a button.
		When read, this file returns the raw number of the actual cpi
		setting reported by the mouse. This number has to be further
		processed to receive the real dpi value:

		===== =====
		VALUE DPI
		===== =====
		1     400
		2     800
		4     1600
		===== =====

With that, the script will know that the description contents will be using
the ReST markup, and parse it accordingly. Right now, what it does, instead,
is to place the description on a code-block, e. g. it will produce this
output for the description:

::

		It is possible to switch the cpi setting of the mouse with the
		press of a button.
		When read, this file returns the raw number of the actual cpi
		setting reported by the mouse. This number has to be further
		processed to receive the real dpi value:

		VALUE DPI
		1     400
		2     800
		4     1600


Greg, 

what do you think?

Thanks,
Mauro

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 04/14] ABI: better identificate tables
  2019-06-19 13:56     ` Mauro Carvalho Chehab
@ 2019-06-19 15:02       ` Greg Kroah-Hartman
  2019-06-19 16:14         ` Mauro Carvalho Chehab
  2019-06-20 12:01         ` Johan Hovold
  0 siblings, 2 replies; 63+ messages in thread
From: Greg Kroah-Hartman @ 2019-06-19 15:02 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Johan Hovold, Linux Doc Mailing List, Mauro Carvalho Chehab,
	Mauro Carvalho Chehab, linux-kernel, Jonathan Corbet,
	Stefan Achatz

On Wed, Jun 19, 2019 at 10:56:33AM -0300, Mauro Carvalho Chehab wrote:
> Hi Johan,
> 
> Em Wed, 19 Jun 2019 14:51:35 +0200
> Johan Hovold <johan@kernel.org> escreveu:
> 
> > On Thu, Jun 13, 2019 at 11:04:10PM -0300, Mauro Carvalho Chehab wrote:
> > > From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> > > 
> > > When parsing via script, it is important to know if the script
> > > should consider a description as a literal block that should
> > > be displayed as-is, or if the description can be considered
> > > as a normal text.
> > > 
> > > Change descriptions to ensure that the preceding line of a table
> > > ends with a colon. That makes easy to identify the need of a
> > > literal block.  
> > 
> > In the cover letter you say that the first four patches of this series,
> > including this one, "fix some ABI descriptions that are violating the
> > syntax described at Documentation/ABI/README". This seems a bit harsh,
> > given that it's you that is now *introducing* a new syntax requirement
> > to assist your script.
> 
> Yeah, what's there at the cover letter doesn't apply to this specific
> patch. The thing is that I wrote this series a lot of time ago (2016/17).
> 
> I revived those per a request at KS ML, as we still need to expose the
> ABI content on some book that will be used by userspace people.
> 
> So, I just rebased it on the top of curent Kernel, add a cover letter
> with the things I remembered and re-sent.
> 
> In the specific case of this patch, the ":" there actually makes sense
> for someone that it is reading it as a text file, and it is an easy
> hack to make it parse better.
> 
> > Specifically, this new requirement isn't documented anywhere AFAICT, so
> > how will anyone adding new ABI descriptions learn about it?
> 
> Yeah, either that or provide an alternative to "Description" tag, to be
> used with more complex ABI descriptions.
> 
> One of the things that occurred to me, back on 2017, is that we should
> have a way to to specify that an specific ABI description would have
> a rich format. Something like:
> 
> What:		/sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/pyra/roccatpyra<minor>/actual_cpi
> Date:		August 2010
> Contact:	Stefan Achatz <erazor_de@users.sourceforge.net>
> RST-Description:
> 		It is possible to switch the cpi setting of the mouse with the
> 		press of a button.
> 		When read, this file returns the raw number of the actual cpi
> 		setting reported by the mouse. This number has to be further
> 		processed to receive the real dpi value:
> 
> 		===== =====
> 		VALUE DPI
> 		===== =====
> 		1     400
> 		2     800
> 		4     1600
> 		===== =====
> 
> With that, the script will know that the description contents will be using
> the ReST markup, and parse it accordingly. Right now, what it does, instead,
> is to place the description on a code-block, e. g. it will produce this
> output for the description:
> 
> ::
> 
> 		It is possible to switch the cpi setting of the mouse with the
> 		press of a button.
> 		When read, this file returns the raw number of the actual cpi
> 		setting reported by the mouse. This number has to be further
> 		processed to receive the real dpi value:
> 
> 		VALUE DPI
> 		1     400
> 		2     800
> 		4     1600
> 
> 
> Greg, 
> 
> what do you think?

I don't know when "Description" and "RST-Description" would be used.
Why not just parse "Description" like rst text and if things are "messy"
we fix them up as found, like you did with the ":" here?  It doesn't
have to be complex, we can always fix them up after-the-fact if new
stuff gets added that doesn't quite parse properly.

Just like we do for most kernel-doc formatting :)

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 04/14] ABI: better identificate tables
  2019-06-19 15:02       ` Greg Kroah-Hartman
@ 2019-06-19 16:14         ` Mauro Carvalho Chehab
  2019-06-20 14:27           ` Mauro Carvalho Chehab
  2019-06-20 12:01         ` Johan Hovold
  1 sibling, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-19 16:14 UTC (permalink / raw)
  To: Greg Kroah-Hartman
  Cc: Johan Hovold, Linux Doc Mailing List, Mauro Carvalho Chehab,
	Mauro Carvalho Chehab, linux-kernel, Jonathan Corbet,
	Stefan Achatz

Em Wed, 19 Jun 2019 17:02:07 +0200
Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:

> On Wed, Jun 19, 2019 at 10:56:33AM -0300, Mauro Carvalho Chehab wrote:
> > Hi Johan,
> > 
> > Em Wed, 19 Jun 2019 14:51:35 +0200
> > Johan Hovold <johan@kernel.org> escreveu:
> >   
> > > On Thu, Jun 13, 2019 at 11:04:10PM -0300, Mauro Carvalho Chehab wrote:  
> > > > From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> > > > 
> > > > When parsing via script, it is important to know if the script
> > > > should consider a description as a literal block that should
> > > > be displayed as-is, or if the description can be considered
> > > > as a normal text.
> > > > 
> > > > Change descriptions to ensure that the preceding line of a table
> > > > ends with a colon. That makes easy to identify the need of a
> > > > literal block.    
> > > 
> > > In the cover letter you say that the first four patches of this series,
> > > including this one, "fix some ABI descriptions that are violating the
> > > syntax described at Documentation/ABI/README". This seems a bit harsh,
> > > given that it's you that is now *introducing* a new syntax requirement
> > > to assist your script.  
> > 
> > Yeah, what's there at the cover letter doesn't apply to this specific
> > patch. The thing is that I wrote this series a lot of time ago (2016/17).
> > 
> > I revived those per a request at KS ML, as we still need to expose the
> > ABI content on some book that will be used by userspace people.
> > 
> > So, I just rebased it on the top of curent Kernel, add a cover letter
> > with the things I remembered and re-sent.
> > 
> > In the specific case of this patch, the ":" there actually makes sense
> > for someone that it is reading it as a text file, and it is an easy
> > hack to make it parse better.
> >   
> > > Specifically, this new requirement isn't documented anywhere AFAICT, so
> > > how will anyone adding new ABI descriptions learn about it?  
> > 
> > Yeah, either that or provide an alternative to "Description" tag, to be
> > used with more complex ABI descriptions.
> > 
> > One of the things that occurred to me, back on 2017, is that we should
> > have a way to to specify that an specific ABI description would have
> > a rich format. Something like:
> > 
> > What:		/sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/pyra/roccatpyra<minor>/actual_cpi
> > Date:		August 2010
> > Contact:	Stefan Achatz <erazor_de@users.sourceforge.net>
> > RST-Description:
> > 		It is possible to switch the cpi setting of the mouse with the
> > 		press of a button.
> > 		When read, this file returns the raw number of the actual cpi
> > 		setting reported by the mouse. This number has to be further
> > 		processed to receive the real dpi value:
> > 
> > 		===== =====
> > 		VALUE DPI
> > 		===== =====
> > 		1     400
> > 		2     800
> > 		4     1600
> > 		===== =====
> > 
> > With that, the script will know that the description contents will be using
> > the ReST markup, and parse it accordingly. Right now, what it does, instead,
> > is to place the description on a code-block, e. g. it will produce this
> > output for the description:
> > 
> > ::
> > 
> > 		It is possible to switch the cpi setting of the mouse with the
> > 		press of a button.
> > 		When read, this file returns the raw number of the actual cpi
> > 		setting reported by the mouse. This number has to be further
> > 		processed to receive the real dpi value:
> > 
> > 		VALUE DPI
> > 		1     400
> > 		2     800
> > 		4     1600
> > 
> > 
> > Greg, 
> > 
> > what do you think?  
> 
> I don't know when "Description" and "RST-Description" would be used.
> Why not just parse "Description" like rst text and if things are "messy"
> we fix them up as found, like you did with the ":" here?  It doesn't
> have to be complex, we can always fix them up after-the-fact if new
> stuff gets added that doesn't quite parse properly.
> 
> Just like we do for most kernel-doc formatting :)

Works for me. Yet, I guess I tried that, back on 2017. 

If I'm not mistaken, the initial patchset to solve the broken things 
won't be small, and will be require a lot of attention in order to
identify what's broken and where.

Btw, one thing is to pass at ReST validation. Another thing is to
produce something that people can read. 

Right now, the pertinent logic at the script I wrote (scripts/get_abi.pl)
is here:

                if (!($desc =~ /^\s*$/)) {
                        if ($desc =~ m/\:\n/ || $desc =~ m/\n[\t ]+/  || $desc =~ m/[\x00-\x08\x0b-\x1f\x7b-\xff]/) {
                                # put everything inside a code block
                                $desc =~ s/\n/\n /g;

                                print "::\n\n";
                                print " $desc\n\n";
                        } else {
                                # Escape any special chars from description
                                $desc =~s/([\x00-\x08\x0b-\x1f\x21-\x2a\x2d\x2f\x3c-\x40\x5c\x5e-\x60\x7b-\xff])/\\$1/g;

                                print "$desc\n\n";
                        }
                }

If it discovers something weird enough, it just places everything
into a comment block. Otherwise, it assumes that it is a plain
text and that any special characters should be escaped.

If the above block is replaced by a simple:

		print "$desc\n\n";

The description content will be handled as a ReST file.

I don't have any time right now to do this change and to handle the
warnings that will start to popup.

Btw, a single replace there is enough to show the amount of problems that
it will rise, as it will basically break Sphinx build with:

	reading sources... [  1%] admin-guide/abi-testing
	reST markup error:
	get_abi.pl rest --dir $srctree/Documentation/ABI/testing:45261: (SEVERE/4) Missing matching underline for section title overline.
	
	==========================
	PCIe Device AER statistics
	These attributes show up under all the devices that are AER capable. These

Thanks,
Mauro

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
index 7bc619b6890c..e75f441afdcd 100755
--- a/scripts/get_abi.pl
+++ b/scripts/get_abi.pl
@@ -288,18 +288,18 @@ sub output_rest {
 		$desc =~ s/\n[\-\*\=\^\~]+\n/\n/g;
 
 		if (!($desc =~ /^\s*$/)) {
-			if ($desc =~ m/\:\n/ || $desc =~ m/\n[\t ]+/  || $desc =~ m/[\x00-\x08\x0b-\x1f\x7b-\xff]/) {
-				# put everything inside a code block
-				$desc =~ s/\n/\n /g;
+#			if ($desc =~ m/\:\n/ || $desc =~ m/\n[\t ]+/  || $desc =~ m/[\x00-\x08\x0b-\x1f\x7b-\xff]/) {
+#				# put everything inside a code block
+#				$desc =~ s/\n/\n /g;
 
-				print "::\n\n";
-				print " $desc\n\n";
-			} else {
-				# Escape any special chars from description
-				$desc =~s/([\x00-\x08\x0b-\x1f\x21-\x2a\x2d\x2f\x3c-\x40\x5c\x5e-\x60\x7b-\xff])/\\$1/g;
+#				print "::\n\n";
+#				print " $desc\n\n";
+#			} else {
+#				# Escape any special chars from description
+#				$desc =~s/([\x00-\x08\x0b-\x1f\x21-\x2a\x2d\x2f\x3c-\x40\x5c\x5e-\x60\x7b-\xff])/\\$1/g;
 
 				print "$desc\n\n";
-			}
+#			}
 		} else {
 			print "DESCRIPTION MISSING for $what\n\n" if (!$data{$what}->{is_file});
 		}


^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-18  8:47             ` Jani Nikula
@ 2019-06-19 16:37               ` Mauro Carvalho Chehab
  2019-06-21 14:27                 ` Mauro Carvalho Chehab
  0 siblings, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-19 16:37 UTC (permalink / raw)
  To: Jani Nikula
  Cc: Greg Kroah-Hartman, Linux Doc Mailing List,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet

Em Tue, 18 Jun 2019 11:47:32 +0300
Jani Nikula <jani.nikula@linux.intel.com> escreveu:

> On Mon, 17 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> > Yeah, I guess it should be possible to do that. How a python script
> > can identify if it was called by Sphinx, or if it was called directly?  
> 
> if __name__ == '__main__':
> 	# run on the command-line, not imported

Ok, when I have some spare time, I may try to convert one script
to python and see how it behaves. 

Thanks,
Mauro

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 04/14] ABI: better identificate tables
  2019-06-19 15:02       ` Greg Kroah-Hartman
  2019-06-19 16:14         ` Mauro Carvalho Chehab
@ 2019-06-20 12:01         ` Johan Hovold
  2019-06-20 12:54           ` Greg Kroah-Hartman
  1 sibling, 1 reply; 63+ messages in thread
From: Johan Hovold @ 2019-06-20 12:01 UTC (permalink / raw)
  To: Greg Kroah-Hartman
  Cc: Mauro Carvalho Chehab, Johan Hovold, Linux Doc Mailing List,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Stefan Achatz

On Wed, Jun 19, 2019 at 05:02:07PM +0200, Greg Kroah-Hartman wrote:
> On Wed, Jun 19, 2019 at 10:56:33AM -0300, Mauro Carvalho Chehab wrote:
> > Hi Johan,
> > 
> > Em Wed, 19 Jun 2019 14:51:35 +0200
> > Johan Hovold <johan@kernel.org> escreveu:
> > 
> > > On Thu, Jun 13, 2019 at 11:04:10PM -0300, Mauro Carvalho Chehab wrote:
> > > > From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> > > > 
> > > > When parsing via script, it is important to know if the script
> > > > should consider a description as a literal block that should
> > > > be displayed as-is, or if the description can be considered
> > > > as a normal text.
> > > > 
> > > > Change descriptions to ensure that the preceding line of a table
> > > > ends with a colon. That makes easy to identify the need of a
> > > > literal block.  
> > > 
> > > In the cover letter you say that the first four patches of this series,
> > > including this one, "fix some ABI descriptions that are violating the
> > > syntax described at Documentation/ABI/README". This seems a bit harsh,
> > > given that it's you that is now *introducing* a new syntax requirement
> > > to assist your script.
> > 
> > Yeah, what's there at the cover letter doesn't apply to this specific
> > patch. The thing is that I wrote this series a lot of time ago (2016/17).

Got it, thanks.

[...]

> > In the specific case of this patch, the ":" there actually makes sense
> > for someone that it is reading it as a text file, and it is an easy
> > hack to make it parse better.

Human readers probably depend on more on tabulation and white space.
When the preceding description wasn't using a colon to begin with (and
you just replace s/\./:/) it can even look weird, but no big deal.

> > > Specifically, this new requirement isn't documented anywhere AFAICT, so
> > > how will anyone adding new ABI descriptions learn about it?
> > 
> > Yeah, either that or provide an alternative to "Description" tag, to be
> > used with more complex ABI descriptions.
> > 
> > One of the things that occurred to me, back on 2017, is that we should
> > have a way to to specify that an specific ABI description would have
> > a rich format. Something like:

[...]

> I don't know when "Description" and "RST-Description" would be used.
> Why not just parse "Description" like rst text and if things are "messy"
> we fix them up as found, like you did with the ":" here?  It doesn't
> have to be complex, we can always fix them up after-the-fact if new
> stuff gets added that doesn't quite parse properly.
> 
> Just like we do for most kernel-doc formatting :)

But kernel-doc has a documented format, which was sort of the point I
was trying to make. If the new get_abi.pl scripts expects a colon I
think it should be mentioned somewhere (e.g. Documentation/ABI/README).

Grepping for attribute entries in linux-next still reveals a number
descriptions that still lack that colon and use varying formatting. More
are bound to be added later, but perhaps that's ok depending on what
you're aiming at here.

Johan

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 04/14] ABI: better identificate tables
  2019-06-20 12:01         ` Johan Hovold
@ 2019-06-20 12:54           ` Greg Kroah-Hartman
  2019-06-20 14:20             ` Mauro Carvalho Chehab
  0 siblings, 1 reply; 63+ messages in thread
From: Greg Kroah-Hartman @ 2019-06-20 12:54 UTC (permalink / raw)
  To: Johan Hovold
  Cc: Mauro Carvalho Chehab, Linux Doc Mailing List,
	Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
	Jonathan Corbet, Stefan Achatz

On Thu, Jun 20, 2019 at 02:01:50PM +0200, Johan Hovold wrote:
> > I don't know when "Description" and "RST-Description" would be used.
> > Why not just parse "Description" like rst text and if things are "messy"
> > we fix them up as found, like you did with the ":" here?  It doesn't
> > have to be complex, we can always fix them up after-the-fact if new
> > stuff gets added that doesn't quite parse properly.
> > 
> > Just like we do for most kernel-doc formatting :)
> 
> But kernel-doc has a documented format, which was sort of the point I
> was trying to make. If the new get_abi.pl scripts expects a colon I
> think it should be mentioned somewhere (e.g. Documentation/ABI/README).
> 
> Grepping for attribute entries in linux-next still reveals a number
> descriptions that still lack that colon and use varying formatting. More
> are bound to be added later, but perhaps that's ok depending on what
> you're aiming at here.

I'm aiming for "good enough" to start with, and then we can work through
the exceptions.

But given that Mauro hasn't resent the script that does the conversion
of the files, I don't know if that will even matter... {hint}

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 04/14] ABI: better identificate tables
  2019-06-20 12:54           ` Greg Kroah-Hartman
@ 2019-06-20 14:20             ` Mauro Carvalho Chehab
  2019-06-20 16:29               ` Greg Kroah-Hartman
  0 siblings, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-20 14:20 UTC (permalink / raw)
  To: Greg Kroah-Hartman
  Cc: Johan Hovold, Linux Doc Mailing List, Mauro Carvalho Chehab,
	Mauro Carvalho Chehab, linux-kernel, Jonathan Corbet,
	Stefan Achatz

Em Thu, 20 Jun 2019 14:54:13 +0200
Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:

> On Thu, Jun 20, 2019 at 02:01:50PM +0200, Johan Hovold wrote:
> > > I don't know when "Description" and "RST-Description" would be used.
> > > Why not just parse "Description" like rst text and if things are "messy"
> > > we fix them up as found, like you did with the ":" here?  It doesn't
> > > have to be complex, we can always fix them up after-the-fact if new
> > > stuff gets added that doesn't quite parse properly.
> > > 
> > > Just like we do for most kernel-doc formatting :)  
> > 
> > But kernel-doc has a documented format, which was sort of the point I
> > was trying to make. If the new get_abi.pl scripts expects a colon I
> > think it should be mentioned somewhere (e.g. Documentation/ABI/README).
> > 
> > Grepping for attribute entries in linux-next still reveals a number
> > descriptions that still lack that colon and use varying formatting. More
> > are bound to be added later, but perhaps that's ok depending on what
> > you're aiming at here.  
> 
> I'm aiming for "good enough" to start with, and then we can work through
> the exceptions.
> 
> But given that Mauro hasn't resent the script that does the conversion
> of the files, I don't know if that will even matter... {hint}

It sounds I missed something... are you expecting a new version? 

If so, what changes are you expecting?

Thanks,
Mauro

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 04/14] ABI: better identificate tables
  2019-06-19 16:14         ` Mauro Carvalho Chehab
@ 2019-06-20 14:27           ` Mauro Carvalho Chehab
  2019-06-21  7:21             ` Greg Kroah-Hartman
  0 siblings, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-20 14:27 UTC (permalink / raw)
  To: Greg Kroah-Hartman
  Cc: Johan Hovold, Linux Doc Mailing List, Mauro Carvalho Chehab,
	Mauro Carvalho Chehab, linux-kernel, Jonathan Corbet,
	Stefan Achatz

Em Wed, 19 Jun 2019 13:14:08 -0300
Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:

> Em Wed, 19 Jun 2019 17:02:07 +0200
> Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:
> 
> > On Wed, Jun 19, 2019 at 10:56:33AM -0300, Mauro Carvalho Chehab wrote:  
> > > Hi Johan,
> > > 
> > > Em Wed, 19 Jun 2019 14:51:35 +0200
> > > Johan Hovold <johan@kernel.org> escreveu:
> > >     
> > > > On Thu, Jun 13, 2019 at 11:04:10PM -0300, Mauro Carvalho Chehab wrote:    
> > > > > From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> > > > > 
> > > > > When parsing via script, it is important to know if the script
> > > > > should consider a description as a literal block that should
> > > > > be displayed as-is, or if the description can be considered
> > > > > as a normal text.
> > > > > 
> > > > > Change descriptions to ensure that the preceding line of a table
> > > > > ends with a colon. That makes easy to identify the need of a
> > > > > literal block.      
> > > > 
> > > > In the cover letter you say that the first four patches of this series,
> > > > including this one, "fix some ABI descriptions that are violating the
> > > > syntax described at Documentation/ABI/README". This seems a bit harsh,
> > > > given that it's you that is now *introducing* a new syntax requirement
> > > > to assist your script.    
> > > 
> > > Yeah, what's there at the cover letter doesn't apply to this specific
> > > patch. The thing is that I wrote this series a lot of time ago (2016/17).
> > > 
> > > I revived those per a request at KS ML, as we still need to expose the
> > > ABI content on some book that will be used by userspace people.
> > > 
> > > So, I just rebased it on the top of curent Kernel, add a cover letter
> > > with the things I remembered and re-sent.
> > > 
> > > In the specific case of this patch, the ":" there actually makes sense
> > > for someone that it is reading it as a text file, and it is an easy
> > > hack to make it parse better.
> > >     
> > > > Specifically, this new requirement isn't documented anywhere AFAICT, so
> > > > how will anyone adding new ABI descriptions learn about it?    
> > > 
> > > Yeah, either that or provide an alternative to "Description" tag, to be
> > > used with more complex ABI descriptions.
> > > 
> > > One of the things that occurred to me, back on 2017, is that we should
> > > have a way to to specify that an specific ABI description would have
> > > a rich format. Something like:
> > > 
> > > What:		/sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/pyra/roccatpyra<minor>/actual_cpi
> > > Date:		August 2010
> > > Contact:	Stefan Achatz <erazor_de@users.sourceforge.net>
> > > RST-Description:
> > > 		It is possible to switch the cpi setting of the mouse with the
> > > 		press of a button.
> > > 		When read, this file returns the raw number of the actual cpi
> > > 		setting reported by the mouse. This number has to be further
> > > 		processed to receive the real dpi value:
> > > 
> > > 		===== =====
> > > 		VALUE DPI
> > > 		===== =====
> > > 		1     400
> > > 		2     800
> > > 		4     1600
> > > 		===== =====
> > > 
> > > With that, the script will know that the description contents will be using
> > > the ReST markup, and parse it accordingly. Right now, what it does, instead,
> > > is to place the description on a code-block, e. g. it will produce this
> > > output for the description:
> > > 
> > > ::
> > > 
> > > 		It is possible to switch the cpi setting of the mouse with the
> > > 		press of a button.
> > > 		When read, this file returns the raw number of the actual cpi
> > > 		setting reported by the mouse. This number has to be further
> > > 		processed to receive the real dpi value:
> > > 
> > > 		VALUE DPI
> > > 		1     400
> > > 		2     800
> > > 		4     1600
> > > 
> > > 
> > > Greg, 
> > > 
> > > what do you think?    
> > 
> > I don't know when "Description" and "RST-Description" would be used.
> > Why not just parse "Description" like rst text and if things are "messy"
> > we fix them up as found, like you did with the ":" here?  It doesn't
> > have to be complex, we can always fix them up after-the-fact if new
> > stuff gets added that doesn't quite parse properly.
> > 
> > Just like we do for most kernel-doc formatting :)  
> 
> Works for me. Yet, I guess I tried that, back on 2017. 
> 
> If I'm not mistaken, the initial patchset to solve the broken things 
> won't be small, and will be require a lot of attention in order to
> identify what's broken and where.
> 
> Btw, one thing is to pass at ReST validation. Another thing is to
> produce something that people can read. 
> 
> Right now, the pertinent logic at the script I wrote (scripts/get_abi.pl)
> is here:
> 
>                 if (!($desc =~ /^\s*$/)) {
>                         if ($desc =~ m/\:\n/ || $desc =~ m/\n[\t ]+/  || $desc =~ m/[\x00-\x08\x0b-\x1f\x7b-\xff]/) {
>                                 # put everything inside a code block
>                                 $desc =~ s/\n/\n /g;
> 
>                                 print "::\n\n";
>                                 print " $desc\n\n";
>                         } else {
>                                 # Escape any special chars from description
>                                 $desc =~s/([\x00-\x08\x0b-\x1f\x21-\x2a\x2d\x2f\x3c-\x40\x5c\x5e-\x60\x7b-\xff])/\\$1/g;
> 
>                                 print "$desc\n\n";
>                         }
>                 }
> 
> If it discovers something weird enough, it just places everything
> into a comment block. Otherwise, it assumes that it is a plain
> text and that any special characters should be escaped.
> 
> If the above block is replaced by a simple:
> 
> 		print "$desc\n\n";
> 
> The description content will be handled as a ReST file.
> 
> I don't have any time right now to do this change and to handle the
> warnings that will start to popup.
> 
> Btw, a single replace there is enough to show the amount of problems that
> it will rise, as it will basically break Sphinx build with:
> 
> 	reading sources... [  1%] admin-guide/abi-testing
> 	reST markup error:
> 	get_abi.pl rest --dir $srctree/Documentation/ABI/testing:45261: (SEVERE/4) Missing matching underline for section title overline.
> 	
> 	==========================
> 	PCIe Device AER statistics
> 	These attributes show up under all the devices that are AER capable. These

To be clear here: the problem with the above is that ReST has zero
tolerance and actually behaves like a crash, if it receives something like:

   ==========================
   PCIe Device AER statistics
   ==========================

For it to work, it has to have zero spaces before ===..=== line, e. g.:

==========================
PCIe Device AER statistics
==========================

Ok, maybe we could try to teach the parser a way to identify the initial
spacing of the first description line and remove that amount of 
spaces/tabs for the following lines, but it may require some heuristics.

Thanks,
Mauro

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 04/14] ABI: better identificate tables
  2019-06-20 14:20             ` Mauro Carvalho Chehab
@ 2019-06-20 16:29               ` Greg Kroah-Hartman
  2019-06-20 17:16                 ` Mauro Carvalho Chehab
  0 siblings, 1 reply; 63+ messages in thread
From: Greg Kroah-Hartman @ 2019-06-20 16:29 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Johan Hovold, Linux Doc Mailing List, Mauro Carvalho Chehab,
	Mauro Carvalho Chehab, linux-kernel, Jonathan Corbet,
	Stefan Achatz

On Thu, Jun 20, 2019 at 11:20:34AM -0300, Mauro Carvalho Chehab wrote:
> Em Thu, 20 Jun 2019 14:54:13 +0200
> Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:
> 
> > On Thu, Jun 20, 2019 at 02:01:50PM +0200, Johan Hovold wrote:
> > > > I don't know when "Description" and "RST-Description" would be used.
> > > > Why not just parse "Description" like rst text and if things are "messy"
> > > > we fix them up as found, like you did with the ":" here?  It doesn't
> > > > have to be complex, we can always fix them up after-the-fact if new
> > > > stuff gets added that doesn't quite parse properly.
> > > > 
> > > > Just like we do for most kernel-doc formatting :)  
> > > 
> > > But kernel-doc has a documented format, which was sort of the point I
> > > was trying to make. If the new get_abi.pl scripts expects a colon I
> > > think it should be mentioned somewhere (e.g. Documentation/ABI/README).
> > > 
> > > Grepping for attribute entries in linux-next still reveals a number
> > > descriptions that still lack that colon and use varying formatting. More
> > > are bound to be added later, but perhaps that's ok depending on what
> > > you're aiming at here.  
> > 
> > I'm aiming for "good enough" to start with, and then we can work through
> > the exceptions.
> > 
> > But given that Mauro hasn't resent the script that does the conversion
> > of the files, I don't know if that will even matter... {hint}
> 
> It sounds I missed something... are you expecting a new version? 

Yes, the last round of patches didn't have a SPDX header on the script,
so I couldn't add it to the tree :(

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 04/14] ABI: better identificate tables
  2019-06-20 16:29               ` Greg Kroah-Hartman
@ 2019-06-20 17:16                 ` Mauro Carvalho Chehab
  2019-06-20 17:55                   ` Greg Kroah-Hartman
  0 siblings, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-20 17:16 UTC (permalink / raw)
  To: Greg Kroah-Hartman
  Cc: Johan Hovold, Linux Doc Mailing List, Mauro Carvalho Chehab,
	Mauro Carvalho Chehab, linux-kernel, Jonathan Corbet,
	Stefan Achatz

Em Thu, 20 Jun 2019 18:29:45 +0200
Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:

> On Thu, Jun 20, 2019 at 11:20:34AM -0300, Mauro Carvalho Chehab wrote:
> > Em Thu, 20 Jun 2019 14:54:13 +0200
> > Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:
> >   
> > > On Thu, Jun 20, 2019 at 02:01:50PM +0200, Johan Hovold wrote:  
> > > > > I don't know when "Description" and "RST-Description" would be used.
> > > > > Why not just parse "Description" like rst text and if things are "messy"
> > > > > we fix them up as found, like you did with the ":" here?  It doesn't
> > > > > have to be complex, we can always fix them up after-the-fact if new
> > > > > stuff gets added that doesn't quite parse properly.
> > > > > 
> > > > > Just like we do for most kernel-doc formatting :)    
> > > > 
> > > > But kernel-doc has a documented format, which was sort of the point I
> > > > was trying to make. If the new get_abi.pl scripts expects a colon I
> > > > think it should be mentioned somewhere (e.g. Documentation/ABI/README).
> > > > 
> > > > Grepping for attribute entries in linux-next still reveals a number
> > > > descriptions that still lack that colon and use varying formatting. More
> > > > are bound to be added later, but perhaps that's ok depending on what
> > > > you're aiming at here.    
> > > 
> > > I'm aiming for "good enough" to start with, and then we can work through
> > > the exceptions.
> > > 
> > > But given that Mauro hasn't resent the script that does the conversion
> > > of the files, I don't know if that will even matter... {hint}  
> > 
> > It sounds I missed something... are you expecting a new version?   
> 
> Yes, the last round of patches didn't have a SPDX header on the script,
> so I couldn't add it to the tree :(

I could swear I sent you a version with SPDX on it... anyway, I'm
re-sending the hole thing. The SPDX header addition is on a separate
patch.


Thanks,
Mauro

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 04/14] ABI: better identificate tables
  2019-06-20 17:16                 ` Mauro Carvalho Chehab
@ 2019-06-20 17:55                   ` Greg Kroah-Hartman
  0 siblings, 0 replies; 63+ messages in thread
From: Greg Kroah-Hartman @ 2019-06-20 17:55 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Johan Hovold, Linux Doc Mailing List, Mauro Carvalho Chehab,
	Mauro Carvalho Chehab, linux-kernel, Jonathan Corbet,
	Stefan Achatz

On Thu, Jun 20, 2019 at 02:16:28PM -0300, Mauro Carvalho Chehab wrote:
> Em Thu, 20 Jun 2019 18:29:45 +0200
> Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:
> 
> > On Thu, Jun 20, 2019 at 11:20:34AM -0300, Mauro Carvalho Chehab wrote:
> > > Em Thu, 20 Jun 2019 14:54:13 +0200
> > > Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:
> > >   
> > > > On Thu, Jun 20, 2019 at 02:01:50PM +0200, Johan Hovold wrote:  
> > > > > > I don't know when "Description" and "RST-Description" would be used.
> > > > > > Why not just parse "Description" like rst text and if things are "messy"
> > > > > > we fix them up as found, like you did with the ":" here?  It doesn't
> > > > > > have to be complex, we can always fix them up after-the-fact if new
> > > > > > stuff gets added that doesn't quite parse properly.
> > > > > > 
> > > > > > Just like we do for most kernel-doc formatting :)    
> > > > > 
> > > > > But kernel-doc has a documented format, which was sort of the point I
> > > > > was trying to make. If the new get_abi.pl scripts expects a colon I
> > > > > think it should be mentioned somewhere (e.g. Documentation/ABI/README).
> > > > > 
> > > > > Grepping for attribute entries in linux-next still reveals a number
> > > > > descriptions that still lack that colon and use varying formatting. More
> > > > > are bound to be added later, but perhaps that's ok depending on what
> > > > > you're aiming at here.    
> > > > 
> > > > I'm aiming for "good enough" to start with, and then we can work through
> > > > the exceptions.
> > > > 
> > > > But given that Mauro hasn't resent the script that does the conversion
> > > > of the files, I don't know if that will even matter... {hint}  
> > > 
> > > It sounds I missed something... are you expecting a new version?   
> > 
> > Yes, the last round of patches didn't have a SPDX header on the script,
> > so I couldn't add it to the tree :(
> 
> I could swear I sent you a version with SPDX on it... anyway, I'm
> re-sending the hole thing. The SPDX header addition is on a separate
> patch.

Ah, you didn't cc: me on the whole thread this time, that's why I missed
it last time.  I sucked it off lkml now, will look at it in the morning.

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 04/14] ABI: better identificate tables
  2019-06-20 14:27           ` Mauro Carvalho Chehab
@ 2019-06-21  7:21             ` Greg Kroah-Hartman
  2019-06-21  9:49               ` Mauro Carvalho Chehab
  0 siblings, 1 reply; 63+ messages in thread
From: Greg Kroah-Hartman @ 2019-06-21  7:21 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Johan Hovold, Linux Doc Mailing List, Mauro Carvalho Chehab,
	Mauro Carvalho Chehab, linux-kernel, Jonathan Corbet,
	Stefan Achatz

On Thu, Jun 20, 2019 at 11:27:49AM -0300, Mauro Carvalho Chehab wrote:
> Em Wed, 19 Jun 2019 13:14:08 -0300
> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> 
> > Em Wed, 19 Jun 2019 17:02:07 +0200
> > Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:
> > 
> > > On Wed, Jun 19, 2019 at 10:56:33AM -0300, Mauro Carvalho Chehab wrote:  
> > > > Hi Johan,
> > > > 
> > > > Em Wed, 19 Jun 2019 14:51:35 +0200
> > > > Johan Hovold <johan@kernel.org> escreveu:
> > > >     
> > > > > On Thu, Jun 13, 2019 at 11:04:10PM -0300, Mauro Carvalho Chehab wrote:    
> > > > > > From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> > > > > > 
> > > > > > When parsing via script, it is important to know if the script
> > > > > > should consider a description as a literal block that should
> > > > > > be displayed as-is, or if the description can be considered
> > > > > > as a normal text.
> > > > > > 
> > > > > > Change descriptions to ensure that the preceding line of a table
> > > > > > ends with a colon. That makes easy to identify the need of a
> > > > > > literal block.      
> > > > > 
> > > > > In the cover letter you say that the first four patches of this series,
> > > > > including this one, "fix some ABI descriptions that are violating the
> > > > > syntax described at Documentation/ABI/README". This seems a bit harsh,
> > > > > given that it's you that is now *introducing* a new syntax requirement
> > > > > to assist your script.    
> > > > 
> > > > Yeah, what's there at the cover letter doesn't apply to this specific
> > > > patch. The thing is that I wrote this series a lot of time ago (2016/17).
> > > > 
> > > > I revived those per a request at KS ML, as we still need to expose the
> > > > ABI content on some book that will be used by userspace people.
> > > > 
> > > > So, I just rebased it on the top of curent Kernel, add a cover letter
> > > > with the things I remembered and re-sent.
> > > > 
> > > > In the specific case of this patch, the ":" there actually makes sense
> > > > for someone that it is reading it as a text file, and it is an easy
> > > > hack to make it parse better.
> > > >     
> > > > > Specifically, this new requirement isn't documented anywhere AFAICT, so
> > > > > how will anyone adding new ABI descriptions learn about it?    
> > > > 
> > > > Yeah, either that or provide an alternative to "Description" tag, to be
> > > > used with more complex ABI descriptions.
> > > > 
> > > > One of the things that occurred to me, back on 2017, is that we should
> > > > have a way to to specify that an specific ABI description would have
> > > > a rich format. Something like:
> > > > 
> > > > What:		/sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/pyra/roccatpyra<minor>/actual_cpi
> > > > Date:		August 2010
> > > > Contact:	Stefan Achatz <erazor_de@users.sourceforge.net>
> > > > RST-Description:
> > > > 		It is possible to switch the cpi setting of the mouse with the
> > > > 		press of a button.
> > > > 		When read, this file returns the raw number of the actual cpi
> > > > 		setting reported by the mouse. This number has to be further
> > > > 		processed to receive the real dpi value:
> > > > 
> > > > 		===== =====
> > > > 		VALUE DPI
> > > > 		===== =====
> > > > 		1     400
> > > > 		2     800
> > > > 		4     1600
> > > > 		===== =====
> > > > 
> > > > With that, the script will know that the description contents will be using
> > > > the ReST markup, and parse it accordingly. Right now, what it does, instead,
> > > > is to place the description on a code-block, e. g. it will produce this
> > > > output for the description:
> > > > 
> > > > ::
> > > > 
> > > > 		It is possible to switch the cpi setting of the mouse with the
> > > > 		press of a button.
> > > > 		When read, this file returns the raw number of the actual cpi
> > > > 		setting reported by the mouse. This number has to be further
> > > > 		processed to receive the real dpi value:
> > > > 
> > > > 		VALUE DPI
> > > > 		1     400
> > > > 		2     800
> > > > 		4     1600
> > > > 
> > > > 
> > > > Greg, 
> > > > 
> > > > what do you think?    
> > > 
> > > I don't know when "Description" and "RST-Description" would be used.
> > > Why not just parse "Description" like rst text and if things are "messy"
> > > we fix them up as found, like you did with the ":" here?  It doesn't
> > > have to be complex, we can always fix them up after-the-fact if new
> > > stuff gets added that doesn't quite parse properly.
> > > 
> > > Just like we do for most kernel-doc formatting :)  
> > 
> > Works for me. Yet, I guess I tried that, back on 2017. 
> > 
> > If I'm not mistaken, the initial patchset to solve the broken things 
> > won't be small, and will be require a lot of attention in order to
> > identify what's broken and where.
> > 
> > Btw, one thing is to pass at ReST validation. Another thing is to
> > produce something that people can read. 
> > 
> > Right now, the pertinent logic at the script I wrote (scripts/get_abi.pl)
> > is here:
> > 
> >                 if (!($desc =~ /^\s*$/)) {
> >                         if ($desc =~ m/\:\n/ || $desc =~ m/\n[\t ]+/  || $desc =~ m/[\x00-\x08\x0b-\x1f\x7b-\xff]/) {
> >                                 # put everything inside a code block
> >                                 $desc =~ s/\n/\n /g;
> > 
> >                                 print "::\n\n";
> >                                 print " $desc\n\n";
> >                         } else {
> >                                 # Escape any special chars from description
> >                                 $desc =~s/([\x00-\x08\x0b-\x1f\x21-\x2a\x2d\x2f\x3c-\x40\x5c\x5e-\x60\x7b-\xff])/\\$1/g;
> > 
> >                                 print "$desc\n\n";
> >                         }
> >                 }
> > 
> > If it discovers something weird enough, it just places everything
> > into a comment block. Otherwise, it assumes that it is a plain
> > text and that any special characters should be escaped.
> > 
> > If the above block is replaced by a simple:
> > 
> > 		print "$desc\n\n";
> > 
> > The description content will be handled as a ReST file.
> > 
> > I don't have any time right now to do this change and to handle the
> > warnings that will start to popup.
> > 
> > Btw, a single replace there is enough to show the amount of problems that
> > it will rise, as it will basically break Sphinx build with:
> > 
> > 	reading sources... [  1%] admin-guide/abi-testing
> > 	reST markup error:
> > 	get_abi.pl rest --dir $srctree/Documentation/ABI/testing:45261: (SEVERE/4) Missing matching underline for section title overline.
> > 	
> > 	==========================
> > 	PCIe Device AER statistics
> > 	These attributes show up under all the devices that are AER capable. These
> 
> To be clear here: the problem with the above is that ReST has zero
> tolerance and actually behaves like a crash, if it receives something like:
> 
>    ==========================
>    PCIe Device AER statistics
>    ==========================
> 
> For it to work, it has to have zero spaces before ===..=== line, e. g.:
> 
> ==========================
> PCIe Device AER statistics
> ==========================
> 
> Ok, maybe we could try to teach the parser a way to identify the initial
> spacing of the first description line and remove that amount of 
> spaces/tabs for the following lines, but it may require some heuristics.

Or we can clean this type of thing up by hand :)

Let's see how bad this gets, the documentation in these files should not
be very complex as they _should_ only be one-value-per-file, but as you
have shown in this very example, that rule is violated :(

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 04/14] ABI: better identificate tables
  2019-06-21  7:21             ` Greg Kroah-Hartman
@ 2019-06-21  9:49               ` Mauro Carvalho Chehab
  0 siblings, 0 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-21  9:49 UTC (permalink / raw)
  To: Greg Kroah-Hartman
  Cc: Johan Hovold, Linux Doc Mailing List, Mauro Carvalho Chehab,
	Mauro Carvalho Chehab, linux-kernel, Jonathan Corbet,
	Stefan Achatz

Em Fri, 21 Jun 2019 09:21:52 +0200
Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:

> On Thu, Jun 20, 2019 at 11:27:49AM -0300, Mauro Carvalho Chehab wrote:
> > Em Wed, 19 Jun 2019 13:14:08 -0300
> > Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> >   
> > > Em Wed, 19 Jun 2019 17:02:07 +0200
> > > Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:
> > >   
> > > > On Wed, Jun 19, 2019 at 10:56:33AM -0300, Mauro Carvalho Chehab wrote:    
> > > > > Hi Johan,
> > > > > 
> > > > > Em Wed, 19 Jun 2019 14:51:35 +0200
> > > > > Johan Hovold <johan@kernel.org> escreveu:
> > > > >       
> > > > > > On Thu, Jun 13, 2019 at 11:04:10PM -0300, Mauro Carvalho Chehab wrote:      
> > > > > > > From: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> > > > > > > 
> > > > > > > When parsing via script, it is important to know if the script
> > > > > > > should consider a description as a literal block that should
> > > > > > > be displayed as-is, or if the description can be considered
> > > > > > > as a normal text.
> > > > > > > 
> > > > > > > Change descriptions to ensure that the preceding line of a table
> > > > > > > ends with a colon. That makes easy to identify the need of a
> > > > > > > literal block.        
> > > > > > 
> > > > > > In the cover letter you say that the first four patches of this series,
> > > > > > including this one, "fix some ABI descriptions that are violating the
> > > > > > syntax described at Documentation/ABI/README". This seems a bit harsh,
> > > > > > given that it's you that is now *introducing* a new syntax requirement
> > > > > > to assist your script.      
> > > > > 
> > > > > Yeah, what's there at the cover letter doesn't apply to this specific
> > > > > patch. The thing is that I wrote this series a lot of time ago (2016/17).
> > > > > 
> > > > > I revived those per a request at KS ML, as we still need to expose the
> > > > > ABI content on some book that will be used by userspace people.
> > > > > 
> > > > > So, I just rebased it on the top of curent Kernel, add a cover letter
> > > > > with the things I remembered and re-sent.
> > > > > 
> > > > > In the specific case of this patch, the ":" there actually makes sense
> > > > > for someone that it is reading it as a text file, and it is an easy
> > > > > hack to make it parse better.
> > > > >       
> > > > > > Specifically, this new requirement isn't documented anywhere AFAICT, so
> > > > > > how will anyone adding new ABI descriptions learn about it?      
> > > > > 
> > > > > Yeah, either that or provide an alternative to "Description" tag, to be
> > > > > used with more complex ABI descriptions.
> > > > > 
> > > > > One of the things that occurred to me, back on 2017, is that we should
> > > > > have a way to to specify that an specific ABI description would have
> > > > > a rich format. Something like:
> > > > > 
> > > > > What:		/sys/bus/usb/devices/<busnum>-<devnum>:<config num>.<interface num>/<hid-bus>:<vendor-id>:<product-id>.<num>/pyra/roccatpyra<minor>/actual_cpi
> > > > > Date:		August 2010
> > > > > Contact:	Stefan Achatz <erazor_de@users.sourceforge.net>
> > > > > RST-Description:
> > > > > 		It is possible to switch the cpi setting of the mouse with the
> > > > > 		press of a button.
> > > > > 		When read, this file returns the raw number of the actual cpi
> > > > > 		setting reported by the mouse. This number has to be further
> > > > > 		processed to receive the real dpi value:
> > > > > 
> > > > > 		===== =====
> > > > > 		VALUE DPI
> > > > > 		===== =====
> > > > > 		1     400
> > > > > 		2     800
> > > > > 		4     1600
> > > > > 		===== =====
> > > > > 
> > > > > With that, the script will know that the description contents will be using
> > > > > the ReST markup, and parse it accordingly. Right now, what it does, instead,
> > > > > is to place the description on a code-block, e. g. it will produce this
> > > > > output for the description:
> > > > > 
> > > > > ::
> > > > > 
> > > > > 		It is possible to switch the cpi setting of the mouse with the
> > > > > 		press of a button.
> > > > > 		When read, this file returns the raw number of the actual cpi
> > > > > 		setting reported by the mouse. This number has to be further
> > > > > 		processed to receive the real dpi value:
> > > > > 
> > > > > 		VALUE DPI
> > > > > 		1     400
> > > > > 		2     800
> > > > > 		4     1600
> > > > > 
> > > > > 
> > > > > Greg, 
> > > > > 
> > > > > what do you think?      
> > > > 
> > > > I don't know when "Description" and "RST-Description" would be used.
> > > > Why not just parse "Description" like rst text and if things are "messy"
> > > > we fix them up as found, like you did with the ":" here?  It doesn't
> > > > have to be complex, we can always fix them up after-the-fact if new
> > > > stuff gets added that doesn't quite parse properly.
> > > > 
> > > > Just like we do for most kernel-doc formatting :)    
> > > 
> > > Works for me. Yet, I guess I tried that, back on 2017. 
> > > 
> > > If I'm not mistaken, the initial patchset to solve the broken things 
> > > won't be small, and will be require a lot of attention in order to
> > > identify what's broken and where.
> > > 
> > > Btw, one thing is to pass at ReST validation. Another thing is to
> > > produce something that people can read. 
> > > 
> > > Right now, the pertinent logic at the script I wrote (scripts/get_abi.pl)
> > > is here:
> > > 
> > >                 if (!($desc =~ /^\s*$/)) {
> > >                         if ($desc =~ m/\:\n/ || $desc =~ m/\n[\t ]+/  || $desc =~ m/[\x00-\x08\x0b-\x1f\x7b-\xff]/) {
> > >                                 # put everything inside a code block
> > >                                 $desc =~ s/\n/\n /g;
> > > 
> > >                                 print "::\n\n";
> > >                                 print " $desc\n\n";
> > >                         } else {
> > >                                 # Escape any special chars from description
> > >                                 $desc =~s/([\x00-\x08\x0b-\x1f\x21-\x2a\x2d\x2f\x3c-\x40\x5c\x5e-\x60\x7b-\xff])/\\$1/g;
> > > 
> > >                                 print "$desc\n\n";
> > >                         }
> > >                 }
> > > 
> > > If it discovers something weird enough, it just places everything
> > > into a comment block. Otherwise, it assumes that it is a plain
> > > text and that any special characters should be escaped.
> > > 
> > > If the above block is replaced by a simple:
> > > 
> > > 		print "$desc\n\n";
> > > 
> > > The description content will be handled as a ReST file.
> > > 
> > > I don't have any time right now to do this change and to handle the
> > > warnings that will start to popup.
> > > 
> > > Btw, a single replace there is enough to show the amount of problems that
> > > it will rise, as it will basically break Sphinx build with:
> > > 
> > > 	reading sources... [  1%] admin-guide/abi-testing
> > > 	reST markup error:
> > > 	get_abi.pl rest --dir $srctree/Documentation/ABI/testing:45261: (SEVERE/4) Missing matching underline for section title overline.
> > > 	
> > > 	==========================
> > > 	PCIe Device AER statistics
> > > 	These attributes show up under all the devices that are AER capable. These  
> > 
> > To be clear here: the problem with the above is that ReST has zero
> > tolerance and actually behaves like a crash, if it receives something like:
> > 
> >    ==========================
> >    PCIe Device AER statistics
> >    ==========================
> > 
> > For it to work, it has to have zero spaces before ===..=== line, e. g.:
> > 
> > ==========================
> > PCIe Device AER statistics
> > ==========================
> > 
> > Ok, maybe we could try to teach the parser a way to identify the initial
> > spacing of the first description line and remove that amount of 
> > spaces/tabs for the following lines, but it may require some heuristics.  
> 
> Or we can clean this type of thing up by hand :)
> 
> Let's see how bad this gets, the documentation in these files should not
> be very complex as they _should_ only be one-value-per-file, but as you
> have shown in this very example, that rule is violated :(

Hi Greg,

Doing a manual review - still incomplete (just stable/ dir) - and I didn't
try to run the script over ABI file yet - this is what I found so
far[1]: 17 files over 37 needed adjustments (45%).

That trick of detecting a complex file when it finds a ':' and parse the
hole description as a literal block solves most of them.

From the changes below, this file:
	Documentation/ABI/stable/sysfs-driver-mlxreg-io

Found a "creative way" to place the "What:" field avoiding long lines...

At least this one should be fixed, as the parser won't handle its
content right (it won't complain, but the result output will be wrong).

[1] I also added a note at README that files should be ReST-compatible


diff --git a/Documentation/ABI/README b/Documentation/ABI/README
index 3121029dce21..8bac9cb09a6d 100644
--- a/Documentation/ABI/README
+++ b/Documentation/ABI/README
@@ -32,7 +32,7 @@ The different levels of stability are:
 	layout of the files below for details on how to do this.)
 
   obsolete/
-  	This directory documents interfaces that are still remaining in
+	This directory documents interfaces that are still remaining in
 	the kernel, but are marked to be removed at some later point in
 	time.  The description of the interface will document the reason
 	why it is obsolete and when it can be expected to be removed.
@@ -58,6 +58,14 @@ Users:		All users of this interface who wish to be notified when
 		be changed further.
 
 
+Note:
+   The fields should be use a simple notation, compatible with ReST markup.
+   Also, the file **should not** have a top-level index, like::
+
+	===
+	foo
+	===
+
 How things move between levels:
 
 Interfaces in stable may move to obsolete, as long as the proper
diff --git a/Documentation/ABI/stable/firewire-cdev b/Documentation/ABI/stable/firewire-cdev
index f72ed653878a..d8df3004c318 100644
--- a/Documentation/ABI/stable/firewire-cdev
+++ b/Documentation/ABI/stable/firewire-cdev
@@ -20,6 +20,7 @@ Description:
 			  - Query node ID
 			  - Query maximum speed of the path between this node
 			    and local node
+
 		  - The 1394 bus (i.e. "card") to which the node is attached to:
 			  - Isochronous stream transmission and reception
 			  - Asynchronous stream transmission and reception
@@ -31,6 +32,7 @@ Description:
 			    manager
 			  - Query cycle time
 			  - Bus reset initiation, bus reset event reception
+
 		  - All 1394 buses:
 			  - Allocation of IEEE 1212 address ranges on the local
 			    link layers, reception of inbound requests to such
diff --git a/Documentation/ABI/stable/sysfs-acpi-pmprofile b/Documentation/ABI/stable/sysfs-acpi-pmprofile
index 964c7a8afb26..3b0e34eef83c 100644
--- a/Documentation/ABI/stable/sysfs-acpi-pmprofile
+++ b/Documentation/ABI/stable/sysfs-acpi-pmprofile
@@ -11,12 +11,15 @@ Values:         For possible values see ACPI specification:
 		Field: Preferred_PM_Profile
 
 		Currently these values are defined by spec:
-		0 Unspecified
-		1 Desktop
-		2 Mobile
-		3 Workstation
-		4 Enterprise Server
-		5 SOHO Server
-		6 Appliance PC
-		7 Performance Server
+
+		== =================
+		0  Unspecified
+		1  Desktop
+		2  Mobile
+		3  Workstation
+		4  Enterprise Server
+		5  SOHO Server
+		6  Appliance PC
+		7  Performance Server
 		>7 Reserved
+		== =================
diff --git a/Documentation/ABI/stable/sysfs-bus-firewire b/Documentation/ABI/stable/sysfs-bus-firewire
index 41e5a0cd1e3e..0c8c023855b6 100644
--- a/Documentation/ABI/stable/sysfs-bus-firewire
+++ b/Documentation/ABI/stable/sysfs-bus-firewire
@@ -125,7 +125,9 @@ Description:
 		Read-only attribute, immutable during the target's lifetime.
 		Format, as exposed by firewire-sbp2 since 2.6.22, May 2007:
 		Colon-separated hexadecimal string representations of
+
 			u64 EUI-64 : u24 directory_ID : u16 LUN
+
 		without 0x prefixes, without whitespace.  The former sbp2 driver
 		(removed in 2.6.37 after being superseded by firewire-sbp2) used
 		a somewhat shorter format which was not as close to SAM.
diff --git a/Documentation/ABI/stable/sysfs-bus-nvmem b/Documentation/ABI/stable/sysfs-bus-nvmem
index 9ffba8576f7b..c399323f37de 100644
--- a/Documentation/ABI/stable/sysfs-bus-nvmem
+++ b/Documentation/ABI/stable/sysfs-bus-nvmem
@@ -9,13 +9,14 @@ Description:
 		Note: This file is only present if CONFIG_NVMEM_SYSFS
 		is enabled
 
-		ex:
-		hexdump /sys/bus/nvmem/devices/qfprom0/nvmem
+		ex::
 
-		0000000 0000 0000 0000 0000 0000 0000 0000 0000
-		*
-		00000a0 db10 2240 0000 e000 0c00 0c00 0000 0c00
-		0000000 0000 0000 0000 0000 0000 0000 0000 0000
-		...
-		*
-		0001000
+		  hexdump /sys/bus/nvmem/devices/qfprom0/nvmem
+
+		  0000000 0000 0000 0000 0000 0000 0000 0000 0000
+		  *
+		  00000a0 db10 2240 0000 e000 0c00 0c00 0000 0c00
+		  0000000 0000 0000 0000 0000 0000 0000 0000 0000
+		  ...
+		  *
+		  0001000
diff --git a/Documentation/ABI/stable/sysfs-bus-usb b/Documentation/ABI/stable/sysfs-bus-usb
index b832eeff9999..cad4bc232520 100644
--- a/Documentation/ABI/stable/sysfs-bus-usb
+++ b/Documentation/ABI/stable/sysfs-bus-usb
@@ -50,8 +50,10 @@ Description:
 
 		Tools can use this file and the connected_duration file to
 		compute the percentage of time that a device has been active.
-		For example,
-		echo $((100 * `cat active_duration` / `cat connected_duration`))
+		For example::
+
+		  echo $((100 * `cat active_duration` / `cat connected_duration`))
+
 		will give an integer percentage.  Note that this does not
 		account for counter wrap.
 Users:
diff --git a/Documentation/ABI/stable/sysfs-class-backlight b/Documentation/ABI/stable/sysfs-class-backlight
index 70302f370e7e..023fb52645f8 100644
--- a/Documentation/ABI/stable/sysfs-class-backlight
+++ b/Documentation/ABI/stable/sysfs-class-backlight
@@ -4,6 +4,7 @@ KernelVersion:	2.6.12
 Contact:	Richard Purdie <rpurdie@rpsys.net>
 Description:
 		Control BACKLIGHT power, values are FB_BLANK_* from fb.h
+
 		 - FB_BLANK_UNBLANK (0)   : power on.
 		 - FB_BLANK_POWERDOWN (4) : power off
 Users:		HAL
diff --git a/Documentation/ABI/stable/sysfs-class-infiniband b/Documentation/ABI/stable/sysfs-class-infiniband
index 17211ceb9bf4..3f58952a8ccb 100644
--- a/Documentation/ABI/stable/sysfs-class-infiniband
+++ b/Documentation/ABI/stable/sysfs-class-infiniband
@@ -8,12 +8,14 @@ Date:		Apr, 2005
 KernelVersion:	v2.6.12
 Contact:	linux-rdma@vger.kernel.org
 Description:
+		=============== ===========================================
 		node_type:	(RO) Node type (CA, RNIC, usNIC, usNIC UDP,
 				switch or router)
 
 		node_guid:	(RO) Node GUID
 
 		sys_image_guid:	(RO) System image GUID
+		=============== ===========================================
 
 
 What:		/sys/class/infiniband/<device>/node_desc
@@ -47,6 +49,7 @@ KernelVersion:	v2.6.12
 Contact:	linux-rdma@vger.kernel.org
 Description:
 
+		=============== ===============================================
 		lid:		(RO) Port LID
 
 		rate:		(RO) Port data rate (active width * active
@@ -66,8 +69,9 @@ Description:
 
 		cap_mask:	(RO) Port capability mask. 2 bits here are
 				settable- IsCommunicationManagementSupported
-				(set when CM module is loaded) and IsSM (set via
-				open of issmN file).
+				(set when CM module is loaded) and IsSM (set
+				via open of issmN file).
+		=============== ===============================================
 
 
 What:		/sys/class/infiniband/<device>/ports/<port-num>/link_layer
@@ -143,7 +147,7 @@ Description:
 		poor link signal integrity
 
 		Data info:
-		---------
+		----------
 
 		port_xmit_data: (RO) Total number of data octets, divided by 4
 		(lanes), transmitted on all VLs. This is 64 bit counter
@@ -177,7 +181,7 @@ Description:
 		packets with errors.
 
 		Misc info:
-		---------
+		----------
 
 		port_xmit_discards: (RO) Total number of outbound packets
 		discarded by the port because the port is down or congested.
@@ -244,9 +248,11 @@ Description:
 		two umad devices and two issm devices, while a switch will have
 		one device of each type (for switch port 0).
 
+		======= =====================================
 		ibdev:	(RO) Show Infiniband (IB) device name
 
 		port:	(RO) Display port number
+		======= =====================================
 
 
 What:		/sys/class/infiniband_mad/abi_version
@@ -281,10 +287,12 @@ Date:		Sept, 2005
 KernelVersion:	v2.6.14
 Contact:	linux-rdma@vger.kernel.org
 Description:
+		=============== ===========================================
 		ibdev:		(RO) Display Infiniband (IB) device name
 
 		abi_version:	(RO) Show ABI version of IB device specific
 				interfaces.
+		=============== ===========================================
 
 
 What:		/sys/class/infiniband_verbs/abi_version
@@ -306,12 +314,14 @@ Date:		Apr, 2005
 KernelVersion:	v2.6.12
 Contact:	linux-rdma@vger.kernel.org
 Description:
+		=============== ================================================
 		hw_rev:		(RO) Hardware revision number
 
 		hca_type:	(RO) Host Channel Adapter type: MT23108, MT25208
 				(MT23108 compat mode), MT25208 or MT25204
 
 		board_id:	(RO) Manufacturing board ID
+		=============== ================================================
 
 
 sysfs interface for Chelsio T3 RDMA Driver (cxgb3)
@@ -324,6 +334,7 @@ Date:		Feb, 2007
 KernelVersion:	v2.6.21
 Contact:	linux-rdma@vger.kernel.org
 Description:
+		=============== ==============================================
 		hw_rev:		(RO) Hardware revision number
 
 		hca_type:	(RO) HCA type. Here it is a driver short name.
@@ -331,6 +342,7 @@ Description:
 				driver structure (e.g.  pci_driver::name).
 
 		board_id:	(RO) Manufacturing board id
+		=============== ==============================================
 
 
 sysfs interface for Mellanox ConnectX HCA IB driver (mlx4)
@@ -343,11 +355,13 @@ Date:		Sep, 2007
 KernelVersion:	v2.6.24
 Contact:	linux-rdma@vger.kernel.org
 Description:
+		=============== ===============================
 		hw_rev:		(RO) Hardware revision number
 
 		hca_type:	(RO) Host channel adapter type
 
 		board_id:	(RO) Manufacturing board ID
+		=============== ===============================
 
 
 What:		/sys/class/infiniband/mlx4_X/iov/ports/<port-num>/gids/<n>
@@ -373,6 +387,7 @@ Description:
 		example, ports/1/pkeys/10 contains the value at index 10 in port
 		1's P_Key table.
 
+		======================= ==========================================
 		gids/<n>:		(RO) The physical port gids n = 0..127
 
 		admin_guids/<n>:	(RW) Allows examining or changing the
@@ -401,6 +416,7 @@ Description:
 					guest, whenever it uses its pkey index
 					1, will actually be using the real pkey
 					index 10.
+		======================= ==========================================
 
 
 What:		/sys/class/infiniband/mlx4_X/iov/<pci-slot-num>/ports/<m>/smi_enabled
@@ -412,12 +428,14 @@ Description:
 		Enabling QP0 on VFs for selected VF/port. By default, no VFs are
 		enabled for QP0 operation.
 
-		smi_enabled:	(RO) Indicates whether smi is currently enabled
-				for the indicated VF/port
+		================= ==== ===========================================
+		smi_enabled:	  (RO) Indicates whether smi is currently enabled
+				       for the indicated VF/port
 
-		enable_smi_admin:(RW) Used by the admin to request that smi
-				capability be enabled or disabled for the
-				indicated VF/port. 0 = disable, 1 = enable.
+		enable_smi_admin: (RW) Used by the admin to request that smi
+				       capability be enabled or disabled for the
+				       indicated VF/port. 0 = disable, 1 = enable.
+		================= ==== ===========================================
 
 		The requested enablement will occur at the next reset of the VF
 		(e.g. driver restart on the VM which owns the VF).
@@ -433,11 +451,13 @@ Date:		Feb, 2008
 KernelVersion:	v2.6.25
 Contact:	linux-rdma@vger.kernel.org
 Description:
+		=============== ==== ==================================
 		hw_rev:		(RO) Hardware revision number
 
 		hca_type:	(RO) Host Channel Adapter type (NEX020)
 
 		board_id:	(RO) Manufacturing board id
+		=============== ==== ==================================
 
 
 sysfs interface for Chelsio T4/T5 RDMA driver (cxgb4)
@@ -451,6 +471,7 @@ KernelVersion:	v2.6.35
 Contact:	linux-rdma@vger.kernel.org
 Description:
 
+		=============== =============================================
 		hw_rev:		(RO) Hardware revision number
 
 		hca_type:	(RO) Driver short name. Should normally match
@@ -459,6 +480,7 @@ Description:
 
 		board_id:	(RO) Manufacturing board id. (Vendor + device
 				information)
+		=============== =============================================
 
 
 sysfs interface for Intel IB driver qib
@@ -479,6 +501,7 @@ Date:		May, 2010
 KernelVersion:	v2.6.35
 Contact:	linux-rdma@vger.kernel.org
 Description:
+		=============== ======================================================
 		version:	(RO) Display version information of installed software
 				and drivers.
 
@@ -505,6 +528,7 @@ Description:
 		chip_reset:	(WO) Reset the chip if possible by writing
 				"reset" to this file. Only allowed if no user
 				contexts are open that use chip resources.
+		=============== ======================================================
 
 
 What:		/sys/class/infiniband/qibX/ports/N/sl2vl/[0-15]
@@ -524,6 +548,7 @@ Contact:	linux-rdma@vger.kernel.org
 Description:
 		Per-port congestion control. Both are binary attributes.
 
+		=============== ================================================
 		cc_table_bin:	(RO) Congestion control table size followed by
 				table entries.
 
@@ -532,6 +557,7 @@ Description:
 				congestion entries - increase, timer, event log
 				trigger threshold and the minimum injection rate
 				delay.
+		=============== ================================================
 
 What:		/sys/class/infiniband/qibX/ports/N/linkstate/loopback
 What:		/sys/class/infiniband/qibX/ports/N/linkstate/led_override
@@ -544,6 +570,7 @@ Contact:	linux-rdma@vger.kernel.org
 Description:
 		[to be documented]
 
+		=============== ===============================================
 		loopback:	(WO)
 		led_override:	(WO)
 		hrtbt_enable:	(RW)
@@ -554,6 +581,7 @@ Description:
 				errors. Possible states are- "Initted",
 				"Present", "IB_link_up", "IB_configured" or
 				"Fatal_Hardware_Error".
+		=============== ===============================================
 
 What:		/sys/class/infiniband/qibX/ports/N/diag_counters/rc_resends
 What:		/sys/class/infiniband/qibX/ports/N/diag_counters/seq_naks
@@ -602,6 +630,7 @@ Contact:	Christian Benvenuti <benve@cisco.com>,
 		linux-rdma@vger.kernel.org
 Description:
 
+		=============== ===============================================
 		board_id:	(RO) Manufacturing board id
 
 		config:		(RO) Report the configuration for this PF
@@ -614,6 +643,7 @@ Description:
 
 		iface:		(RO) Shows which network interface this usNIC
 				entry is associated to (visible with ifconfig).
+		=============== ===============================================
 
 What:		/sys/class/infiniband/usnic_X/qpn/summary
 What:		/sys/class/infiniband/usnic_X/qpn/context
@@ -658,6 +688,7 @@ Date:		May, 2016
 KernelVersion:	v4.6
 Contact:	linux-rdma@vger.kernel.org
 Description:
+		=============== ==== ========================================
 		hw_rev:		(RO) Hardware revision number
 
 		board_id:	(RO) Manufacturing board id
@@ -676,6 +707,7 @@ Description:
 				available.
 
 		tempsense:	(RO) Thermal sense information
+		=============== ==== ========================================
 
 
 What:		/sys/class/infiniband/hfi1_X/ports/N/CCMgtA/cc_settings_bin
@@ -687,19 +719,21 @@ Contact:	linux-rdma@vger.kernel.org
 Description:
 		Per-port congestion control.
 
-		cc_table_bin:	(RO) CCA tables used by PSM2 Congestion control
+		=============== ================================================
+		cc_table_bin	(RO) CCA tables used by PSM2 Congestion control
 				table size followed by table entries. Binary
 				attribute.
 
-		cc_settings_bin:(RO) Congestion settings: port control, control
+		cc_settings_bin (RO) Congestion settings: port control, control
 				map and an array of 16 entries for the
 				congestion entries - increase, timer, event log
 				trigger threshold and the minimum injection rate
 				delay. Binary attribute.
 
-		cc_prescan:	(RW) enable prescanning for faster BECN
+		cc_prescan	(RW) enable prescanning for faster BECN
 				response. Write "on" to enable and "off" to
 				disable.
+		=============== ================================================
 
 What:		/sys/class/infiniband/hfi1_X/ports/N/sc2vl/[0-31]
 What:		/sys/class/infiniband/hfi1_X/ports/N/sl2sc/[0-31]
@@ -708,11 +742,13 @@ Date:		May, 2016
 KernelVersion:	v4.6
 Contact:	linux-rdma@vger.kernel.org
 Description:
+		=============== ===================================================
 		sc2vl/:		(RO) 32 files (0 - 31) used to translate sl->vl
 
 		sl2sc/:		(RO) 32 files (0 - 31) used to translate sl->sc
 
 		vl2mtu/:	(RO) 16 files (0 - 15) used to determine MTU for vl
+		=============== ===================================================
 
 
 What:		/sys/class/infiniband/hfi1_X/sdma_N/cpu_list
@@ -723,26 +759,28 @@ Contact:	linux-rdma@vger.kernel.org
 Description:
 		sdma<N>/ contains one directory per sdma engine (0 - 15)
 
+		=============== ==============================================
 		cpu_list:	(RW) List of cpus for user-process to sdma
 				engine assignment.
 
 		vl:		(RO) Displays the virtual lane (vl) the sdma
 				engine maps to.
+		=============== ==============================================
 
 		This interface gives the user control on the affinity settings
 		for the device. As an example, to set an sdma engine irq
 		affinity and thread affinity of a user processes to use the
 		sdma engine, which is "near" in terms of NUMA configuration, or
-		physical cpu location, the user will do:
+		physical cpu location, the user will do::
 
-		echo "3" > /proc/irq/<N>/smp_affinity_list
-		echo "4-7" > /sys/devices/.../sdma3/cpu_list
-		cat /sys/devices/.../sdma3/vl
-		0
-		echo "8" > /proc/irq/<M>/smp_affinity_list
-		echo "9-12" > /sys/devices/.../sdma4/cpu_list
-		cat /sys/devices/.../sdma4/vl
-		1
+		  echo "3" > /proc/irq/<N>/smp_affinity_list
+		  echo "4-7" > /sys/devices/.../sdma3/cpu_list
+		  cat /sys/devices/.../sdma3/vl
+		  0
+		  echo "8" > /proc/irq/<M>/smp_affinity_list
+		  echo "9-12" > /sys/devices/.../sdma4/cpu_list
+		  cat /sys/devices/.../sdma4/vl
+		  1
 
 		to make sure that when a process runs on cpus 4,5,6, or 7, and
 		uses vl=0, then sdma engine 3 is selected by the driver, and
@@ -764,11 +802,13 @@ Date:		Jan, 2016
 KernelVersion:	v4.10
 Contact:	linux-rdma@vger.kernel.org
 Description:
+		=============== ==== ========================
 		hw_rev:		(RO) Hardware revision number
 
 		hca_type:	(RO) Show HCA type (I40IW)
 
 		board_id:	(RO) I40IW board ID
+		=============== ==== ========================
 
 
 sysfs interface for QLogic qedr NIC Driver
@@ -781,9 +821,11 @@ KernelVersion:	v4.10
 Contact:	linux-rdma@vger.kernel.org
 Description:
 
+		=============== ==== ========================
 		hw_rev:		(RO) Hardware revision number
 
 		hca_type:	(RO) Display HCA type
+		=============== ==== ========================
 
 
 sysfs interface for VMware Paravirtual RDMA driver
@@ -797,11 +839,13 @@ KernelVersion:	v4.10
 Contact:	linux-rdma@vger.kernel.org
 Description:
 
+		=============== ==== =====================================
 		hw_rev:		(RO) Hardware revision number
 
 		hca_type:	(RO) Host channel adapter type
 
 		board_id:	(RO) Display PVRDMA manufacturing board ID
+		=============== ==== =====================================
 
 
 sysfs interface for Broadcom NetXtreme-E RoCE driver
@@ -813,6 +857,8 @@ Date:		Feb, 2017
 KernelVersion:	v4.11
 Contact:	linux-rdma@vger.kernel.org
 Description:
+		=============== ==== =========================
 		hw_rev:		(RO) Hardware revision number
 
 		hca_type:	(RO) Host channel adapter type
+		=============== ==== =========================
diff --git a/Documentation/ABI/stable/sysfs-class-rfkill b/Documentation/ABI/stable/sysfs-class-rfkill
index 80151a409d67..370254b6296a 100644
--- a/Documentation/ABI/stable/sysfs-class-rfkill
+++ b/Documentation/ABI/stable/sysfs-class-rfkill
@@ -54,6 +54,7 @@ Description: 	Current state of the transmitter.
 		through this interface. There will likely be another attempt to
 		remove it in the future.
 Values: 	A numeric value.
+
 		0: RFKILL_STATE_SOFT_BLOCKED
 			transmitter is turned off by software
 		1: RFKILL_STATE_UNBLOCKED
@@ -69,6 +70,7 @@ KernelVersion	v2.6.34
 Contact:	linux-wireless@vger.kernel.org
 Description: 	Current hardblock state. This file is read only.
 Values: 	A numeric value.
+
 		0: inactive
 			The transmitter is (potentially) active.
 		1: active
@@ -82,6 +84,7 @@ KernelVersion	v2.6.34
 Contact:	linux-wireless@vger.kernel.org
 Description:	Current softblock state. This file is read and write.
 Values: 	A numeric value.
+
 		0: inactive
 			The transmitter is (potentially) active.
 		1: active
diff --git a/Documentation/ABI/stable/sysfs-class-tpm b/Documentation/ABI/stable/sysfs-class-tpm
index c0e23830f56a..bbee8899a90e 100644
--- a/Documentation/ABI/stable/sysfs-class-tpm
+++ b/Documentation/ABI/stable/sysfs-class-tpm
@@ -32,11 +32,11 @@ KernelVersion:	2.6.12
 Contact:	tpmdd-devel@lists.sf.net
 Description:	The "caps" property contains TPM manufacturer and version info.
 
-		Example output:
+		Example output::
 
-		Manufacturer: 0x53544d20
-		TCG version: 1.2
-		Firmware version: 8.16
+		  Manufacturer: 0x53544d20
+		  TCG version: 1.2
+		  Firmware version: 8.16
 
 		Manufacturer is a hex dump of the 4 byte manufacturer info
 		space in a TPM. TCG version shows the TCG TPM spec level that
@@ -54,9 +54,9 @@ Description:	The "durations" property shows the 3 vendor-specific values
 		any longer than necessary before starting to poll for a
 		result.
 
-		Example output:
+		Example output::
 
-		3015000 4508000 180995000 [original]
+		  3015000 4508000 180995000 [original]
 
 		Here the short, medium and long durations are displayed in
 		usecs. "[original]" indicates that the values are displayed
@@ -92,14 +92,14 @@ Description:	The "pcrs" property will dump the current value of all Platform
 		values may be constantly changing, the output is only valid
 		for a snapshot in time.
 
-		Example output:
+		Example output::
 
-		PCR-00: 3A 3F 78 0F 11 A4 B4 99 69 FC AA 80 CD 6E 39 57 C3 3B 22 75
-		PCR-01: 3A 3F 78 0F 11 A4 B4 99 69 FC AA 80 CD 6E 39 57 C3 3B 22 75
-		PCR-02: 3A 3F 78 0F 11 A4 B4 99 69 FC AA 80 CD 6E 39 57 C3 3B 22 75
-		PCR-03: 3A 3F 78 0F 11 A4 B4 99 69 FC AA 80 CD 6E 39 57 C3 3B 22 75
-		PCR-04: 3A 3F 78 0F 11 A4 B4 99 69 FC AA 80 CD 6E 39 57 C3 3B 22 75
-		...
+		  PCR-00: 3A 3F 78 0F 11 A4 B4 99 69 FC AA 80 CD 6E 39 57 C3 3B 22 75
+		  PCR-01: 3A 3F 78 0F 11 A4 B4 99 69 FC AA 80 CD 6E 39 57 C3 3B 22 75
+		  PCR-02: 3A 3F 78 0F 11 A4 B4 99 69 FC AA 80 CD 6E 39 57 C3 3B 22 75
+		  PCR-03: 3A 3F 78 0F 11 A4 B4 99 69 FC AA 80 CD 6E 39 57 C3 3B 22 75
+		  PCR-04: 3A 3F 78 0F 11 A4 B4 99 69 FC AA 80 CD 6E 39 57 C3 3B 22 75
+		  ...
 
 		The number of PCRs and hex bytes needed to represent a PCR
 		value will vary depending on TPM chip version. For TPM 1.1 and
@@ -119,44 +119,44 @@ Description:	The "pubek" property will return the TPM's public endorsement
 		ated at TPM manufacture time and exists for the life of the
 		chip.
 
-		Example output:
-
-		Algorithm: 00 00 00 01
-		Encscheme: 00 03
-		Sigscheme: 00 01
-		Parameters: 00 00 08 00 00 00 00 02 00 00 00 00
-		Modulus length: 256
-		Modulus:
-		B4 76 41 82 C9 20 2C 10 18 40 BC 8B E5 44 4C 6C
-		3A B2 92 0C A4 9B 2A 83 EB 5C 12 85 04 48 A0 B6
-		1E E4 81 84 CE B2 F2 45 1C F0 85 99 61 02 4D EB
-		86 C4 F7 F3 29 60 52 93 6B B2 E5 AB 8B A9 09 E3
-		D7 0E 7D CA 41 BF 43 07 65 86 3C 8C 13 7A D0 8B
-		82 5E 96 0B F8 1F 5F 34 06 DA A2 52 C1 A9 D5 26
-		0F F4 04 4B D9 3F 2D F2 AC 2F 74 64 1F 8B CD 3E
-		1E 30 38 6C 70 63 69 AB E2 50 DF 49 05 2E E1 8D
-		6F 78 44 DA 57 43 69 EE 76 6C 38 8A E9 8E A3 F0
-		A7 1F 3C A8 D0 12 15 3E CA 0E BD FA 24 CD 33 C6
-		47 AE A4 18 83 8E 22 39 75 93 86 E6 FD 66 48 B6
-		10 AD 94 14 65 F9 6A 17 78 BD 16 53 84 30 BF 70
-		E0 DC 65 FD 3C C6 B0 1E BF B9 C1 B5 6C EF B1 3A
-		F8 28 05 83 62 26 11 DC B4 6B 5A 97 FF 32 26 B6
-		F7 02 71 CF 15 AE 16 DD D1 C1 8E A8 CF 9B 50 7B
-		C3 91 FF 44 1E CF 7C 39 FE 17 77 21 20 BD CE 9B
-
-		Possible values:
-
-		Algorithm:	TPM_ALG_RSA			(1)
-		Encscheme:	TPM_ES_RSAESPKCSv15		(2)
+		Example output::
+
+		  Algorithm: 00 00 00 01
+		  Encscheme: 00 03
+		  Sigscheme: 00 01
+		  Parameters: 00 00 08 00 00 00 00 02 00 00 00 00
+		  Modulus length: 256
+		  Modulus:
+		  B4 76 41 82 C9 20 2C 10 18 40 BC 8B E5 44 4C 6C
+		  3A B2 92 0C A4 9B 2A 83 EB 5C 12 85 04 48 A0 B6
+		  1E E4 81 84 CE B2 F2 45 1C F0 85 99 61 02 4D EB
+		  86 C4 F7 F3 29 60 52 93 6B B2 E5 AB 8B A9 09 E3
+		  D7 0E 7D CA 41 BF 43 07 65 86 3C 8C 13 7A D0 8B
+		  82 5E 96 0B F8 1F 5F 34 06 DA A2 52 C1 A9 D5 26
+		  0F F4 04 4B D9 3F 2D F2 AC 2F 74 64 1F 8B CD 3E
+		  1E 30 38 6C 70 63 69 AB E2 50 DF 49 05 2E E1 8D
+		  6F 78 44 DA 57 43 69 EE 76 6C 38 8A E9 8E A3 F0
+		  A7 1F 3C A8 D0 12 15 3E CA 0E BD FA 24 CD 33 C6
+		  47 AE A4 18 83 8E 22 39 75 93 86 E6 FD 66 48 B6
+		  10 AD 94 14 65 F9 6A 17 78 BD 16 53 84 30 BF 70
+		  E0 DC 65 FD 3C C6 B0 1E BF B9 C1 B5 6C EF B1 3A
+		  F8 28 05 83 62 26 11 DC B4 6B 5A 97 FF 32 26 B6
+		  F7 02 71 CF 15 AE 16 DD D1 C1 8E A8 CF 9B 50 7B
+		  C3 91 FF 44 1E CF 7C 39 FE 17 77 21 20 BD CE 9B
+
+		Possible values::
+
+		  Algorithm:	TPM_ALG_RSA			(1)
+		  Encscheme:	TPM_ES_RSAESPKCSv15		(2)
 				TPM_ES_RSAESOAEP_SHA1_MGF1	(3)
-		Sigscheme:	TPM_SS_NONE			(1)
-		Parameters, a byte string of 3 u32 values:
+		  Sigscheme:	TPM_SS_NONE			(1)
+		  Parameters, a byte string of 3 u32 values:
 			Key Length (bits):	00 00 08 00	(2048)
 			Num primes:		00 00 00 02	(2)
 			Exponent Size:		00 00 00 00	(0 means the
 								 default exp)
-		Modulus Length: 256 (bytes)
-		Modulus:	The 256 byte Endorsement Key modulus
+		  Modulus Length: 256 (bytes)
+		  Modulus:	The 256 byte Endorsement Key modulus
 
 What:		/sys/class/tpm/tpmX/device/temp_deactivated
 Date:		April 2006
@@ -176,9 +176,9 @@ Description:	The "timeouts" property shows the 4 vendor-specific values
 		timeouts is defined by the TPM interface spec that the chip
 		conforms to.
 
-		Example output:
+		Example output::
 
-		750000 750000 750000 750000 [original]
+		  750000 750000 750000 750000 [original]
 
 		The four timeout values are shown in usecs, with a trailing
 		"[original]" or "[adjusted]" depending on whether the values
diff --git a/Documentation/ABI/stable/sysfs-devices b/Documentation/ABI/stable/sysfs-devices
index 4404bd9b96c1..42bf1eab5677 100644
--- a/Documentation/ABI/stable/sysfs-devices
+++ b/Documentation/ABI/stable/sysfs-devices
@@ -1,5 +1,6 @@
-# Note: This documents additional properties of any device beyond what
-# is documented in Documentation/admin-guide/sysfs-rules.rst
+Note:
+  This documents additional properties of any device beyond what
+  is documented in Documentation/admin-guide/sysfs-rules.rst
 
 What:		/sys/devices/*/of_node
 Date:		February 2015
diff --git a/Documentation/ABI/stable/sysfs-driver-ib_srp b/Documentation/ABI/stable/sysfs-driver-ib_srp
index 7049a2b50359..2d706b4900ce 100644
--- a/Documentation/ABI/stable/sysfs-driver-ib_srp
+++ b/Documentation/ABI/stable/sysfs-driver-ib_srp
@@ -6,6 +6,7 @@ Description:	Interface for making ib_srp connect to a new target.
 		One can request ib_srp to connect to a new target by writing
 		a comma-separated list of login parameters to this sysfs
 		attribute. The supported parameters are:
+
 		* id_ext, a 16-digit hexadecimal number specifying the eight
 		  byte identifier extension in the 16-byte SRP target port
 		  identifier. The target port identifier is sent by ib_srp
diff --git a/Documentation/ABI/stable/sysfs-driver-mlxreg-io b/Documentation/ABI/stable/sysfs-driver-mlxreg-io
index 156319fc5b80..3544968f43cc 100644
--- a/Documentation/ABI/stable/sysfs-driver-mlxreg-io
+++ b/Documentation/ABI/stable/sysfs-driver-mlxreg-io
@@ -1,5 +1,4 @@
-What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/
-							asic_health
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/asic_health
 
 Date:		June 2018
 KernelVersion:	4.19
@@ -9,9 +8,8 @@ Description:	This file shows ASIC health status. The possible values are:
 
 		The files are read only.
 
-What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/
-							cpld1_version
-							cpld2_version
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/cpld1_version
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/cpld2_version
 Date:		June 2018
 KernelVersion:	4.19
 Contact:	Vadim Pasternak <vadimpmellanox.com>
@@ -20,8 +18,7 @@ Description:	These files show with which CPLD versions have been burned
 
 		The files are read only.
 
-What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/
-							fan_dir
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/fan_dir
 
 Date:		December 2018
 KernelVersion:	5.0
@@ -32,8 +29,7 @@ Description:	This file shows the system fans direction:
 
 		The files are read only.
 
-What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/
-							jtag_enable
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/jtag_enable
 
 Date:		November 2018
 KernelVersion:	5.0
@@ -43,8 +39,7 @@ Description:	These files show with which CPLD versions have been burned
 
 		The files are read only.
 
-What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/
-							jtag_enable
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/jtag_enable
 
 Date:		November 2018
 KernelVersion:	5.0
@@ -87,16 +82,15 @@ Description:	These files allow asserting system power cycling, switching
 
 		The files are write only.
 
-What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/
-							reset_aux_pwr_or_ref
-							reset_asic_thermal
-							reset_hotswap_or_halt
-							reset_hotswap_or_wd
-							reset_fw_reset
-							reset_long_pb
-							reset_main_pwr_fail
-							reset_short_pb
-							reset_sw_reset
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/reset_aux_pwr_or_ref
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/reset_asic_thermal
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/reset_hotswap_or_halt
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/reset_hotswap_or_wd
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/reset_fw_reset
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/reset_long_pb
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/reset_main_pwr_fail
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/reset_short_pb
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/reset_sw_reset
 Date:		June 2018
 KernelVersion:	4.19
 Contact:	Vadim Pasternak <vadimpmellanox.com>
@@ -110,11 +104,10 @@ Description:	These files show the system reset cause, as following: power
 
 		The files are read only.
 
-What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/
-						reset_comex_pwr_fail
-						reset_from_comex
-						reset_system
-						reset_voltmon_upgrade_fail
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/reset_comex_pwr_fail
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/reset_from_comex
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/reset_system
+What:		/sys/devices/platform/mlxplat/mlxreg-io/hwmon/hwmon*/reset_voltmon_upgrade_fail
 
 Date:		November 2018
 KernelVersion:	5.0
diff --git a/Documentation/ABI/stable/sysfs-firmware-efi-vars b/Documentation/ABI/stable/sysfs-firmware-efi-vars
index 5def20b9019e..46ccd233e359 100644
--- a/Documentation/ABI/stable/sysfs-firmware-efi-vars
+++ b/Documentation/ABI/stable/sysfs-firmware-efi-vars
@@ -17,6 +17,7 @@ Description:
 		directory has a name of the form "<key>-<vendor guid>"
 		and contains the following files:
 
+		=============== ========================================
 		attributes:	A read-only text file enumerating the
 				EFI variable flags.  Potential values
 				include:
@@ -59,12 +60,14 @@ Description:
 
 		size:		As ASCII representation of the size of
 				the variable's value.
+		=============== ========================================
 
 
 		In addition, two other magic binary files are provided
 		in the top-level directory and are used for adding and
 		removing variables:
 
+		=============== ========================================
 		new_var:	Takes a "struct efi_variable" and
 				instructs the EFI firmware to create a
 				new variable.
@@ -73,3 +76,4 @@ Description:
 				instructs the EFI firmware to remove any
 				variable that has a matching vendor GUID
 				and variable key name.
+		=============== ========================================
diff --git a/Documentation/ABI/stable/sysfs-firmware-opal-dump b/Documentation/ABI/stable/sysfs-firmware-opal-dump
index 32fe7f5c4880..1f74f45327ba 100644
--- a/Documentation/ABI/stable/sysfs-firmware-opal-dump
+++ b/Documentation/ABI/stable/sysfs-firmware-opal-dump
@@ -7,6 +7,7 @@ Description:
 
 		This is only for the powerpc/powernv platform.
 
+		=============== ===============================================
 		initiate_dump:	When '1' is written to it,
 				we will initiate a dump.
 				Read this file for supported commands.
@@ -19,8 +20,11 @@ Description:
 				and ID of the dump, use the id and type files.
 				Do not rely on any particular size of dump
 				type or dump id.
+		=============== ===============================================
 
 		Each dump has the following files:
+
+		=============== ===============================================
 		id:		An ASCII representation of the dump ID
 				in hex (e.g. '0x01')
 		type:		An ASCII representation of the type of
@@ -39,3 +43,4 @@ Description:
 				inaccessible.
 				Reading this file will get a list of
 				supported actions.
+		=============== ===============================================
diff --git a/Documentation/ABI/stable/sysfs-firmware-opal-elog b/Documentation/ABI/stable/sysfs-firmware-opal-elog
index 2536434d49d0..7c8a61a2d005 100644
--- a/Documentation/ABI/stable/sysfs-firmware-opal-elog
+++ b/Documentation/ABI/stable/sysfs-firmware-opal-elog
@@ -38,6 +38,7 @@ Description:
 		For each log entry (directory), there are the following
 		files:
 
+		==============  ================================================
 		id:		An ASCII representation of the ID of the
 				error log, in hex - e.g. "0x01".
 
@@ -58,3 +59,4 @@ Description:
 				entry will be removed from sysfs.
 				Reading this file will list the supported
 				operations (currently just acknowledge).
+		==============  ================================================
diff --git a/Documentation/ABI/stable/sysfs-hypervisor-xen b/Documentation/ABI/stable/sysfs-hypervisor-xen
index 3cf5cdfcd9a8..01bd46be261e 100644
--- a/Documentation/ABI/stable/sysfs-hypervisor-xen
+++ b/Documentation/ABI/stable/sysfs-hypervisor-xen
@@ -33,6 +33,7 @@ Description:	If running under Xen:
 		Space separated list of supported guest system types. Each type
 		is in the format: <class>-<major>.<minor>-<arch>
 		With:
+			======== ============================================
 			<class>: "xen" -- x86: paravirtualized, arm: standard
 				 "hvm" -- x86 only: fully virtualized
 			<major>: major guest interface version
@@ -43,6 +44,7 @@ Description:	If running under Xen:
 				 "x86_64": 64 bit x86 guest
 				 "armv7l": 32 bit arm guest
 				 "aarch64": 64 bit arm guest
+			======== ============================================
 
 What:		/sys/hypervisor/properties/changeset
 Date:		March 2009
diff --git a/Documentation/ABI/stable/vdso b/Documentation/ABI/stable/vdso
index 55406ec8a35a..73ed1240a5c0 100644
--- a/Documentation/ABI/stable/vdso
+++ b/Documentation/ABI/stable/vdso
@@ -23,6 +23,7 @@ Unless otherwise noted, the set of symbols with any given version and the
 ABI of those symbols is considered stable.  It may vary across architectures,
 though.
 
-(As of this writing, this ABI documentation as been confirmed for x86_64.
+Note:
+ As of this writing, this ABI documentation as been confirmed for x86_64.
  The maintainers of the other vDSO-using architectures should confirm
- that it is correct for their architecture.)
+ that it is correct for their architecture.





Thanks,
Mauro

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-19 16:37               ` Mauro Carvalho Chehab
@ 2019-06-21 14:27                 ` Mauro Carvalho Chehab
  2019-06-27  9:48                   ` Jani Nikula
  0 siblings, 1 reply; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-21 14:27 UTC (permalink / raw)
  To: Jani Nikula
  Cc: Greg Kroah-Hartman, Linux Doc Mailing List, linux-kernel,
	Jonathan Corbet

Em Wed, 19 Jun 2019 13:37:39 -0300
Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:

> Em Tue, 18 Jun 2019 11:47:32 +0300
> Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> 
> > On Mon, 17 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:  
> > > Yeah, I guess it should be possible to do that. How a python script
> > > can identify if it was called by Sphinx, or if it was called directly?    
> > 
> > if __name__ == '__main__':
> > 	# run on the command-line, not imported  
> 
> Ok, when I have some spare time, I may try to convert one script
> to python and see how it behaves. 

Did a quick test here... 

Probably I'm doing something wrong (as I'm a rookie with Python), but,
in order to be able to use the same script as command line and as an Sphinx
extension, everything that it is currently there should be "escaped"
by an:

	if __name__ != '__main__':

As event the class definition:

    class KernelCmd(Directive):

depends on:

	from docutils.parsers.rst import directives, Directive

With is only required when one needs to parse ReST - e. g. only
when the script runs as a Sphinx extension.

If this is right, as we want a script that can run by command line
to parse and search inside ABI files, at the end of the day, it will
be a lot easier to maintain if the parser script is different from the
Sphinx extension. 

If so, as the Sphinx extension script will need to call a parsing script
anyway, it doesn't matter the language of the script with will be
doing the ABI file parsing.

See the enclosing file. I didn't add anything from the ABI parsing
script yet. It was just changed in order to not generate an error
when trying to run it from command line.


Thanks,
Mauro
#!/usr/bin/env python3
# coding=utf-8
# SPDX-License-Identifier: GPL-2.0
#
u"""
    kernel-abi
    ~~~~~~~~~~

    Implementation of the ``kernel-abi`` reST-directive.

    :copyright:  Copyright (C) 2016  Markus Heiser
    :copyright:  Copyright (C) 2016-2019  Mauro Carvalho Chehab
    :maintained-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
    :license:    GPL Version 2, June 1991 see Linux/COPYING for details.

    The ``kernel-abi`` (:py:class:`KernelCmd`) directive calls the
    scripts/get_abi.pl script to parse the Kernel ABI files.

    Overview of directive's argument and options.

    .. code-block:: rst

        .. kernel-abi:: <ABI directory location>
            :debug:

    The argument ``<ABI directory location>`` is required. It contains the
    location of the ABI files to be parsed.

    ``debug``
      Inserts a code-block with the *raw* reST. Sometimes it is helpful to see
      what reST is generated.

"""

import codecs
import os
import subprocess
import sys

from os import path

if __name__ != '__main__':
    from docutils import nodes, statemachine
    from docutils.statemachine import ViewList
    from docutils.parsers.rst import directives, Directive
    from docutils.utils.error_reporting import ErrorString

    #
    # AutodocReporter is only good up to Sphinx 1.7
    #
    import sphinx

    Use_SSI = sphinx.__version__[:3] >= '1.7'
    if Use_SSI:
        from sphinx.util.docutils import switch_source_input
    else:
        from sphinx.ext.autodoc import AutodocReporter

__version__  = '1.0'

if __name__ != '__main__':
    def setup(app):

        app.add_directive("kernel-abi", KernelCmd)
        return dict(
            version = __version__
            , parallel_read_safe = True
            , parallel_write_safe = True
        )

    class KernelCmd(Directive):

        u"""KernelABI (``kernel-abi``) directive"""

        required_arguments = 1
        optional_arguments = 2
        has_content = False
        final_argument_whitespace = True

        option_spec = {
            "debug"     : directives.flag,
            "rst"       : directives.unchanged
        }

        def warn(self, message, **replace):
            replace["fname"]   = self.state.document.current_source
            replace["line_no"] = replace.get("line_no", self.lineno)
            message = ("%(fname)s:%(line_no)s: [kernel-abi WARN] : " + message) % replace
            self.state.document.settings.env.app.warn(message, prefix="")

        def run(self):

            doc = self.state.document
            if not doc.settings.file_insertion_enabled:
                raise self.warning("docutils: file insertion disabled")

            env = doc.settings.env
            cwd = path.dirname(doc.current_source)
            fname = self.arguments[0]

            cmd = "get_abi.pl rest --dir "
            cmd += fname

            if 'rst' in self.options:
                cmd += " --rst-source"

            srctree = path.abspath(os.environ["srctree"])

            # extend PATH with $(srctree)/scripts
            path_env = os.pathsep.join([
                srctree + os.sep + "scripts",
                os.environ["PATH"]
            ])
            shell_env = os.environ.copy()
            shell_env["PATH"]    = path_env
            shell_env["srctree"] = srctree

            lines = self.runCmd(cmd, shell=True, cwd=cwd, env=shell_env)
            nodeList = self.nestedParse(lines, self.arguments[0])
            return nodeList

        def runCmd(self, cmd, **kwargs):
            u"""Run command ``cmd`` and return it's stdout as unicode."""

            try:
                proc = subprocess.Popen(
                    cmd
                    , stdout = subprocess.PIPE
                    , stderr = subprocess.PIPE
                    , **kwargs
                )
                out, err = proc.communicate()

                out, err = codecs.decode(out, 'utf-8'), codecs.decode(err, 'utf-8')

                if proc.returncode != 0:
                    raise self.severe(
                        u"command '%s' failed with return code %d"
                        % (cmd, proc.returncode)
                    )
            except OSError as exc:
                raise self.severe(u"problems with '%s' directive: %s."
                                % (self.name, ErrorString(exc)))
            return out

        def nestedParse(self, lines, fname):
            content = ViewList()
            node    = nodes.section()

            if "debug" in self.options:
                code_block = "\n\n.. code-block:: rst\n    :linenos:\n"
                for l in lines.split("\n"):
                    code_block += "\n    " + l
                lines = code_block + "\n\n"

            for c, l in enumerate(lines.split("\n")):
                content.append(l, fname, c)

            buf  = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter

            if Use_SSI:
                with switch_source_input(self.state, content):
                    self.state.nested_parse(content, 0, node, match_titles=1)
            else:
                self.state.memo.title_styles  = []
                self.state.memo.section_level = 0
                self.state.memo.reporter      = AutodocReporter(content, self.state.memo.reporter)
                try:
                    self.state.nested_parse(content, 0, node, match_titles=1)
                finally:
                    self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf

            return node.children

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-21 14:27                 ` Mauro Carvalho Chehab
@ 2019-06-27  9:48                   ` Jani Nikula
  2019-06-27 10:52                     ` Mauro Carvalho Chehab
  0 siblings, 1 reply; 63+ messages in thread
From: Jani Nikula @ 2019-06-27  9:48 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Greg Kroah-Hartman, Linux Doc Mailing List, linux-kernel,
	Jonathan Corbet

On Fri, 21 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> Em Wed, 19 Jun 2019 13:37:39 -0300
> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
>
>> Em Tue, 18 Jun 2019 11:47:32 +0300
>> Jani Nikula <jani.nikula@linux.intel.com> escreveu:
>> 
>> > On Mon, 17 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:  
>> > > Yeah, I guess it should be possible to do that. How a python script
>> > > can identify if it was called by Sphinx, or if it was called directly?    
>> > 
>> > if __name__ == '__main__':
>> > 	# run on the command-line, not imported  
>> 
>> Ok, when I have some spare time, I may try to convert one script
>> to python and see how it behaves. 
>
> Did a quick test here... 
>
> Probably I'm doing something wrong (as I'm a rookie with Python), but,
> in order to be able to use the same script as command line and as an Sphinx
> extension, everything that it is currently there should be "escaped"
> by an:
>
> 	if __name__ != '__main__':
>
> As event the class definition:
>
>     class KernelCmd(Directive):
>
> depends on:
>
> 	from docutils.parsers.rst import directives, Directive
>
> With is only required when one needs to parse ReST - e. g. only
> when the script runs as a Sphinx extension.
>
> If this is right, as we want a script that can run by command line
> to parse and search inside ABI files, at the end of the day, it will
> be a lot easier to maintain if the parser script is different from the
> Sphinx extension. 

Split it into two files, one has the nuts and bolts of parsing and has
the "if __name__ == '__main__':" bit to run on the command line, and the
other interfaces with Sphinx and imports the parser.

> If so, as the Sphinx extension script will need to call a parsing script
> anyway, it doesn't matter the language of the script with will be
> doing the ABI file parsing.

Calling the parser using an API will be easier to use, maintain and
extend than using pipes, with all the interleaved sideband information
to adjust line numbers and whatnot.

BR,
Jani.



>
> See the enclosing file. I didn't add anything from the ABI parsing
> script yet. It was just changed in order to not generate an error
> when trying to run it from command line.
>
>
> Thanks,
> Mauro
> #!/usr/bin/env python3
> # coding=utf-8
> # SPDX-License-Identifier: GPL-2.0
> #
> u"""
>     kernel-abi
>     ~~~~~~~~~~
>
>     Implementation of the ``kernel-abi`` reST-directive.
>
>     :copyright:  Copyright (C) 2016  Markus Heiser
>     :copyright:  Copyright (C) 2016-2019  Mauro Carvalho Chehab
>     :maintained-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
>     :license:    GPL Version 2, June 1991 see Linux/COPYING for details.
>
>     The ``kernel-abi`` (:py:class:`KernelCmd`) directive calls the
>     scripts/get_abi.pl script to parse the Kernel ABI files.
>
>     Overview of directive's argument and options.
>
>     .. code-block:: rst
>
>         .. kernel-abi:: <ABI directory location>
>             :debug:
>
>     The argument ``<ABI directory location>`` is required. It contains the
>     location of the ABI files to be parsed.
>
>     ``debug``
>       Inserts a code-block with the *raw* reST. Sometimes it is helpful to see
>       what reST is generated.
>
> """
>
> import codecs
> import os
> import subprocess
> import sys
>
> from os import path
>
> if __name__ != '__main__':
>     from docutils import nodes, statemachine
>     from docutils.statemachine import ViewList
>     from docutils.parsers.rst import directives, Directive
>     from docutils.utils.error_reporting import ErrorString
>
>     #
>     # AutodocReporter is only good up to Sphinx 1.7
>     #
>     import sphinx
>
>     Use_SSI = sphinx.__version__[:3] >= '1.7'
>     if Use_SSI:
>         from sphinx.util.docutils import switch_source_input
>     else:
>         from sphinx.ext.autodoc import AutodocReporter
>
> __version__  = '1.0'
>
> if __name__ != '__main__':
>     def setup(app):
>
>         app.add_directive("kernel-abi", KernelCmd)
>         return dict(
>             version = __version__
>             , parallel_read_safe = True
>             , parallel_write_safe = True
>         )
>
>     class KernelCmd(Directive):
>
>         u"""KernelABI (``kernel-abi``) directive"""
>
>         required_arguments = 1
>         optional_arguments = 2
>         has_content = False
>         final_argument_whitespace = True
>
>         option_spec = {
>             "debug"     : directives.flag,
>             "rst"       : directives.unchanged
>         }
>
>         def warn(self, message, **replace):
>             replace["fname"]   = self.state.document.current_source
>             replace["line_no"] = replace.get("line_no", self.lineno)
>             message = ("%(fname)s:%(line_no)s: [kernel-abi WARN] : " + message) % replace
>             self.state.document.settings.env.app.warn(message, prefix="")
>
>         def run(self):
>
>             doc = self.state.document
>             if not doc.settings.file_insertion_enabled:
>                 raise self.warning("docutils: file insertion disabled")
>
>             env = doc.settings.env
>             cwd = path.dirname(doc.current_source)
>             fname = self.arguments[0]
>
>             cmd = "get_abi.pl rest --dir "
>             cmd += fname
>
>             if 'rst' in self.options:
>                 cmd += " --rst-source"
>
>             srctree = path.abspath(os.environ["srctree"])
>
>             # extend PATH with $(srctree)/scripts
>             path_env = os.pathsep.join([
>                 srctree + os.sep + "scripts",
>                 os.environ["PATH"]
>             ])
>             shell_env = os.environ.copy()
>             shell_env["PATH"]    = path_env
>             shell_env["srctree"] = srctree
>
>             lines = self.runCmd(cmd, shell=True, cwd=cwd, env=shell_env)
>             nodeList = self.nestedParse(lines, self.arguments[0])
>             return nodeList
>
>         def runCmd(self, cmd, **kwargs):
>             u"""Run command ``cmd`` and return it's stdout as unicode."""
>
>             try:
>                 proc = subprocess.Popen(
>                     cmd
>                     , stdout = subprocess.PIPE
>                     , stderr = subprocess.PIPE
>                     , **kwargs
>                 )
>                 out, err = proc.communicate()
>
>                 out, err = codecs.decode(out, 'utf-8'), codecs.decode(err, 'utf-8')
>
>                 if proc.returncode != 0:
>                     raise self.severe(
>                         u"command '%s' failed with return code %d"
>                         % (cmd, proc.returncode)
>                     )
>             except OSError as exc:
>                 raise self.severe(u"problems with '%s' directive: %s."
>                                 % (self.name, ErrorString(exc)))
>             return out
>
>         def nestedParse(self, lines, fname):
>             content = ViewList()
>             node    = nodes.section()
>
>             if "debug" in self.options:
>                 code_block = "\n\n.. code-block:: rst\n    :linenos:\n"
>                 for l in lines.split("\n"):
>                     code_block += "\n    " + l
>                 lines = code_block + "\n\n"
>
>             for c, l in enumerate(lines.split("\n")):
>                 content.append(l, fname, c)
>
>             buf  = self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter
>
>             if Use_SSI:
>                 with switch_source_input(self.state, content):
>                     self.state.nested_parse(content, 0, node, match_titles=1)
>             else:
>                 self.state.memo.title_styles  = []
>                 self.state.memo.section_level = 0
>                 self.state.memo.reporter      = AutodocReporter(content, self.state.memo.reporter)
>                 try:
>                     self.state.nested_parse(content, 0, node, match_titles=1)
>                 finally:
>                     self.state.memo.title_styles, self.state.memo.section_level, self.state.memo.reporter = buf
>
>             return node.children

-- 
Jani Nikula, Intel Open Source Graphics Center

^ permalink raw reply	[flat|nested] 63+ messages in thread

* Re: [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book
  2019-06-27  9:48                   ` Jani Nikula
@ 2019-06-27 10:52                     ` Mauro Carvalho Chehab
  0 siblings, 0 replies; 63+ messages in thread
From: Mauro Carvalho Chehab @ 2019-06-27 10:52 UTC (permalink / raw)
  To: Jani Nikula
  Cc: Greg Kroah-Hartman, Linux Doc Mailing List, linux-kernel,
	Jonathan Corbet

Em Thu, 27 Jun 2019 12:48:12 +0300
Jani Nikula <jani.nikula@linux.intel.com> escreveu:

> On Fri, 21 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:
> > Em Wed, 19 Jun 2019 13:37:39 -0300
> > Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> >  
> >> Em Tue, 18 Jun 2019 11:47:32 +0300
> >> Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> >>   
> >> > On Mon, 17 Jun 2019, Mauro Carvalho Chehab <mchehab+samsung@kernel.org> wrote:    
> >> > > Yeah, I guess it should be possible to do that. How a python script
> >> > > can identify if it was called by Sphinx, or if it was called directly?      
> >> > 
> >> > if __name__ == '__main__':
> >> > 	# run on the command-line, not imported    
> >> 
> >> Ok, when I have some spare time, I may try to convert one script
> >> to python and see how it behaves.   
> >
> > Did a quick test here... 
> >
> > Probably I'm doing something wrong (as I'm a rookie with Python), but,
> > in order to be able to use the same script as command line and as an Sphinx
> > extension, everything that it is currently there should be "escaped"
> > by an:
> >
> > 	if __name__ != '__main__':
> >
> > As event the class definition:
> >
> >     class KernelCmd(Directive):
> >
> > depends on:
> >
> > 	from docutils.parsers.rst import directives, Directive
> >
> > With is only required when one needs to parse ReST - e. g. only
> > when the script runs as a Sphinx extension.
> >
> > If this is right, as we want a script that can run by command line
> > to parse and search inside ABI files, at the end of the day, it will
> > be a lot easier to maintain if the parser script is different from the
> > Sphinx extension.   
> 
> Split it into two files, one has the nuts and bolts of parsing and has
> the "if __name__ == '__main__':" bit to run on the command line, and the
> other interfaces with Sphinx and imports the parser.

It seems we have an agreement here: the best is indeed to have two
files, one with the Documentation/ABI parser, and another one with the 
Sphinx extension...

> 
> > If so, as the Sphinx extension script will need to call a parsing script
> > anyway, it doesn't matter the language of the script with will be
> > doing the ABI file parsing.  
> 
> Calling the parser using an API will be easier to use, maintain and
> extend than using pipes, with all the interleaved sideband information
> to adjust line numbers and whatnot.

... and here is where we have two different views.

From debug PoV, the Documentation/ABI parser script should be able to
print the results by a command line call. This is also a feature
that it is useful for the users: to be able to seek for a symbol
and output its ABI description. So, the "stdout" output will be
there anyway.

The only extra data for the extension side is the file name where
the information came and the line number.

From maintainership PoV, adding the sideband API for file+line is
one line at the parser script (a print) and two lines at the Sphinx
extension (a regex expression and a match line). That's simple to
maintain.

It is also simple to verify both sides independently, as what
you see when running the parser script is what you get at the
extension.

If we add a new ABI between the parser script and the extension
script, this would require to also maintain the ABI, and would
make harder to identify problems on ABI problems.

-

Another advantage of having those independent is that the
language of the parsing script can be different. Not being
python is a big advantage for me, as perl is a lot more
intuitive and easier to write parser scripts for my eyes.

I can write a perl parsing script in a matter of minutes.
It takes me a lot more time to do the same with python, and then
ensure that it will work with two similar but different languages
(python2 and python3) [1].

[1] btw, with that regards, I still don't know how to teach a
    python script that it should "prefer" to run with python3 but would
    fall back to python2. Adding this shebang:
	# /usr/bin/env python
    just do the opposite - at least on Fedora


Thanks,
Mauro

^ permalink raw reply	[flat|nested] 63+ messages in thread

end of thread, other threads:[~2019-06-27 10:52 UTC | newest]

Thread overview: 63+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-06-14  2:04 [PATCH 00/14] Add support to generate ABI documentation at admin-guide Mauro Carvalho Chehab
2019-06-14  2:04 ` [PATCH 01/14] ABI: fix some syntax issues at the ABI database Mauro Carvalho Chehab
2019-06-14  7:20   ` Andrew Donnellan
2019-06-14 16:16     ` Greg Kroah-Hartman
2019-06-14  8:30   ` Rafael J. Wysocki
2019-06-14  2:04 ` [PATCH 02/14] ABI: sysfs-driver-hid: the "What" field doesn't parse fine Mauro Carvalho Chehab
2019-06-14  2:04 ` [PATCH 03/14] ABI: sysfs-class-uwb_rc: remove a duplicated incomplete entry Mauro Carvalho Chehab
2019-06-14  2:04 ` [PATCH 04/14] ABI: better identificate tables Mauro Carvalho Chehab
2019-06-19 12:51   ` Johan Hovold
2019-06-19 13:56     ` Mauro Carvalho Chehab
2019-06-19 15:02       ` Greg Kroah-Hartman
2019-06-19 16:14         ` Mauro Carvalho Chehab
2019-06-20 14:27           ` Mauro Carvalho Chehab
2019-06-21  7:21             ` Greg Kroah-Hartman
2019-06-21  9:49               ` Mauro Carvalho Chehab
2019-06-20 12:01         ` Johan Hovold
2019-06-20 12:54           ` Greg Kroah-Hartman
2019-06-20 14:20             ` Mauro Carvalho Chehab
2019-06-20 16:29               ` Greg Kroah-Hartman
2019-06-20 17:16                 ` Mauro Carvalho Chehab
2019-06-20 17:55                   ` Greg Kroah-Hartman
2019-06-14  2:04 ` [PATCH 05/14] scripts: add an script to parse the ABI files Mauro Carvalho Chehab
2019-06-14 13:31   ` Jani Nikula
2019-06-14 13:39     ` Greg Kroah-Hartman
2019-06-14 13:58       ` Mauro Carvalho Chehab
2019-06-14 14:00       ` Jani Nikula
2019-06-14 14:14         ` Jonathan Corbet
2019-06-14 16:24         ` Greg Kroah-Hartman
2019-06-14  2:04 ` [PATCH 06/14] scripts/get_abi.pl: parse files with text at beginning Mauro Carvalho Chehab
2019-06-14  2:04 ` [PATCH 07/14] scripts/get_abi.pl: avoid use literal blocks when not needed Mauro Carvalho Chehab
2019-06-14  2:04 ` [PATCH 08/14] scripts/get_abi.pl: split label naming from xref logic Mauro Carvalho Chehab
2019-06-14  2:04 ` [PATCH 09/14] scripts/get_abi.pl: add support for searching for ABI symbols Mauro Carvalho Chehab
2019-06-14  2:04 ` [PATCH 10/14] scripts/get_abi.pl: represent what in tables Mauro Carvalho Chehab
2019-06-14  2:04 ` [PATCH 11/14] scripts/get_abi.pl: fix parse issues with some files Mauro Carvalho Chehab
2019-06-14  2:04 ` [PATCH 12/14] doc-rst: add ABI documentation to the admin-guide book Mauro Carvalho Chehab
2019-06-14 13:42   ` Jani Nikula
2019-06-14 14:06     ` Greg Kroah-Hartman
2019-06-14 15:27       ` Mauro Carvalho Chehab
2019-06-17 12:36         ` Jani Nikula
2019-06-17 12:54           ` Greg Kroah-Hartman
2019-06-17 13:48             ` Jonathan Corbet
2019-06-17 13:50             ` Jani Nikula
2019-06-17 13:51           ` Mauro Carvalho Chehab
2019-06-18  8:47             ` Jani Nikula
2019-06-19 16:37               ` Mauro Carvalho Chehab
2019-06-21 14:27                 ` Mauro Carvalho Chehab
2019-06-27  9:48                   ` Jani Nikula
2019-06-27 10:52                     ` Mauro Carvalho Chehab
2019-06-14 14:10     ` Markus Heiser
2019-06-14 14:15       ` Jonathan Corbet
2019-06-16 16:04         ` Markus Heiser
2019-06-17  9:11           ` Mauro Carvalho Chehab
2019-06-17 16:31             ` Markus Heiser
2019-06-14  2:04 ` [PATCH 13/14] sphinx/kernel_abi.py: make it compatible with Sphinx 1.7+ Mauro Carvalho Chehab
2019-06-14  2:04 ` [PATCH 14/14] docs: sphinx/kernel_abi.py: fix UTF-8 support Mauro Carvalho Chehab
2019-06-14 16:18   ` Greg Kroah-Hartman
2019-06-14 16:25     ` Mauro Carvalho Chehab
2019-06-15  6:16       ` Greg Kroah-Hartman
2019-06-16 15:43       ` Markus Heiser
2019-06-17  9:16         ` Mauro Carvalho Chehab
2019-06-17 13:56           ` Jonathan Corbet
2019-06-17 15:55             ` Mauro Carvalho Chehab
2019-06-14 16:20 ` [PATCH 00/14] Add support to generate ABI documentation at admin-guide Greg Kroah-Hartman

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).