LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* [PATCH] input: Add a detailed multi-touch finger data report protocol
@ 2008-11-07 11:15 Henrik Rydberg
  2008-11-07 13:34 ` J.R. Mauro
                   ` (3 more replies)
  0 siblings, 4 replies; 22+ messages in thread
From: Henrik Rydberg @ 2008-11-07 11:15 UTC (permalink / raw)
  To: Dmitry Torokhov, Andrew Morton, linux-input; +Cc: linux-kernel

In order to utilize the full power of the new multi-touch devices, a
way to report detailed finger data to user space is needed. This patch
adds a multi-touch (MT) protocol which allows drivers to report details
for an arbitrary number of fingers.

BTN_MT_REPORT_PACKET

The driver presses this button to indicate the start of a packet of
finger data. The button is released after the whole packet has been
reported.

BTN_MT_REPORT_FINGER

The driver presses this button to indicate the start of a new
finger. The button is released when the finger has been reported.

In order to stay compatible with existing applications, the data
reported in a finger packet must not be recognized as single-touch
events. Therefore, a specific set of ABS_MT events is used instead.

Signed-off-by: Henrik Rydberg <rydberg@euromail.se>
---
 include/linux/input.h |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/include/linux/input.h b/include/linux/input.h
index b86fb55..c82dd42 100644
--- a/include/linux/input.h
+++ b/include/linux/input.h
@@ -590,6 +590,9 @@ struct input_absinfo {
 #define KEY_NUMERIC_STAR	0x20a
 #define KEY_NUMERIC_POUND	0x20b

+#define BTN_MT_REPORT_PACKET	0x210	/* report multitouch packet data */
+#define BTN_MT_REPORT_FINGER	0x211	/* report multitouch finger data */
+
 /* We avoid low common keys in module aliases so they don't get huge. */
 #define KEY_MIN_INTERESTING	KEY_MUTE
 #define KEY_MAX			0x2ff
@@ -642,6 +645,15 @@ struct input_absinfo {
 #define ABS_TOOL_WIDTH		0x1c
 #define ABS_VOLUME		0x20
 #define ABS_MISC		0x28
+#define ABS_MT_TOUCH		0x30	/* Diameter of touching circle */
+#define ABS_MT_TOUCH_MAJOR	0x30	/* Major axis of touching ellipse */
+#define ABS_MT_TOUCH_MINOR	0x31	/* Minor axis of touching ellipse */
+#define ABS_MT_WIDTH		0x32	/* Diameter of approaching circle */
+#define ABS_MT_WIDTH_MAJOR	0x32	/* Major axis of approaching ellipse */
+#define ABS_MT_WIDTH_MINOR	0x33	/* Minor axis of approaching ellipse */
+#define ABS_MT_ORIENTATION	0x34	/* Ellipse orientation */
+#define ABS_MT_POSITION_X	0x35	/* Center X ellipse position */
+#define ABS_MT_POSITION_Y	0x36	/* Center Y ellipse position */
 #define ABS_MAX			0x3f
 #define ABS_CNT			(ABS_MAX+1)

-- 
1.5.6.3


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-07 11:15 [PATCH] input: Add a detailed multi-touch finger data report protocol Henrik Rydberg
@ 2008-11-07 13:34 ` J.R. Mauro
  2008-11-07 14:26   ` Henrik Rydberg
  2008-11-08  0:51 ` Arjan Opmeer
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 22+ messages in thread
From: J.R. Mauro @ 2008-11-07 13:34 UTC (permalink / raw)
  To: Henrik Rydberg; +Cc: Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel

On Fri, Nov 7, 2008 at 6:15 AM, Henrik Rydberg <rydberg@euromail.se> wrote:
> In order to utilize the full power of the new multi-touch devices, a
> way to report detailed finger data to user space is needed. This patch
> adds a multi-touch (MT) protocol which allows drivers to report details
> for an arbitrary number of fingers.

This is a great idea, Henrik. How will things like X11 go about
leveraging this functionality?

>
> BTN_MT_REPORT_PACKET
>
> The driver presses this button to indicate the start of a packet of
> finger data. The button is released after the whole packet has been
> reported.
>
> BTN_MT_REPORT_FINGER
>
> The driver presses this button to indicate the start of a new
> finger. The button is released when the finger has been reported.
>
> In order to stay compatible with existing applications, the data
> reported in a finger packet must not be recognized as single-touch
> events. Therefore, a specific set of ABS_MT events is used instead.
>
> Signed-off-by: Henrik Rydberg <rydberg@euromail.se>
> ---
>  include/linux/input.h |   12 ++++++++++++
>  1 files changed, 12 insertions(+), 0 deletions(-)
>
> diff --git a/include/linux/input.h b/include/linux/input.h
> index b86fb55..c82dd42 100644
> --- a/include/linux/input.h
> +++ b/include/linux/input.h
> @@ -590,6 +590,9 @@ struct input_absinfo {
>  #define KEY_NUMERIC_STAR       0x20a
>  #define KEY_NUMERIC_POUND      0x20b
>
> +#define BTN_MT_REPORT_PACKET   0x210   /* report multitouch packet data */
> +#define BTN_MT_REPORT_FINGER   0x211   /* report multitouch finger data */
> +
>  /* We avoid low common keys in module aliases so they don't get huge. */
>  #define KEY_MIN_INTERESTING    KEY_MUTE
>  #define KEY_MAX                        0x2ff
> @@ -642,6 +645,15 @@ struct input_absinfo {
>  #define ABS_TOOL_WIDTH         0x1c
>  #define ABS_VOLUME             0x20
>  #define ABS_MISC               0x28
> +#define ABS_MT_TOUCH           0x30    /* Diameter of touching circle */
> +#define ABS_MT_TOUCH_MAJOR     0x30    /* Major axis of touching ellipse */
> +#define ABS_MT_TOUCH_MINOR     0x31    /* Minor axis of touching ellipse */
> +#define ABS_MT_WIDTH           0x32    /* Diameter of approaching circle */
> +#define ABS_MT_WIDTH_MAJOR     0x32    /* Major axis of approaching ellipse */
> +#define ABS_MT_WIDTH_MINOR     0x33    /* Minor axis of approaching ellipse */
> +#define ABS_MT_ORIENTATION     0x34    /* Ellipse orientation */
> +#define ABS_MT_POSITION_X      0x35    /* Center X ellipse position */
> +#define ABS_MT_POSITION_Y      0x36    /* Center Y ellipse position */
>  #define ABS_MAX                        0x3f
>  #define ABS_CNT                        (ABS_MAX+1)
>
> --
> 1.5.6.3
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-07 13:34 ` J.R. Mauro
@ 2008-11-07 14:26   ` Henrik Rydberg
  2008-11-07 14:36     ` J.R. Mauro
  0 siblings, 1 reply; 22+ messages in thread
From: Henrik Rydberg @ 2008-11-07 14:26 UTC (permalink / raw)
  To: J.R. Mauro; +Cc: Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel

J.R. Mauro wrote:
> On Fri, Nov 7, 2008 at 6:15 AM, Henrik Rydberg <rydberg@euromail.se> wrote:
>> In order to utilize the full power of the new multi-touch devices, a
>> way to report detailed finger data to user space is needed. This patch
>> adds a multi-touch (MT) protocol which allows drivers to report details
>> for an arbitrary number of fingers.
> 
> This is a great idea, Henrik. How will things like X11 go about
> leveraging this functionality?

What I have in mind is this:

1. Add support for the MT interface in the trackpad drivers, to the extent
they support the information. First out is bcm5974, for which the interface
is designed. A patch is prepared, but I will keep it on hold for a little
while longer.

2. Add support for the MT protocol in Xorg synaptics and/or a new Xorg
multitouch driver. As you might have guessed, I have an experimental
driver, utilizing the MT protocol. It uses proper matching techniques
to obtain smooth multi-finger actions. My plan is to launch this in user
space as soon as there is kernel support.

Henrik


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-07 14:26   ` Henrik Rydberg
@ 2008-11-07 14:36     ` J.R. Mauro
  2008-11-07 14:52       ` Henrik Rydberg
  0 siblings, 1 reply; 22+ messages in thread
From: J.R. Mauro @ 2008-11-07 14:36 UTC (permalink / raw)
  To: Henrik Rydberg; +Cc: Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel

On Fri, Nov 7, 2008 at 9:26 AM, Henrik Rydberg <rydberg@euromail.se> wrote:
> J.R. Mauro wrote:
>> On Fri, Nov 7, 2008 at 6:15 AM, Henrik Rydberg <rydberg@euromail.se> wrote:
>>> In order to utilize the full power of the new multi-touch devices, a
>>> way to report detailed finger data to user space is needed. This patch
>>> adds a multi-touch (MT) protocol which allows drivers to report details
>>> for an arbitrary number of fingers.
>>
>> This is a great idea, Henrik. How will things like X11 go about
>> leveraging this functionality?
>
> What I have in mind is this:
>
> 1. Add support for the MT interface in the trackpad drivers, to the extent
> they support the information. First out is bcm5974, for which the interface
> is designed. A patch is prepared, but I will keep it on hold for a little
> while longer.
>
> 2. Add support for the MT protocol in Xorg synaptics and/or a new Xorg
> multitouch driver. As you might have guessed, I have an experimental
> driver, utilizing the MT protocol. It uses proper matching techniques
> to obtain smooth multi-finger actions. My plan is to launch this in user
> space as soon as there is kernel support.
>
> Henrik
>

Have you worked with MPX? Or was a vanilla-X11 driver simple enough?
How does this all get presented to userspace applications? Multiple
mouse inputs or special stuff that applications would need a rewrite
to leverage?

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-07 14:36     ` J.R. Mauro
@ 2008-11-07 14:52       ` Henrik Rydberg
  2008-11-07 14:57         ` J.R. Mauro
  2008-11-11  4:08         ` Peter Hutterer
  0 siblings, 2 replies; 22+ messages in thread
From: Henrik Rydberg @ 2008-11-07 14:52 UTC (permalink / raw)
  To: J.R. Mauro; +Cc: Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel

> Have you worked with MPX? Or was a vanilla-X11 driver simple enough?
> How does this all get presented to userspace applications? Multiple
> mouse inputs or special stuff that applications would need a rewrite
> to leverage?

MPX is about making use of multiple pointers. The MT protocol is about
making use of multiple fingers. It can in principle be used as input
to MPX applications; I will bring it up with Peter Hutterer. However,
the main aim is to bring advanced gestures to the linux desktop.

Henrik


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-07 14:52       ` Henrik Rydberg
@ 2008-11-07 14:57         ` J.R. Mauro
  2008-11-07 15:07           ` Henrik Rydberg
  2008-11-11  4:08         ` Peter Hutterer
  1 sibling, 1 reply; 22+ messages in thread
From: J.R. Mauro @ 2008-11-07 14:57 UTC (permalink / raw)
  To: Henrik Rydberg; +Cc: Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel

On Fri, Nov 7, 2008 at 9:52 AM, Henrik Rydberg <rydberg@euromail.se> wrote:
>> Have you worked with MPX? Or was a vanilla-X11 driver simple enough?
>> How does this all get presented to userspace applications? Multiple
>> mouse inputs or special stuff that applications would need a rewrite
>> to leverage?
>
> MPX is about making use of multiple pointers. The MT protocol is about
> making use of multiple fingers. It can in principle be used as input
> to MPX applications; I will bring it up with Peter Hutterer. However,
> the main aim is to bring advanced gestures to the linux desktop.
>
> Henrik

I think this approach is cleaner than "fooling" userspace into seeing
a bunch of mice.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-07 14:57         ` J.R. Mauro
@ 2008-11-07 15:07           ` Henrik Rydberg
  0 siblings, 0 replies; 22+ messages in thread
From: Henrik Rydberg @ 2008-11-07 15:07 UTC (permalink / raw)
  To: J.R. Mauro; +Cc: Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel


> I think this approach is cleaner than "fooling" userspace into seeing
> a bunch of mice.

The many-input-device approach has problems with synchronicity. In
particular when envisioning the appearance of more advanced chips,
with finger matching in hardware. Currently, no chip with open or
sufficient documentation has these features, but it is probably
only a matter of time.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-07 11:15 [PATCH] input: Add a detailed multi-touch finger data report protocol Henrik Rydberg
  2008-11-07 13:34 ` J.R. Mauro
@ 2008-11-08  0:51 ` Arjan Opmeer
  2008-11-08 10:43   ` Henrik Rydberg
  2008-11-11  4:20 ` Peter Hutterer
  2008-11-19 16:37 ` Dmitry Torokhov
  3 siblings, 1 reply; 22+ messages in thread
From: Arjan Opmeer @ 2008-11-08  0:51 UTC (permalink / raw)
  To: Henrik Rydberg; +Cc: Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel


Hi Henrik,

On Fri, Nov 07, 2008 at 12:15:29PM +0100, Henrik Rydberg wrote:
> In order to utilize the full power of the new multi-touch devices, a way
> to report detailed finger data to user space is needed. This patch adds a
> multi-touch (MT) protocol which allows drivers to report details for an
> arbitrary number of fingers.

Good. Something like this is also needed for the Elantech touchpad which can
report the coordinates of two fingers on the pad simultaneously. This allows
for the well known two finger zoom and rotate gestures.

> BTN_MT_REPORT_PACKET
> 
> The driver presses this button to indicate the start of a packet of finger
> data. The button is released after the whole packet has been reported.
> 
> BTN_MT_REPORT_FINGER
> 
> The driver presses this button to indicate the start of a new finger. The
> button is released when the finger has been reported.

Hmm, why not use a new event type to report multi-touch data? I find the use
of fake buttons here kind of ugly.

> +#define ABS_MT_TOUCH		0x30	/* Diameter of touching circle */
> +#define ABS_MT_TOUCH_MAJOR	0x30	/* Major axis of touching ellipse */
> +#define ABS_MT_TOUCH_MINOR	0x31	/* Minor axis of touching ellipse */
> +#define ABS_MT_WIDTH		0x32	/* Diameter of approaching circle */
> +#define ABS_MT_WIDTH_MAJOR	0x32	/* Major axis of approaching ellipse */
> +#define ABS_MT_WIDTH_MINOR	0x33	/* Minor axis of approaching ellipse */
> +#define ABS_MT_ORIENTATION	0x34	/* Ellipse orientation */
> +#define ABS_MT_POSITION_X	0x35	/* Center X ellipse position */
> +#define ABS_MT_POSITION_Y	0x36	/* Center Y ellipse position */

Does your hardware provide all this data, or do you want to do the
calculations for these numbers in a kernel driver? Isn't it much easier to
do these kind of calculations in user space where you can make liberal use
of floating point numbers?

In other words, are you not overdesigning this interface or targetting it
too much to some specific hardware you have in mind? It doesn't seem to fit
the Elantech touchpad too well, for example.


Arjan


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-08  0:51 ` Arjan Opmeer
@ 2008-11-08 10:43   ` Henrik Rydberg
  2008-11-08 11:29     ` Arjan Opmeer
  0 siblings, 1 reply; 22+ messages in thread
From: Henrik Rydberg @ 2008-11-08 10:43 UTC (permalink / raw)
  To: Arjan Opmeer; +Cc: Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel

[..]

>> BTN_MT_REPORT_PACKET
[..]
>> BTN_MT_REPORT_FINGER
> 
> Hmm, why not use a new event type to report multi-touch data? I find the use
> of fake buttons here kind of ugly.

Good point, I will revise.

>> +#define ABS_MT_TOUCH		0x30	/* Diameter of touching circle */
>> +#define ABS_MT_TOUCH_MAJOR	0x30	/* Major axis of touching ellipse */
>> +#define ABS_MT_TOUCH_MINOR	0x31	/* Minor axis of touching ellipse */
>> +#define ABS_MT_WIDTH		0x32	/* Diameter of approaching circle */
>> +#define ABS_MT_WIDTH_MAJOR	0x32	/* Major axis of approaching ellipse */
>> +#define ABS_MT_WIDTH_MINOR	0x33	/* Minor axis of approaching ellipse */
>> +#define ABS_MT_ORIENTATION	0x34	/* Ellipse orientation */
>> +#define ABS_MT_POSITION_X	0x35	/* Center X ellipse position */
>> +#define ABS_MT_POSITION_Y	0x36	/* Center Y ellipse position */
> 
> Does your hardware provide all this data, or do you want to do the
> calculations for these numbers in a kernel driver?

These are all numbers directly from the bcm5974 USB interface. There is
no computation involved.

> Isn't it much easier to
> do these kind of calculations in user space where you can make liberal use
> of floating point numbers?

For chips that cannot produce the above events, one simply reports less data.
If orientation data cannot be obtained, leave out TOUCH_MINOR, WIDTH_MINOR
and ORIENTATION. If finger width cannot be obtained, leave out WIDTH. The
minimum set of data is TOUCH, POSITION_X and POSITION_Y. If touch pressure
information is not given in trackpad dimension units, use a specific range.
I believe all current drivers are able to produce this data.

> In other words, are you not overdesigning this interface or targetting it
> too much to some specific hardware you have in mind?

It is designed to bring advanced gestures to the linux desktop. It can be
implemented on different levels depending on hardware capabilities.


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-08 10:43   ` Henrik Rydberg
@ 2008-11-08 11:29     ` Arjan Opmeer
  2008-11-08 12:02       ` Henrik Rydberg
  2008-11-11  4:13       ` Peter Hutterer
  0 siblings, 2 replies; 22+ messages in thread
From: Arjan Opmeer @ 2008-11-08 11:29 UTC (permalink / raw)
  To: Henrik Rydberg; +Cc: Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel


Hi Henrik,

On Sat, Nov 08, 2008 at 11:43:32AM +0100, Henrik Rydberg wrote:
> 
> For chips that cannot produce the above events, one simply reports less
> data. [...]
> The minimum set of data is TOUCH, POSITION_X and POSITION_Y. If touch
> pressure information is not given in trackpad dimension units, use a
> specific range.

The Elantech touchpad does not seem to be able to report pressure, only
finger coordinates.

> I believe all current drivers are able to produce this data.

The Elantech driver is not... :)

> It is designed to bring advanced gestures to the linux desktop.

Certainly. Have you already thought about the rest of the path from driver
to application, ie kernel driver -> X.org driver -> X events -> toolkit
events -> application?

For instance does the X.org driver interpret the data and emit a zoom or
rotate event based on the finger position and movement? As far as I know
those event do not exist and would have to be added to X.org as well as the
toolkits like GTK+ and Qt. It could be a long implementation process... :(


Arjan


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-08 11:29     ` Arjan Opmeer
@ 2008-11-08 12:02       ` Henrik Rydberg
  2008-11-11  4:13       ` Peter Hutterer
  1 sibling, 0 replies; 22+ messages in thread
From: Henrik Rydberg @ 2008-11-08 12:02 UTC (permalink / raw)
  To: Arjan Opmeer; +Cc: Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel

>> The minimum set of data is TOUCH, POSITION_X and POSITION_Y. If touch
>> pressure information is not given in trackpad dimension units, use a
>> specific range.
> 
> The Elantech touchpad does not seem to be able to report pressure, only
> finger coordinates.

I was envisioning a binary TOUCH value, but it is arguably redundant.

>> I believe all current drivers are able to produce this data.
> 
> The Elantech driver is not... :)

So only POSITION_X and POSITION_Y for elantech, then :-)

>> It is designed to bring advanced gestures to the linux desktop.
> 
> Certainly. Have you already thought about the rest of the path from driver
> to application, ie kernel driver -> X.org driver -> X events -> toolkit
> events -> application?

I am involved in the synaptics Xorg driver. I have an experimental Xorg
driver utilizing the multi-touch interface. The plan is to map stable
gestures to the existing toolkit events, removing the need for programs
like syndaemon.

> For instance does the X.org driver interpret the data and emit a zoom or
> rotate event based on the finger position and movement? As far as I know
> those event do not exist and would have to be added to X.org as well as the
> toolkits like GTK+ and Qt. It could be a long implementation process... :(

Every journey begins with a step.

Henrik


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-07 14:52       ` Henrik Rydberg
  2008-11-07 14:57         ` J.R. Mauro
@ 2008-11-11  4:08         ` Peter Hutterer
  1 sibling, 0 replies; 22+ messages in thread
From: Peter Hutterer @ 2008-11-11  4:08 UTC (permalink / raw)
  To: Henrik Rydberg
  Cc: J.R. Mauro, Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel

On Fri, Nov 07, 2008 at 03:52:24PM +0100, Henrik Rydberg wrote:
> > Have you worked with MPX? Or was a vanilla-X11 driver simple enough?
> > How does this all get presented to userspace applications? Multiple
> > mouse inputs or special stuff that applications would need a rewrite
> > to leverage?
> 
> MPX is about making use of multiple pointers. The MT protocol is about
> making use of multiple fingers. It can in principle be used as input
> to MPX applications; I will bring it up with Peter Hutterer. However,
> the main aim is to bring advanced gestures to the linux desktop.

MPX is about making use of multiple input devices simultaneously and most of
the groundwork to get MT devices going is already there. From X's point of
view we need a bunch of new events to be sent to clients (and the bells and
whistles around it). The hard bit here is deciding which client to send the
events to.

I'd also say that an MT device should be presented as a single device with
multiple input points rather than as multiple separate devices.
This does require to rewrite applications (or at least slot some intermediate
layer in between), but having multitouch look like multi-mouse is a really bad
idea.

Cheers,
  Peter

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-08 11:29     ` Arjan Opmeer
  2008-11-08 12:02       ` Henrik Rydberg
@ 2008-11-11  4:13       ` Peter Hutterer
  1 sibling, 0 replies; 22+ messages in thread
From: Peter Hutterer @ 2008-11-11  4:13 UTC (permalink / raw)
  To: Arjan Opmeer
  Cc: Henrik Rydberg, Dmitry Torokhov, Andrew Morton, linux-input,
	linux-kernel

On Sat, Nov 08, 2008 at 12:29:02PM +0100, Arjan Opmeer wrote:
> > It is designed to bring advanced gestures to the linux desktop.
> 
> Certainly. Have you already thought about the rest of the path from driver
> to application, ie kernel driver -> X.org driver -> X events -> toolkit
> events -> application?
> 
> For instance does the X.org driver interpret the data and emit a zoom or
> rotate event based on the finger position and movement? As far as I know
> those event do not exist and would have to be added to X.org as well as the
> toolkits like GTK+ and Qt. It could be a long implementation process... :(

IMHO the server or driver shouldn't do anything with the event but pass it on
to the client. What the events then mean semantically depends on the client
(or the toolkit). This replicates pretty much what we do with mouse events and
although it puts the burden on the client side to do anything useful, it makes
it easier to develop, debug and also to do non-standard interpretation of
gestures.

The X server's job is basically just to pick the right client to send the
events to, and of course augment the events with X specific data (such as the
window).

Cheers,
  Peter

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-07 11:15 [PATCH] input: Add a detailed multi-touch finger data report protocol Henrik Rydberg
  2008-11-07 13:34 ` J.R. Mauro
  2008-11-08  0:51 ` Arjan Opmeer
@ 2008-11-11  4:20 ` Peter Hutterer
  2008-11-11 11:19   ` Henrik Rydberg
  2008-11-19 16:37 ` Dmitry Torokhov
  3 siblings, 1 reply; 22+ messages in thread
From: Peter Hutterer @ 2008-11-11  4:20 UTC (permalink / raw)
  To: Henrik Rydberg; +Cc: Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel

On Fri, Nov 07, 2008 at 12:15:29PM +0100, Henrik Rydberg wrote:
> In order to utilize the full power of the new multi-touch devices, a
> way to report detailed finger data to user space is needed. This patch
> adds a multi-touch (MT) protocol which allows drivers to report details
> for an arbitrary number of fingers.

[...]

>  /* We avoid low common keys in module aliases so they don't get huge. */
>  #define KEY_MIN_INTERESTING	KEY_MUTE
>  #define KEY_MAX			0x2ff
> @@ -642,6 +645,15 @@ struct input_absinfo {
>  #define ABS_TOOL_WIDTH		0x1c
>  #define ABS_VOLUME		0x20
>  #define ABS_MISC		0x28
> +#define ABS_MT_TOUCH		0x30	/* Diameter of touching circle */
> +#define ABS_MT_TOUCH_MAJOR	0x30	/* Major axis of touching ellipse */
> +#define ABS_MT_TOUCH_MINOR	0x31	/* Minor axis of touching ellipse */
> +#define ABS_MT_WIDTH		0x32	/* Diameter of approaching circle */
> +#define ABS_MT_WIDTH_MAJOR	0x32	/* Major axis of approaching ellipse */
> +#define ABS_MT_WIDTH_MINOR	0x33	/* Minor axis of approaching ellipse */
> +#define ABS_MT_ORIENTATION	0x34	/* Ellipse orientation */
> +#define ABS_MT_POSITION_X	0x35	/* Center X ellipse position */
> +#define ABS_MT_POSITION_Y	0x36	/* Center Y ellipse position */
>  #define ABS_MAX			0x3f
>  #define ABS_CNT			(ABS_MAX+1)

is there hardware that can do finger identification? (i.e. thumb vs. index
finger)? Should we accommodate for this?
 
Cheers,
  Peter

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-11  4:20 ` Peter Hutterer
@ 2008-11-11 11:19   ` Henrik Rydberg
  0 siblings, 0 replies; 22+ messages in thread
From: Henrik Rydberg @ 2008-11-11 11:19 UTC (permalink / raw)
  To: Peter Hutterer; +Cc: Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel

Peter Hutterer wrote:

[...]

> is there hardware that can do finger identification? (i.e. thumb vs. index
> finger)? Should we accommodate for this?

I believe we should start with events that fit the general idea of
detailed finger information, and which can be produced by at least one
existing kernel driver, so that we can test it immediately. I believe
the proposed set pretty much covers it. I would love to be wrong. :-)

Regarding identification, one of the harder problems involved in
making use of finger data is that of matching an anonymous finger at a
certain position to an identified finger, tagged with a number.  This
is very important in order to know if the fingers moved, which finger
did the tapping, how much rotation was made, etc. Generally, this is
the (euclidian) bipartite matching problem, and is one of the major
computations a multi-touch X driver needs to perform. I can imagine
such identification features eventually ending up on a chip. Maybe
someone more knowledgeable in hardware can give us a hint.


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-07 11:15 [PATCH] input: Add a detailed multi-touch finger data report protocol Henrik Rydberg
                   ` (2 preceding siblings ...)
  2008-11-11  4:20 ` Peter Hutterer
@ 2008-11-19 16:37 ` Dmitry Torokhov
  2008-11-19 16:54   ` Jim Gettys
                     ` (2 more replies)
  3 siblings, 3 replies; 22+ messages in thread
From: Dmitry Torokhov @ 2008-11-19 16:37 UTC (permalink / raw)
  To: Henrik Rydberg; +Cc: Andrew Morton, linux-input, linux-kernel

Hi Henrik,

On Fri, Nov 07, 2008 at 12:15:29PM +0100, Henrik Rydberg wrote:
> In order to utilize the full power of the new multi-touch devices, a
> way to report detailed finger data to user space is needed. This patch
> adds a multi-touch (MT) protocol which allows drivers to report details
> for an arbitrary number of fingers.
> 
> BTN_MT_REPORT_PACKET
> 
> The driver presses this button to indicate the start of a packet of
> finger data. The button is released after the whole packet has been
> reported.
> 
> BTN_MT_REPORT_FINGER
> 
> The driver presses this button to indicate the start of a new
> finger. The button is released when the finger has been reported.
> 
> In order to stay compatible with existing applications, the data
> reported in a finger packet must not be recognized as single-touch
> events. Therefore, a specific set of ABS_MT events is used instead.
> 
> Signed-off-by: Henrik Rydberg <rydberg@euromail.se>
> ---
>  include/linux/input.h |   12 ++++++++++++
>  1 files changed, 12 insertions(+), 0 deletions(-)
> 
> diff --git a/include/linux/input.h b/include/linux/input.h
> index b86fb55..c82dd42 100644
> --- a/include/linux/input.h
> +++ b/include/linux/input.h
> @@ -590,6 +590,9 @@ struct input_absinfo {
>  #define KEY_NUMERIC_STAR	0x20a
>  #define KEY_NUMERIC_POUND	0x20b
> 
> +#define BTN_MT_REPORT_PACKET	0x210	/* report multitouch packet data */
> +#define BTN_MT_REPORT_FINGER	0x211	/* report multitouch finger data */
> +
>  /* We avoid low common keys in module aliases so they don't get huge. */
>  #define KEY_MIN_INTERESTING	KEY_MUTE
>  #define KEY_MAX			0x2ff
> @@ -642,6 +645,15 @@ struct input_absinfo {
>  #define ABS_TOOL_WIDTH		0x1c
>  #define ABS_VOLUME		0x20
>  #define ABS_MISC		0x28
> +#define ABS_MT_TOUCH		0x30	/* Diameter of touching circle */
> +#define ABS_MT_TOUCH_MAJOR	0x30	/* Major axis of touching ellipse */
> +#define ABS_MT_TOUCH_MINOR	0x31	/* Minor axis of touching ellipse */
> +#define ABS_MT_WIDTH		0x32	/* Diameter of approaching circle */
> +#define ABS_MT_WIDTH_MAJOR	0x32	/* Major axis of approaching ellipse */
> +#define ABS_MT_WIDTH_MINOR	0x33	/* Minor axis of approaching ellipse */
> +#define ABS_MT_ORIENTATION	0x34	/* Ellipse orientation */
> +#define ABS_MT_POSITION_X	0x35	/* Center X ellipse position */
> +#define ABS_MT_POSITION_Y	0x36	/* Center Y ellipse position */
>  #define ABS_MAX			0x3f
>  #define ABS_CNT			(ABS_MAX+1)
> 

Sorry for the long silence. I don't think utilizing button events for
this is a good idea. I'd rather just start reporting extended touch
events for a finger and signal end of sub-packet with something like
EV_SYN/SYN_MT_REPORT.

-- 
Dmitry

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-19 16:37 ` Dmitry Torokhov
@ 2008-11-19 16:54   ` Jim Gettys
  2008-11-19 23:34     ` Henrik Rydberg
  2008-11-19 23:31   ` Henrik Rydberg
  2008-11-21 14:44   ` Henrik Rydberg
  2 siblings, 1 reply; 22+ messages in thread
From: Jim Gettys @ 2008-11-19 16:54 UTC (permalink / raw)
  To: Dmitry Torokhov; +Cc: Henrik Rydberg, Andrew Morton, linux-input, linux-kernel

On Wed, 2008-11-19 at 11:37 -0500, Dmitry Torokhov wrote:
> Hi Henrik,
> 
> On Fri, Nov 07, 2008 at 12:15:29PM +0100, Henrik Rydberg wrote:
> > In order to utilize the full power of the new multi-touch devices, a
> > way to report detailed finger data to user space is needed. This patch
> > adds a multi-touch (MT) protocol which allows drivers to report details
> > for an arbitrary number of fingers.
> > 
> > BTN_MT_REPORT_PACKET
> > 
> > The driver presses this button to indicate the start of a packet of
> > finger data. The button is released after the whole packet has been
> > reported.
> > 
> > BTN_MT_REPORT_FINGER
> > 
> > The driver presses this button to indicate the start of a new
> > finger. The button is released when the finger has been reported.
> > 
> > In order to stay compatible with existing applications, the data
> > reported in a finger packet must not be recognized as single-touch
> > events. Therefore, a specific set of ABS_MT events is used instead.
> > 
> > Signed-off-by: Henrik Rydberg <rydberg@euromail.se>
> > ---
> >  include/linux/input.h |   12 ++++++++++++
> >  1 files changed, 12 insertions(+), 0 deletions(-)
> > 
> > diff --git a/include/linux/input.h b/include/linux/input.h
> > index b86fb55..c82dd42 100644
> > --- a/include/linux/input.h
> > +++ b/include/linux/input.h
> > @@ -590,6 +590,9 @@ struct input_absinfo {
> >  #define KEY_NUMERIC_STAR	0x20a
> >  #define KEY_NUMERIC_POUND	0x20b
> > 
> > +#define BTN_MT_REPORT_PACKET	0x210	/* report multitouch packet data */
> > +#define BTN_MT_REPORT_FINGER	0x211	/* report multitouch finger data */
> > +
> >  /* We avoid low common keys in module aliases so they don't get huge. */
> >  #define KEY_MIN_INTERESTING	KEY_MUTE
> >  #define KEY_MAX			0x2ff
> > @@ -642,6 +645,15 @@ struct input_absinfo {
> >  #define ABS_TOOL_WIDTH		0x1c
> >  #define ABS_VOLUME		0x20
> >  #define ABS_MISC		0x28
> > +#define ABS_MT_TOUCH		0x30	/* Diameter of touching circle */
> > +#define ABS_MT_TOUCH_MAJOR	0x30	/* Major axis of touching ellipse */
> > +#define ABS_MT_TOUCH_MINOR	0x31	/* Minor axis of touching ellipse */
> > +#define ABS_MT_WIDTH		0x32	/* Diameter of approaching circle */
> > +#define ABS_MT_WIDTH_MAJOR	0x32	/* Major axis of approaching ellipse */
> > +#define ABS_MT_WIDTH_MINOR	0x33	/* Minor axis of approaching ellipse */
> > +#define ABS_MT_ORIENTATION	0x34	/* Ellipse orientation */
> > +#define ABS_MT_POSITION_X	0x35	/* Center X ellipse position */
> > +#define ABS_MT_POSITION_Y	0x36	/* Center Y ellipse position */
> >  #define ABS_MAX			0x3f
> >  #define ABS_CNT			(ABS_MAX+1)
> > 
> 
> Sorry for the long silence. I don't think utilizing button events for
> this is a good idea. I'd rather just start reporting extended touch
> events for a finger and signal end of sub-packet with something like
> EV_SYN/SYN_MT_REPORT.

I agree.

Also, I'm still concerned about using elipses. I doubt very much any
hardware will ever be reporting elipses; more likely are parallelograms,
or trapezoids, other simple geometric figures, rather than a center with
size and orientation.

There are reasons Cairo uses aligned traps internally when tessellating
figures....
                                 - Jim



> 
-- 
Jim Gettys <jg@laptop.org>
One Laptop Per Child


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-19 16:37 ` Dmitry Torokhov
  2008-11-19 16:54   ` Jim Gettys
@ 2008-11-19 23:31   ` Henrik Rydberg
  2008-11-21 14:44   ` Henrik Rydberg
  2 siblings, 0 replies; 22+ messages in thread
From: Henrik Rydberg @ 2008-11-19 23:31 UTC (permalink / raw)
  To: Dmitry Torokhov; +Cc: Andrew Morton, linux-input, linux-kernel

Dmitry Torokhov wrote:
> Hi Henrik,
> 
> On Fri, Nov 07, 2008 at 12:15:29PM +0100, Henrik Rydberg wrote:
>> In order to utilize the full power of the new multi-touch devices, a
>> way to report detailed finger data to user space is needed. This patch
>> adds a multi-touch (MT) protocol which allows drivers to report details
>> for an arbitrary number of fingers.
>>
[...]
> 
> Sorry for the long silence. I don't think utilizing button events for
> this is a good idea. I'd rather just start reporting extended touch
> events for a finger and signal end of sub-packet with something like
> EV_SYN/SYN_MT_REPORT.
> 

Long time no see :-)

Yes, there has been quite a bit of feedback on this patch, and using
EV_SYN the way you suggest came up. Thanks - will redo.

Henrik


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-19 16:54   ` Jim Gettys
@ 2008-11-19 23:34     ` Henrik Rydberg
  2008-11-21 14:43       ` Jim Gettys
  0 siblings, 1 reply; 22+ messages in thread
From: Henrik Rydberg @ 2008-11-19 23:34 UTC (permalink / raw)
  To: jg; +Cc: Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel

Jim Gettys wrote:
[...]
> Also, I'm still concerned about using elipses. I doubt very much any
> hardware will ever be reporting elipses; more likely are parallelograms,
> or trapezoids, other simple geometric figures, rather than a center with
> size and orientation.

The bcm5974 chip is doing precisely this.

Henrik


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-19 23:34     ` Henrik Rydberg
@ 2008-11-21 14:43       ` Jim Gettys
  0 siblings, 0 replies; 22+ messages in thread
From: Jim Gettys @ 2008-11-21 14:43 UTC (permalink / raw)
  To: Henrik Rydberg; +Cc: Dmitry Torokhov, Andrew Morton, linux-input, linux-kernel

Interesting....  I know of the a different case: a device which
certainly does not report ellipses, rather reports x, y, dx, dy.
                          - Jim


On Thu, 2008-11-20 at 00:34 +0100, Henrik Rydberg wrote:
> Jim Gettys wrote:
> [...]
> > Also, I'm still concerned about using elipses. I doubt very much any
> > hardware will ever be reporting elipses; more likely are parallelograms,
> > or trapezoids, other simple geometric figures, rather than a center with
> > size and orientation.
> 
> The bcm5974 chip is doing precisely this.
> 
> Henrik
> 
-- 
Jim Gettys <jg@laptop.org>
One Laptop Per Child


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-19 16:37 ` Dmitry Torokhov
  2008-11-19 16:54   ` Jim Gettys
  2008-11-19 23:31   ` Henrik Rydberg
@ 2008-11-21 14:44   ` Henrik Rydberg
  2008-11-21 16:04     ` Dmitry Torokhov
  2 siblings, 1 reply; 22+ messages in thread
From: Henrik Rydberg @ 2008-11-21 14:44 UTC (permalink / raw)
  To: Dmitry Torokhov; +Cc: Henrik Rydberg, Andrew Morton, linux-input, linux-kernel

Dmitry,

My patch is all wrong. It would certainly benefit from using the
EV_SYN/SYN_MT_REPORT, but the defuzz mechanism of the EV_ABS class
simply will not work when sending data for several different fingers
using the same event.

I can see two clear alternatives: either add a new event class, EV_MT,
which sends all data without trying to limit the bandwidth, or forget
about the whole serial-finger-data idea and expand the current EV_ABS
class with an array of MT finger data [1]. I would very much appreciate
your input on this. Maybe there is a third option.

Cheers,
Henrik

[1] An example of a straight-forward addition of EV_ABS/ABS_MT events:

#define ABS_MT_FINGER_CNT	10
#define ABS_MT_PROPERTY_CNT	12

#define ABS_MT_FINGER_START	0x30
#define ABS_MT_TOUCH(x)		(ABS_MT_FINGER_START + 0 * ABS_MT_FINGER_CNT + x)
#define ABS_MT_WIDTH(x)		(ABS_MT_FINGER_START + 1 * ABS_MT_FINGER_CNT + x)
...
#define ABS_MT_FINGER_END	0xa8 /* = ABS_MT_FINGER_START + ABS_MT_PROPERTY_CNT * ABS_MT_FINGER_CNT */


^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [PATCH] input: Add a detailed multi-touch finger data report protocol
  2008-11-21 14:44   ` Henrik Rydberg
@ 2008-11-21 16:04     ` Dmitry Torokhov
  0 siblings, 0 replies; 22+ messages in thread
From: Dmitry Torokhov @ 2008-11-21 16:04 UTC (permalink / raw)
  To: Henrik Rydberg; +Cc: Henrik Rydberg, Andrew Morton, linux-input, linux-kernel

Hi Henrik,

On Fri, Nov 21, 2008 at 03:44:44PM +0100, Henrik Rydberg wrote:
> Dmitry,
> 
> My patch is all wrong. It would certainly benefit from using the
> EV_SYN/SYN_MT_REPORT, but the defuzz mechanism of the EV_ABS class
> simply will not work when sending data for several different fingers
> using the same event.
> 
> I can see two clear alternatives: either add a new event class, EV_MT,
> which sends all data without trying to limit the bandwidth, or forget
> about the whole serial-finger-data idea and expand the current EV_ABS
> class with an array of MT finger data [1]. I would very much appreciate
> your input on this. Maybe there is a third option.
> 

I don't like the idea of a new class since the finger data is a part
of absolute device state (in my opinion anyway). I am not sure if we
really need to pre-allocate space for 10 fingers either. Could we just
mark certain EV_ABS events as uncacheable in the input core? It could
be implemented with one static bitmap.

-- 
Dmitry

^ permalink raw reply	[flat|nested] 22+ messages in thread

end of thread, other threads:[~2008-11-21 16:04 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-11-07 11:15 [PATCH] input: Add a detailed multi-touch finger data report protocol Henrik Rydberg
2008-11-07 13:34 ` J.R. Mauro
2008-11-07 14:26   ` Henrik Rydberg
2008-11-07 14:36     ` J.R. Mauro
2008-11-07 14:52       ` Henrik Rydberg
2008-11-07 14:57         ` J.R. Mauro
2008-11-07 15:07           ` Henrik Rydberg
2008-11-11  4:08         ` Peter Hutterer
2008-11-08  0:51 ` Arjan Opmeer
2008-11-08 10:43   ` Henrik Rydberg
2008-11-08 11:29     ` Arjan Opmeer
2008-11-08 12:02       ` Henrik Rydberg
2008-11-11  4:13       ` Peter Hutterer
2008-11-11  4:20 ` Peter Hutterer
2008-11-11 11:19   ` Henrik Rydberg
2008-11-19 16:37 ` Dmitry Torokhov
2008-11-19 16:54   ` Jim Gettys
2008-11-19 23:34     ` Henrik Rydberg
2008-11-21 14:43       ` Jim Gettys
2008-11-19 23:31   ` Henrik Rydberg
2008-11-21 14:44   ` Henrik Rydberg
2008-11-21 16:04     ` Dmitry Torokhov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).