Re: Failing to attach bond(created with two interfaces from different NICs) to a bridge

[Florian Fainelli <f.fainelli@gmail.com>]

On 9/6/2020 10:13 AM, Jakub Kicinski wrote:
> On Sun, 6 Sep 2020 14:12:49 +0300 Ido Schimmel wrote:
>> On Thu, Sep 03, 2020 at 12:14:28PM -0700, Jakub Kicinski wrote:
>>> On Thu, 3 Sep 2020 12:52:25 +0530 Vasundhara Volam wrote:
>>>> Hello Jiri,
>>>>
>>>> After the following set of upstream commits, the user fails to attach
>>>> a bond to the bridge, if the user creates the bond with two interfaces
>>>> from different bnxt_en NICs. Previously bnxt_en driver does not
>>>> advertise the switch_id for legacy mode as part of
>>>> ndo_get_port_parent_id cb but with the following patches, switch_id is
>>>> returned even in legacy mode which is causing the failure.
>>>>
>>>> ---------------
>>>> 7e1146e8c10c00f859843817da8ecc5d902ea409 net: devlink: introduce
>>>> devlink_compat_switch_id_get() helper
>>>> 6605a226781eb1224c2dcf974a39eea11862b864 bnxt: pass switch ID through
>>>> devlink_port_attrs_set()
>>>> 56d9f4e8f70e6f47ad4da7640753cf95ae51a356 bnxt: remove
>>>> ndo_get_port_parent_id implementation for physical ports
>>>> ----------------
>>>>
>>>> As there is a plan to get rid of ndo_get_port_parent_id in future, I
>>>> think there is a need to fix devlink_compat_switch_id_get() to return
>>>> the switch_id only when device is in SWITCHDEV mode and this effects
>>>> all the NICs.
>>>>
>>>> Please let me know your thoughts. Thank you.
>>>
>>> I'm not Jiri, but I'd think that hiding switch_id from devices should
>>> not be the solution here. Especially that no NICs offload bridging
>>> today.
>>>
>>> Could you describe the team/bridge failure in detail, I'm not that
>>> familiar with this code.
>>
>> Maybe:
>>
>> br_add_slave()
>> 	br_add_if()
>> 		nbp_switchdev_mark_set()
>> 			dev_get_port_parent_id()
>>
>> I believe the last call will return '-ENODATA' because the two bnxt
>> netdevs member in the bond have different switch IDs. Perhaps the
>> function can be changed to return '-EOPNOTSUPP' when it's called for an
>> upper device that have multiple parent IDs beneath it:
>>
>> diff --git a/net/core/dev.c b/net/core/dev.c
>> index d42c9ea0c3c0..7932594ca437 100644
>> --- a/net/core/dev.c
>> +++ b/net/core/dev.c
>> @@ -8646,7 +8646,7 @@ int dev_get_port_parent_id(struct net_device *dev,
>>                  if (!first.id_len)
>>                          first = *ppid;
>>                  else if (memcmp(&first, ppid, sizeof(*ppid)))
>> -                       return -ENODATA;
>> +                       return -EOPNOTSUPP;
>>          }
>>   
>>          return err;
> 
> LGTM, or we could make bridge ignore ENODATA (in case the distinctions
> is useful?)
> 
> I was searching for the early versions of Florian's patch set but
> I can't find it :( Florian, do you remember if there was a reason to
> fail bridge in this case?

v3: https://patchwork.kernel.org/patch/10798697/
v2: https://lore.kernel.org/patchwork/patch/1038907/
v1: 
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1921358.html

I went back to check the tree before 
d6abc5969463359c366d459247b90366fcd6f5c5 and the logic for return 
-ENODATA was copied from switchdev_port_attr_get():

...
           /* Switch device port(s) may be stacked under
            * bond/team/vlan dev, so recurse down to get attr on
            * each port.  Return -ENODATA if attr values don't
            * compare across ports.
            */

           netdev_for_each_lower_dev(dev, lower_dev, iter) {
                   err = switchdev_port_attr_get(lower_dev, attr);
                   if (err)
                           break;
                   if (first.id == SWITCHDEV_ATTR_ID_UNDEFINED)
                           first = *attr;
                   else if (memcmp(&first, attr, sizeof(*attr)))
                           return -ENODATA;
           }

           return err;
   }
   EXPORT_SYMBOL_GPL(switchdev_port_attr_get);

the bridge code would specifically treat -EOPNOTSUPP as success and 
return early, whereas other error code would be treated as a failure.

Jiri or Ido, do you remember the reason for return -ENODATA here?
-- 
Florian