Failing to attach bond(created with two interfaces from different NICs) to a bridge

Failing to attach bond(created with two interfaces from different NICs) to a bridge

[Vasundhara Volam]

Hello Jiri,

After the following set of upstream commits, the user fails to attach
a bond to the bridge, if the user creates the bond with two interfaces
from different bnxt_en NICs. Previously bnxt_en driver does not
advertise the switch_id for legacy mode as part of
ndo_get_port_parent_id cb but with the following patches, switch_id is
returned even in legacy mode which is causing the failure.

---------------
7e1146e8c10c00f859843817da8ecc5d902ea409 net: devlink: introduce
devlink_compat_switch_id_get() helper
6605a226781eb1224c2dcf974a39eea11862b864 bnxt: pass switch ID through
devlink_port_attrs_set()
56d9f4e8f70e6f47ad4da7640753cf95ae51a356 bnxt: remove
ndo_get_port_parent_id implementation for physical ports
----------------

As there is a plan to get rid of ndo_get_port_parent_id in future, I
think there is a need to fix devlink_compat_switch_id_get() to return
the switch_id only when device is in SWITCHDEV mode and this effects
all the NICs.

Please let me know your thoughts. Thank you.

Thanks,
Vasundhara

Re: Failing to attach bond(created with two interfaces from different NICs) to a bridge

[Jakub Kicinski]

On Thu, 3 Sep 2020 12:52:25 +0530 Vasundhara Volam wrote:
> Hello Jiri,
> 
> After the following set of upstream commits, the user fails to attach
> a bond to the bridge, if the user creates the bond with two interfaces
> from different bnxt_en NICs. Previously bnxt_en driver does not
> advertise the switch_id for legacy mode as part of
> ndo_get_port_parent_id cb but with the following patches, switch_id is
> returned even in legacy mode which is causing the failure.
> 
> ---------------
> 7e1146e8c10c00f859843817da8ecc5d902ea409 net: devlink: introduce
> devlink_compat_switch_id_get() helper
> 6605a226781eb1224c2dcf974a39eea11862b864 bnxt: pass switch ID through
> devlink_port_attrs_set()
> 56d9f4e8f70e6f47ad4da7640753cf95ae51a356 bnxt: remove
> ndo_get_port_parent_id implementation for physical ports
> ----------------
> 
> As there is a plan to get rid of ndo_get_port_parent_id in future, I
> think there is a need to fix devlink_compat_switch_id_get() to return
> the switch_id only when device is in SWITCHDEV mode and this effects
> all the NICs.
> 
> Please let me know your thoughts. Thank you.

I'm not Jiri, but I'd think that hiding switch_id from devices should
not be the solution here. Especially that no NICs offload bridging
today. 

Could you describe the team/bridge failure in detail, I'm not that
familiar with this code.

Re: Failing to attach bond(created with two interfaces from different NICs) to a bridge

[Ido Schimmel]

On Thu, Sep 03, 2020 at 12:14:28PM -0700, Jakub Kicinski wrote:
> On Thu, 3 Sep 2020 12:52:25 +0530 Vasundhara Volam wrote:
> > Hello Jiri,
> > 
> > After the following set of upstream commits, the user fails to attach
> > a bond to the bridge, if the user creates the bond with two interfaces
> > from different bnxt_en NICs. Previously bnxt_en driver does not
> > advertise the switch_id for legacy mode as part of
> > ndo_get_port_parent_id cb but with the following patches, switch_id is
> > returned even in legacy mode which is causing the failure.
> > 
> > ---------------
> > 7e1146e8c10c00f859843817da8ecc5d902ea409 net: devlink: introduce
> > devlink_compat_switch_id_get() helper
> > 6605a226781eb1224c2dcf974a39eea11862b864 bnxt: pass switch ID through
> > devlink_port_attrs_set()
> > 56d9f4e8f70e6f47ad4da7640753cf95ae51a356 bnxt: remove
> > ndo_get_port_parent_id implementation for physical ports
> > ----------------
> > 
> > As there is a plan to get rid of ndo_get_port_parent_id in future, I
> > think there is a need to fix devlink_compat_switch_id_get() to return
> > the switch_id only when device is in SWITCHDEV mode and this effects
> > all the NICs.
> > 
> > Please let me know your thoughts. Thank you.
> 
> I'm not Jiri, but I'd think that hiding switch_id from devices should
> not be the solution here. Especially that no NICs offload bridging
> today. 
> 
> Could you describe the team/bridge failure in detail, I'm not that
> familiar with this code.

Maybe:

br_add_slave()
	br_add_if()
		nbp_switchdev_mark_set()
			dev_get_port_parent_id()

I believe the last call will return '-ENODATA' because the two bnxt
netdevs member in the bond have different switch IDs. Perhaps the
function can be changed to return '-EOPNOTSUPP' when it's called for an
upper device that have multiple parent IDs beneath it:

diff --git a/net/core/dev.c b/net/core/dev.c
index d42c9ea0c3c0..7932594ca437 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -8646,7 +8646,7 @@ int dev_get_port_parent_id(struct net_device *dev,
                if (!first.id_len)
                        first = *ppid;
                else if (memcmp(&first, ppid, sizeof(*ppid)))
-                       return -ENODATA;
+                       return -EOPNOTSUPP;
        }
 
        return err;

Re: Failing to attach bond(created with two interfaces from different NICs) to a bridge

[Jakub Kicinski]

On Sun, 6 Sep 2020 14:12:49 +0300 Ido Schimmel wrote:
> On Thu, Sep 03, 2020 at 12:14:28PM -0700, Jakub Kicinski wrote:
> > On Thu, 3 Sep 2020 12:52:25 +0530 Vasundhara Volam wrote:  
> > > Hello Jiri,
> > > 
> > > After the following set of upstream commits, the user fails to attach
> > > a bond to the bridge, if the user creates the bond with two interfaces
> > > from different bnxt_en NICs. Previously bnxt_en driver does not
> > > advertise the switch_id for legacy mode as part of
> > > ndo_get_port_parent_id cb but with the following patches, switch_id is
> > > returned even in legacy mode which is causing the failure.
> > > 
> > > ---------------
> > > 7e1146e8c10c00f859843817da8ecc5d902ea409 net: devlink: introduce
> > > devlink_compat_switch_id_get() helper
> > > 6605a226781eb1224c2dcf974a39eea11862b864 bnxt: pass switch ID through
> > > devlink_port_attrs_set()
> > > 56d9f4e8f70e6f47ad4da7640753cf95ae51a356 bnxt: remove
> > > ndo_get_port_parent_id implementation for physical ports
> > > ----------------
> > > 
> > > As there is a plan to get rid of ndo_get_port_parent_id in future, I
> > > think there is a need to fix devlink_compat_switch_id_get() to return
> > > the switch_id only when device is in SWITCHDEV mode and this effects
> > > all the NICs.
> > > 
> > > Please let me know your thoughts. Thank you.  
> > 
> > I'm not Jiri, but I'd think that hiding switch_id from devices should
> > not be the solution here. Especially that no NICs offload bridging
> > today. 
> > 
> > Could you describe the team/bridge failure in detail, I'm not that
> > familiar with this code.  
> 
> Maybe:
> 
> br_add_slave()
> 	br_add_if()
> 		nbp_switchdev_mark_set()
> 			dev_get_port_parent_id()
> 
> I believe the last call will return '-ENODATA' because the two bnxt
> netdevs member in the bond have different switch IDs. Perhaps the 
> function can be changed to return '-EOPNOTSUPP' when it's called for an
> upper device that have multiple parent IDs beneath it:
> 
> diff --git a/net/core/dev.c b/net/core/dev.c
> index d42c9ea0c3c0..7932594ca437 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -8646,7 +8646,7 @@ int dev_get_port_parent_id(struct net_device *dev,
>                 if (!first.id_len)
>                         first = *ppid;
>                 else if (memcmp(&first, ppid, sizeof(*ppid)))
> -                       return -ENODATA;
> +                       return -EOPNOTSUPP;
>         }
>  
>         return err;

LGTM, or we could make bridge ignore ENODATA (in case the distinctions
is useful?)

I was searching for the early versions of Florian's patch set but
I can't find it :( Florian, do you remember if there was a reason to
fail bridge in this case?

Re: Failing to attach bond(created with two interfaces from different NICs) to a bridge

[Florian Fainelli]

On 9/6/2020 10:13 AM, Jakub Kicinski wrote:
> On Sun, 6 Sep 2020 14:12:49 +0300 Ido Schimmel wrote:
>> On Thu, Sep 03, 2020 at 12:14:28PM -0700, Jakub Kicinski wrote:
>>> On Thu, 3 Sep 2020 12:52:25 +0530 Vasundhara Volam wrote:
>>>> Hello Jiri,
>>>>
>>>> After the following set of upstream commits, the user fails to attach
>>>> a bond to the bridge, if the user creates the bond with two interfaces
>>>> from different bnxt_en NICs. Previously bnxt_en driver does not
>>>> advertise the switch_id for legacy mode as part of
>>>> ndo_get_port_parent_id cb but with the following patches, switch_id is
>>>> returned even in legacy mode which is causing the failure.
>>>>
>>>> ---------------
>>>> 7e1146e8c10c00f859843817da8ecc5d902ea409 net: devlink: introduce
>>>> devlink_compat_switch_id_get() helper
>>>> 6605a226781eb1224c2dcf974a39eea11862b864 bnxt: pass switch ID through
>>>> devlink_port_attrs_set()
>>>> 56d9f4e8f70e6f47ad4da7640753cf95ae51a356 bnxt: remove
>>>> ndo_get_port_parent_id implementation for physical ports
>>>> ----------------
>>>>
>>>> As there is a plan to get rid of ndo_get_port_parent_id in future, I
>>>> think there is a need to fix devlink_compat_switch_id_get() to return
>>>> the switch_id only when device is in SWITCHDEV mode and this effects
>>>> all the NICs.
>>>>
>>>> Please let me know your thoughts. Thank you.
>>>
>>> I'm not Jiri, but I'd think that hiding switch_id from devices should
>>> not be the solution here. Especially that no NICs offload bridging
>>> today.
>>>
>>> Could you describe the team/bridge failure in detail, I'm not that
>>> familiar with this code.
>>
>> Maybe:
>>
>> br_add_slave()
>> 	br_add_if()
>> 		nbp_switchdev_mark_set()
>> 			dev_get_port_parent_id()
>>
>> I believe the last call will return '-ENODATA' because the two bnxt
>> netdevs member in the bond have different switch IDs. Perhaps the
>> function can be changed to return '-EOPNOTSUPP' when it's called for an
>> upper device that have multiple parent IDs beneath it:
>>
>> diff --git a/net/core/dev.c b/net/core/dev.c
>> index d42c9ea0c3c0..7932594ca437 100644
>> --- a/net/core/dev.c
>> +++ b/net/core/dev.c
>> @@ -8646,7 +8646,7 @@ int dev_get_port_parent_id(struct net_device *dev,
>>                  if (!first.id_len)
>>                          first = *ppid;
>>                  else if (memcmp(&first, ppid, sizeof(*ppid)))
>> -                       return -ENODATA;
>> +                       return -EOPNOTSUPP;
>>          }
>>   
>>          return err;
> 
> LGTM, or we could make bridge ignore ENODATA (in case the distinctions
> is useful?)
> 
> I was searching for the early versions of Florian's patch set but
> I can't find it :( Florian, do you remember if there was a reason to
> fail bridge in this case?

v3: https://patchwork.kernel.org/patch/10798697/
v2: https://lore.kernel.org/patchwork/patch/1038907/
v1: 
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1921358.html

I went back to check the tree before 
d6abc5969463359c366d459247b90366fcd6f5c5 and the logic for return 
-ENODATA was copied from switchdev_port_attr_get():

...
           /* Switch device port(s) may be stacked under
            * bond/team/vlan dev, so recurse down to get attr on
            * each port.  Return -ENODATA if attr values don't
            * compare across ports.
            */

           netdev_for_each_lower_dev(dev, lower_dev, iter) {
                   err = switchdev_port_attr_get(lower_dev, attr);
                   if (err)
                           break;
                   if (first.id == SWITCHDEV_ATTR_ID_UNDEFINED)
                           first = *attr;
                   else if (memcmp(&first, attr, sizeof(*attr)))
                           return -ENODATA;
           }

           return err;
   }
   EXPORT_SYMBOL_GPL(switchdev_port_attr_get);

the bridge code would specifically treat -EOPNOTSUPP as success and 
return early, whereas other error code would be treated as a failure.

Jiri or Ido, do you remember the reason for return -ENODATA here?
-- 
Florian

Re: Failing to attach bond(created with two interfaces from different NICs) to a bridge

[Ido Schimmel]

On Sun, Sep 06, 2020 at 12:23:23PM -0700, Florian Fainelli wrote:
> 
> 
> On 9/6/2020 10:13 AM, Jakub Kicinski wrote:
> > On Sun, 6 Sep 2020 14:12:49 +0300 Ido Schimmel wrote:
> > > On Thu, Sep 03, 2020 at 12:14:28PM -0700, Jakub Kicinski wrote:
> > > > On Thu, 3 Sep 2020 12:52:25 +0530 Vasundhara Volam wrote:
> > > > > Hello Jiri,
> > > > > 
> > > > > After the following set of upstream commits, the user fails to attach
> > > > > a bond to the bridge, if the user creates the bond with two interfaces
> > > > > from different bnxt_en NICs. Previously bnxt_en driver does not
> > > > > advertise the switch_id for legacy mode as part of
> > > > > ndo_get_port_parent_id cb but with the following patches, switch_id is
> > > > > returned even in legacy mode which is causing the failure.
> > > > > 
> > > > > ---------------
> > > > > 7e1146e8c10c00f859843817da8ecc5d902ea409 net: devlink: introduce
> > > > > devlink_compat_switch_id_get() helper
> > > > > 6605a226781eb1224c2dcf974a39eea11862b864 bnxt: pass switch ID through
> > > > > devlink_port_attrs_set()
> > > > > 56d9f4e8f70e6f47ad4da7640753cf95ae51a356 bnxt: remove
> > > > > ndo_get_port_parent_id implementation for physical ports
> > > > > ----------------
> > > > > 
> > > > > As there is a plan to get rid of ndo_get_port_parent_id in future, I
> > > > > think there is a need to fix devlink_compat_switch_id_get() to return
> > > > > the switch_id only when device is in SWITCHDEV mode and this effects
> > > > > all the NICs.
> > > > > 
> > > > > Please let me know your thoughts. Thank you.
> > > > 
> > > > I'm not Jiri, but I'd think that hiding switch_id from devices should
> > > > not be the solution here. Especially that no NICs offload bridging
> > > > today.
> > > > 
> > > > Could you describe the team/bridge failure in detail, I'm not that
> > > > familiar with this code.
> > > 
> > > Maybe:
> > > 
> > > br_add_slave()
> > > 	br_add_if()
> > > 		nbp_switchdev_mark_set()
> > > 			dev_get_port_parent_id()
> > > 
> > > I believe the last call will return '-ENODATA' because the two bnxt
> > > netdevs member in the bond have different switch IDs. Perhaps the
> > > function can be changed to return '-EOPNOTSUPP' when it's called for an
> > > upper device that have multiple parent IDs beneath it:
> > > 
> > > diff --git a/net/core/dev.c b/net/core/dev.c
> > > index d42c9ea0c3c0..7932594ca437 100644
> > > --- a/net/core/dev.c
> > > +++ b/net/core/dev.c
> > > @@ -8646,7 +8646,7 @@ int dev_get_port_parent_id(struct net_device *dev,
> > >                  if (!first.id_len)
> > >                          first = *ppid;
> > >                  else if (memcmp(&first, ppid, sizeof(*ppid)))
> > > -                       return -ENODATA;
> > > +                       return -EOPNOTSUPP;
> > >          }
> > >          return err;
> > 
> > LGTM, or we could make bridge ignore ENODATA (in case the distinctions
> > is useful?)
> > 
> > I was searching for the early versions of Florian's patch set but
> > I can't find it :( Florian, do you remember if there was a reason to
> > fail bridge in this case?
> 
> v3: https://patchwork.kernel.org/patch/10798697/
> v2: https://lore.kernel.org/patchwork/patch/1038907/
> v1:
> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1921358.html
> 
> I went back to check the tree before
> d6abc5969463359c366d459247b90366fcd6f5c5 and the logic for return -ENODATA
> was copied from switchdev_port_attr_get():
> 
> ...
>           /* Switch device port(s) may be stacked under
>            * bond/team/vlan dev, so recurse down to get attr on
>            * each port.  Return -ENODATA if attr values don't
>            * compare across ports.
>            */
> 
>           netdev_for_each_lower_dev(dev, lower_dev, iter) {
>                   err = switchdev_port_attr_get(lower_dev, attr);
>                   if (err)
>                           break;
>                   if (first.id == SWITCHDEV_ATTR_ID_UNDEFINED)
>                           first = *attr;
>                   else if (memcmp(&first, attr, sizeof(*attr)))
>                           return -ENODATA;
>           }
> 
>           return err;
>   }
>   EXPORT_SYMBOL_GPL(switchdev_port_attr_get);
> 
> the bridge code would specifically treat -EOPNOTSUPP as success and return
> early, whereas other error code would be treated as a failure.
> 
> Jiri or Ido, do you remember the reason for return -ENODATA here?

I don't know about the past, but I checked all the current callers of
dev_get_port_parent_id() and I think the proposed change should be OK:

1. nbp_switchdev_mark_set(): Current use case. Does not seem to be a
problem

2. dev_get_port_parent_id(): Recursive call

3. netdev_port_same_parent_id(): Unaffected by this change

4. phys_switch_id_show(): Likewise. Does not recurse

5. rtnl_phys_switch_id_fill(): Likewise

6. vif_add: Does not check the error code

I can test the patch in our regression and submit later this week unless
you have a better suggestion. Please let me know.

Thanks

Re: Failing to attach bond(created with two interfaces from different NICs) to a bridge

[Jakub Kicinski]

On Mon, 7 Sep 2020 10:36:35 +0300 Ido Schimmel wrote:
> I can test the patch in our regression and submit later this week unless
> you have a better suggestion. Please let me know.

That'd be great!

Re: Failing to attach bond(created with two interfaces from different NICs) to a bridge

[Vasundhara Volam]

On Mon, Sep 7, 2020 at 1:06 PM Ido Schimmel <idosch@idosch.org> wrote:
>
> On Sun, Sep 06, 2020 at 12:23:23PM -0700, Florian Fainelli wrote:
> >
> >
> > On 9/6/2020 10:13 AM, Jakub Kicinski wrote:
> > > On Sun, 6 Sep 2020 14:12:49 +0300 Ido Schimmel wrote:
> > > > On Thu, Sep 03, 2020 at 12:14:28PM -0700, Jakub Kicinski wrote:
> > > > > On Thu, 3 Sep 2020 12:52:25 +0530 Vasundhara Volam wrote:
> > > > > > Hello Jiri,
> > > > > >
> > > > > > After the following set of upstream commits, the user fails to attach
> > > > > > a bond to the bridge, if the user creates the bond with two interfaces
> > > > > > from different bnxt_en NICs. Previously bnxt_en driver does not
> > > > > > advertise the switch_id for legacy mode as part of
> > > > > > ndo_get_port_parent_id cb but with the following patches, switch_id is
> > > > > > returned even in legacy mode which is causing the failure.
> > > > > >
> > > > > > ---------------
> > > > > > 7e1146e8c10c00f859843817da8ecc5d902ea409 net: devlink: introduce
> > > > > > devlink_compat_switch_id_get() helper
> > > > > > 6605a226781eb1224c2dcf974a39eea11862b864 bnxt: pass switch ID through
> > > > > > devlink_port_attrs_set()
> > > > > > 56d9f4e8f70e6f47ad4da7640753cf95ae51a356 bnxt: remove
> > > > > > ndo_get_port_parent_id implementation for physical ports
> > > > > > ----------------
> > > > > >
> > > > > > As there is a plan to get rid of ndo_get_port_parent_id in future, I
> > > > > > think there is a need to fix devlink_compat_switch_id_get() to return
> > > > > > the switch_id only when device is in SWITCHDEV mode and this effects
> > > > > > all the NICs.
> > > > > >
> > > > > > Please let me know your thoughts. Thank you.
> > > > >
> > > > > I'm not Jiri, but I'd think that hiding switch_id from devices should
> > > > > not be the solution here. Especially that no NICs offload bridging
> > > > > today.
> > > > >
> > > > > Could you describe the team/bridge failure in detail, I'm not that
> > > > > familiar with this code.
> > > >
> > > > Maybe:
> > > >
> > > > br_add_slave()
> > > >   br_add_if()
> > > >           nbp_switchdev_mark_set()
> > > >                   dev_get_port_parent_id()
> > > >
> > > > I believe the last call will return '-ENODATA' because the two bnxt
> > > > netdevs member in the bond have different switch IDs. Perhaps the
> > > > function can be changed to return '-EOPNOTSUPP' when it's called for an
> > > > upper device that have multiple parent IDs beneath it:
> > > >
> > > > diff --git a/net/core/dev.c b/net/core/dev.c
> > > > index d42c9ea0c3c0..7932594ca437 100644
> > > > --- a/net/core/dev.c
> > > > +++ b/net/core/dev.c
> > > > @@ -8646,7 +8646,7 @@ int dev_get_port_parent_id(struct net_device *dev,
> > > >                  if (!first.id_len)
> > > >                          first = *ppid;
> > > >                  else if (memcmp(&first, ppid, sizeof(*ppid)))
> > > > -                       return -ENODATA;
> > > > +                       return -EOPNOTSUPP;
> > > >          }
> > > >          return err;
> > >
> > > LGTM, or we could make bridge ignore ENODATA (in case the distinctions
> > > is useful?)
> > >
> > > I was searching for the early versions of Florian's patch set but
> > > I can't find it :( Florian, do you remember if there was a reason to
> > > fail bridge in this case?
> >
> > v3: https://patchwork.kernel.org/patch/10798697/
> > v2: https://lore.kernel.org/patchwork/patch/1038907/
> > v1:
> > https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1921358.html
> >
> > I went back to check the tree before
> > d6abc5969463359c366d459247b90366fcd6f5c5 and the logic for return -ENODATA
> > was copied from switchdev_port_attr_get():
> >
> > ...
> >           /* Switch device port(s) may be stacked under
> >            * bond/team/vlan dev, so recurse down to get attr on
> >            * each port.  Return -ENODATA if attr values don't
> >            * compare across ports.
> >            */
> >
> >           netdev_for_each_lower_dev(dev, lower_dev, iter) {
> >                   err = switchdev_port_attr_get(lower_dev, attr);
> >                   if (err)
> >                           break;
> >                   if (first.id == SWITCHDEV_ATTR_ID_UNDEFINED)
> >                           first = *attr;
> >                   else if (memcmp(&first, attr, sizeof(*attr)))
> >                           return -ENODATA;
> >           }
> >
> >           return err;
> >   }
> >   EXPORT_SYMBOL_GPL(switchdev_port_attr_get);
> >
> > the bridge code would specifically treat -EOPNOTSUPP as success and return
> > early, whereas other error code would be treated as a failure.
> >
> > Jiri or Ido, do you remember the reason for return -ENODATA here?
>
> I don't know about the past, but I checked all the current callers of
> dev_get_port_parent_id() and I think the proposed change should be OK:
>
> 1. nbp_switchdev_mark_set(): Current use case. Does not seem to be a
> problem
>
> 2. dev_get_port_parent_id(): Recursive call
>
> 3. netdev_port_same_parent_id(): Unaffected by this change
>
> 4. phys_switch_id_show(): Likewise. Does not recurse
>
> 5. rtnl_phys_switch_id_fill(): Likewise
>
> 6. vif_add: Does not check the error code
>
> I can test the patch in our regression and submit later this week unless
> you have a better suggestion. Please let me know.
Thanks, sounds good.

>
> Thanks

Re: Failing to attach bond(created with two interfaces from different NICs) to a bridge

[Vasundhara Volam]

On Fri, Sep 4, 2020 at 12:44 AM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Thu, 3 Sep 2020 12:52:25 +0530 Vasundhara Volam wrote:
> > Hello Jiri,
> >
> > After the following set of upstream commits, the user fails to attach
> > a bond to the bridge, if the user creates the bond with two interfaces
> > from different bnxt_en NICs. Previously bnxt_en driver does not
> > advertise the switch_id for legacy mode as part of
> > ndo_get_port_parent_id cb but with the following patches, switch_id is
> > returned even in legacy mode which is causing the failure.
> >
> > ---------------
> > 7e1146e8c10c00f859843817da8ecc5d902ea409 net: devlink: introduce
> > devlink_compat_switch_id_get() helper
> > 6605a226781eb1224c2dcf974a39eea11862b864 bnxt: pass switch ID through
> > devlink_port_attrs_set()
> > 56d9f4e8f70e6f47ad4da7640753cf95ae51a356 bnxt: remove
> > ndo_get_port_parent_id implementation for physical ports
> > ----------------
> >
> > As there is a plan to get rid of ndo_get_port_parent_id in future, I
> > think there is a need to fix devlink_compat_switch_id_get() to return
> > the switch_id only when device is in SWITCHDEV mode and this effects
> > all the NICs.
> >
> > Please let me know your thoughts. Thank you.
>
> I'm not Jiri, but I'd think that hiding switch_id from devices should
> not be the solution here. Especially that no NICs offload bridging
> today.
>
> Could you describe the team/bridge failure in detail, I'm not that
> familiar with this code.

I am copying the Redhat kernel team analysis in crisp to clarify. Hope
this helps.

----
When a new bridge port is being set up in br_add_if(),
nbp_switchdev_mark_set() is called to get the Switch ID of the new
port (in this case, the bond). Prior to devlink_compat_switch_id()
changes, the bond's lower devs (the bnxt ports) do not report a Switch
ID (EOPNOTSUPP) when device is not in SWITCHDEV mode, so this activity
is moot.

Prior to the devlink_compat_switch_id() changes,
switchdev_port_attr_get() is set and points to
bnxt_swdev_port_attr_get(). It simply calls bnxt_port_attr_get():

8358 static const struct switchdev_ops bnxt_switchdev_ops = {
8359         .switchdev_port_attr_get        = bnxt_swdev_port_attr_get
8360 };

8352 static int bnxt_swdev_port_attr_get(struct net_device *dev,
8353                                     struct switchdev_attr *attr)
8354 {
8355         return bnxt_port_attr_get(netdev_priv(dev), attr);
8356 }

- Basically, bnxt_port_attr_get() is either going to copy the switch
ID into the passed-in switchdev_attr struct OR return -EOPNOTSUPP for
legacy mode:

8332 int bnxt_port_attr_get(struct bnxt *bp, struct switchdev_attr *attr)
8333 {
8334         if (bp->eswitch_mode != DEVLINK_ESWITCH_MODE_SWITCHDEV)
8335                 return -EOPNOTSUPP;
8336
8337         /* The PF and it's VF-reps only support the switchdev framework */
8338         if (!BNXT_PF(bp))
8339                 return -EOPNOTSUPP;
8340
8341         switch (attr->id) {
8342         case SWITCHDEV_ATTR_ID_PORT_PARENT_ID:
8343                 attr->u.ppid.id_len = sizeof(bp->switch_id);
8344                 memcpy(attr->u.ppid.id, bp->switch_id,
attr->u.ppid.id_len);
8345                 break;
8346         default:
8347                 return -EOPNOTSUPP;
8348         }
8349         return 0;
8350 }


However, now the bnxt driver provides an ID via its new
ndo_get_devlink_port() handler. Logic in dev_get_port_parent_id()
returns ENODATA if the bond's ports do not all have the same switch
identifier (here, phys_switch_id).

Now nbp_switchdev_mark_set() only uses dev_get_port_parent_id(), which
calls devlink_compat_switch_id_get and when it comes to the bnxt
devices the call to devlink_compat_switch_id_get() actually returns a
useful value. But of course, the Switch ID of two physically separate
cards is not expected to be the same so the overall result is the
ENODATA and the bond is failed to attach to the bridge..
---------

I am typing the above and saw mail from Ido Schimmel in parallel.

Thanks,
Vasundhara