Re: Failing to attach bond(created with two interfaces from different NICs) to a bridge

[Ido Schimmel <idosch@idosch.org>]

On Sun, Sep 06, 2020 at 12:23:23PM -0700, Florian Fainelli wrote:
> 
> 
> On 9/6/2020 10:13 AM, Jakub Kicinski wrote:
> > On Sun, 6 Sep 2020 14:12:49 +0300 Ido Schimmel wrote:
> > > On Thu, Sep 03, 2020 at 12:14:28PM -0700, Jakub Kicinski wrote:
> > > > On Thu, 3 Sep 2020 12:52:25 +0530 Vasundhara Volam wrote:
> > > > > Hello Jiri,
> > > > > 
> > > > > After the following set of upstream commits, the user fails to attach
> > > > > a bond to the bridge, if the user creates the bond with two interfaces
> > > > > from different bnxt_en NICs. Previously bnxt_en driver does not
> > > > > advertise the switch_id for legacy mode as part of
> > > > > ndo_get_port_parent_id cb but with the following patches, switch_id is
> > > > > returned even in legacy mode which is causing the failure.
> > > > > 
> > > > > ---------------
> > > > > 7e1146e8c10c00f859843817da8ecc5d902ea409 net: devlink: introduce
> > > > > devlink_compat_switch_id_get() helper
> > > > > 6605a226781eb1224c2dcf974a39eea11862b864 bnxt: pass switch ID through
> > > > > devlink_port_attrs_set()
> > > > > 56d9f4e8f70e6f47ad4da7640753cf95ae51a356 bnxt: remove
> > > > > ndo_get_port_parent_id implementation for physical ports
> > > > > ----------------
> > > > > 
> > > > > As there is a plan to get rid of ndo_get_port_parent_id in future, I
> > > > > think there is a need to fix devlink_compat_switch_id_get() to return
> > > > > the switch_id only when device is in SWITCHDEV mode and this effects
> > > > > all the NICs.
> > > > > 
> > > > > Please let me know your thoughts. Thank you.
> > > > 
> > > > I'm not Jiri, but I'd think that hiding switch_id from devices should
> > > > not be the solution here. Especially that no NICs offload bridging
> > > > today.
> > > > 
> > > > Could you describe the team/bridge failure in detail, I'm not that
> > > > familiar with this code.
> > > 
> > > Maybe:
> > > 
> > > br_add_slave()
> > > 	br_add_if()
> > > 		nbp_switchdev_mark_set()
> > > 			dev_get_port_parent_id()
> > > 
> > > I believe the last call will return '-ENODATA' because the two bnxt
> > > netdevs member in the bond have different switch IDs. Perhaps the
> > > function can be changed to return '-EOPNOTSUPP' when it's called for an
> > > upper device that have multiple parent IDs beneath it:
> > > 
> > > diff --git a/net/core/dev.c b/net/core/dev.c
> > > index d42c9ea0c3c0..7932594ca437 100644
> > > --- a/net/core/dev.c
> > > +++ b/net/core/dev.c
> > > @@ -8646,7 +8646,7 @@ int dev_get_port_parent_id(struct net_device *dev,
> > >                  if (!first.id_len)
> > >                          first = *ppid;
> > >                  else if (memcmp(&first, ppid, sizeof(*ppid)))
> > > -                       return -ENODATA;
> > > +                       return -EOPNOTSUPP;
> > >          }
> > >          return err;
> > 
> > LGTM, or we could make bridge ignore ENODATA (in case the distinctions
> > is useful?)
> > 
> > I was searching for the early versions of Florian's patch set but
> > I can't find it :( Florian, do you remember if there was a reason to
> > fail bridge in this case?
> 
> v3: https://patchwork.kernel.org/patch/10798697/
> v2: https://lore.kernel.org/patchwork/patch/1038907/
> v1:
> https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1921358.html
> 
> I went back to check the tree before
> d6abc5969463359c366d459247b90366fcd6f5c5 and the logic for return -ENODATA
> was copied from switchdev_port_attr_get():
> 
> ...
>           /* Switch device port(s) may be stacked under
>            * bond/team/vlan dev, so recurse down to get attr on
>            * each port.  Return -ENODATA if attr values don't
>            * compare across ports.
>            */
> 
>           netdev_for_each_lower_dev(dev, lower_dev, iter) {
>                   err = switchdev_port_attr_get(lower_dev, attr);
>                   if (err)
>                           break;
>                   if (first.id == SWITCHDEV_ATTR_ID_UNDEFINED)
>                           first = *attr;
>                   else if (memcmp(&first, attr, sizeof(*attr)))
>                           return -ENODATA;
>           }
> 
>           return err;
>   }
>   EXPORT_SYMBOL_GPL(switchdev_port_attr_get);
> 
> the bridge code would specifically treat -EOPNOTSUPP as success and return
> early, whereas other error code would be treated as a failure.
> 
> Jiri or Ido, do you remember the reason for return -ENODATA here?

I don't know about the past, but I checked all the current callers of
dev_get_port_parent_id() and I think the proposed change should be OK:

1. nbp_switchdev_mark_set(): Current use case. Does not seem to be a
problem

2. dev_get_port_parent_id(): Recursive call

3. netdev_port_same_parent_id(): Unaffected by this change

4. phys_switch_id_show(): Likewise. Does not recurse

5. rtnl_phys_switch_id_fill(): Likewise

6. vif_add: Does not check the error code

I can test the patch in our regression and submit later this week unless
you have a better suggestion. Please let me know.

Thanks