Netdev Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Dario Alcocer <dalcocer@helixd.com>
To: Andrew Lunn <andrew@lunn.ch>
Cc: netdev@vger.kernel.org
Subject: Re: Marvell switch port shows LOWERLAYERDOWN, ping fails
Date: Mon, 26 Jul 2021 18:39:58 -0700	[thread overview]
Message-ID: <527bcc43-d99c-f86e-29b0-2b4773226e38@helixd.com> (raw)
In-Reply-To: <bcd589bd-eeb4-478c-127b-13f613fdfebc@helixd.com>

On 7/24/21 7:36 PM, Dario Alcocer wrote:
> On 7/24/21 7:26 PM, Dario Alcocer wrote:
>> On 7/24/21 10:34 AM, Andrew Lunn wrote:
>>> You might want to enable dbg prints in driver/nets/phy/phy.c, so you
>>> can see the state machine changes.
>>
>> Great suggestion. I added the following to the boot options:
>>
>> dyndbg="file net/dsa/* +p; file drivers/net/phy/phy.c +p"
>>
>> The relevant messages collected from the system log are below. 
>> Interestingly, all of the ports go from UP to NOLINK. In addition, 
>> "breaking chain for DSA event 7" is reported, once for each port.

Andrew,

As I mentioned before, the system log shows that each switch port link 
state goes from UP to NOLINK when "ip link set DEVNAME up" runs.

Since you suggested it's probably a PHY problem, I used kernel tracing 
to track PHY-related calls in the following files:

* drivers/net/phy/phylink.c
* drivers/net/phy/marvell.c
* net/dsa/port.c

I set up the kernel tracing before trying to bring up the lan1 interface:

root@dali:~# mount -t tracefs tracefs /sys/kernel/tracing
root@dali:~# echo 0 > /sys/kernel/tracing/tracing_on
root@dali:~# echo function_graph > /sys/kernel/tracing/current_tracer
root@dali:~# echo phylink_\* > /sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo dsa_port_phylink_\* >> 
/sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo m88e1510_probe >> /sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo marvell_config_init >> 
/sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo m88e1510_config_aneg >> 
/sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo marvell_read_status >> 
/sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo marvell_ack_interrupt >> 
/sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo marvell_config_intr >> 
/sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo m88e1121_did_interrupt >> 
/sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo genphy_resume >> /sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo genphy_suspend >> /sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo marvell_read_page >> /sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo marvell_write_page >> 
/sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo marvell_get_sset_count >> 
/sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo marvell_get_strings >> 
/sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo m88e1540_get_tunable >> 
/sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo m88e1540_set_tunable >> 
/sys/kernel/tracing/set_ftrace_filter
root@dali:~# echo 1 > /sys/kernel/tracing/tracing_on

I then tried bringing up lan1 again to see which functions would be 
called, then stopped kernel tracing:

root@dali:~# ip addr add 192.0.2.1/24 dev lan1
root@dali:~# ip link set lan1 up
[  511.763909] mv88e6085 stmmac-0:1a lan1: configuring for phy/gmii link 
mode
[  511.773082] 8021q: adding VLAN 0 to HW filter on device lan1
root@dali:~# echo 0 > /sys/kernel/tracing/tracing_on

I then dumped the trace buffer:

root@dali:~# cat /sys/kernel/tracing/trace
# tracer: function_graph
#
# CPU  DURATION                  FUNCTION CALLS
# |     |   |                     |   |   |   |
  1) + 10.900 us   |  phylink_ethtool_ksettings_get();
  1)   5.220 us    |  phylink_ethtool_ksettings_get();
  1)               |  phylink_ethtool_ksettings_get() {
  1)   2.060 us    |    phylink_get_fixed_state();
  1)   1.890 us    |    phylink_get_ksettings();
  1) + 11.740 us   |  }
  1)   6.760 us    |  phylink_ethtool_ksettings_get();
  1)   4.890 us    |  phylink_ethtool_ksettings_get();
  1)   4.670 us    |  phylink_ethtool_ksettings_get();
  1) + 11.560 us   |  phylink_ethtool_ksettings_get();
  1)   5.060 us    |  phylink_ethtool_ksettings_get();
  1)               |  phylink_ethtool_ksettings_get() {
  1)   2.010 us    |    phylink_get_fixed_state();
  1)   1.810 us    |    phylink_get_ksettings();
  1) + 12.140 us   |  }
  1)   5.890 us    |  phylink_ethtool_ksettings_get();
  1)   5.300 us    |  phylink_ethtool_ksettings_get();
  1)   4.630 us    |  phylink_ethtool_ksettings_get();
  1) + 10.070 us   |  phylink_ethtool_ksettings_get();
  1)   5.100 us    |  phylink_ethtool_ksettings_get();
  1)               |  phylink_ethtool_ksettings_get() {
  1)   1.910 us    |    phylink_get_fixed_state();
  1)   1.840 us    |    phylink_get_ksettings();
  1) + 12.120 us   |  }
  1)   5.910 us    |  phylink_ethtool_ksettings_get();
  1)   5.410 us    |  phylink_ethtool_ksettings_get();
  1)   4.630 us    |  phylink_ethtool_ksettings_get();
  1) + 10.560 us   |  phylink_ethtool_ksettings_get();
  1)   4.620 us    |  phylink_ethtool_ksettings_get();
  1)               |  phylink_ethtool_ksettings_get() {
  1)   1.920 us    |    phylink_get_fixed_state();
  1)   1.930 us    |    phylink_get_ksettings();
  1) + 12.370 us   |  }
  1)   5.290 us    |  phylink_ethtool_ksettings_get();
  1)   4.570 us    |  phylink_ethtool_ksettings_get();
  1)   4.500 us    |  phylink_ethtool_ksettings_get();
  0)               |  phylink_start() {
  0)   2.510 us    |    phylink_resolve_flow();
  0)               |    phylink_mac_config() {
  0)   2.260 us    |      dsa_port_phylink_mac_config();
  0)   5.830 us    |    }
  0) + 14.740 us   |    phylink_run_resolve.part.0();
  0)               |    genphy_resume() {
  ------------------------------------------
  0)     ip-626     =>   kworker-20
  ------------------------------------------

  0)               |  phylink_resolve() {
  0)   1.860 us    |    phylink_resolve_flow();
  0)   6.800 us    |  }
  ------------------------------------------
  0)   kworker-20   =>     ip-626
  ------------------------------------------

  0) # 1818.330 us |    } /* genphy_resume */
  0) # 8730.320 us |  } /* phylink_start */
  ------------------------------------------
  0)     ip-626     =>   kworker-20
  ------------------------------------------

  0)               |  m88e1510_config_aneg() {
  0) # 1871.880 us |    marvell_read_page();
  0) # 1807.480 us |    marvell_write_page();
  0) # 1804.300 us |    marvell_write_page();
  0) * 42803.07 us |  }
  0) * 13283.95 us |  marvell_read_status();
  0)               |  phylink_phy_change() {
  0)   3.990 us    |    phylink_run_resolve.part.0();
  0)   8.930 us    |  }
  0)               |  phylink_resolve() {
  0)   2.110 us    |    phylink_resolve_flow();
  0)   6.320 us    |  }
  1) + 11.280 us   |  phylink_ethtool_ksettings_get();
  1)   4.560 us    |  phylink_ethtool_ksettings_get();
  1)               |  phylink_ethtool_ksettings_get() {
  1)   2.180 us    |    phylink_get_fixed_state();
  1)   1.960 us    |    phylink_get_ksettings();
  1) + 12.620 us   |  }
  1)   5.290 us    |  phylink_ethtool_ksettings_get();
  1)   4.980 us    |  phylink_ethtool_ksettings_get();
  1)   5.000 us    |  phylink_ethtool_ksettings_get();
  1) + 10.430 us   |  phylink_ethtool_ksettings_get();
  1)   4.860 us    |  phylink_ethtool_ksettings_get();
  1)               |  phylink_ethtool_ksettings_get() {
  1)   2.070 us    |    phylink_get_fixed_state();
  1)   1.950 us    |    phylink_get_ksettings();
  1) + 12.090 us   |  }
  1)   5.310 us    |  phylink_ethtool_ksettings_get();
  1)   5.110 us    |  phylink_ethtool_ksettings_get();
  1)   4.730 us    |  phylink_ethtool_ksettings_get();

I filtered the output to see which specific calls are made when checking 
the port link status:

root@dali:~# cat /sys/kernel/tracing/trace | fgrep '|' | cut -d '|' -f 2 
| sort | uniq

       dsa_port_phylink_mac_config();
     genphy_resume() {
     marvell_read_page();
     marvell_write_page();
     phylink_get_fixed_state();
     phylink_get_ksettings();
     phylink_mac_config() {
     phylink_resolve_flow();
     phylink_run_resolve.part.0();
     }
     } /* genphy_resume */
   m88e1510_config_aneg() {
   marvell_read_status();
   phylink_ethtool_ksettings_get() {
   phylink_ethtool_ksettings_get();
   phylink_phy_change() {
   phylink_resolve() {
   phylink_start() {
   }
   } /* phylink_start */
root@dali:~#

I will focus on adding more tracing to these specific functions, in 
hopes of narrowing down the link issue further.

Let me know if you have any other suggestions, in case I missed something.

Thanks!


  reply	other threads:[~2021-07-27  1:40 UTC|newest]

Thread overview: 27+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-07-22 22:55 Dario Alcocer
2021-07-23 13:41 ` Andrew Lunn
2021-07-23 18:21   ` Dario Alcocer
2021-07-23 18:25     ` Andrew Lunn
2021-07-23 18:36       ` Dario Alcocer
2021-07-23 21:58         ` Dario Alcocer
2021-07-24 17:34           ` Andrew Lunn
2021-07-25  2:26             ` Dario Alcocer
2021-07-25  2:36               ` Dario Alcocer
2021-07-27  1:39                 ` Dario Alcocer [this message]
2021-07-28 18:07                   ` Dario Alcocer
2021-07-28 18:23                     ` Andrew Lunn
2021-07-28 18:33                       ` Dario Alcocer
2021-07-28 19:24                         ` Andrew Lunn
2021-07-28 19:37                           ` Dario Alcocer
2021-07-28 20:47                             ` Andrew Lunn
2021-07-28 20:54                               ` Dario Alcocer
2021-07-28 21:09                             ` Andrew Lunn
2021-08-05 21:44                             ` Dario Alcocer
2021-08-06 16:03                               ` Dario Alcocer
2021-08-06 23:46                                 ` Dario Alcocer
2021-08-07 18:57                                   ` Andrew Lunn
2021-08-09 16:28                                     ` Dario Alcocer
2021-08-10 20:58                                       ` Dario Alcocer
2021-08-10 22:13                                         ` Andrew Lunn
2021-08-11 13:16                                           ` Dario Alcocer
2021-08-07 17:44                               ` Andrew Lunn

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=527bcc43-d99c-f86e-29b0-2b4773226e38@helixd.com \
    --to=dalcocer@helixd.com \
    --cc=andrew@lunn.ch \
    --cc=netdev@vger.kernel.org \
    --subject='Re: Marvell switch port shows LOWERLAYERDOWN, ping fails' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).