It's been a couple weeks now since I've had an outage, so I don't have any specific evidence to report.
I did make one change shortly after the last outage, which was to make sure that my DHCP client sends DHCPREQUESTs every couple hours or so.
Based on reading the Juniper docs and seeing that the Sonic router has never sent an ARP request, it seems clear that the Juniper router is snooping some set of packets to fill its ARP cache. I would have expected it to be snooping ARP packets, but that doesn't explain the failure mode I'm observing. It seems that it is, instead, snooping DHCP requests.
That hypothesis is backed up by these data points:
Jan 05 12:18:32 DHCPDISCOVER/DHCPOFFER/DHCPREQUEST/DHCPACK
Jan 05 12:25:07 Down (ARP requests go unanswered)
Jan 05 17:33:36 DHCPREQUEST/DHCPACK
Jan 05 17:33:36 Up (ARP requests answered again)
Also, I'm not sure what exactly the hypothetical DHCP snooping is doing. It's not just filling the ARP cache on the Juniper side, since the Juniper still sends IPv4 traffic to me even when it stops answering ACKs. I'm guessing that there's an entry in some interior routing protocol that's falling out without sufficient DHCP requests coming in. And that's stopping even my ARP requests from making it to the IP next hop.
In any event, it clearly manifests as a bug on the Sonic side to stop answering ARP requests, regardless of how long it's been since the last DHCP request. (DHCP is supposed to layer above ARP.)
I did make one change shortly after the last outage, which was to make sure that my DHCP client sends DHCPREQUESTs every couple hours or so.
Based on reading the Juniper docs and seeing that the Sonic router has never sent an ARP request, it seems clear that the Juniper router is snooping some set of packets to fill its ARP cache. I would have expected it to be snooping ARP packets, but that doesn't explain the failure mode I'm observing. It seems that it is, instead, snooping DHCP requests.
That hypothesis is backed up by these data points:
- My router's ARP requests continue, but the Sonic router stops answering them at some point.
- My router sending a DHCP request causes the Sonic router to answer ARP requests again.
- If my router sends DHCP renewal requests every two to three hours, the problem seems not to happen. Maybe.
Jan 05 12:18:32 DHCPDISCOVER/DHCPOFFER/DHCPREQUEST/DHCPACK
Jan 05 12:25:07 Down (ARP requests go unanswered)
Jan 05 17:33:36 DHCPREQUEST/DHCPACK
Jan 05 17:33:36 Up (ARP requests answered again)
Also, I'm not sure what exactly the hypothetical DHCP snooping is doing. It's not just filling the ARP cache on the Juniper side, since the Juniper still sends IPv4 traffic to me even when it stops answering ACKs. I'm guessing that there's an entry in some interior routing protocol that's falling out without sufficient DHCP requests coming in. And that's stopping even my ARP requests from making it to the IP next hop.
In any event, it clearly manifests as a bug on the Sonic side to stop answering ARP requests, regardless of how long it's been since the last DHCP request. (DHCP is supposed to layer above ARP.)
Statistics: Posted by gadams — Tue Feb 04, 2025 1:14 pm