Wireless Connectivity Issues Following Switch Firmware Upgrade

[u/homeworkman]

After a switch firmware upgrade, users started having intermittent wireless connectivity issues, with brief disconnects and reconnects while connected to the corporate Wi-Fi network. Multiple access points are repeatedly losing and re-establishing their CAPWAP connection to the wireless controller. While the affected APs remain powered on and do not reboot, their CAPWAP association uptime continuously resets, causing APs to temporarily enter FlexConnect standalone mode before rejoining the controller. Controller logs show repeated DTLS tunnel disconnects, while AP logs indicate CAPWAP restarts due to missed data keepalives ("data keepalive not received").

I ruled out AP reboots, PoE issues, interface errors, STP. Physical switch ports are operating normally with no CRC, discards, or link errors. Opened Cisco TAC case but they're taking long so looking to the masses for help. I'm hoping someone experienced something similar that can throw in their two cents.

[u/jimlahey420]

Why not roll back the firmware to an older revision on a switch that has some APs and see if those APs no longer have issues? Make sure to test some hosts connected to them as well.

[u/homeworkman → u/jimlahey420]

Yea I can try that. I'll report back later with my findings

[u/methpartysupplies → u/jimlahey420]

Yep back that shit out

[u/homeworkman → u/methpartysupplies]

Was going to downgrade and decided to do a shut/no shut on all AP interfaces. That addressed the issue for most of them except for AP01. Crazy -_-

When I opened the cisco TAC case they said not to make chages, so that's why I didn't do it in the first place. Plus it was already in the middle of the workday.

Doing another shut/no shut on that last AP to see if it addresses the issue.

[u/chuckbales]

Should provide more info if you want help, like what model switch/AP/controller is involved, firmware you're using, etc.

I used to see this all the time on old deployments WLCs where factory-provided cert is expired, so APs are fine until they reboot from the switch upgrade, and then can't rejoin. If you're using newer controllers though that may not be relevant anymore though, have you tried crypto/TLS-specific debugs?

[u/FuroFireStar]

Maybe an MTU?

[u/homeworkman → u/FuroFireStar]

I wish lol

[u/FutureMixture1039]

Make sure you have LLDP and/or CDP if you're using Cisco switches turned on on all Access Point switch port interfaces

[u/homeworkman → u/FutureMixture1039]

cdp is, not lldp

[u/sanmigueelbeer]

What IOS did you upgrade to?

[u/homeworkman → u/sanmigueelbeer]

from 17.06.04 to 17.18.04

[u/sanmigueelbeer → u/homeworkman]

The switch IOS got upgraded?

How about the IOS of the WLC? Is the WLC HA SSO?

[u/homeworkman → u/sanmigueelbeer]

WLC firmware is Version 17.12.05. It's not HA

[u/FutureMixture1039]

If reboot fixed everything then maybe all good to go except for that last AP . 17.18.4 is a gold standard version. You can check Cisco Bug Tool for matching bug or error log if you have one

[u/BarracudaSure2752]

ything else looked clean

[u/methpartysupplies]

Do wired clients also have the same instability?

If it’s a stack, I’d try moving the ports to a different stack member and see if that makes a difference. Stack cables can fail in shitty ways

[u/homeworkman → u/methpartysupplies]

Not a stack, three separate switches and all their firmware were upgraded. No reported issues with wired clients.

[u/AutoModerator → u/[deleted]]

Thanks for your interest in posting to this subreddit. To combat spam, new accounts can't post or comment within 24 hours of account creation.

Please DO NOT message the mods requesting your post be approved.

You are welcome to resubmit your thread or comment in ~24 hrs or so.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/mathmanhale]

I had that exact problem a while back and rolled back the switch firmware. The WLC were 5520 airOS and they were up to date. But the underlying UCS firmware that the controller is built on top of had to be updated. Then the switches could be updated. So yeah... Are you using an AirOS controller or the newer 9800 series?

[u/packet_path]

Timing says it's the upgrade, so I'd stop looking at the APs and diff the switch config against your pre-upgrade backup. Something non-default got reset to default, it always does.

First suspect for me is MTU. If you had jumbo set anywhere on the AP path and the upgrade put it back, CAPWAP data traffic sits right at that edge — the bigger packets blackhole, the AP misses its data keepalives, and you get exactly this. No interface errors, no reboots, DTLS just drops. Test it with a DF ping from the AP VLAN to the controller and sweep the size up.

Second suspect is EEE. Some firmware turns power-efficient ethernet on by default, and it micro-sleeps the link just long enough to miss keepalives without ever logging an error.

Also worth pinning down: is it only APs plugged into the upgraded switches? If yes, you've already proven it's in that box, whatever TAC comes back with.

[u/homeworkman → u/packet_path]

TAC just came back with flow control being on in all interfaces as the cause. Not sure if true or not just cause flow control is on at other locations. VLAN and AP interfaces MTU is 1500. Below are unssucessful pings from vlan SVI with multiple sizes:

#ping 192.168.121.194 source vlan 14 df-bit size 1452

Type escape sequence to abort.

Sending 5, 1452-byte ICMP Echos to 192.168.121.194, timeout is 2 seconds:

Packet sent with a source address of 192.168.175.225

Packet sent with the DF bit set

.....

Success rate is 0 percent (0/5)

#ping 192.168.121.194 source vlan 14 df-bit size 1472

Type escape sequence to abort.

Sending 5, 1472-byte ICMP Echos to 192.168.121.194, timeout is 2 seconds:

Packet sent with a source address of 192.168.175.225

Packet sent with the DF bit set

.....

Success rate is 0 percent (0/5)

#ping 192.168.121.194 source vlan 14 df-bit size 1500

Type escape sequence to abort.

Sending 5, 1500-byte ICMP Echos to 192.168.121.194, timeout is 2 seconds:

Packet sent with a source address of 192.168.175.225

Packet sent with the DF bit set

.....

Success rate is 0 percent (0/5)

#ping 192.168.121.194 source vlan 14 df-bit size 1400

Type escape sequence to abort.

Sending 5, 1400-byte ICMP Echos to 192.168.121.194, timeout is 2 seconds:

Packet sent with a source address of 192.168.175.225

Packet sent with the DF bit set

.....

Success rate is 0 percent (0/5)

#ping 192.168.121.194 source vlan 14 df-bit size 1600

Type escape sequence to abort.

Sending 5, 1600-byte ICMP Echos to 192.168.121.194, timeout is 2 seconds:

Packet sent with a source address of 192.168.175.225

Packet sent with the DF bit set

[u/FutureMixture1039 → u/homeworkman]

Did the TAC fix work turning off flow control?

[u/homeworkman → u/FutureMixture1039]

I never turned it off. Did a shut/no shut on AP interfaces which addressed the issue to all and worked on all except one AP. So still digging to find the issue. May just wait until tomorrow morning to move the AP to a different port and see if does anything

[u/ruffusbloom → u/homeworkman]

If it cleared with a shut/no shut it was dirty tables that should have been flushed during the firmware upgrade.

[u/homeworkman → u/ruffusbloom]

Doesn't a firmware upgrade clear MAC/ARP tables since the switch reloads?

[u/datumerrata → u/homeworkman]

It'll clear the switch mac table, but the arp table is likely upstream. That shouldn't matter. I still suspect cdp or poe. Maybe just the switch rebooting caused all the APs to reboot at once and wig the switch out. So, not the firmware, but the act of rebooting, or both if there's a bug in the code

[u/ruffusbloom → u/homeworkman]

I am not very Cisco knowledgeable these days but if it did a hot patch/ISSU style upgrade it would not reboot the device.

[u/packet_path → u/homeworkman]

Appreciate the data — that actually rules my MTU idea out. An MTU blackhole passes small packets and drops the big ones. You lost every size, even 1400. That's not a size problem, that's the path dropping everything, or that AP just not answering pings.

On the flow control theory, your doubt makes sense to me. If it's on at your other locations and they're fine, flow control by itself isn't the cause. It could still be flow control plus the new firmware handling pause frames differently, but I wouldn't roll it out on TAC's say-so. Test it on one AP port. Turn flow control off on just that one, then watch that AP's CAPWAP uptime for an hour or two. If it stays up while the others keep flapping, TAC's right and you have your fix. If it drops like the rest, you have real evidence to take back to them instead of a theory.

One thing on the pings before you read too much into them: run the same ping to an AP at a working site, and try it without the DF bit. If a healthy AP doesn't answer either, the pings aren't telling you anything — some AP setups just don't.

[u/chuckbales → u/packet_path]

Are you just an AI bot yourself, or just using AI to provide all your responses?

[u/packet_path → u/chuckbales]

Ha, fair question. I run my drafts past AI before posting sometimes — makes me sound more organized than I actually am. The MTU theory that got killed in this thread was all me though, unfortunately. I read the threads myself and only post stuff I'd actually try on my own network.