The setting

New office, all brand new macbook pros (M2) and intermittent, non-deterministic wifi issues.

Most clients cannot connect to wifi
Some can connect and cannot route
Some can connect and disassociate after a minute or two
Some work fine and never experience any issues.

The wifi set-up was a brand new network on new hardware (meraki) which is where the issues started. After flipping every switch and turning every conceivable combination of options on and off again, we resorted to standing up a 2nd wifi network with different hardware (unifi). The new wifi network is behaving the same way. Users can connect to wifi but connection drops randomly, pages don’t load, nothing works, or sometimes it does. No rhyme or reason.

We started brainstorming and sweeping the area for offending wifi signals, rogue WAPs and even malicious actors doing some network poisoning or de-auth attacks. We did manage to find someone's bevvy machine broadcasting a hidden SSID with high strength. Its owner was pretty surprised when we told him precisely where in his office it was. But disabling that didn't do any good. The thing it ended up being was so silly we didn't even take it seriously at first.

You all know about airplay and airdrop right? Apple has these magical network behaviors that "just work". Well we all rightly mistrust magic now.

The fix

Ready for this? The issue wasn’t the wifi network at all. Those shiny new macbook pros are using a hidden network interface card for baked-in airplay and airdrop. Killing the airplay interface fixes the wifi issues instantly:

#sudo ifconfig awdl0 down

That line punched into the terminal fixes the issue. The craziest thing is that because this bug exists in a broadcast card, it tanks wifi for other devices nearby!

So that fixes it right? Nope. Clients report recurrence of the problem. It turns out Apple re-enables the interface every 15 minutes.

The cost

A week of our lives, a week of troubleshooting 12-18hrs a day (I was at the office hard wiring desks at 1:30AM the first night). Oh and flipping to ethernet is nontrivial because apple took our ethernet ports in 2015. We scoured the city for usbc to ethernet adapters.

A dozen highly trained people working on it. A dozen more requests for help to anyone we could think of to review our new network config, or sweep our environment for interference. A hundred people in our office were unproductive during opening week. Deals lost, millions of dollars in revenue left on the table. All because Apple screwed up their "magic" interface and as of this writing 8 days later still hasn't noticed or if they noticed, hasn't admitted it.

The scope

Are we alone? UCLA is having the same problem, as is Yahoo, as are a half dozen mac heavy shops in San Francisco. It's probably a global phenomenon. It is probably exacerbated by close contact to other apple users and our new office had close neighbors and quite a bit of wifi coverage to a busy street. Maybe we were the first and the worst, but in no way were we the only ones affected.