Success or learning vs repeated failure
The successful among us are always either winning or learning.
The Failures among us keep repeating them over and over again.
The difference between success and failure? Learning from your mistakes.
The best lessons are the painful lessons we learn from personal experience.
The next best are the ones we learn from the mistakes of others.
Let's get the best of both worlds by taking personally the breaches that happen in our community so we can really feel an ownership of that pain and use it to motivate us.
That Pipeline company from a few years back (I'll redact the name, but I think everyone remembers)
This is a really great (and terrible) one.
It's almost hard to know where to begin. Let's begin at the end when everyone became aware of the problem. Gas prices were going up. Why? Supply chain issues. Why? REDACTED pipeline had to stop delivering fuel. Why? Ransomware in their billing system. Ouch. How did that all come to pass?
In the beginning
Hackers broke in through their VPN.
Oh man, that's rough. Anyone who has set up a VPN knows it's a pain. The good ones can cost so much they won't even tell you the price before you agree to sit through a sales call. The free ones are devilishly difficult to get right (anyone who has done it understands the pain of days troubleshooting iptables rules in openvpn), the automation is insufficient (and weeks scripting account setup and teardown), everyone's boss wants it delivered yesterday for negative eleven dollars, that one sales guy will literally stab you if you make his life any more difficult. Oh and once you get everything working, you need to touch every machine (hope you got mdm bro) and then brace yourself for the deluge of tech support calls real and imagined. (the website is down dude, is it the vpn?)
So maybe you take shortcuts. Everyone's done it. Well REDACTED Pipeline did. And there were quite a string of them. The IT guy set everyone with the same VPN password. Which was his password, which he reused elsewhere. It was probably his "I don't give a crap if other people know this" password. Right? Because he's handing it out to every employee at the company?
The password was of course leaked in a credential compromise from another company.
And then everything fell apart.
Let's stop for a moment and talk lessons.
So far this might just sound like a tragic tale of woe. But it's chock full of lessons.
Consider password policies.
This is exactly why they exist.
Because well-meaning people need the guidance of rules to know which corners not to cut.
A password policy might not seem like your most important thing in the world (I've had startup CEOs say we're delaying all security policies for a year, it's just not our priority) but it would have saved REDACTED Pipeline. I've heard arguments like "We trust our people to make proper judgment calls" Well not this IT guy, and he certainly should have known better. Again it was almost certainly the fault of the leader for failing to grant the time, budget or guidance to do the job right, but it also illustrates that without a password policy you can't expect people to do the right thing. And you certainly can't expect them to do the right thing in the face of conflicting obligations.
So that's lesson one. We’ll go into detail about what's in a good password policy a little later.
What's the next lesson? Probably that everyone needs an offboarding checklist.
When that IT person left the company someone should have looked at what he had access to and said "crap, we need to rotate a half dozen passwords INCLUDING THE SHARED VPN PASSWORD"
But since the reused and shared password leaked and was not rotated when the owner left, the hackers could just casually waltz in pretending to be legitimate users they got past the perimeter.
(Can you just imagine the glee on the hacker's face when he cross referenced an unrelated password dump and turned it into vpn access into the company's corp network? Did he shout the classic hacker line "I'm in!" or just sit there stupefied at how easy it was?)
Once the attackers were in on the flat corp network they took their time to find a juicy target (ok it probably took five minutes), decided on the machines in the finance department, pivoted, Crypto locked the machines and demanded ransom.
And then REDACTED Pipeline had to cease operations because they couldn't bill anyone for their gas anymore. I mean beside the concept that if your business is worth anything at all it's worth securing?
OK, but a lesson that is specific and actionable? Sure. Segment your network. Your network gear probably already supports it. Takes some time and testing, but is not super hard.
What if there was a different vlan for finance? And the only people who get access to it are the finance people? You can do this in a dumb way (dumb is fast and sometimes fast is way smarter than "I'll do this right when I have time") by assigning the specific ports and VPN accounts to the correct vlans. Or in "zero trust" way with 802.1x where machine certificates are pushed to each machine and vlans are assigned based on those.
How about lessons about ransomware in general? Sure Backups, and ransomware canaries.
Ok so let's refresh and dig in.
Prevent the initial break-in:
Create a password policy (you can take this one it can literally be this simple):
- Use a password manager because it makes everything below easier
- No password reuse. Ever.
- No shared passwords.
- Rotate passwords if they're shared in unencrypted channels
- Eliminate user access when they leave the company (this includes rotating any shared passwords, but you don't have any because #3 right?)
Create an offboarding checklist and use it
Let's pause here for a moment. If you have the above policies building the VPN like he did would absolutely not fly. Because it's directly in violation of a bunch of rules and furthermore is going to incur a massive amount of maintenance debt (boss, you want to rotate everyone's vpn password when I quit?) Having the above policies now guides the implementation of the VPN system. The way that is most compliant with the above? An IDP backed VPN. Easy setup, no password reuse, no shared passwords. Supports password rotation and 2FA. But it's a little more expensive. Except on the pro "side" it's compliant with the rules so it's obviously the best choice. But only in a world where those policies exist.
Creating the policies gives the junior people the guidance and ammunition to do the right thing even when their bosses are pushing for corner cutting and fast results.This is where the value of having good initial policies shines. It guides your future choices.
OK, but having a policy that no one follows is not super useful. And heck, someone needs to push for the policies to exist to begin with?
Who does it all?
Creating and enforcing the policy and internal audits.
- You gotta have someone in charge of security. They don't have to be spectacular, but it has to be their job. Pick your sharpest most pedantic IT person and staple a security engineer hat to his head. He or she can go Google for what needs to be in a Password policy. It's not rocket surgery, but someone has to actually own it.
Prevent the pivot
Segment your network with vlans
Segment your network. There's no reason under the sun that other departments need to reach Finance. If sharing needs to happen both hosts allowed inbound to a fileserver protects departments from each other. (It is possible to pivot in that environment, but it is harder). Bonus points if you get 802.1x certificate based vlan access working
Find your flaws before the baddies do
Security audits vs audit + remediate:
REDACTED Pipeline paid for a security audit just months before the attack. We can guarantee that the audit had some juicy TODOs. Had REDACTED Pipeline paid for remediation as well as a paper report, the top 5 here would have been done and the attack would never have happened.
So I suppose the lesson here is:
Never pay for a security audit that doesn't include remediation
Catching the baddies before they get ya
Stand up an intrusion detection system. You can buy a turnkey one that's quite reasonable. There are even a few free open source examples. (Remember that you pay for "free" in engineering time). Your security engineer will love dickering around with it, setting up signals of compromise, tracking down suspicious patterns. The day he finds something really suspicious he'll be so thrilled that it finally worked he'll forget to be scared that someone got in.
Hope for the best, plan for the worst
Telling your ransomware gang to get stuffed (Because Backups):
If you have good offsite backups you can tell your ransomware gang to go get stuffed. Isolate the intrusion, nuke the machines and restore from backup. Your IT staff is going to have a rea-lllly long night but you'll be back in business in a day. Slip them some overtime and a gift card and count yourself lucky you had the foresight to avoid a $1-4M ransom.
Stop it before it goes big
Huntress has a great ransomware canary. If your data is sensitive or your service is critical you might consider it.
Every control I mentioned above is a hallmark of a mature security program.
You can have them all for free or mostly free. None of it required an advanced degree (we just talked it through from a single breach report), None of it is even all that hard to implement or even all that expensive.
All it needs from us ( us meaning anyone who cares about security and privacy) is to take it personally, internalize the lessons learned and do better next time. The REDACTED Pipeline hack affected us all. Let's all take the painful lessons to heart and use that pain to move us forward.