MSP Ignition! Podcast – Ransomware Costs HOW Much!?
In this episode, Axcient CISO, Joshua Foltz, sits down with Jeff to dive into the mind of a bad guy. He describes numerous ways that hackers can access your data, how to protect yourself, and the shocking cost of ransomware.
You can get the #MSPIgnition podcast on Apple Podcasts, Google Play, or stream it on the Axcient website by listening to the full audio version below 👇
[00:00:12] Jeff: Hello everyone, and welcome to another episode of the MSP Ignition! Podcast, brought to you by Axcient. It’s not about the backup, it’s about the recovery. Axcient, protect everything.
Be sure to subscribe to us on iTunes, and please give us a review while you’re there. You can also follow us on Twitter, Facebook, LinkedIn, and all that other social media stuff. I’m your host Jeff Cummings.
Today’s topic is so scary. I think it might’ve been more fitting to post this podcast on Halloween. Nefarious characters are now targeting MSPs, no longer settling for one single target, but the keys to a much larger kingdom by hacking you the MSPs. Who better to help us understand what MSPs need to do in order to protect themselves, but a security expert. That expert joining us today is Joshua Foltz. Thank you, for joining us, Joshua.
[00:01:02] Joshua: Thank you, Jeff.
[00:01:02] Jeff: How are you doing today, man?
[00:01:03] Joshua: I’m good.
[00:01:04] Jeff: You sound so stoic. Just like a security guy should. So serious. If you’re the guy that’s guarding the kingdom, you have to keep that straight face all the time off.
[00:01:11] Joshua: That’s my smiley face.
[00:01:12] Jeff: [laughs] Am I seeing you smile right now? That’s awesome. Also joining us today is Cline, [unintelligible 00:01:18]. One of my favorite wines. It’s Sarah. I’m pretty excited about that. Kelly’s actually joining us for a drink today too, huh, Kelly?
[00:01:26] Kelly: Hey y’all.
[00:01:26] Jeff: All right, Kelly, our producer, keeping us honest, and cutting all the bad stuff that I do, that you guys never get to hear. Let’s get right into it, Joshua. We’ve seen a lot in the news that now these hackers are really targeting MSPs. They know that when they get to the MSP, they get to the business. They get to more than one business. They get to all the businesses.
Something was interesting that when you and I were talking about this topic, I said, “Let’s call it, ‘what if you get hacked’ or ‘You’ve been hacked, now what?” You basically, said to me, “Chances are somebody’s already in, or trying to get in, and they don’t even realize it.” Is that accurate, or is that just what a security expert does to create a little security in their own job?
[00:02:11] Joshua: We talk about FUD all the time. Fear, uncertainty, and doubt. There’s definitely some of that, but what I find is that a lot of technical people are overconfident in their abilities, in their skills. They believe because they’re smart, and they’re technical, that they’re not subject to attacks. That is absolutely not the case. Some of the most smartest people I’ve known have had some of the worst breaches I’ve ever seen.
[00:02:34] Jeff: One analogy that I used recently was, the mechanic who’s owned cars always broken down, where MSPs are focused on providing security to their end clients and don’t think about the security that they themselves need to implement. Do you think that some of it?
[00:02:52] Joshua: I think that’s some of it. I think in addition to that, security is not IT. That’s a common misconception. I talk to executives all the time, I talked to MSPs. Just because you’re responsible to make sure machine runs a computer, or a network, or make sure phones are working, and things like that, it’s not the same skillset, as making sure it can’t be compromised. Hackers typically have a creative mindset. I’d probably put them more in the category of artists, than in technical people.
[00:03:22] Jeff: Oh, wow. Really?
[00:03:24] Joshua: What happens is, I always use the analogy that an IT person’s job is to clean a room, make it look nice, put everything in place. Then I come in, and I throw everything on the floor. I’m like, “I threw everything on the floor. Why did you let me do that?” That’s the difference between a security person and an IT person.
IT, you want order you want nice clean-cut lines. You want things the way they should be. You have rules you can follow. Security’s totally outside of all those bounds. It’s coming outside the lines and seeing what you can do with it. Computers don’t know how to respond to that. That’s why security vulnerabilities exist. You’re giving a computer something it does not know what to do with.
[00:04:06] Jeff: You talked a little bit about the RMM as being a– is that the spot you would target? Do you think like a hacker, or do you think like, “This is how I would get in?” You and I talked about that a little bit, but would that be your entry point? How would you get in?
[00:04:24] Joshua: Absolutely. I started in IT years ago as a sys admin and network admin. Very similar to a lot of our MSPs. We wore a lot of hats. We kept getting compromised all the time. That excited me. I started looking at how to do this. I started stimulating these different attacks and learning about him. That’s how I got started. Yes, absolutely.
We’re seeing attacks from these big nation states. Think of an army of 200 elite hackers going after one of our MSPs. Those are difficult odds to try to combat against. What I thought we’d do today is just walk through that.
If I were targeting an MSP, if I wanted to get to a customer system, or I wanted to get to all of their customers, what would I do? The most critical spot in any MSP is their RMM because I can do anything with that. I can change your passwords, I can steal every file on your system. I can do whatever I want in your environment, and I can even cover my tracks. Owning the RMM is the trophy or the gold standard, that we’ll talk about today.
There’s a thousand different ways to hack things and gather data, and create a breach, and things like that. I see these nation states going after the RMM when they’re attacking MSPs.
[00:05:42] Jeff: You think if you were a hacker, that’s where you would start?
[00:05:44] Joshua: Absolutely.
[00:05:44] Jeff: Okay. All right.
[00:05:45] Joshua: I’ll go for the gold, soon you’ll find on the way.
[00:05:47] Jeff: Yes, but if I’m an MSP, I’m thinking, “Hey man, I’m using an RMM that’s used by everybody and it’s world class and the name is really popular.” We won’t put any RMMs out there. “Dude, I’m fine. I’m protected.”
[00:06:01] Joshua: Yes. That’s not the case. If I were to attack an RMM, the very first thing I need to do is gain access to that RMM. A lot of MSPs will hide that behind a firewall. If you’re using a cloud-based firewall RMM, then it’s already done. I can already get to that login screen. That’s the first challenge I have to overcome in MSP scenario.
Let’s say, for example, that it’s inside an internal network, it’s behind a VPN, and we have good solid firewalls in place, good network security, and things like that.
[00:06:34] Jeff: Is that what you would typically see? Do you think that’s what you would typically see most of the time?
[00:06:38] Joshua: I think so. I think most environments are pretty good at their perimeter protection. They have firewalls in place. They’re behind a single IP address. Internal networks are separate from external. You’re not putting your RMM on the Internet, hopefully. You can, but I’ve certainly seen it. I think, for the most part, hiding it behind your gates, is the right approach. I think that’s what most MSPs are doing.
[00:07:06] Jeff: If you’re not doing that, then that’s–
[00:07:08] Joshua: That’d be step one.
[00:07:09] Jeff: That’s the step one.
[00:07:10] Joshua: Yes. Keep it away from me [crosstalk] a bad guy, because anytime I have access to an interface, there’s a lot that I can do with it. There’s different ways I can compromise that. We’ll go into those in a minute. If I had to get to it, that would mean that I have to be on your network. I can do that by compromising a VPN. I can do that by compromising your WiFi, or I can do that by compromising a back door that your admins may have done. It’s really, really common for an administrator to say, “Okay, if my firewall fails, if my VPN fails, and I’m at home, I don’t want to come into the office.” They come up with some super creative way to–
[00:07:45] Jeff: [laughs] Like hide the key in the rock outside your house? Is that what we’re talking about here?
[00:07:49] Joshua: If I can find the hide a key, then that’s super easy. It’s really not that hard to find those things. I can point a few scanners at your environment. I can hide their existence, and I can profile every port and protocol that your environment can expose to me. I can even find new ones by researching the Internet for your name, or identifiers that have to do with it. I can look at your ISP and see what range they’re in, and I can look for those. I can identify these little back doors that admins hide.
Let’s say, your admin’s super smart and realizes that’s not the way to do it. I’ll probably just drive by your office and get close enough either to the WiFi, or go across the street, and create a cantenna. Do you know what a cantenna is?
[00:08:31] Jeff: No, I don’t. It sounds interesting.
[00:08:34] Joshua: It looks like a Pringles can because often they’re made out of Pringles cans. For 50 bucks of parts, and a Pringles can, I can hit your WiFi from over a mile away. I just have to point it at your building, and I can get enough signal that I can create a connection that I can get on your network.
[00:08:48] Jeff: Which is shocking because I can’t even get my WiFi in my own apartment one bedroom away, by the way, so it’s pretty cool.
[00:08:54] Joshua: Exactly. WiFi is extremely vulnerable. They just came out with a new–
[00:08:59] Jeff: Yes, it’s all encrypted and stuff, right?
[00:09:01] Joshua: No.
[00:09:02] Jeff: No?
[00:09:03] Joshua: Well, it’s encrypted. Basically, here’s how I would hack a WiFi.
[00:09:07] Jeff: You’re taking away my own ball [unintelligible 00:09:09] security right now, by the way.
[00:09:10] Joshua: Basically, I’d hit your WiFi router with what’s called a Deauth. I would basically, reset your WiFi router remotely. Almost every WiFi router in the world will allow me to do this. I’ll hit it with a Dauth and it resets it, and then everything that was connected to that WiFi will now try to connect again. In order for it to connect, it has to send its credentials. Think about a username and password. When all those things are trying to reconnect, I’m capturing those packets. I’m capturing a copy of them. They’re all encrypted. I can’t see them. I can’t understand them, but all I’ll have to do is, take one of those packets, and try sending it until something will let me in.
The router knows that this encrypted stream is legitimate, and so all I’m doing is duplicating a legitimate encrypted stream. Now I’m on your WiFi networks, Now, I’m on your network, and now I can get to that login screen.
[00:10:00] Jeff: Wow. You’re freaking me out here a little bit. You can essentially spoof a signal to get access to someone’s entire network?
[00:10:12] Joshua: Think about a system that’s overloaded. That’s what I’m doing. I’m overloading your WiFi network so that it resets itself and then everything has to connect again and all I’m doing is watching– [crosstalk]
[00:10:20] Jeff: [crosstalk] -grabbing something. That’s amazing.
[00:10:24] Joshua: Right. Now, I’m on your network. Now, I can get to your RMM. I can see it. There’s other ways that I could do it. I could do a phishing attack and try to trick one of your admins or something, but there’s different ways to get on the network but that’s my favorite, is that WiFi attack.
Now, that I have access to your RMM, I have to authenticate into it. There’s a username and password in most environments so I have to get past that in order to gain access just like an admin would within the RMM. There’s a few different ways I could do that. The first I would do is if I could get on to a system. Let’s say I use malware to get in or phishing attack or some watering hole attack or something like that. I’ve compromised the system, I’m going to try to capture your password database. I’d say a good portion of MSPs are using a KeePass. We’ll use that as an example.
KeePass is a great program but if I can get that database and I can download it to myself, I can crack that within a amount of time. One of the side effects of all of this bitcoin craze and everyone trying to create bitcoin, we’ve created an extremely powerful CPUs, they’re GPUs, actually.
We use the graphics processors and we can process through tremendous amounts of data. What that means is, is for us password crackers, that means we can crack passwords much, much faster.
I was just reading yesterday on a rig that someone put together that has, I think, it’s 25 GPUs or something like that. It can crack an eight-character password that follows all the rules we’ve been taught. Uppercase, lowercase, special characters, spaces, and numbers. We can crack an eight-character password within five and a half hours on this system.
[00:12:07] Jeff: Even with special characters and everything.
[00:12:08] Joshua: Exactly. Yes.
[00:12:09] Jeff: Wait. Hold on a minute. Let me see here. P-A-S-S-W-O-R-D. I’m at eight so I’m just right there on the cusp, essentially. All right.
[00:12:19] Joshua: Yours would go much faster [crosstalk] dictionary.
[00:12:21] Jeff: I’m going to go faster than five hours. What it was password1!?
[00:12:29] Joshua: The LinkedIn breach, we found that almost everyone adds a one or an exclamation mark after their password.
[00:12:34] Jeff: Really? I don’t [crosstalk] anyone.
[00:12:37] Joshua: That’s a terrible way to deal with this. Most password crackers will try that. Now, we’ve cracked the password. There’s other ways to do it. For example, if you’re using an old version of KeePass like before 1.17, it has a vulnerability in it. I can exploit that vulnerability. I can get to the database. Now, I have all your passwords.
[00:12:56] Jeff: Wait, a password manager has a vulnerability which allows people access to your passwords? All right. Great.
[00:13:03] Joshua: Any software in the world, whether it’s an RMM or a password or whatever, they are full vulnerabilities. Once somebody finds them and discloses them responsibly, then we go and fix them. Keeping software up to date is a big part of the resolution to this. I tried to get to those passwords. If I couldn’t, then I might try to find a vulnerability in the RMM itself.
A good example of this, ConnectWise had one a few months back. Once again, it’s software. Software has vulnerabilities. I could potentially try to get in the middle just like I did with the WiFi where I get between your communication with the RMM and I just watch for a few days.
Someone’s going to log in to the RMM to support one of their customers and if I can get a man in the middle position, let’s say they use weak SSL or TLS or something like that, then I can capture that password and I don’t have to have the password, I just need the hash so that I can duplicate it so that I can pretend to be you making a legitimate request into the RMM itself.
[00:14:03] Jeff: What’s a hash?
[00:14:08] Joshua: You know not to expose your passwords. Somebody comes to you on the internet and says, “Hey, what’s your password?” Most people know not to do that. Most software writers know not to keep that password in the clear which means that if I’m sitting somewhere on the network, I can see that password, that’s using insecure protocol, not using any [unintelligible 00:14:26] as an example, but the idea is that instead of sending an actual password, we send the hash of the password. The hash is just a one way– Do you remember one-way math way back to sixth grade where you could do a math problem one direction, but you can do it the other way.
[00:14:42] Jeff: I don’t even remember current math much less one-way math back in the old days. No.
[00:14:47] Joshua: That’s what hash is in theory. I can create a hash but I can’t create the original from the hash. Whereas encryption is different. Encryption, I can go back and create the original, if I have the decryption key. Hash is one way, if that makes sense.
I have to capture your credentials or capture the hash. Something else I might do if I’ve compromised like a Windows system. Windows has a database on it called SAM and I can crack SAM in a pretty reasonable amount of time depending on how complex your hashes are. Once again, eight-character passwords are pretty easy to get.
If I can get passwords, that’s great. If I get into your RMM through a vulnerability, that’s great. If I have access to the RMM system itself, there may be a database on there that I can go look at and see if it has passwords encrypted or if they’re hashed, is there some other way to do it? Hash is even though they’re one way, we can come up with the original passwords from hash by throwing it into a rainbow table. Someone figured out a long time ago, I can’t undo the math here, but if I make a table that has every possible hash for everything up to 14 characters, all I have to do is reference that table so they build these huge databases up in the cloud. Most of this, you can get to without paying for anything.
[00:16:08] Jeff: Wait, so you’re saying that tools, you’ve been describing so far are either cheap or free. Is that accurate? Did I get that right?
[00:16:17] Joshua: As far as what I’m using as a hacker? Absolutely.
[00:16:19] Jeff: Yes. In the series of hacking steps you’ve been sharing, most of the stuff is either free or low-cost.
[00:16:27] Joshua: Yes. If I’m new to this, let’s say I’m a what we call script kiddie. Somebody who’s new to it. They’re like running tools but they don’t really understand that hack things or break out of bounds or something like that. I can go pay on the dark web using bitcoin and go pay to have malware written for me. I can go pay to have ransomware written for me. The last time I looked, I could get a custom ransomware piece of malware for $150. It’s guaranteed not to be caught by antivirus or any of these EDRs.
[00:16:59] Jeff: You’re saying once you’re in, you pay $150, you’re in and you’ve got the keys to the kingdom.
[00:17:08] Joshua: That puts me on the network or gives me credentials or it gives me the ability to capture those passwords or locks a system or something like that. A malware–[crosstalk]
[00:17:15] Jeff: For $150. Wow.
[00:17:17] Joshua: Yes. I don’t even have to know how to write code.
[00:17:20] Jeff: You just have to have bad intentions.
[00:17:22] Joshua: Exactly, and be willing to do it. There’s lots of other ways. We hear about all the specter and meltdown and then there’s this new MDS attacks. If I’m on a virtualized system, let’s say our customer, our partner uses ESXI. They have VMware and they’ve got their RMM on one of those and then I’m able to compromise another machine on that same VMware. As of yesterday, if I could compromise one of those machines, I can read the memory of every machine in that VMware environment.
There’s all these different ways once I’m on the network to capture these credentials so that I could log in to the RMM itself.
[00:18:07] Jeff: This is freaking me out a little bit that we should have definitely done this on Halloween, can’t we? You’re basically saying– I love the fact, not in a good way, but I love the fact that you’re thinking like a hacker and you’re saying, “My whole goal is to get to this RMM. Once I get to this RMM, I feel like I’m okay.” That is the master key, right?
[00:18:31] Joshua: It is. You have to realize that I’m compromising lots of things along the way. I’m compromising operating systems that may have HIPAA data on them or PCI data. I’m compromising databases. I’m compromising your antivirus infrastructure. I might own your firewall on the process. I just talked about getting past your WiFI credentials and so now being able to get on your WiFi anytime I want. I might just distribute that information out to other hackers. I might be a generous hacker and I threw all the stuff on the dark web. Anybody that wants to get on your WiFI and never pay for WiFI again can use that login that I’ve just created.
[00:19:08] Jeff: Throwing your machines in the bitcoining mining machines and stuff like that, stuff that we hear.
[00:19:13] Joshua: That’s exactly it. I’m going into your room and I’m making a mess. That’s ultimately the situation you’re dealing with.
Once I can get into an RMM. Once I have access inside that barrier, that username password barrier. Some of the older ones probably have SQL injection issues which are really easy to get past things like that. Now, I’m authenticated as you. I’m authenticated as your admin or I’m authenticated as one of your individuals. Getting to the client system is usually very easy. I would say a good percentage of our managed service providers store the domain credentials for their customers within the RMM.
As long as I can get past that barrier for the RMM, now I can do anything I want on any customer system in their environment because I’m admin.
[00:20:07] Jeff: Now you’re admin on their systems, and you’re admin on their client systems as well. That’s essentially what’s happened here.
[00:20:16] Joshua: That’s exactly it. I control the MSP environment. I control the MSP network, and I control the client network and the client machines, and I can do whatever I want with those because I am an admin because MSPs have to be admins to do their job.
[00:20:31] Jeff: To administrate their customers systems. Wow, that’s crazy. This is so crazy. That easy?
[00:20:39] Joshua: That easy. It’s not easy. There’s a lot of complexity to it, but it’s not that hard either. There’s YouTube video, everything that I’ve mentioned, there’s a YouTube video on how to do it.
[00:20:50] Jeff: We know that we’ve got these nation-state hackers. Then you just have just your locals, your rowdy locals, and the people in the US that are doing it. They want to get in there. Are they just dropping ransomware all of the place? What are these guys do when they get in there?
[00:21:12] Joshua: They might be. Ransomware is the new cool thing in the hacker world. It’s a really easy way to monetize what you’re doing. Most hackers now, and it used to be, if you’ve seen the movie Hackers like Angelina Jolie back in the day. It used to be that hacking was cool and trendy and everybody wanted to do it. Most hackers nowadays want to make money, they want to make good money, and they can do that by stealing your credit card data. They can do that by stealing your information and selling it to people who have other ideas with it. Selling all your clients emails, could be valuable to another hacker that’s out there that’s going to take your clients emails and compare them to a list of known vulnerable of known credentials that were compromised from LinkedIn and from these other different breaches that happened over the years.
Then, they’ll go login to your clients’ Gmail or your clients’ bank or your clients’ credit card account. They’re looking for emails to work with. The goal is data, I want to capture data, I want to capture information, and I want to make money off of it, and there’s hundreds of ways to do this.
I even read something– This was earlier this year where there was a ransomware attack on a medical provider. I think it was a hospital. The provider refused to pay, and so the hacker took and went on Kickstarter and start asking for money from the people who are compromised. It said, “I’m going to release your information if you don’t pay me money on this Kickstarter campaign.” Hackers are incredibly innovative. Now we have crowdfunding for hacking because they’re trying to put pressure on this– [crosstalk]
[00:22:55] Jeff: This hospital to pay up the cash.
[00:22:57] Joshua: Exactly, right.
[00:22:57] Jeff: What does study show? How often when you pay up the cash to even, I’ve heard some crazy numbers, but how often when you pay up the cash to actually even get the resolution that you hoped for?
[00:23:07] Joshua: I’ve seen numbers from 70% to 90% will get your information back, but there’s no way those numbers can be accurate. Most statistics are made up, but I would guess probably 50% of CryptoLocker or ransomware is paid, and nobody tells anybody it happened. They just do it.
[00:23:29] Jeff: This is just so crazy. One of my big concerns and I was with a group of people, not too long ago, talking about this was, as MSPs step into the MSSP realm, they’re becoming more and more security experts, and a lot of small to medium-sized businesses are outsourcing the security of their systems to MSPs, and so if word gets out that MSPs have vulnerabilities as well, and when you unlock an MSP, you unlock 30, 50, 100, 200 small businesses, it puts them at more risk. This is a pretty serious issue.
[00:24:06] Joshua: It is, and it’s increasing, it’s not decreasing. You’re seeing these nation-state attackers that are armies of hackers going after MSP is because they’re a central point of attack.
[00:24:18] Jeff: Wow. All right. Well, let’s take a little break. When we come back, Josh, I’d like to hear how long have you been doing this, man? That sounds pretty crazy. I want to talk a little bit about your background here. When we return, let’s talk about like, now that you’ve scared the heck out of everybody, what are we going to do about all this and how we’re going to make it better, sound good?
[00:24:39] Joshua: Sounds good.
[00:24:39] Jeff: All right.[music]
[00:24:48] Jeff: All right, we are back. Just as a quick reminder, please take a few moments to subscribe and review the podcast on iTunes. I think Kelly is going to start putting us on some other social media podcasty platforms here, but make sure you share this podcast on social media that really helps us get support for the show. We want to keep bringing in guests and keep doing this. Thank you so much for listening and we encourage you guys to subscribe and share.
All right, we’re back. Joshua, let’s talk a little bit here. We didn’t really get it into your background. How long have you been doing? It sounds like you’ve been doing security for a long time. How long have you been doing this, man?
[00:25:24] Joshua: That’s 17 years at this point.
[00:25:26] Jeff: 17 years. I didn’t even realize that people cared about this 17 years ago.
[00:25:30] Joshua: They didn’t pay for it.
[00:25:31] Jeff: Your interest where you got curious about it, and you were like, “Hey, I love this creativity.” I’ve never heard it described as an artist before. What did you do? What was the path that you got here? What makes you really security expert?
[00:25:44] Joshua: A lot of is just exposure. Most of the vulnerabilities that you see today are based on older vulnerabilities. Someone has come up with a creative spin on or a different way of doing it, but ultimately, like I said, your goal is to confuse an application or an operating system or something like that into doing something it’s not accustomed to doing. I have a personality where it’s like a river, it dries, if it hits a rock, just goes right around, doesn’t worry much about it, and that’s my nature. I try to find holes, and if that’s not a hole, well I’ll find another hole, and I’ll just keep spreading wide until I find the right hole, and then I’ll go through that hole, and that’s ultimately what a hacker does.
[00:26:24] Jeff: That’s what a hacker does. You’re thinking, “How do I get into my own systems? Or how do I get into client system? Or how do I get into somebody that you’re consulting with system?” You’re like, “Let me think about all the different ways I get in.” Then that helps you figure out how to block those ways, and they’re busy constantly looking for new ways, and you’re constantly looking for those new ways and making sure your block them, and that’s pretty crazy.
What really freaked me out is that you said, “Hey, Jeff, you want to talk about these various scenarios, but chances are, somebody might already be in there and you don’t even know about it.” Before we get it to what can be done about it, is that true or is that just fear mongering? Like, “What’s going on here, man?” It’s like, somebody might already be in your system and you don’t even know it or do they just hang around and wait? What’s that all about?
[00:27:13] Joshua: I’m not a big fan of the message that you’ve already been owned. That somebody is already in your system, and you just don’t know that they’re there, but what is accurate is most technical people are overconfident in the security of their environment, and so if someone wanted to be in your environment, there’s probably not a lot you could do to stop them without building a really comprehensive security environment, like we’ve done here at Axcient.
The goal and security, so there’s offense, there’s defense, Red team, Blue team, things like that. The ultimate goal of security defense is to make the data more difficult to get to than it’s worth. If it takes me as a hacker, four weeks to get to your credit cards, and I can only get $500 off your credit card, because you have no credit limit, well, then that’s not worth my time, but if you’re a bank and you have thousands of credit cards, you’re an MSP and you have access to hundreds of victims, well, then that starts to become worth my time. Making sure you have these layers of security in places is really your best defense.
[00:28:20] Jeff: You almost sound like sometimes you’re describing like Home Security. They’re like, “If you’ve got an alarm system and a dog and you’ve got the sign on the window says, ‘I’ve an alarm system.’ They’re like, ‘It’s probably not worth it.” Is this the same concept?
[00:28:30] Joshua: It really is. I had an instructor tell me, this was 20 years ago when I was learning this stuff. “The only way you can secure a system is if you are surrounded with concrete and buried 100 feet in the dirt, then now you have a secure computer system, but now it’s not usable.” There’s always this balance between usability and security, and you have to strike the right balance based on what you’re trying to protect.
As MSPs, people are starting to realize that your environment is valuable, and that’s why we’re starting to see an increasing attack in the MSP space. When we talk with MSPs, we’re out of shows and things like that. That’s a constant messages. These attacks that are coming against me are more complex than anything I’ve ever seen. I don’t know how to stop them, I don’t know where they’re coming from. I don’t know what to do about it, and that’s ultimately where you start, is these layers of defense.
[00:29:24] Jeff: What can be done? We’ve talked about some of the basic layers of defense, let’s get to the heart of it here like, “What do I do?” Oh my God, you just described how easy it is to get in my system and if somebody really, really wants in, they can get in, I got to make it as difficult as possible. What do I do here? Now I’m not just thinking about me, I’m thinking about all– I’m a caretaker for so many other companies, which puts heavy is the head that wears the crown. What can be done? What do we do? Is it just too overwhelming you’re like, “Dude, it’s going to happen one day?” Well, what’s the answer there?
[00:30:00] Joshua: No, I think a lot of it is education. There’s a portion of MSPs that are sitting on their worlds. They’re good at what they do. They know how to make great environments. They’re good at email, they’re good at active directory. They’re good at solving these typical problems and securities coming out of left field. The complexity is weird because the security community or the hacker community has been building on these complex attacks for years and years and years. They just haven’t been targeting MSPs.
The first thing to get in mind is, one, is I’m not perfect and I shouldn’t expect to be perfect, and I shouldn’t expect that whatever I’ve done is enough. There’s always a way to compromise a system. I don’t care how good you are, how smart you are, security is not a form of perfection. You can’t perfectly secure a system, and so you need a backup. That’s first and foremost. That’s what we do at Axcient.
We want a backup in case everything goes wrong, that saves face with your customers. It saves face with you and your environment. It keeps your business up and running. I read about a Colorado company recently that had to go out of business because they got ransomware, they couldn’t restore anything. They had paid the fee and the decryption key didn’t work so they closed their business. Their business is no more.
[00:31:21] Jeff: That’s an example where they paid and still didn’t win anyway. Still didn’t get their money back.
[00:31:25] Joshua: Right. You want to make sure you have backups in place, you want to make sure those backups are good and that there is close to the data as possible. I talk with MSPs all the time. They’re like, “Well, you know, I’ve got this whole disk backup. I’ve got [unintelligible 00:31:37] BRC or data or one of these image-based encryptions.” What they don’t realize is, we’re seeing attacks that are ransomware plus hacker. Ransomware gets in, encrypts everything, it creates a backdoor. The hacker gets in and goes and deletes all the data off the backup server.
They go into your [unintelligible 00:31:55] or your data server and they delete everything because they have your password. The software has to let you in to delete data. I get calls from customers that are like, “We’ve been hacked. We lost our password because we use the same password on every admin system, and every one of our admins has them.”
[00:32:14] Jeff: Kelly and I share a password. It’s password123 is that we decided–
[00:32:17] Kelly: Exclamation point.
[00:32:18] Jeff: We do the exclamation point at the end. All right. Feeling pretty good about that exclamation point, not.
[00:32:24] Joshua: Exactly, right. To pull back data, and Axcient is one of the only companies I know that can actually– that will actually do this. You wipe out every machine in your environment, you wipe out every image-based backup server in your environment. These are the data servers and the [unintelligible 00:32:41] servers. As long as that data is backed up to the cloud, we can restore within 14 days.
We do that, but it’s petabytes of data. Bringing your customer back online with four petabytes of data, it takes a really long time. If you had an anchor, some sort of file system, sync server, I can have clients up and running while I’m restoring servers and less critical things, so your RTO, your RPO, all that stuff is really important.
[00:33:09] Jeff: Regardless of whether you’re using an Axcient product or not, you should at least understand what it’s going to take to bring that back and have that mapped out and have it prepared, right?
[00:33:21] Joshua: You should test it.
[00:33:22] Jeff: You should definitely test it. I feel like it’s probably that last piece, it’s a little bit tough, is making the time to test it. After hearing you, it sounds like– you even said to me like they’re probably, eventually going to get in. If they want to get in-
[00:33:38] Joshua: Absolutely.
[00:33:38] Jeff: -they’re probably going to get in. You better make sure that you’re testing your systems, especially because protecting your customers is what you’ve been hired to do, which means you have to protect yourself. Get your backups, make sure that they work. What else? What else would you tell them?
[00:33:54] Joshua: It’s layers of security. When I ran through a scenario with you, like how to get to the RMM, I ran through different paths that I could take to get to the same ultimate goal. To be on the defense side, blue team or protecting your environment before your customers get to it, first you need to protect yourself and then you can protect your customers. Because if they can get to you, it doesn’t matter how well you protect your customers, you have access as an MSP.
You want to protect yourself. You want to do that with layers of security. That’s perimeter protection, like firewalls. That’s endpoint protection, like your antivirus. It’s first protecting and then detecting. If something’s able to get past your controls, you want to know what happened, how it happened, when it happened, where it happened, so you can fix it next time, and to understand what happened.
When I was consulting, I had a client came back to me and they’re like, “Yes, we got malware on this machine.” It’s a company that dealt with DNA. What they would do is they would take DNA from their customers, they would store it in a database, and one of their database servers got malware on it. I was like, “How did you handle that?” “Well, we wiped the server and we brought it back up.” I was like, “Did you lose any data?” They’re like, “Well, we don’t deal with any sensitive data.” “But you have DNA.” At some point somebody’s going to be able to make it–
[00:35:12] Jeff: That’s pretty sensitive.[crosstalk]
[00:35:15] Joshua: Can’t do it today, but that’s really sensitive. I was very frustrated by the situation because they didn’t understand-
[00:35:24] Jeff: The gravity-
[00:35:25] Joshua: -the gravity.
[00:35:25] Jeff: -of what happened well.
[00:35:26] Joshua: Having visibility into syslog and these types of things, and forensic capability to look on an endpoint, it’s exactly what happened. All that stuff’s important. To protect, you detect, you back up. These are your big stages.
On the protect side, that’s the best scenario. If you can just stop the attack from happening in the first place, it’s super easy. You do that through layers. A framework is a really good way to do this. There’s so many different ways that an attacker can get to your system. If you use a framework like a enter for information security as a top 20, you just go through that top 20 and make sure that there aren’t big gaps in your security defenses.
[00:36:06] Jeff: Wait, so hold on. I want to stop you there because I want to hear about a resource. Is this a website that they go to and they can kind of go through a checklist? Is that what you’re describing?
[00:36:15] Joshua: Exactly, yes. The reason I like it is because there’s only 20 items. Most of the checklists that are out there in security are 200, 300, 400 different items. This one starts with 20.
[00:36:24] Jeff: It seem like onerous, right?
[00:36:27] Joshua: Overwhelming.
[00:36:26] Jeff: If you think you have to do 200 items, you almost don’t do it. There’s a spot where you stop doing it because it’s too big, right?
[00:36:32] Joshua: Yes. This one has the same level of rigor, but it starts with, “Okay, let’s make sure my passwords are strong. Let’s make sure I have a good way to store passwords. Let’s make sure people are educated on my passwords.” It goes through these top 20. Then once you’ve got the 20 down, then you get better at the 20 and then you get better at the 20, you get better at the 20. You go from a level one to level five. By the time you get to level five, you’re dealing with the same level of control that you would if you open up a spreadsheet with 400 controls, but it’s not as overwhelming.
[00:37:01] Jeff: Wow, so you can grow to the point where-
[00:37:03] Joshua: Exactly, right.
[00:37:04] Jeff: -your security just gets stronger, but there’s no better time to start the now. Wow. This was scary but illuminating. It sounds like it’s something that with the right thought and the right level of effort, you can at least get started on it. No better time than now. Josh, this has been a really great conversation, man.
[00:37:22] Joshua: It’s fun. I love this stuff.
[00:37:23] Jeff: Yes. You say it’s fun. I’m scared. I’m changing all my passwords to be in like 50 characters with not an exclamation mark at the end. Right, Kelly?
[00:37:31] Kelly: 321.
[00:37:32] Jeff: 321. Yes. I’m going just flip that– [crosstalk]
[00:37:33] Joshua: That’s actually why I’m here, to improve Jeff’s password.
[00:37:36] Jeff: Yes. The whole purpose of this podcast was to improve my password. Well, dude, thanks so much for joining us, man. This was great. We’re going to have to probably bring you back in because I know you know a lot about training, internal training for security and things like that. We’re going to talk about this again. This was a really great topic. All right. Thanks a lot, Josh. Cheers, man.
[END OF AUDIO]