[00:01.2] Hi, I'm Craig Rowland, founder of Sandfly Security. Today we're going to talk about immutable process binaries on Linux and why they're almost always a threat. Now, an immutable flag on Linux is basically something that you could set on a system file in the file system that says essentially you cannot modify or delete whatever it is I have this bit set on. [00:20.3] So commonly on Linux this is done to protect files against accidental deletion or modification. If the immutable flag is set on a file, even the root user cannot delete or modify that file until the flag is removed and then at that point they could do what they need to it. [00:36.3] But for what we need to know today, malware often will set an immutable flag on a binary in order to protect itself from simple removal. Now this is a pretty blunt tool. Typically I only see it on low grade malware. But we have seen some more sophisticated malware use this tactic in order to hang around. [00:54.0] But the main thing here to understand is that if you see a process binary running with an immutable flag, almost always is going to be bad news. I've never once seen it used in a legitimate way. It can be done, I suppose, but most of the time it's bad news. So we're going to go over here to the Sandfly browser, and Sandfly is an agentless Linux endpoint detection response product. [01:14.1] We also do SSH key tracking, password auditing and more. But for today, what we did is we scanned a Raspberry PI because we work perfectly fine in embedded Linux systems. And on that system we got two alerts. The two alerts for today that we're going to discuss is first of all, we found a fine binary that was hidden under the bin directory that was marked as basically it's being hidden. [01:35.7] We know that this is actually something we don't ever want to see. We discussed that in another video. But the main reason we're here is that we found a process binary called dot config and that process binary is marked here is this process name .config with PID 1884 is marked as immutable. [01:52.9] Malware often masks its binary marks its binaries as immutable so that they are harder to delete from the disk and can maintain persistence. Yep, that's pretty much what they do. Essentially it's like I said, a very crude mechanism. It can actually prevent automated configuration management tools from removing the file and maybe some, even some admins out there who don't know anything about the immutable file. [02:13.8] But the basic thing here to understand is that if you see a process binary running with an immutable flag, probably up to no good. Some of the raw data if we go here to the raw results data, we can see information about that process. We got our command line here with this unusual flag. We can also go down here, we can see some of our flag set. [02:30.0] So for instance here this immutable flag is set to true. So we know that this process is basically something that we need to investigate. Which is what we're going to do. So we're going to go over to the screen terminal over here. The process was under /usr/bin/.config. So first of all the process PID is 1884. [02:46.7] So we'll go to our /proc directory 1884. I'll do an "ls -al" here and we're just going to list out kind of what we see here. And, we see everything about this basic product process here, current working directory. We see the exe here. I'll make a copy of that. [03:03.2] And what we can do now is with this process exe path I can now go and I can see everything about that file. User bin config I could see here is made on December 29. Assuming that they didn't time stomp this, this date would probably give you pretty good indication about when the system might have been compromised. [03:22.1] But there is a special command a lot of admins might not know about it and that's called "lsattr" which is short for list attributes. And again, if we put in our process path here, we do an "lsattr" on it. We see here this flag here called "i". That "i" is essentially means immutable, which means that this file cannot be removed. [03:39.6] Now you notice here, I'm the root user, I'm going to try to delete that file /usr/bin/.config and let's watch what happens here when I do that. Oh, it says operation not permitted, right. So until that file is actually removed with the change attribute flag, it can't do anything. [03:57.4] Now how do you do that? You, you basically do do this change attribute (chattr) "-i" for removing the immutable bit and I would do it on my file. And again now if I run my "lsattr" on it, I could see that my immutable flag is missing. And if I were to remove that file, which I don't recommend you doing, but for here, for demonstration purposes, assuming that you copy the file and blah, blah, you're investigating the host. At this point now, now I can remove the file like normal. [04:22.0] But if that mutable flag set, you won't be able to do it. So basically, again, kind of a crude technique, but we do see it and what's great about it is essentially the attackers are announcing with a huge red banner, I am malicious. If you see a process binary running with an immutable flag, virtually always, it's going to be something that's going to be bad news and you should investigate that host. [04:42.1] So they're actually doing you a favor by marking it as immutable, believe it or not. And once you realize how to remove the flag and all that, it's actually a very simple thing to get around. So anyway, that's it on immutable process binaries. You shouldn't see them on Linux, and if you do, you got to look at them, handle it like an incident response, and move on from there. [05:00.7] So I'm Craig Rowland from Sandfly Security. Thanks again for watching my video. Check out our channel for more short video topics just like this on Linux Intrusion and Detection and Incident Response. Hope you enjoyed this video. Please leave a comment, let us know what you like and dislike. Thanks again and bye.