How to Detect Obfuscated Malware on Your Server

Traditional malware detecting methods are ineffective against the hackers’ new techniques – deploying malware that is obfuscated.

That’s why an innovative new approach is required in server security to stop these new threats.

History of Malware Detecting Technologies

The constant battle between malware creators and anti-malware companies leads to the invention of more and more sophisticated techniques on both sides.

Understanding the trends and the recent threats is vital if you’d like to keep your servers safe. 

File Signature-Based Detection Techniques

Legacy malware was a simple PHP file with regular coding and usually used good standards.

They were not created to hide, so when this kind of malware started to kick in, there were quite good tools to detect and remove them.

Detecting a clean malware code is pretty easy and when you have a collection of known malicious code, you just need to simply match them with every newly created or modified file.

Hashing function (MD5, SHAx hashing)

However, finding exact matches between the collection of clean malware code and the files is very resource hungry.

That’s why a similar, but faster solution was created. 

With hashing function, you can give a string or a file and it’ll generate a fixed-length string.

Every time the code is the same, it’ll generate the same hash from that code. The most well-known hashing techniques are MD5 and SHAx. 

Hackers realized that it’s quite easy to find the backdoors with these detection methods.

It is enough to change 1 byte, for example, add a space and the hash will be completely different so anti-malware tools won’t recognize it. 

Pattern Matching Detection Techniques

The next solution was pattern matching. This technique is based on creating some strings and trying to match them on the file. For example, find the “eval” word in the file.

Pattern matching has a lot of disadvantages. You can expect a high false-positive rate but also the false-negative rate is pretty high at the same time.

When the hacker knows that you use pattern-matching detection mechanisms, they can try to change the code.

For example, modify eval to EvAl. It’ll be the same code and will run the same way but you can avoid the detection.

Rule-Based Detection Techniques (Yara)

Yara is specially built to write rule-based signatures and it is widely used by cyber defense systems.

The rule-based detection techniques try to fix the problems of the classic pattern-matching mechanism, but we found out that it’s actually a pattern-matching just with a little steroid and it still has disadvantages (difficulty of writing new rules, high false-positive rate, etc).

Detecting Obfuscated Malware

All the above-mentioned techniques are ineffective in detecting obfuscated malware. Hackers know it, that’s why this new malware type is getting into the foreground.

What is Code Obfuscation?

Obfuscation means converting a clean code into a new one. The obfuscated code will give the exact same result as the original code, however this way the source code will not be readable to human eyes.

It is usually used for checkout codes, banking, licensing, etc.

The problem is that you can’t tell whether obfuscated codes are malicious or not because of their unreadability.

The New Approach

BitNinja Server Security experimented with this topic a lot and developed a brand new detecting method which is not like any other solution found on the market currently.

This new method is based on the structure of the source code.

When you rely on the structure of malware, you can expect a very low false-positive rate because malware’s structure can not be the same as a legitimate file’s structure.

Otherwise, valid codes could be used for malicious purposes.

The false-negative rate is also very low with the structure-based detection because it is no matter how the hacker modifies the code (add spaces, new lines, etc), the structure of the program must be the same. 

Source Structure-Based Detection

In the first step, BitNinja detects if the obfuscation method was used in a file. At this phase, the system doesn’t determine yet whether the obfuscated code is malware or not.

Because in order to figure this out, the code needs to be run to see its purpose. And of course, running a possible malicious code on your server is not safe. 

Behavior-Based Detection (Sandboxing)

So the next step is to run the code in a sandbox farm and inspect the behavior of the code (e.g. generated network traffic, newly created files, etc).

Based on these behavior signatures, we can find out if the code was legit or malicious. It is an important step because quarantining all the obfuscated files is not a good idea.

Several valid files use obfuscation techniques too as discussed above.

Deobfuscation

By running the code in a sandbox, Bitninja will try to deobfuscate the code, and after that regular matching mechanisms can be used to find out the intention of the code.

If you would like to know more about how BitNinja works, you can find the details here.

About the Author

George Egri
CEO & Founder at BitNinja
15+ years of experience as shared web hosting CEO, 7 years of experience in server and website security.
CEO of BitNinja.io.

Also Read: A Complete Malware Analysis Tutorials, Cheatsheet & Tools list for Security Professionals