Friday, April 19, 2024

How to Detect Obfuscated Malware on Your Server

Traditional malware detecting methods are ineffective against the hackers’ new techniques – deploying malware that is obfuscated.

That’s why an innovative new approach is required in server security to stop these new threats.

History of Malware Detecting Technologies

The constant battle between malware creators and anti-malware companies leads to the invention of more and more sophisticated techniques on both sides.

Understanding the trends and the recent threats is vital if you’d like to keep your servers safe. 

File Signature-Based Detection Techniques

Legacy malware was a simple PHP file with regular coding and usually used good standards.

They were not created to hide, so when this kind of malware started to kick in, there were quite good tools to detect and remove them.

Detecting a clean malware code is pretty easy and when you have a collection of known malicious code, you just need to simply match them with every newly created or modified file.

server security

Hashing function (MD5, SHAx hashing)

However, finding exact matches between the collection of clean malware code and the files is very resource hungry.

That’s why a similar, but faster solution was created. 

With hashing function, you can give a string or a file and it’ll generate a fixed-length string.

Every time the code is the same, it’ll generate the same hash from that code. The most well-known hashing techniques are MD5 and SHAx. 

Hackers realized that it’s quite easy to find the backdoors with these detection methods.

It is enough to change 1 byte, for example, add a space and the hash will be completely different so anti-malware tools won’t recognize it. 

Pattern Matching Detection Techniques

The next solution was pattern matching. This technique is based on creating some strings and trying to match them on the file. For example, find the “eval” word in the file.

Pattern matching has a lot of disadvantages. You can expect a high false-positive rate but also the false-negative rate is pretty high at the same time.

When the hacker knows that you use pattern-matching detection mechanisms, they can try to change the code.

For example, modify eval to EvAl. It’ll be the same code and will run the same way but you can avoid the detection.

Rule-Based Detection Techniques (Yara)

Yara is specially built to write rule-based signatures and it is widely used by cyber defense systems.

The rule-based detection techniques try to fix the problems of the classic pattern-matching mechanism, but we found out that it’s actually a pattern-matching just with a little steroid and it still has disadvantages (difficulty of writing new rules, high false-positive rate, etc).

Detecting Obfuscated Malware

All the above-mentioned techniques are ineffective in detecting obfuscated malware. Hackers know it, that’s why this new malware type is getting into the foreground.

What is Code Obfuscation?

Obfuscation means converting a clean code into a new one. The obfuscated code will give the exact same result as the original code, however this way the source code will not be readable to human eyes.

It is usually used for checkout codes, banking, licensing, etc.

The problem is that you can’t tell whether obfuscated codes are malicious or not because of their unreadability.

obfuscated malware

The New Approach

BitNinja Server Security experimented with this topic a lot and developed a brand new detecting method which is not like any other solution found on the market currently.

This new method is based on the structure of the source code.

When you rely on the structure of malware, you can expect a very low false-positive rate because malware’s structure can not be the same as a legitimate file’s structure.

Otherwise, valid codes could be used for malicious purposes.

The false-negative rate is also very low with the structure-based detection because it is no matter how the hacker modifies the code (add spaces, new lines, etc), the structure of the program must be the same. 

Source Structure-Based Detection

In the first step, BitNinja detects if the obfuscation method was used in a file. At this phase, the system doesn’t determine yet whether the obfuscated code is malware or not.

Because in order to figure this out, the code needs to be run to see its purpose. And of course, running a possible malicious code on your server is not safe. 

Behavior-Based Detection (Sandboxing)

So the next step is to run the code in a sandbox farm and inspect the behavior of the code (e.g. generated network traffic, newly created files, etc).

Based on these behavior signatures, we can find out if the code was legit or malicious. It is an important step because quarantining all the obfuscated files is not a good idea.

Several valid files use obfuscation techniques too as discussed above.


By running the code in a sandbox, Bitninja will try to deobfuscate the code, and after that regular matching mechanisms can be used to find out the intention of the code.

If you would like to know more about how BitNinja works, you can find the details here.

About the Author

George Egri
CEO & Founder at BitNinja
15+ years of experience as shared web hosting CEO, 7 years of experience in server and website security.
CEO of

Also Read: A Complete Malware Analysis Tutorials, Cheatsheet & Tools list for Security Professionals


Latest articles

Palo Alto ZeroDay Exploited in The Wild Following PoC Release

Palo Alto Networks has disclosed a critical vulnerability within its PAN-OS operating system, identified...

FIN7 Hackers Attacking IT Employees Of Automotive Industry

IT employees in the automotive industry are often targeted by hackers because they have...

Russian APT44 – The Most Notorious Cyber Sabotage Group Globally

As Russia's invasion of Ukraine enters its third year, the formidable Sandworm (aka FROZENBARENTS,...

SoumniBot Exploiting Android Manifest Flaws to Evade Detection

A new banker, SoumniBot, has recently been identified. It targets Korean users and is...

LeSlipFrancais Data Breach: Customers’ Personal Information Exposed

LeSlipFrancais, the renowned French underwear brand, has confirmed a data breach impacting its customer...

Cisco Hypershield: AI-Powered Hyper-Distributed Security for Data Center

Cisco has unveiled its latest innovation, Cisco Hypershield, marking a milestone in cybersecurity.This groundbreaking...

Phishing-as-a-Service Platform LabHost Seized by Authorities

Authorities have dismantled LabHost, a notorious cybercrime platform that facilitated widespread phishing attacks across...


Mastering WAAP/WAF ROI Analysis

As the importance of compliance and safeguarding critical websites and APIs grows, Web Application and API Protection (WAAP) solutions play an integral role.
Key takeaways include:

  • Pricing models
  • Cost Estimation
  • ROI Calculation

Related Articles