What is Encryption in Malware? – Understand From Basics to XOR

Malware commonly encrypts its traffic (stolen data sent to a command-and-control server) and internal strings (like URLs and configurations) to prevent security systems from recognizing malicious content. 

Cryptography fundamentals, classical ciphers, bitwise operations, XOR functions, and XOR cipher detection and decryption techniques, and showcase a practical example of how to decrypt malware C2 communication encrypted with XOR. 

Common encryption methods:

• XOR
• Rc4
• AES
• DES
• 3DES (Tripple DES)

Main concepts in encryption are:

• Plaintext is the raw data that hasn’t been encrypted and can be read and understood without processing.
• Ciphertext is the encrypted data that looks like a bunch of random letters or bytes and can’t be read. It’s what the coding process turns into.
• An encryption algorithm is a set of rules that turns plaintext into ciphertext. Several substitutions and permutations are usually used in plain text.
• A key is a piece of data, usually a string of letters or numbers, used with an encryption algorithm to secure and decrypt data.

Analyze Encrypted Traffic in ANY.RUN with HTTPS MTIM Proxy - Register for Free

Fundamentals of Encryption:

The encryption algorithm dictates how the data is manipulated, typically through substitutions and permutations, allowing decryption with the correct key. 

There are two encryption methods: symmetric and asymmetric, whereas symmetric encryption utilizes a single key for both encryption and decryption, making it faster and simpler, which necessitates a secure channel for key exchange. 

Asymmetric encryption, also known as public-key cryptography, addresses this by using two mathematically linked keys: a public key for encryption, freely distributable, and a private key for decryption, kept confidential, which allows for more secure key exchange but comes at the cost of increased complexity and slower processing times. 

The breakdown will explore encryption concepts through a Lego-like approach, building towards understanding XOR with simple substitution, where a cleartext message is transformed based on a key. 

Mutating the message at the bit level, introducing bitwise operations, and synthesizing concepts result in a comprehensive understanding of XOR, a fundamental operation in cryptography. 

Simple substitution ciphers replace plaintext characters with alternative symbols according to a key, like swapping letters for emojis, as the concept underlies all encryption. 

Caesar ciphers, a type of simple substitution, shift plaintext letters by a fixed amount. 

Caesar ciphers are vulnerable due to predictable patterns and unchanged symbol frequencies, where encryption relies on substituting symbols based on a defined rule. 

The Vigenère cipher demonstrates the concept of using a key to mutate plaintext for encryption. A keyword generates multiple Caesar ciphers, applying a shift based on the corresponding key letter’s position (A = 0).

Repetition arises when the key is shorter than the message, making the cipher vulnerable. This vulnerability, due to key brevity, is also applicable to modern symmetric encryption methods like XOR. 

ANY.RUN explored bitwise operations, specifically XOR, for their role in encryption, where XOR operates on individual bits, returning 1 if only one input bit is 1, allowing to combine a message (plaintext) with a key using XOR, resulting in an encrypted ciphertext. 

Decryption without the key is computationally impossible because the key controls the bit-wise changes and is thrown away after a single use (one-time pad). This shows how bitwise operations can be used to encrypt data securely.

The XOR cipher encrypts data by performing a bitwise XOR operation between the plaintext and a secret key, where each bit in the plaintext is flipped (0 to 1 or 1 to 0) if the corresponding bit in the key is 1. 

By XORing the ciphertext with the same key once more, decryption is possible. This exposes the cipher’s flaw when using short, repetitive keys. In the hexadecimal representation of the encrypted data, it appears as repeating patterns of zeros, indicating a potential XOR encryption. 

ANY.RUN analysis revealed a process sending a suspicious GET request for a .mp4 file, where the requested content, exhibiting repetitive patterns of 5s and 3s, suggested XOR encryption. While the key likely involved a sequence of 5s and 3s, its exact length remained unknown. 

Downloading the executable from it and examining it in dnSpy could reveal the encryption function and the key itself. Once obtained, the key could decrypt the downloaded file (potentially malware) using tools like CyberChef. 

Quickly analyze malware dynamically and statically in ANY.RUN sandbox  - Register for free

What is ANY.RUN?

ANY.RUN is a cloud-based malware lab that does most of the work for security teams. 400,000 professionals use ANY.RUN platform every day to look into events and speed up threat research on Linux and Windows cloud VMs.

Advantages of ANY.RUN

• Real-time Detection: ANY.RUN can find malware and instantly identify many malware families using YARA and Suricata rules within about 40 seconds of posting a file.
• Interactive Malware Analysis: ANY.RUN differs from many automated options because it lets you connect with the virtual machine from your browser. This live feature helps stop zero-day vulnerabilities and advanced malware that can get past signature-based protection.
• Value for money: ANY.RUN’s cloud-based nature makes it a cost-effective option for businesses since your DevOps team doesn’t have to do any setup or support work.
• Best for onboarding new security team members: ANY. RUN’s easy-to-use interface allows even new SOC researchers to quickly learn to examine malware and identify signs of compromise (IOCs).