What is Encryption in Malware? – Understand From Basics to XOR

Malware commonly encrypts its traffic (stolen data sent to a command-and-control server) and internal strings (like URLs and configurations) to prevent security systems from recognizing malicious content. 

Cryptography fundamentals, classical ciphers, bitwise operations, XOR functions, and XOR cipher detection and decryption techniques, and showcase a practical example of how to decrypt malware C2 communication encrypted with XOR. 

Common encryption methods:

  • XOR
  • Rc4
  • AES
  • DES
  • 3DES (Tripple DES)

Main concepts in encryption are:

  • Plaintext is the raw data that hasn’t been encrypted and can be read and understood without processing.
  • Ciphertext is the encrypted data that looks like a bunch of random letters or bytes and can’t be read. It’s what the coding process turns into.
  • An encryption algorithm is a set of rules that turns plaintext into ciphertext. Several substitutions and permutations are usually used in plain text.
  • A key is a piece of data, usually a string of letters or numbers, used with an encryption algorithm to secure and decrypt data.

Analyze Encrypted Traffic in ANY.RUN with HTTPS MTIM Proxy - Register for Free

Fundamentals of Encryption:

The encryption algorithm dictates how the data is manipulated, typically through substitutions and permutations, allowing decryption with the correct key. 

There are two encryption methods: symmetric and asymmetric, whereas symmetric encryption utilizes a single key for both encryption and decryption, making it faster and simpler, which necessitates a secure channel for key exchange. 

Asymmetric encryption, also known as public-key cryptography, addresses this by using two mathematically linked keys: a public key for encryption, freely distributable, and a private key for decryption, kept confidential, which allows for more secure key exchange but comes at the cost of increased complexity and slower processing times. 

The breakdown will explore encryption concepts through a Lego-like approach, building towards understanding XOR with simple substitution, where a cleartext message is transformed based on a key. 

Mutating the message at the bit level, introducing bitwise operations, and synthesizing concepts result in a comprehensive understanding of XOR, a fundamental operation in cryptography. 

Simple substitution ciphers replace plaintext characters with alternative symbols according to a key, like swapping letters for emojis, as the concept underlies all encryption. 

Caesar ciphers, a type of simple substitution, shift plaintext letters by a fixed amount. 

Example of Caesar Cipher

Caesar ciphers are vulnerable due to predictable patterns and unchanged symbol frequencies, where encryption relies on substituting symbols based on a defined rule. 

The Vigenère cipher demonstrates the concept of using a key to mutate plaintext for encryption. A keyword generates multiple Caesar ciphers, applying a shift based on the corresponding key letter’s position (A = 0).

Conversion of Plain text into Key, and Ciphertext

Repetition arises when the key is shorter than the message, making the cipher vulnerable. This vulnerability, due to key brevity, is also applicable to modern symmetric encryption methods like XOR. 

ANY.RUN explored bitwise operations, specifically XOR, for their role in encryption, where XOR operates on individual bits, returning 1 if only one input bit is 1, allowing to combine a message (plaintext) with a key using XOR, resulting in an encrypted ciphertext. 

Conversion of plain text to 8-bit binary

Decryption without the key is computationally impossible because the key controls the bit-wise changes and is thrown away after a single use (one-time pad). This shows how bitwise operations can be used to encrypt data securely.

The XOR cipher encrypts data by performing a bitwise XOR operation between the plaintext and a secret key, where each bit in the plaintext is flipped (0 to 1 or 1 to 0) if the corresponding bit in the key is 1. 

Cyberchef Screenshot

By XORing the ciphertext with the same key once more, decryption is possible. This exposes the cipher’s flaw when using short, repetitive keys. In the hexadecimal representation of the encrypted data, it appears as repeating patterns of zeros, indicating a potential XOR encryption. 

Any.Run Interface

ANY.RUN analysis revealed a process sending a suspicious GET request for a .mp4 file, where the requested content, exhibiting repetitive patterns of 5s and 3s, suggested XOR encryption. While the key likely involved a sequence of 5s and 3s, its exact length remained unknown. 

Downloading the executable from it and examining it in dnSpy could reveal the encryption function and the key itself. Once obtained, the key could decrypt the downloaded file (potentially malware) using tools like CyberChef. 

Quickly analyze malware dynamically and statically in ANY.RUN sandbox  - Register for free

What is ANY.RUN?

ANY.RUN is a cloud-based malware lab that does most of the work for security teams. 400,000 professionals use ANY.RUN platform every day to look into events and speed up threat research on Linux and Windows cloud VMs.

Advantages of ANY.RUN

  • Real-time Detection: ANY.RUN can find malware and instantly identify many malware families using YARA and Suricata rules within about 40 seconds of posting a file.
  • Interactive Malware Analysis: ANY.RUN differs from many automated options because it lets you connect with the virtual machine from your browser. This live feature helps stop zero-day vulnerabilities and advanced malware that can get past signature-based protection.
  • Value for money: ANY.RUN’s cloud-based nature makes it a cost-effective option for businesses since your DevOps team doesn’t have to do any setup or support work.
  • Best for onboarding new security team members: ANY. RUN’s easy-to-use interface allows even new SOC researchers to quickly learn to examine malware and identify signs of compromise (IOCs).

BALAJI is an Ex-Security Researcher (Threat Research Labs) at Comodo Cybersecurity. Editor-in-Chief & Co-Founder - Cyber Security News & GBHackers On Security.

Recent Posts

Hackers Claiming Dettol Data Breach: 453,646 users Impacted

A significant data breach has been reported by a threat actor known as 'Hana,' who claims to have compromised the…

2 days ago

CrowdStrike Update Triggers Widespread Windows BSOD Crashes

A recent update from cybersecurity firm CrowdStrike has caused significant disruptions for Windows users, leading to widespread reports of Blue…

2 days ago

Operation Spincaster Disrupts Approval Phishing Technique that Drains Victim’s Wallets

Chainalysis has launched Operation Spincaster, an initiative to disrupt approval phishing scams that have drained billions from victims' wallets. This…

2 days ago

Octo Tempest Know for Attacking VMWare ESXi Servers Added RansomHub & Qilin to Its Arsenal

Threat actors often attack VMware ESXi servers since they accommodate many virtual machines, which link to a variety of systems…

3 days ago

TAG-100 Actors Using Open-Source Tools To Attack Gov & Private Orgs

Hackers exploit open-source tools to execute attacks because they are readily available, well-documented, and often have extensive community support, making…

3 days ago

macOS Users Beware Of Weaponized Meeting App From North Korean Hackers

Meeting apps are often targeted and turned into weapons by hackers as they are largely employed for communication and collaboration,…

3 days ago