Irdeto - partially hacked encoding. The second version of the encoding (Irdeto 2).

Continuing on in our Encryption 101 series, where we gave and demonstrated encryption techniques using, we now look at what it takes to break an encryption. In order for something as powerful as encryption to break, there needs to be some kind of secret flaw.

That flaw is often a result of an error in implementation. There are a number of things that can go wrong for someone who is implementing encryption. What’s difficult is being able to identify and analyze the methods a programmer used for encryption and look for any weaknesses to exploit. These weaknesses can be anything from weak encryption algorithms and weak key generators to server-side vulnerabilities and leaked keys. Locating encryption algorithms Before you can even attempt to find the weakness, you must first know what was the encryption algorithm being used.

A lot of times, it’s as simple as looking at the API calls. If this is the case, it can be quite simple to identify the algorithm. This was the case for the previous ShiOne walkthrough. There are times, however, where the encryption is statically compiled into the malware or even a custom written encryption algorithm is used. When this is the case, you must be able to understand the inner workings of encryption algorithms to be able to identify code.

A file’s content will be encrypted and written back into the file, so a quick method to narrow down the general region where the encryption lies is to simply xref the ReadFile and WriteFile API calls. The encryption implementation will likely be performed between these two points.

Identifying encryption code When looking for statically compiled encryption code, as we mentioned, you will not have the luxury of searching for any API calls. A basic understanding of some of the low-level details of how these encryption algorithms work will be necessary. Starting off, below, we have the high-level flow of AES algorithm. In general, most synchronous encryption algorithms have a similar flow to this; the differences may be the types of mathematical operations performed, but the core concepts remain the same. So, understanding AES will be enough of a starting point to help identify other types going forward in a real-world analysis.

With AES, being that it is a symmetric encryption algorithm, it performs a series of mathematical and logical operations on three things working together: • Plaintext data to be encrypted • Static bytes that are part of the algorithm (lookup table) • The key used for encryption Depending on the flavor of AES and key size, the flow will be slightly different. In the picture above, you see a loop involving a few blocks: • Add key • Shift rows • Sub bytes • Mix columns What is happening in these steps is the file data is read into a matrix of a fixed number of bytes. In this case, it’s 16 bytes, but depending on the algorithm, it could be anything. Here are the rounds of steps: • The add key round XORs the key data against the matrix of input data. • The shift rows round rolls the data using a shift operation. What I mean by rolling is the following: 4 5 2 1. If the roll shifted left one count, it would become 5 2 1 4. Rolled again it would become 2 1 4 5.