Category: AI

On the Coming Industrialisation of Exploit Generation with LLMs

seanhnLeave a comment

Recently I ran an experiment where I built agents on top of Opus 4.5 and GPT-5.2 and then challenged them to write exploits for a zeroday vulnerability in the QuickJS Javascript interpreter. I added a variety of modern exploit mitigations, various constraints (like assuming an unknown heap starting state, or forbidding hardcoded offsets in the exploits) and different objectives (spawn a shell, write a file, connect back to a command and control server). The agents succeeded in building over 40 distinct exploits across 6 different scenarios, and GPT-5.2 solved every scenario. Opus 4.5 solved all but two. I’ve put a technical write-up of the experiments and the results on Github, as well as the code to reproduce the experiments.

In this post I’m going to focus on the main conclusion I’ve drawn from this work, which is that we should prepare for the industrialisation of many of the constituent parts of offensive cyber security. We should start assuming that in the near future the limiting factor on a state or group’s ability to develop exploits, break into networks, escalate privileges and remain in those networks, is going to be their token throughput over time, and not the number of hackers they employ. Nothing is certain, but we would be better off having wasted effort thinking through this scenario and have it not happen, than be unprepared if it does.

A Brief Overview of the Experiment

All of the code to re-run the experiments, a detailed write-up of them, and the raw data the agents produced are on Github, but just to give a flavour of what the agents accomplished:

  1. Both agents turned the QuickJS vulnerability into an ‘API’ to allow them to read and arbitrarily modify the address space of the target process. As the vulnerability is a zeroday with no public exploits for it, this capability had to be developed by the agents through reading source code, debugging and trial and error. A sample of the notable exploits is here and I have written up one of them in detail here.
  2. They solved most challenges in less than an hour and relatively cheaply. I set a token limit of 30M per agent run and ran ten runs per agent. This was more than enough to solve all but the hardest task. With Opus 4.5 30M total tokens (input and output) ends up costing about $30 USD.
  3. In the hardest task I challenged GPT-5.2 it to figure out how to write a specified string to a specified path on disk, while the following protections were enabled: address space layout randomisation, non-executable memory, full RELRO, fine-grained CFI on the QuickJS binary, hardware-enforced shadow-stack, a seccomp sandbox to prevent shell execution, and a build of QuickJS where I had stripped all functionality in it for accessing the operating system and file system. To write a file you need to chain multiple function calls, but the shadow-stack prevents ROP and the sandbox prevents simply spawning a shell process to solve the problem. GPT-5.2 came up with a clever solution involving chaining 7 function calls through glibc’s exit handler mechanism. The full exploit is here and an explanation of the solution is here. It took the agent 50M tokens and just over 3 hours to solve this, for a cost of about $50 for that agent run. (As I was running four agents in parallel the true cost was closer to $150).

Before going on there are two important caveats that need to be kept in mind with these experiments:

  1. While QuickJS is a real Javascript interpreter, it is an order of magnitude less code, and at least an order of magnitude less complex, than the Javascript interpreters in Chrome and Firefox. We can observe the exploits produced for QuickJS and the manner in which they were produced and conclude, as I have, that it appears that LLMs are likely to solve these problems either now or in the near future, but we can’t say definitively that they can without spending the tokens and seeing it happen.
  2. The exploits generated do not demonstrate novel, generic breaks in any of the protection mechanisms. They take advantage of known flaws in those protection mechanisms and gaps that exist in real deployments of them. These are the same gaps that human exploit developers take advantage of, as they also typically do not come up with novel breaks of exploit mitigations for each exploit. I’ve explained those gaps in detail here. What is novel are the overall exploit chains. This is true by definition as the QuickJS vulnerability was previously unknown until I found it (or, more correctly: my Opus 4.5 vulnerability discovery agent found it). The approach GPT-5.2 took to solving the hardest challenge mentioned above was also novel to me at least, and I haven’t been able to find any example of it written down online. However, I wouldn’t be surprised if it’s known by CTF players and professional exploit developers, and just not written down anywhere.

The Industrialisation of Intrusion

By ‘industrialisation’ I mean that the ability of an organisation to complete a task will be limited by the number of tokens they can throw at that task. In order for a task to be ‘industrialised’ in this way it needs two things:

  1. An LLM-based agent must be able to search the solution space. It must have an environment in which to operate, appropriate tools, and not require human assistance. The ability to do true ‘search’, and cover more of the solution space as more tokens are spent also requires some baseline capability from the model to process information, react to it, and make sensible decisions that move the search forward. It looks like Opus 4.5 and GPT-5.2 possess this in my experiments. It will be interesting to see how they do against a much larger space, like v8 or Firefox.
  2. The agent must have some way to verify its solution. The verifier needs to be accurate, fast and again not involve a human.

Exploit development is the ideal case for industrialisation. An environment is easy to construct, the tools required to help solve it are well understood, and verification is straightforward. I have written up the verification process I used for the experiments here, but the summary is: an exploit tends to involve building a capability to allow you to do something you shouldn’t be able to do. If, after running the exploit, you can do that thing, then you’ve won. For example, some of the experiments involved writing an exploit to spawn a shell from the Javascript process. To verify this the verification harness starts a listener on a particular local port, runs the Javascript interpreter and then pipes a command into it to run a command line utility that connects to that local port. As the Javascript interpreter has no ability to do any sort of network connections, or spawning of another process in normal execution, you know that if you receive the connect back then the exploit works as the shell that it started has run the command line utility you sent to it.

There is a third attribute of problems in this space that may influence how/when they are industrialisable: if an agent can solve a problem in an offline setting and then use its solution, then it maps to the sort of large scale solution search that models seem to be good at today. If offline search isn’t feasible, and the agent needs to find a solution while interacting with the real environment, and that environment has the attribute that certain actions by the agent permanently terminate the search, then industrialisation may be more difficult. Or, at least, it’s less apparent that the capabilities of current LLMs map directly to problems with this attribute.

There are several tasks involved in cyber intrusions that have this third property: initial access via exploitation, lateral movement, maintaining access, and the use of access to do espionage (i.e. exfiltrate data). You can’t perform the entire search ahead of time and then use the solution. Some amount of search has to take place in the real environment, and that environment is adversarial in that if a wrong action is taken it can terminate the entire search. i.e. the agent is detected and kicked out of the network, and potentially the entire operation is burned. For these tasks I think my current experiments provide less information. They are fundamentally not about trading tokens for search space coverage. That said, if we think we can build models for automating coding and SRE work, then it would seem unusual to think that these sorts of hacking-related tasks are going to be impossible.

Where are we now?

We are already at a point where with vulnerability discovery and exploit development you can trade tokens for real results. There’s evidence for this from the Aardvark project at OpenAI where they have said they’re seeing this sort of result: the more tokens you spend, the more bugs you find, and the better quality those bugs are. You can also see it in my experiments. As the challenges got harder I was able to spend more and more tokens to keep finding solutions. Eventually the limiting factor was my budget, not the models. I would be more surprised if this isn’t industrialised by LLMs, than if it is.

For the other tasks involved in hacking/cyber intrusion we have to speculate. There’s less public information on how LLMs perform on these tasks in real environments (for obvious reasons). We have the report from Anthropic on the Chinese hacking team using their API to orchestrate attacks, so we can at least conclude that organisations are trying to get this to work. One hint that we might not be yet at a place where post-access hacking-related tasks are automated is that there don’t appear to be any companies that have entirely automated SRE work (or at least, that I am aware of).

The types of problems that you encounter if you want to automate the work of SREs, system admins and developers that manage production networks are conceptually similar to those of a hacker operating within an adversary’s network. An agent for SRE can’t just do arbitrary search for solutions without considering the consequences of actions. There are actions that if it takes the search is terminated and it loses permanently (i.e. dropping the production database). While we might not get public confirmation that the hacking-related tasks with this third property are now automatable, we do have a ‘canary’. If there are companies successfully selling agents to automate the work of an SRE, and using general purpose models from frontier labs, then it’s more likely that those same models can be used to automate a variety of hacking-related tasks where an agent needs to operate within the adversary’s network.

Conclusion

These experiments shifted my expectations regarding what is and is not likely to get automated in the cyber domain, and my time line for that. It also left me with a bit of a wish list from the AI companies and other entities doing evaluations.

Right now, I don’t think we have a clear idea of the real abilities of current generation models. The reason for that is that CTF-based evaluations and evaluations using synthetic data or old vulnerabilities just aren’t that informative when your question relates to finding and exploiting zerodays in hard targets. I would strongly urge the teams at frontier labs that are evaluating model capabilities, as well as for AI Security Institutes, to consider evaluating their models against real, hard, targets using zeroday vulnerabilities and reporting those evaluations publicly. With the next major release from a frontier lab I would love to read something like “We spent X billion tokens running our agents against the Linux kernel and Firefox and produced Y exploits“. It doesn’t matter if Y=0. What matters is that X is some very large number. Both companies have strong security teams so it’s entirely possible they are already moving towards this. OpenAI already have the Aardvark project and it would be very helpful to pair that with a project trying to exploit the vulnerabilities they are already finding.

For the AI Security Institutes it’s would be worth spending time identifying gaps in the evaluations that the model companies are doing, and working with them to get those gaps addressed. For example, I’m almost certain that you could drop the firmware from a huge number of IoT devices (routers, IP cameras, etc) into an agent based on Opus 4.5 or GPT-5.2 and get functioning exploits out the other end in less a week of work. It’s not ideal that evaluations focus on CTFs, synthetic environments and old vulnerabilities, but don’t provide this sort of direct assessment against real targets.

In general, if you’re a researcher or engineer, I would encourage you to pick the most interesting exploitation related problem you can think of, spend as many tokens as you can afford on it, and write up the results. You may be surprised by how well it works.

Hopefully the source code for my experiments will be of some use in that.

How I used o3 to find CVE-2025-37899, a remote zeroday vulnerability in the Linux kernel’s SMB implementation

seanhn11 Comments

In this post I’ll show you how I found a zeroday vulnerability in the Linux kernel using OpenAI’s o3 model. I found the vulnerability with nothing more complicated than the o3 API – no scaffolding, no agentic frameworks, no tool use.

Recently I’ve been auditing ksmbd for vulnerabilities. ksmbd is “a linux kernel server which implements SMB3 protocol in kernel space for sharing files over network.“. I started this project specifically to take a break from LLM-related tool development but after the release of o3 I couldn’t resist using the bugs I had found in ksmbd as a quick benchmark of o3’s capabilities. In a future post I’ll discuss o3’s performance across all of those bugs, but here we’ll focus on how o3 found a zeroday vulnerability during my benchmarking. The vulnerability it found is CVE-2025-37899 (fix here), a use-after-free in the handler for the SMB ‘logoff’ command. Understanding the vulnerability requires reasoning about concurrent connections to the server, and how they may share various objects in specific circumstances. o3 was able to comprehend this and spot a location where a particular object that is not referenced counted is freed while still being accessible by another thread. As far as I’m aware, this is the first public discussion of a vulnerability of that nature being found by a LLM.

Before I get into the technical details, the main takeaway from this post is this: with o3 LLMs have made a leap forward in their ability to reason about code, and if you work in vulnerability research you should start paying close attention. If you’re an expert-level vulnerability researcher or exploit developer the machines aren’t about to replace you. In fact, it is quite the opposite: they are now at a stage where they can make you significantly more efficient and effective. If you have a problem that can be represented in fewer than 10k lines of code there is a reasonable chance o3 can either solve it, or help you solve it.

Benchmarking o3 using CVE-2025-37778

Lets first discuss CVE-2025-37778, a vulnerability that I found manually and which I was using as a benchmark for o3’s capabilities when it found the zeroday, CVE-2025-37899.

CVE-2025-37778 is a use-after-free vulnerability. The issue occurs during the Kerberos authentication path when handling a “session setup” request from a remote client. To save us referring to CVE numbers, I will refer to this vulnerability as the “kerberos authentication vulnerability“.

The root cause looks as follows:

static int krb5_authenticate(struct ksmbd_work *work,
			     struct smb2_sess_setup_req *req,
			     struct smb2_sess_setup_rsp *rsp)
{
...
	if (sess->state == SMB2_SESSION_VALID) 
		ksmbd_free_user(sess->user);
	
	retval = ksmbd_krb5_authenticate(sess, in_blob, in_len,
					 out_blob, &out_len);
	if (retval) {
		ksmbd_debug(SMB, "krb5 authentication failed\n");
		return -EINVAL;
	}
...

If krb5_authenticate detects that the session state is SMB2_SESSION_VALID then it frees sess->user. The assumption here appears to be that afterwards either ksmbd_krb5_authenticate will reinitialise it to a new valid value, or that after returning from krb5_authenticate with a return value of -EINVAL that sess->user will not be used elsewhere. As it turns out, this assumption is false. We can force ksmbd_krb5_authenticate to not reinitialise sess->user, and we can access sess->user even if krb5_authenticate returns -EINVAL.

This vulnerability is a nice benchmark for LLM capabilities as:

  1. It is interesting by virtue of being part of the remote attack surface of the Linux kernel.
  2. It is not trivial as it requires:
    • (a) Figuring out how to get sess->state == SMB2_SESSION_VALID in order to trigger the free.
    • (b) Realising that there are paths in ksmbd_krb5_authenticate that do not reinitialise sess->user and reasoning about how to trigger those paths.
    • (c) Realising that there are other parts of the codebase that could potentially access sess->user after it has been freed.
  3. While it is not trivial, it is also not insanely complicated. I could walk a colleague through the entire code-path in 10 minutes, and you don’t really need to understand a lot of auxiliary information about the Linux kernel, the SMB protocol, or the remainder of ksmbd, outside of connection handling and session setup code. I calculated how much code you would need to read at a minimum if you read every ksmbd function called along the path from a packet arriving to the ksmbd module to the vulnerability being triggered, and it works out at about 3.3k LoC.

OK, so we have the vulnerability we want to use for evaluation, now what code do we show the LLM to see if it can find it? My goal here is to evaluate how o3 would perform were it the backend for a hypothetical vulnerability detection system, so we need to ensure we have clarity on how such a system would generate queries to the LLM. In other words, it is no good arbitrary selecting functions to give to the LLM to look at if we can’t clearly describe how an automated system would select those functions. The ideal use of an LLM is we give it all the code from a repository, it ingests it and spits out results. However, due to context window limitations and regressions in performance that occur as the amount of context increases, this isn’t practically possible right now.

Instead, I thought one possible way that an automated tool could generate context for the LLM was through expansion of each SMB command handler individually. So, I gave the LLM the code for the ‘session setup’ command handler, including the code for all functions it calls, and so on, up to a call depth of 3 (this being the depth required to include all of the code necessary to reason about the vulnerability). I also include all of the code for the functions that read data off the wire, parses an incoming request, selects the command handler to run, and then tears down the connection after the handler has completed. Without this the LLM would have to guess at how various data structures were set up and that would lead to more false positives. In the end, this comes out at about 3.3k LoC (~27k tokens), and gives us a benchmark we can use to contrast o3 with prior models. If you’re interested, the code to be analysed is here as a single file, created with the files-to-prompt tool.

The final decision is what prompt to use. You can find the system prompt and the other information I provided to the LLM in the .prompt files in this Github repository. The main points to note are:

  1. I told the LLM to look for use-after-free vulnerabilities.
  2. I gave it a brief, high level overview of what ksmbd is, its architecture, and what its threat model is.
  3. I tried to strongly guide it to not report false positives, and to favour not reporting any bugs over reporting false positives. I have no idea if this helps, but I’d like it to help, so here we are. In fact my entire system prompt is speculative in that I haven’t ran a sufficient number of evaluations to determine if it helps or hinders, so consider it equivalent to me saying a prayer, rather than anything resembling science or engineering. Once I have ran those evaluations I’ll let you know.

To run the query I then use the llm tool (github) like: 

$ llm --sf system_prompt_uafs.prompt                \ 
        -f session_setup_code.prompt                \          
        -f ksmbd_explainer.prompt                   \
        -f session_setup_context_explainer.prompt   \
        -f audit_request.prompt

My experiment harness executes this N times (N=100 for this particular experiment) and saves the results. It’s worth noting, if you rerun this you may not get identical results to me as between running the original experiment and writing this blog post I had removed the file containing the code to be analysed, and had to regenerate it. I believe it is effectively identical, but have not re-run the experiment.

o3 finds the kerberos authentication vulnerability in the benchmark in 8 of the 100 runs. In another 66 of the runs o3 concludes there is no bug present in the code (false negatives), and the remaining 28 reports are false positives. For comparison, Claude Sonnet 3.7 finds it 3 out of 100 runs and Claude Sonnet 3.5 does not find it in 100 runs. So on this benchmark at least we have a 2x-3x improvement in o3 over Claude Sonnet 3.7.

For the curious, I have uploaded a sample report from o3 (here) and Sonnet 3.7 (here). One aspect I found interesting is their presentation of results. With o3 you get something that feels like a human-written bug report, condensed to just present the findings, whereas with Sonnet 3.7 you get something like a stream of thought, or a work log. There are pros and cons to both. o3’s output is typically easier to follow due to its structure and focus. On the other hand, sometimes it is too brief, and clarity suffers.

o3 finds a 0-day (CVE-2025-37899)

Having confirmed that o3 can find the kerberos authentication vulnerability (CVE-2025-37778) when given the code for the session setup command handler, I wanted to see if it could find it if I give it the code for all of the command handlers. This is a harder problem as the command handlers are all found in smb2pdu.c, which has ~9k LoC. However, if o3 can still find vulnerabilities when given all of the handlers in one go then it suggests we can build a more straightforward wrapper for o3 that simply hands it entire files, covering a variety of functionality, rather than going handler by handler.

Combining the code for all of the handlers with the connection setup and teardown code, as well as the command handler dispatch routines, ends up at about 12k LoC (~100k input tokens), and as before I ran the experiment 100 times.

o3 finds the kerberos authentication vulnerability in 1 out of 100 runs with this larger number of input tokens, so a clear drop in performance, but it does still find it. More interestingly however, in the output from the other runs I found a report for a similar, but novel, vulnerability that I did not previously know about. This vulnerability is also due to a free of sess->user, but this time in the session logoff handler.

int smb2_session_logoff(struct ksmbd_work *work)
{
	struct ksmbd_conn *conn = work->conn;
	struct ksmbd_session *sess = work->sess;
	
    ...
	
    ksmbd_close_session_fds(work);
	ksmbd_conn_wait_idle(conn);
    
    ...
	
    down_write(&conn->session_lock);
	sess->state = SMB2_SESSION_EXPIRED;
	up_write(&conn->session_lock);

	if (sess->user) {
		ksmbd_free_user(sess->user);
		sess->user = NULL;
	}
    ...

I’ll let o3 explain the issue:

Short description
=================
While one ksmbd worker thread is still executing requests that use
sess‑>user, another thread that processes an SMB2 LOGOFF for the same
session frees that structure.  No synchronisation protects the pointer,
so the first thread dereferences freed memory – a classic
use‑after‑free that leads to kernel memory corruption and arbitrary code
execution in kernel context.

Affected code path
------------------
1.  A second transport is bound to an existing session  
    (SMB 3.0 or later, `conn->binding == true`).

2.  Worker‑A (running on connection C2) receives any normal request
    (e.g. WRITE).  
       • `smb2_check_user_session()` stores a pointer to the already
         existing `struct ksmbd_session` in `work->sess` and
         increments the session ref‑count, **but it does not take
         any reference on `sess->user`.**  
       • The pointer to `sess` is kept for the whole request
         processing; every helper that needs authentication data
         accesses `sess->user` directly.

3.  Before Worker‑A finishes, Worker‑B (running on the first connection
    C1) processes an SMB2 ​LOGOFF for the same session and executes
    `smb2_session_logoff()` (smb2pdu.c).

Relevant part of smb2_session_logoff()
--------------------------------------

```c
	...
	if (sess->user) {
		ksmbd_free_user(sess->user);   /* (1) frees memory     */
		sess->user = NULL;             /* (2) clear the field  */
	}
	...
```

`ksmbd_free_user()` ultimately ends in simple `kfree(user)`.

4.  Worker‑B **does not wait for the other connections that are still
    using the session**; it only waits for running requests on *its own*
    connection (`ksmbd_conn_wait_idle(conn)`).

5.  Worker‑A continues to execute (e.g. inside `smb2_open()`,
   `smb2_write()`, …) and dereferences `sess->user`:

```c
/* examples (many of them) */
if (user_guest(sess->user))        ← dereference after free
ksmbd_compare_user(sess->user, …)
sess->user->uid
```

Because the memory was already `kfree()`‑ed in step (1) the access is to
freed memory.  Depending on exact timing it is either:

• a use‑after‑free (pointer still points into now‑reused slab object),
  enabling controlled kernel‑memory overwrite, or  
• a NULL–deref (if Worker‑A reads after step (2)), still a DoS.

Reading this report I felt my expectations shift on how helpful AI tools are going to be in vulnerability research. If we were to never progress beyond what o3 can do right now, it would still make sense for everyone working in VR to figure out what parts of their work-flow will benefit from it, and to build the tooling to wire it in. Of course, part of that wiring will be figuring out how to deal with the the signal to noise ratio of ~1:50 in this case, but that’s something we are already making progress at.

One other interesting point of note is that when I found the kerberos authentication vulnerability the fix I proposed was as follows:

diff --git a/fs/smb/server/smb2pdu.c b/fs/smb/server/smb2pdu.c
index d24d95d15d87..57839f9708bb 100644
--- a/fs/smb/server/smb2pdu.c
+++ b/fs/smb/server/smb2pdu.c
@@ -1602,8 +1602,10 @@ static int krb5_authenticate(struct ksmbd_work *work,
 	if (prev_sess_id && prev_sess_id != sess->id)
 		destroy_previous_session(conn, sess->user, prev_sess_id);
 
-	if (sess->state == SMB2_SESSION_VALID)
+	if (sess->state == SMB2_SESSION_VALID) {
 		ksmbd_free_user(sess->user);
+		sess->user = NULL;
+	}
 
 	retval = ksmbd_krb5_authenticate(sess, in_blob, in_len,
 					 out_blob, &out_len);
-- 
2.43.0

When I read o3’s bug report above I realised this was insufficient. The logoff handler already sets sess->user = NULL, but is still vulnerable as the SMB protocol allows two different connections to “bind” to the same session and there is nothing on the kerberos authentication path to prevent another thread making use of sess->user in the short window after it has been freed and before it has been set to NULL. I had already made use of this property to hit a prior vulnerability in ksmbd but I didn’t think of it when considering the kerberos authentication vulnerability.

Having realised this, I went again through o3’s results from searching for the kerberos authentication vulnerability and noticed that in some of its reports it had made the same error as me, in others it had not, and it had realised that setting sess->user = NULL was insufficient to fix the issue due to the possibilities offered by session binding. That is quite cool as it means that had I used o3 to find and fix the original vulnerability I would have, in theory, done a better job than without it. I say ‘in theory’ because right now the false positive to true positive ratio is probably too high to definitely say I would have gone through each report from o3 with the diligence required to spot its solution. Still, that ratio is only going to get better.

Conclusion

LLMs exist at a point in the capability space of program analysis techniques that is far closer to humans than anything else we have seen. Considering the attributes of creativity, flexibility, and generality, LLMs are far more similar to a human code auditor than they are to symbolic execution, abstract interpretation or fuzzing. Since GPT-4 there has been hints of the potential for LLMs in vulnerability research, but the results on real problems have never quite lived up to the hope or the hype. That has changed with o3, and we have a model that can do well enough at code reasoning, Q&A, programming and problem solving that it can genuinely enhance human performance at vulnerability research.

o3 is not infallible. Far from it. There’s still a substantial chance it will generate nonsensical results and frustrate you. What is different, is that for the first time the chance of getting correct results is sufficiently high that it is worth your time and and your effort to try to use it on real problems.

Application optimisation with LLMs: Finding faster, equivalent, software libraries.

seanhn

A few months back I wrote a blog post where I mentioned that the least-effort/highest reward approach to application optimisation is to deploy a whole-system profiler across your clusters, look at the most expensive libraries & processes, and then search Google for faster, equivalent replacements. At Optimyze/Elastic we have had customers of our whole-system profiler use this approach successfully on numerous occasions. The only difficulty with this approach is the amount of Googling for alternatives that you need to do, followed by even more Googling for comparative benchmarks. If only we had a compressed version of all information on the internet, and an interface that allowed for production of free-form text based on that!

As you’ve probably gathered: we do. OpenAI’s, and other’s, Large Language Models (LLMs), have been trained on a huge amount of the internet, including the blurbs describing a significant number of available software libraries, as well as benchmarks comparing them, and plenty of other relevant content in the form of blogs, StackOverflow comments, and Github content.

So why use an LLM as a search interface when you could just Google for the same information? There are a couple of reasons. The first is that an LLM provides a much broader set of capabilities than just search. You can, for example, ask something like “Find me three alternative libraries, list the pros and cons of each, and then based on these pros and cons make a recommendation as to which I should use.”. Thus, we can condense the multistep process of Googling for software and benchmarks, interpreting the results, and coming up with a recommendation, into a single step. The second reason is that because LLMs have a natural language input/output interface, it is far easier to programatically solve the problem and make use of the result than if we had to use Google’s API, scrape web pages, extract information, and then produce a report.

If you’d like to try this out, a couple of days ago I open sourced sysgrok (blog, code), a tool intended to help with automating solutions to systems analysis and understanding. It is organised into a series of subcommands, one of which is called findfaster. The findfaster subcommand uses a prompt that asks the question mentioned above. Here’s an example of using it to find a faster replacement for libjpeg.

sysgrok also has a “–chat” parameter which will drop you into a chat session with the LLM after it has produced an initial response. This can be used to ask for clarification on recommendations, correct mistakes the LLM has made etc. For example, here we ask for replacements for Python’s stdlib JSON library. The LLM responds with three good alternatives (ujson, orjson, simdjson), and recommends we use the best of them (orjson). We then drop into a chat session and ask the LLM how to install and use the library.

asciicast

Usually this works really well, and is the best automated solution to this problem that I have come across. That said, we are using an LLM, so it can go amusingly wrong at times. The most common way for it to fail is the wholesale hallucination of software projects. This is prone to happening when there are limited, or no, alternatives to the target software which provide the same functionality but better performance. A good example of this is libtiff. If you ask findfaster to find you a faster version of libtiff you’ll may be used to consider TurboTiff. TurboTiff is not a software project.

If you use sysgrok and it comes back with a bad recommendation then I’d love to hear about it, as these examples are helpful in refining the prompts. You can open an issue on GitHub here.