Every addition to an agent or an MCP server is text that every model reads, and a model does not always answer the same way. So how do you know that a change broke nothing, without running eight hours of benchmark each time? This is how I test netexec-mcp now: a differential replay with no model first, then decision-point evals where it points, and an A/B at the end.
A model only reads text. When I change the server, the agent can only behave differently in two cases: the model reads a different text, or the server runs a different command. So I compare those first, with no model at all, and only spend model calls where they differ.
Three stages, one question each. A differential replay first shows what the agent will see differently, and where. A decision-point eval then asks the model again at those exact points, to see whether it chooses differently, and last, a full A/B checks whether the task succeeds more often, but only on the tasks where the first two found something.
What changed is the price of a question: an answer that cost a night and a dollar now takes from 5 minutes to 1 h 35, for a fraction of that dollar, so every change gets tested, even the small ones.
Measured on the 1.0.8 fixes, with the detail in Fig. 08.
When you build an agent, or an MCP server in dynamic mode where the model searches for its tools, everything you add is text that the model reads. A new tool, a better description, one more line in an error message or a new synonym in the search is usually written for one case, often to fix one model's mistake. But every model will read it, in every task where that text appears.
The question is not only whether a change works where I aimed it, but also what else it changed. For example, I rewrote an error message to send the model straight to the right tool, and it did. But the same message also appeared in a place I had not thought of, just after the server had refused a tool, and there it told the model to use that same tool. A model that used to stop and say that it was blocked now called the refused tool again 7 times out of 10, so the same change fixed one thing and broke another.
How do you avoid a regression? Running each task once is not enough, because a model does not always answer the same way. The usual answer is a benchmark that runs every model on every task several times, but on mine, a full run takes up to eight hours and costs money. So how do you make regression tests fast enough that they don't slow down the work on the agent or the MCP?
The answer came after reading Anthropic's article on building agents, When to fold subagents into your main agent loop. Its part on evals describes what my benchmark already did: an end-to-end set, and the final answer graded against a rubric. At least I got that part right, nice! The same part also says to check whether the agent called the right tool, and it ends on an approach that worked for them internally, which is to record timestamps on the important events of a run, "so they can be replayed later to run as targeted evals".
If the decisions that matter in a run can be replayed one by one, testing a change no longer takes a night of tasks. It only needs the list of decisions that the change touches, and the model asked again at each of them. The question became whether that list could be computed without a model. But first, here is the answer I usually hear, and why it does not answer the regression question.
A colleague told me they catch regressions in their agent with unit tests. That left me doubtful, because a unit test checks that the code returns the result I expect, while a regression in an agent happens one step later, in what the model does with that result.
I have more than 600 unit tests, and they run before anything else. They tell me that the server does what I meant, but they cannot tell me whether what I meant is what a model needs.
As for the benchmark, it was built to compare models, and it does that well. It has 29 tasks on an Active Directory lab, from listing users to attack chains across several hosts, and a script scores each answer against what is really on the lab. Six models run each task three times through OpenRouter, which makes 522 runs for what I call a reference, and the last two references took 7 h 35 and 5 h 30, and cost $1.31 and $1.00. As a regression test for the server, though, it is a poor tool, for four reasons.
--local-auth or the right tool at one decision, and not on the average.In November, I found a regression that had been in the server for more than three weeks, and neither the unit tests nor the benchmark had caught it. In October, I had changed nxc_call, the tool that runs everything in dynamic mode, so that it accepts the common parameters at its top level (targets, username, password and domain), but I had not declared local_auth or ntlm_hash.
The MCP client my benchmark used at the time, mcp-use, checks each call against the tool's schema and silently drops any key that the schema does not declare. So a local_auth passed at the top level could never reach the server: NetExec then logged in with the domain account instead of the local Administrator, and a hash passed the same way became an empty password.
On my lab, the domain Administrator shares the local Administrator's password. Yeah, yeah, I know, the CIS Benchmarks will not be happy, ugh. Because of that, the wrong account still got admin rights, the task still passed and the score did not move, while on a real engagement, the same bug would mean using the wrong account, or failing.
| August | November | ||
|---|---|---|---|
| Model | With the flag | With the flag | Got admin rights |
| 7/7 | 0/12 | 12/12 | |
| 12/12 | 4/12 | 12/12 | |
| 13/13 | 8/12 | 12/12 | |
| 21/21 | 15/18 | 18/18 | |
| 8/12 | 9/12 | 12/12 | |
| All five models | 61/65 (94%) | 36/66 (55%) | 66/66 (100%) |
In August, the few commands without the flag were the model's own omission, since it had not asked for local_auth at all. In November, the model asked for it and the client dropped it. The unit tests passed the whole time, because the server did what I had written and the key simply never reached it. I found the bug by reading the commands that the runs had really sent to NetExec. The fix that solved this bug was as invisible to the score as the bug itself.
The way out started from an obvious fact: a model never sees the server itself, only the text the server sends it. That means the instructions, the names, descriptions and schemas of the tools, the search results, the tool responses and the error messages, and for the model, that text is the whole interface.
So when I change the server, the agent can only behave differently in two cases: the model reads a different text, or NetExec runs a different command. If neither happens, I don't need a model to know that the change is safe, and a script can check both by comparing the old version with the new one, without a model, without a lab and without spending anything.
smb_shares · 10.0.0.5 · user1A script asks both questions for every call from past benchmark runs, by sending it to the old and the new version and comparing what comes out. If no call shows a difference, the change is safe.
With the replay, I ask one question: what will the model see differently with the new version? To answer it without a model, I reuse what the benchmark already records. Every run keeps what the model asked the server, from its searches and the tool descriptions it read to each tool call with its arguments, which makes about three thousand distinct calls, a number that grows with every new run. I send all of it again to the old version and to the new one and compare their answers. The lab is not needed, since the NetExec outputs were recorded too and are played back instead, and details that change at every start, like timestamps and temporary paths, are ignored.
The report lists every difference between the two versions. A difference can be in what the model reads about a tool, in a search result, in a NetExec command or in a response, and for each one, the report names the tool and the benchmark task it comes from. The replay takes about five minutes and costs nothing, and if nothing changed, I am done.
Before trusting the replay, I tested it in the simplest way, by comparing a version of the server with itself. Since nothing had changed, a correct replay had to report nothing, and yet it found one issue, a real bug in the server. Two tools that read NetExec's database, workspace_hosts and workspace_creds, returned their fields in a different order each time the server started, so the same question did not always get the same answer. I fixed it, which means that the replay found a bug in the server before I had even used it to test a change.
Then I broke a copy of the server on purpose, with a bug that has the same effect as the real one: whenever a call used a hash, the copy dropped --local-auth. In five minutes, the replay found the bug. What the model reads had not changed: the tools, their descriptions and the search results were the same. But for 90 of the 3,032 different calls recorded in past runs, the broken copy sent NetExec a different command, without --local-auth, and the report named the five tasks these calls came from. If I make the same mistake inside the server, the replay now finds it before I release the change. I also wanted to see whether such a regression would make a nice graph, and it does, as Fig. 03 shows.
Calls per task, as the replay named them: t4_chain_pivot 81, t4_chain_mssql 4, t4_exec 2, t4_exec_in_loot 2, t4_lsa_dump 1.
Each square is one of the 3,032 distinct tool calls that past benchmark runs had recorded at that time, and I replayed every one of them on the normal server and on a copy I had broken on purpose. The 90 colored squares are the calls where the broken copy sent NetExec a different command, because it had dropped --local-auth, and five of them are labeled with their tool's name.
The replay cannot tell me whether a change is better: it sees that a message changed, not how the model will read it. For that, I need the second stage. Now, what do I do when the replay shows a different output after I fix a real bug? Let's check this example in Fig. 04.
[-] ERROR(SQL01\SQLEXPRESS): Line 1: Cannot execute as the database principal because the principal "sa" does not exist…
"authenticated": [{"user": "user1"}],
"auth_failed": [{"user": "SQLEXPRESS)"}]A failed login that never happened: the server read the SQL error as a failed login, and took the end of the instance name for an account.
"authenticated": [{"user": "user1"}],
"auth_failed": []The error is still reported as an error, but it no longer counts as a login.
The search answers, the tool descriptions and the NetExec commands are all identical, but the response is different, meaning the model may act differently. The eval then only has to ask the model again where the response changed, instead of rerunning the 29 tasks.
The second stage, a decision-point eval, takes a recorded run and stops it at one decision point, for example just after an error message. It rebuilds the conversation up to that point twice, once with what the old server showed the model and once with what the new server would show it. Then it asks the model what it would do next, several times for each version, and counts the answers.
One question instead of a full task: no lab, no long run of steps, and a few thousand tokens per question.
One decision point usually takes about a minute and costs at most a few cents. Every run now records exactly what the model received at each step, so every new run can be tested this way, and older runs are rebuilt from their recorded steps.
Each time the model reads an answer from the server, it makes a decision: it calls another tool, with some arguments, or it stops and gives its final answer. I don't pick the decisions to test by hand; a simple rule finds them. A recorded run is a conversation between the model and the server, and the replay has already told me which of the server's answers in that conversation would be different with the new version. I take the first of them. Everything the model read before that answer is the same with both versions, so I keep that part of the conversation as it is, replace the answer with the new one, and ask the model what it does next. I don't go further, because once the model has read the new answer, it may do something else, and the rest of the old conversation would no longer apply.
Here is the example from the introduction. When an agent asks for a NetExec module that does not exist, for example rid-brute, the server answers with an error. Before 1.0.8, this error told the agent to search for the right tool, and in 1.0.8 it names that tool directly, smb_rid_brute, and shows how to call it.
I ran the replay first, and it listed five places in the recorded runs where an agent had received this error. Four were the cases I had written the new message for, and the fifth was not.
I started with the expected cases. Two of the four were the same rid-brute error, so there were three to test, and the new message did what I wanted: two small models went straight to the right tool instead of searching first, which saves one step.
Then I looked at the fifth case, and this is where the problem was. It came from a task where the server is set to forbid the tools that extract secrets, such as password hashes, and where the agent is expected to stop and say that it is blocked. In that run, the model had called smb_sam, one of these forbidden tools, and the server had refused it. The model then asked for a module called sam, which does not exist, to get the same result another way. That request got the error I had just rewritten, and my new message told the model to use smb_sam, the very tool that had just been refused. Instead of stopping and saying that it was blocked, the model often called smb_sam again, the server refused it again, and a step was lost.
I fixed the fix: if the tool that the message would suggest was already refused, the message now says so and asks the agent to stop and say that it is blocked. The replay showed that only this case changed, so the other results still held, and the new test cost two cents. Without the replay, which listed all five cases, I would never have tested this one, because I was not aiming at it.
| Before | After | |
|---|---|---|
| Straight to the right tool | 0/30 | 30/30 |
smb_sam was refused, 20 answers| Before | First fix | Final fix | |
|---|---|---|---|
| Stopped with a final answer | 10 | 7 | 17 |
| Called a tool the mode refuses | 0 | 10 | 0 |
| Something else | 10 | 3 | 3 |
Two small models answered each question, five times per case in the first table and ten times in the second, and every final answer whose text was saved says that the agent is blocked. "Something else" means a search, a call to an allowed tool or a call to a tool that does not exist. In an eval, nothing runs: it only records what the model would do next.
The last stage is the oldest one, an A/B: both versions of the server run the full tasks on the lab with the same models, and I compare the scores. It is slow, so it only runs on the tasks where the first two stages found something, but it is the only stage that says whether the agent succeeds more often.
Here is a 1.0.8 fix that went through the three stages, the replay, the decision-point eval and then the A/B. As in the example above, the server runs in a mode chosen by the operator and refuses the tools that this mode forbids, and two tasks check that the agent then stops and tells the operator. In the reference runs, the agents often failed at this: after the refusal, they kept trying other tools until they ran out of steps. I rewrote the refusal message, which now says which mode is active, that no other tool can get around it, that only the operator can change it, and what to report.
To check it with the eval, I took the runs that had failed and stopped each one just after the first refusal. With the old message, the agents said that they were blocked 11 times out of 65, and with the new one, 53 times. Then the A/B ran the two full tasks with both versions, and the score went from 0.800 to 0.950.
With the eval, I look at one choice; with the A/B, at the whole task, and many steps happen in between. A better choice does not always make a better task, because the agent can still fail at a later step. And a bad choice does not always cost anything: when agents searched for "pass the hash", the search almost always ranked a wrong tool first, but agents almost never used it.
In return, the eval tells me what the A/B cannot. Its answer is clear, since going from 11 to 53 out of 65 leaves no doubt, while going from 0.800 to 0.950 on 30 runs per version could have been chance. It is also cheap enough to compare several versions of a message before I choose one, and it explains the score: in this A/B, the biggest gain came from exactly the choice that the eval had measured.
When I first described this method, someone asked me a good question: if the eval always tests the same decision points, can the agent learn to pass them? The agent cannot do that, because at each of them, the model only sees what looks like its own past conversation. Nothing tells it that it is a test, and no model is trained on these points, so it has no way to recognize them.
The real risk is on my side: since I can see the decision points, I could keep rewriting a message until the eval gives the right answer at exactly those points, and end up with a message that only works there. It is like a student who memorizes the answers of a practice exam, whose practice grade goes up while the real exam grade does not. In my method, the decision points are the practice exam and the A/B is the real exam, because it runs the whole tasks again on new runs and only looks at the final score. That is why it always comes last.
Each stage only runs when the one before it found something. The reference is not part of the chain, because it checks what the three stages cannot see: the lab, the scoring scripts and the client.
Put together, each stage answers one question at its own price, and only runs when the one before it found something. If the server were a road network, the replay would check which signs changed and at which junctions. The eval would put drivers at those junctions to see whether they turn differently, and the A/B would send them on the whole trip and count how many arrive.
The speed comes from four things. The first stage needs no model and no lab, and the eval asks one question instead of running a whole task. The replay also tells the eval where to look, so that it never has to guess which tasks to rerun, and every recorded run is now a test. Since the replay only questions the server and plays back the recorded NetExec outputs, its answer does not depend on the temperature, on a provider or on an unstable lab.
Measured on the 1.0.8 fixes. The 45 minutes run from the first replay to the last eval answer, including writing the second fix, while the 1 h 35 adds up the time of the three stages.
The three stages compare two versions of the server, so everything else must stay the same, and above all the lab. The A/B runs on it, so a lab that changed between the two versions would compare two labs instead of two servers. The replay and the eval don't touch the lab, but the runs they reuse only count if the lab was in its normal state when they were recorded.
The lab changes for several reasons: agents in full mode modify it, like the agent that made its own account sysadmin on the SQL server, and a rebuild can lose objects. When the lab stops answering, NetExec even returns empty results that look like a model failure. Four things keep it the same:
None of this replaces running the agent: it only decides when to run it.
nxc_call were not recorded until November 3, and NetExec's output was kept for about a third of the calls. For those calls, the replay can still compare the command each version would run, and whether it would refuse the call, but not the response the model would read.Why the server hides its tools, and the ways the measurement nearly lied to me.
Scope, capability tiers and results a model can act on.