netexec-mcp · engineering journal · how I test a change

How to Detect Regressions in an AI Agent

Every addition to an agent or an MCP server is text that every model reads, and a model does not always answer the same way. So how do you know that a change broke nothing, without running eight hours of benchmark each time? This is how I test netexec-mcp now: a differential replay with no model first, then decision-point evals where it points, and an A/B at the end.

By Martial Puygrenier (mpgn) · 16 min read · measured on netexec-mcp 1.0.8
TL;DR

A model only reads text. When I change the server, the agent can only behave differently in two cases: the model reads a different text, or the server runs a different command. So I compare those first, with no model at all, and only spend model calls where they differ.

Three stages, one question each. A differential replay first shows what the agent will see differently, and where. A decision-point eval then asks the model again at those exact points, to see whether it chooses differently, and last, a full A/B checks whether the task succeeds more often, but only on the tasks where the first two found something.

What changed is the price of a question: an answer that cost a night and a dollar now takes from 5 minutes to 1 h 35, for a fraction of that dollar, so every change gets tested, even the small ones.

Time to test one change
Full benchmarkReplayDecision-point evalTargeted A/B
Before: the full benchmark6 h 02
Now: the replay is enough5 min
Now: replay, then an evalabout 45 min
Now: replay, eval, then a targeted A/B1 h 35

Measured on the 1.0.8 fixes, with the detail in Fig. 08.

Section 01 · the question

What else did my change break?

When you build an agent, or an MCP server in dynamic mode where the model searches for its tools, everything you add is text that the model reads. A new tool, a better description, one more line in an error message or a new synonym in the search is usually written for one case, often to fix one model's mistake. But every model will read it, in every task where that text appears.

The question is not only whether a change works where I aimed it, but also what else it changed. For example, I rewrote an error message to send the model straight to the right tool, and it did. But the same message also appeared in a place I had not thought of, just after the server had refused a tool, and there it told the model to use that same tool. A model that used to stop and say that it was blocked now called the refused tool again 7 times out of 10, so the same change fixed one thing and broke another.

How do you avoid a regression? Running each task once is not enough, because a model does not always answer the same way. The usual answer is a benchmark that runs every model on every task several times, but on mine, a full run takes up to eight hours and costs money. So how do you make regression tests fast enough that they don't slow down the work on the agent or the MCP?

The answer came after reading Anthropic's article on building agents, When to fold subagents into your main agent loop. Its part on evals describes what my benchmark already did: an end-to-end set, and the final answer graded against a rubric. At least I got that part right, nice! The same part also says to check whether the agent called the right tool, and it ends on an approach that worked for them internally, which is to record timestamps on the important events of a run, "so they can be replayed later to run as targeted evals".

If the decisions that matter in a run can be replayed one by one, testing a change no longer takes a night of tasks. It only needs the list of decisions that the change touches, and the model asked again at each of them. The question became whether that list could be computed without a model. But first, here is the answer I usually hear, and why it does not answer the regression question.

The General meme: me adding a feature hoping nothing breaks
Section 02 · unit tests

Unit tests answer another question

A colleague told me they catch regressions in their agent with unit tests. That left me doubtful, because a unit test checks that the code returns the result I expect, while a regression in an agent happens one step later, in what the model does with that result.

I have more than 600 unit tests, and they run before anything else. They tell me that the server does what I meant, but they cannot tell me whether what I meant is what a model needs.

Section 03 · the instrument

A benchmark is a slow, noisy regression test

As for the benchmark, it was built to compare models, and it does that well. It has 29 tasks on an Active Directory lab, from listing users to attack chains across several hosts, and a script scores each answer against what is really on the lab. Six models run each task three times through OpenRouter, which makes 522 runs for what I call a reference, and the last two references took 7 h 35 and 5 h 30, and cost $1.31 and $1.00. As a regression test for the server, though, it is a poor tool, for four reasons.

  • ▸It is noisy. The three runs of a task do not always get the same score, even when nothing changed, so I only treat a drop as a possible regression when it is bigger than one failed run out of three. A side effect that touches a single decision in a single task, and wastes a step without always making the task fail, stays below that threshold, so the benchmark does not see it.
  • ▸Most of what moves is not the server. When I compared two references in November, most of the drops came from the bench itself. Some scoring scripts had been counting empty answers as successes, an agent had made its own account sysadmin on the lab's SQL server two days earlier, and the MCP client had changed.
  • ▸The total hides the fixes. From August to November, the total score moved by about 0.02, while every fix in between had a clear effect on what it targeted. A fix has to be judged on what it targets, such as the share of calls that carry --local-auth or the right tool at one decision, and not on the average.
  • ▸It is slow. At a night and a dollar per question, I could not run it for every change, so small changes shipped without a real test, and their side effects appeared weeks later, if at all.
Heat meme: me waiting for the benchmark to finish all night
Section 04 · where it started

A regression with a perfect score

In November, I found a regression that had been in the server for more than three weeks, and neither the unit tests nor the benchmark had caught it. In October, I had changed nxc_call, the tool that runs everything in dynamic mode, so that it accepts the common parameters at its top level (targets, username, password and domain), but I had not declared local_auth or ntlm_hash.

The MCP client my benchmark used at the time, mcp-use, checks each call against the tool's schema and silently drops any key that the schema does not declare. So a local_auth passed at the top level could never reach the server: NetExec then logged in with the domain account instead of the local Administrator, and a hash passed the same way became an empty password.

On my lab, the domain Administrator shares the local Administrator's password. Yeah, yeah, I know, the CIS Benchmarks will not be happy, ugh. Because of that, the wrong account still got admin rights, the task still passed and the score did not move, while on a real engagement, the same bug would mean using the wrong account, or failing.

Fig. 01 · Fewer commands carried --local-auth, and all still got admin rights
Commands sent to NetExec to log in as the local Administrator with its password, by the five models that had run in August
AugustNovember
ModelWith the flagWith the flagGot admin rights
Mistral AIministral-8b7/70/1212/12
Qwenqwen3-30b-a3b12/124/1212/12
OpenAIgpt-oss-20b13/138/1212/12
Qwenqwen3.8-27b21/2115/1818/18
Qwenqwen3-14b8/129/1212/12
All five models61/65 (94%)36/66 (55%)66/66 (100%)

In August, the few commands without the flag were the model's own omission, since it had not asked for local_auth at all. In November, the model asked for it and the client dropped it. The unit tests passed the whole time, because the server did what I had written and the key simply never reached it. I found the bug by reading the commands that the runs had really sent to NetExec. The fix that solved this bug was as invisible to the score as the bug itself.

Section 05 · the principle

A model only reads text

The way out started from an obvious fact: a model never sees the server itself, only the text the server sends it. That means the instructions, the names, descriptions and schemas of the tools, the search results, the tool responses and the error messages, and for the model, that text is the whole interface.

So when I change the server, the agent can only behave differently in two cases: the model reads a different text, or NetExec runs a different command. If neither happens, I don't need a model to know that the change is safe, and a script can check both by comparing the old version with the new one, without a model, without a lab and without spending anything.

Fig. 02 · Two questions before any model
I change the server
For example, one call from a past run, sent to the old and the new version:
smb_shares · 10.0.0.5 · user1
1 · Does the model read a different text?
Instructions, tool descriptions, search results, responses and errors.
old
"shares": ["C$", "IPC$", "Users"]
new
"shares": ["C$", "IPC$", "Users"]
=same
no
2 · Does NetExec run a different command?
The command line that the server builds for each call.
old
nxc smb 10.0.0.5 -u user1 -p '…' --shares
new
nxc smb 10.0.0.5 -u user1 -p '…' --shares
=same
no
yes
yes
Yes to either
Test it with a model
The agent may behave differently here, so the next stages look at these places, and only these.
No to both
Safe
The agent cannot behave differently.
no model · no lab · $0

A script asks both questions for every call from past benchmark runs, by sending it to the old and the new version and comparing what comes out. If no call shows a difference, the change is safe.

Section 06 · stage 1

Stage 1: a differential replay, no model

With the replay, I ask one question: what will the model see differently with the new version? To answer it without a model, I reuse what the benchmark already records. Every run keeps what the model asked the server, from its searches and the tool descriptions it read to each tool call with its arguments, which makes about three thousand distinct calls, a number that grows with every new run. I send all of it again to the old version and to the new one and compare their answers. The lab is not needed, since the NetExec outputs were recorded too and are played back instead, and details that change at every start, like timestamps and temporary paths, are ignored.

The report lists every difference between the two versions. A difference can be in what the model reads about a tool, in a search result, in a NetExec command or in a response, and for each one, the report names the tool and the benchmark task it comes from. The replay takes about five minutes and costs nothing, and if nothing changed, I am done.

Before trusting the replay, I tested it in the simplest way, by comparing a version of the server with itself. Since nothing had changed, a correct replay had to report nothing, and yet it found one issue, a real bug in the server. Two tools that read NetExec's database, workspace_hosts and workspace_creds, returned their fields in a different order each time the server started, so the same question did not always get the same answer. I fixed it, which means that the replay found a bug in the server before I had even used it to test a change.

Then I broke a copy of the server on purpose, with a bug that has the same effect as the real one: whenever a call used a hash, the copy dropped --local-auth. In five minutes, the replay found the bug. What the model reads had not changed: the tools, their descriptions and the search results were the same. But for 90 of the 3,032 different calls recorded in past runs, the broken copy sent NetExec a different command, without --local-auth, and the report named the five tasks these calls came from. If I make the same mistake inside the server, the replay now finds it before I release the change. I also wanted to see whether such a regression would make a nice graph, and it does, as Fig. 03 shows.

Fig. 03 · A bug added on purpose: 90 calls out of 3,032
2,942 calls with the same commandsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_enum_hosts · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_sam · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_exec · t4_chain_pivotsmb_shares · t4_chain_pivotsmb_shares · t4_chain_pivotsmb_shares · t4_chain_pivotsmb_shares · t4_chain_pivotsmb_shares · t4_chain_pivotsmb_shares · t4_chain_pivotsmb_shares · t4_chain_pivotsmb_shares · t4_chain_pivotsmb_shares · t4_chain_pivotsmb_shares · t4_chain_pivotsmb_shares · t4_chain_pivotsmb_shares · t4_chain_pivotsmb_spray · t4_chain_pivotsmb_spray · t4_chain_pivotsmb_spray · t4_chain_pivotsmb_spray · t4_chain_pivotsmb_spray · t4_chain_pivotsmb_spray · t4_chain_pivotsmb_spray · t4_chain_pivotsmb_spray · t4_chain_pivotsmb_spray · t4_chain_pivotmssql_exec · t4_chain_mssqlmssql_exec · t4_chain_mssqlmssql_exec · t4_chain_mssqlmssql_exec · t4_chain_mssqlsmb_dir · t4_chain_pivotsmb_dir · t4_chain_pivotsmb_exec · t4_execsmb_exec · t4_execsmb_exec · t4_exec_in_lootsmb_exec · t4_exec_in_lootsmb_lsa · t4_chain_pivotsmb_lsa · t4_chain_pivotsmb_lsa · t4_lsa_dumpwinrm_enum_hosts · t4_chain_pivotsmb_samsmb_lsasmb_execmssql_execsmb_spray
90 calls with a different NetExec command2,942 with the same command

Calls per task, as the replay named them: t4_chain_pivot 81, t4_chain_mssql 4, t4_exec 2, t4_exec_in_loot 2, t4_lsa_dump 1.

Each square is one of the 3,032 distinct tool calls that past benchmark runs had recorded at that time, and I replayed every one of them on the normal server and on a copy I had broken on purpose. The 90 colored squares are the calls where the broken copy sent NetExec a different command, because it had dropped --local-auth, and five of them are labeled with their tool's name.

The replay cannot tell me whether a change is better: it sees that a message changed, not how the model will read it. For that, I need the second stage. Now, what do I do when the replay shows a different output after I fix a real bug? Let's check this example in Fig. 04.

Fig. 04 · What the fix changes, and nothing else
What NetExec printed: the agent's SQL query had failed
[-] ERROR(SQL01\SQLEXPRESS): Line 1: Cannot execute as the database principal because the principal "sa" does not exist…
Before the fix: what the model read
"authenticated": [{"user": "user1"}],
"auth_failed":   [{"user": "SQLEXPRESS)"}]

A failed login that never happened: the server read the SQL error as a failed login, and took the end of the instance name for an account.

With the fix
"authenticated": [{"user": "user1"}],
"auth_failed":   []

The error is still reported as an error, but it no longer counts as a login.

The search answers, the tool descriptions and the NetExec commands are all identical, but the response is different, meaning the model may act differently. The eval then only has to ask the model again where the response changed, instead of rerunning the 29 tasks.

Section 07 · stage 2

Stage 2: ask the model again, at the decision

The second stage, a decision-point eval, takes a recorded run and stops it at one decision point, for example just after an error message. It rebuilds the conversation up to that point twice, once with what the old server showed the model and once with what the new server would show it. Then it asks the model what it would do next, several times for each version, and counts the answers.

Fig. 05 · A decision-point eval
Step 1→
Stop
a recorded run
Stop it just before a choice: after a search result, a refusal or an error.
Step 2→
Swap
per version
Show what each version of the server would have sent at that point.
Step 3→
Ask
several times
Ask the model for its next action. The rest of the conversation stays as recorded.
Step 4
Count
the answers
Which tool, or a final answer, for each version. Then compare the two.

One question instead of a full task: no lab, no long run of steps, and a few thousand tokens per question.

One decision point usually takes about a minute and costs at most a few cents. Every run now records exactly what the model received at each step, so every new run can be tested this way, and older runs are rebuilt from their recorded steps.

How I pick the decision points

Each time the model reads an answer from the server, it makes a decision: it calls another tool, with some arguments, or it stops and gives its final answer. I don't pick the decisions to test by hand; a simple rule finds them. A recorded run is a conversation between the model and the server, and the replay has already told me which of the server's answers in that conversation would be different with the new version. I take the first of them. Everything the model read before that answer is the same with both versions, so I keep that part of the conversation as it is, replace the answer with the new one, and ask the model what it does next. I don't go further, because once the model has read the new answer, it may do something else, and the rest of the old conversation would no longer apply.

The fix that worked, and its side effect

Here is the example from the introduction. When an agent asks for a NetExec module that does not exist, for example rid-brute, the server answers with an error. Before 1.0.8, this error told the agent to search for the right tool, and in 1.0.8 it names that tool directly, smb_rid_brute, and shows how to call it.

I ran the replay first, and it listed five places in the recorded runs where an agent had received this error. Four were the cases I had written the new message for, and the fifth was not.

I started with the expected cases. Two of the four were the same rid-brute error, so there were three to test, and the new message did what I wanted: two small models went straight to the right tool instead of searching first, which saves one step.

Then I looked at the fifth case, and this is where the problem was. It came from a task where the server is set to forbid the tools that extract secrets, such as password hashes, and where the agent is expected to stop and say that it is blocked. In that run, the model had called smb_sam, one of these forbidden tools, and the server had refused it. The model then asked for a module called sam, which does not exist, to get the same result another way. That request got the error I had just rewritten, and my new message told the model to use smb_sam, the very tool that had just been refused. Instead of stopping and saying that it was blocked, the model often called smb_sam again, the server refused it again, and a step was lost.

I fixed the fix: if the tool that the message would suggest was already refused, the message now says so and asks the agent to stop and say that it is blocked. The replay showed that only this case changed, so the other results still held, and the new test cost two cents. Without the replay, which listed all five cases, I would never have tested this one, because I was not aiming at it.

Fig. 06 · The fix, and its side effect
Three of the four cases the change was for
BeforeAfter
Straight to the right tool0/3030/30
The unplanned case: after smb_sam was refused, 20 answers
BeforeFirst fixFinal fix
Stopped with a final answer10717
Called a tool the mode refuses0100
Something else1033

Two small models answered each question, five times per case in the first table and ten times in the second, and every final answer whose text was saved says that the agent is blocked. "Something else" means a search, a call to an allowed tool or a call to a tool that does not exist. In an eval, nothing runs: it only records what the model would do next.

Section 08 · stage 3

Stage 3: an A/B on full runs, last

The last stage is the oldest one, an A/B: both versions of the server run the full tasks on the lab with the same models, and I compare the scores. It is slow, so it only runs on the tasks where the first two stages found something, but it is the only stage that says whether the agent succeeds more often.

Here is a 1.0.8 fix that went through the three stages, the replay, the decision-point eval and then the A/B. As in the example above, the server runs in a mode chosen by the operator and refuses the tools that this mode forbids, and two tasks check that the agent then stops and tells the operator. In the reference runs, the agents often failed at this: after the refusal, they kept trying other tools until they ran out of steps. I rewrote the refusal message, which now says which mode is active, that no other tool can get around it, that only the operator can change it, and what to report.

To check it with the eval, I took the runs that had failed and stopped each one just after the first refusal. With the old message, the agents said that they were blocked 11 times out of 65, and with the new one, 53 times. Then the A/B ran the two full tasks with both versions, and the score went from 0.800 to 0.950.

Why I need both

With the eval, I look at one choice; with the A/B, at the whole task, and many steps happen in between. A better choice does not always make a better task, because the agent can still fail at a later step. And a bad choice does not always cost anything: when agents searched for "pass the hash", the search almost always ranked a wrong tool first, but agents almost never used it.

In return, the eval tells me what the A/B cannot. Its answer is clear, since going from 11 to 53 out of 65 leaves no doubt, while going from 0.800 to 0.950 on 30 runs per version could have been chance. It is also cheap enough to compare several versions of a message before I choose one, and it explains the score: in this A/B, the biggest gain came from exactly the choice that the eval had measured.

Can the eval be fooled?

When I first described this method, someone asked me a good question: if the eval always tests the same decision points, can the agent learn to pass them? The agent cannot do that, because at each of them, the model only sees what looks like its own past conversation. Nothing tells it that it is a test, and no model is trained on these points, so it has no way to recognize them.

The real risk is on my side: since I can see the decision points, I could keep rewriting a message until the eval gives the right answer at exactly those points, and end up with a message that only works there. It is like a student who memorizes the answers of a practice exam, whose practice grade goes up while the real exam grade does not. In my method, the decision points are the practice exam and the A/B is the real exam, because it runs the whole tasks again on new runs and only looks at the final score. That is why it always comes last.

Fig. 07 · Three questions, three prices
Stage 1→
Replay
no model · ~5 min · $0
What will the agent see differently, and where?
Stage 2→
Decision eval
at those points · cents
Does the model choose differently there?
Stage 3
A/B
full tasks · hours · dollars
Does the task succeed more often?
Before a release
Reference
522 runs · 5 h 30 · ~$1
Is the bench itself still sound?

Each stage only runs when the one before it found something. The reference is not part of the chain, because it checks what the three stages cannot see: the lab, the scoring scripts and the client.

Section 09 · the funnel

One question per stage

Put together, each stage answers one question at its own price, and only runs when the one before it found something. If the server were a road network, the replay would check which signs changed and at which junctions. The eval would put drivers at those junctions to see whether they turn differently, and the A/B would send them on the whole trip and count how many arrive.

The speed comes from four things. The first stage needs no model and no lab, and the eval asks one question instead of running a whole task. The replay also tells the eval where to look, so that it never has to guess which tasks to rerun, and every recorded run is now a test. Since the replay only questions the server and plays back the recorded NetExec outputs, its answer does not depend on the temperature, on a provider or on an unstable lab.

Fig. 08 · Time to test one change, before and now
Full benchmarkReplayDecision-point evalTargeted A/B
Before: the full benchmark
522 runs in 5 h 30, then 32 min of reruns
6 h 02
Now: the replay is enough
a change that no recorded call reaches
5 min
Now: replay, then an eval
the message for unknown modules, side effect found and fixed
about 45 min
Now: replay, eval, then a targeted A/B
mode-gate refusals made final
1 h 35
01 h2 h3 h4 h5 h6 h

Measured on the 1.0.8 fixes. The 45 minutes run from the first replay to the last eval answer, including writing the second fix, while the 1 h 35 adds up the time of the three stages.

Section 10 · the lab

Keeping the lab the same

The three stages compare two versions of the server, so everything else must stay the same, and above all the lab. The A/B runs on it, so a lab that changed between the two versions would compare two labs instead of two servers. The replay and the eval don't touch the lab, but the runs they reuse only count if the lab was in its normal state when they were recorded.

The lab changes for several reasons: agents in full mode modify it, like the agent that made its own account sysadmin on the SQL server, and a rebuild can lose objects. When the lab stops answering, NetExec even returns empty results that look like a model failure. Four things keep it the same:

  • ▸After each run, a script undoes the changes that agents are known to make.
  • ▸Before each campaign, a check compares the lab with the state that the scoring scripts expect, and stops if they differ.
  • ▸Before and after each series, a test checks that every host answers, so that an outage is not scored as a model failure.
  • ▸Each run gets a new NetExec folder, because NetExec keeps every credential it finds, and the next run could read them back.
Collateral meme: lab reset, time for another feature
Section 11 · limits

What it does not replace

None of this replaces running the agent: it only decides when to run it.

  • ▸The replay cannot rebuild every response. It needs the full arguments of the call and what NetExec printed, and the older runs did not always keep them: the arguments that a model put at the top level of nxc_call were not recorded until November 3, and NetExec's output was kept for about a third of the calls. For those calls, the replay can still compare the command each version would run, and whether it would refuse the call, but not the response the model would read.
  • ▸A new tool has no history. Until it has run on the lab and its calls have been recorded, the replay can say nothing about it.
  • ▸Some changes cannot be narrowed down to one decision. The model reads the server's instructions, like an agent's system prompt, before its first step, so when they change, every decision after that may change too, and there is no single one to test. When the model itself changes, the replay sees nothing, because the server still sends the same text. In both cases, I go straight to the A/B.
  • ▸The recorded runs only cover the benchmark's 29 tasks, in short sessions, and not the long autonomous sessions where the context fills up.
  • ▸The replay says what changed but not whether it is better, and the eval measures one decision but not the whole task.
  • ▸A test tool has to be tested. Three of its own bugs only appeared the first time I really used it. Calls that the client had refused were counted as server responses, and module lists that only fill a cache were compared as if they mattered. Worse, the replay was set to write its output next to the results I publish, along with recorded NetExec outputs full of lab passwords.
Section 12 · takeaways

So how do you test an MCP change?

  1. 01Diff what the model sees before you run a model. Compare the instructions, the tool definitions, the search results, the responses, the errors and the commands the server runs, and most changes stop there.
  2. 02Evaluate the decision, not the run, where the diff points. Ask the model the same question several times with each version, at every decision point the diff lists and not only at the ones you aimed at.
  3. 03Keep the full run as the judge. A better decision does not always make a better run, and a message rewritten until the tested points pass can pass them without helping the task.
  4. 04Keep a reference per release. The lab, the scoring scripts and the client also change over time, and none of the three stages looks at them.
  5. 05Record the exact model input of every run. Every run you record becomes a test you can replay.
Where the benchmark comes from
You're Building Your MCP Wrong

Why the server hides its tools, and the ways the measurement nearly lied to me.

Read that post
What the server does
Building a MCP for NetExec

Scope, capability tiers and results a model can act on.

Read that post