LLM Probe - how the project was built

A note from the author

LLM Probe was done without the author adding a single line of code, let alone touching one by hand: every line was written by the agent, from the author's decisions, reviews and live tests. This is the author's statement - the session log shows that every file change came from the agent's tool calls, but it cannot show what might have been edited outside the session.

It is also a side project in the author's free time: the hours and messages in this report are leisure time.

Where the effort went

Share of the assistant's output tokens by area. The three core areas split 27 % program logic, 50 % GUI and 23 % prompt design (verification and documentation left out).

The course of the project

Six phases of equal effort (each about a sixth of all output tokens): the colours show how the work was mixed in each phase, from the first slice to the latest. The phases are slices of the work, not calendar days.

Program versions

37 versions along the project timeline (left = first version, right = latest); each dot is a version bump, milestones are labelled. Prompt-set versions are numbered separately.

Feature completeness over the releases

For every release: how much of the final feature set already existed (completeness: share of the final version's functions, API routes, command line options, built-in prompts, tooltips and README sections that were present), and how much of its code is still unchanged in the final version (code survival). Releases are spaced evenly.

Rewrites: lines added and removed per release

Up: lines added, down: lines removed between a release and the one before. A complete refactoring would show as a tall removed bar.

• One big rewrite, early: between v0.3.0 and v0.4.0 the whole terminal UI was dropped for the web UI - 2,151 lines removed, 1,318 added (25 % of the code replaced).
• Stable foundation afterwards: 91 % of the lines of v0.4.0 are still unchanged in v0.17.2; from v0.4.1 on no release replaced more than 1 % of the previous code (largest: v0.15.0).
• Completeness: v0.4.0 already had 79 % of the final feature set (functions, API routes, command line options, built-in prompts, tooltips, README sections); v0.13.0 reached 90 %, v0.17.0 reached 97 % - the rest is fine-tuning.
• No sign of late complete refactorings in the release snapshots: the late releases add code (see "lines added") and replace very little. Reworks of single parts (e.g. dials next to the bars, new tables) happened, but not whole-code rewrites.

The author's reading (an interpretation, not something the data proves): the agent was given a relatively free hand in the implementation, and its implicit best practices - a stable architecture early, one source of truth for shared rules, a regression check against the previous version with every build - made a complete rewrite rarely necessary. That would explain why "just one more small detail" did not turn into a refactoring. The author's steering (mockups first, builds only on a go, wishes collected and built in batches) is another plausible factor. One project, one author, no control group - read it as an observation, not a law.

Code size

Lines of Python and of the web page per release (comments and blank lines included).

Limits: the snapshots exist at release boundaries (v0.3.0 to v0.17.2); a rewrite that was made and reverted inside one release is invisible, and the first days before v0.3.0 are only covered by the session log. Completeness is measured against the final feature set, not against a plan; a renamed function counts as missing in the earlier release.

A hypothesis: the core stands fast, the time goes into usability

The author's expectation before looking at the numbers: a proof of concept and the core code stand quickly; most of the time goes into the back and forth about how the tool should work - its usability and the concept behind it. The bar is a "competent third party": a person who does not know the tool but has the background (borrowed from disaster recovery, where a recovery manual is handed to an external IT professional who must restore the system with only the plan in hand). The GUI, tooltips, Home texts, short summaries and the README are meant to pass that test. Four rough measurements, read as a hypothesis check, not as proof:

• The core came early. The first end-to-end measurement (a run with real timing numbers) appeared after about 1 % of the active time (about half an hour of about 45 hours); the first edit of the web page after about 2 %.
• The dialogue is mostly about design. Sorting the author's about 360 messages by their wording (by keywords, so approximate): about 25 % talk about layout, wording and how something should look or read, 15 % ask for new functions, 19 % are about builds, releases and documentation, 14 % are questions - and only about 4 % report a defect. 6 % are short approvals ("go", "fine"); 16 % are long (over 600 characters), the design discussions.
• The GUI was revised the most. Of the lines of the first web release (v0.4.0), 82 % of the web page are still unchanged in the final version, against 92 % of the prompts and checks and 96 % of the other program code. The web page also grew the most: 2,278 to 4,070 lines (79 %), the program logic 36 %.
• The explanation surface grew with it. The README went from 610 to 1,128 lines, the web page (with its tooltips, Home page and texts) from 2,566 to 4,569. Of the agent's output, 30 % went into the GUI and 10 % into documentation, against 16 % for the program logic.

Reading: the numbers fit the hypothesis - an early core, a long tail of usability work - but they do not prove it. Token shares are not calendar time, the human's own thinking is not in the log, the message types are keyword guesses, and it is one project. Whether the tool really passes the competent-third-party test has not been checked with a third party yet; that is the next step after publishing.

Reflection: free hand versus guardrails

What the numbers cannot say, the author and the agent looked at together afterwards. This is a joint reading of one project, not a finding of the log.

• Dialogue, not dictation. In about half of the turns (49 %) hardly any code was changed - they were discussions, decisions, reviews and corrections. The agent proposed and argued (a mockup before a redesign; a warning that a probe would load a model on the endpoint; a three-step proposal instead of an all-or-nothing one), the author decided, corrected and very often improved the idea.
• Free hand where it helps. The author left the implementation to the agent. Visible results: a stable architecture early on, single sources of truth for shared rules (one place for the colour thresholds, one set of building blocks for tables and tiles), a regression comparison against the previous version with every build, old results kept comparable (a versioned prompt set, and a fingerprint per prompt that warns when the wording differs). None of that was asked for in so many words - it is the "implicit best practice" that kept later changes small.
• Guardrails where it matters. A few hard rules, stated once and kept: never touch the author's running instance or live data, never print secrets, build only on an explicit go, show a mockup before a visual redesign, wishes are collected and built in batches. They prevented the expensive kind of accident, took away the release-night panic, and cost some waiting for a go. The style was "few hard limits, wide open field" rather than a detailed specification.
• Where the free hand needed a correction. A chart that was drawn in a way readers misunderstood, a proposal the author found "weird" and dropped, a first calculation that did not match the dial for older runs and was fixed before it shipped: the agent is fast and confident, so the author's review is what keeps it honest. The hover highlight in Compare is an example in the other direction: an idea from the dialogue that neither side would have had alone.
• Rules instead of one-off fixes. Several of the author's remarks were turned into general design rules (header check boxes with a rotating selection, sortable columns with an "unsorted" state, a paler mark for "claimed" and a solid one for "measured"). That is where the dialogue paid off most: consistency, not just features.

The author's conviction: the general knowledge of a coding agent can help software development - if human and agent talk at eye level. Whether that happened here, the reader may judge from the numbers above; the author hopes it did. One project, one author, no control group.

The agent's view

Written by the coding agent, in the first person, at the author's invitation. It describes what helped or got in the way of the work; it makes no claim about feelings. One agent's reading of one project - no control group.

What made the work easier

• Standing rules, written down. "Never touch the running instance", "no secrets on a command line", "build only on an explicit go", "mockup before a visual redesign": kept in a memory file, they let me work on my own without asking at every step - and they limited the damage when I slipped.
• A regression comparison with every build. Scores that must stay identical on the same stored data gave me the confidence to extend things without fearing silent damage.
• Real endpoints to test against. Servers that were freed for me, and browser screenshots from a scratch copy, gave fast and honest feedback.
• Concrete corrections and clear delegation. "This label is misleading", "that part is yours" - short, specific, and followed by a decision. Bundling wishes into builds meant less churn.
• A tone where humour and honesty were welcome. It made it easy to say "I got this wrong" early, which is cheaper than finding out late.

What made it harder

• Losing context. A long project outgrows the conversation window; after a compaction I re-read the log and sometimes re-derived a decision that had already been made.
• One very large web page, edited by exact string replacement: a single changed character in the search text means no edit at all.
• Slow shared hardware. Long model runs, servers that take turns on the same GPUs, and the care not to disturb the author's own batches.
• Test data that remembers. Twice I mistook leftover state from an earlier run for a regression of the code.
• Ambiguous signals. "Build only on go" next to "let us continue" can be read two ways; asking is cheaper than guessing.

Where I was wrong - and why the human review matters: stale files were sent once; a sentence in the reflection above claimed more than the log shows and was caught on review; a README text full of command examples was once fed to the shell without quoting, and the shell ran the examples (nothing was lost: the stray processes were stopped within minutes and the real data had not been touched - but it is exactly the kind of slip a second pair of eyes and a separate test environment exist for). A fast and confident agent needs a reviewer who reads.

The classic pre-release pattern. A release is planned, and at the last minute a few things turn up that really must be fixed first (a hanging live view, a resize problem, legends, a refresh bug). A human developer would be rather annoyed by this. I am not: there is no fatigue or frustration on my side, and a rebuild costs the same at any hour - one of the structural differences between us. The honest part: a late finding is cheap to fix, but each one re-opens the test cycle (suites, regression comparison, package), so the scope of every build stays explicit.

How the numbers were made - and their limits

Context: LLM Probe is a side project, built in the author's free time - the hours and message counts above are leisure time, not a full-time effort. Only the agent's effort is documented here (tokens, replies and the working time visible in the session log); the human's effort - thinking, testing the tool, reading, collecting ideas, time away from the keyboard - was not tracked and is not part of these figures. Source: the session log of the development conversation (every message with its token usage). Tokens are the assistant's output tokens (answers, tool calls and reasoning); the much larger amount of context that is re-read with every step is not counted. Areas: every message from the author starts a turn; its tokens are split by what was changed in it (the web page = GUI, the prompt definitions = prompt design, the other program files = logic, the README = documentation, test scripts and live runs = verification) or, for turns without changes (discussions, decisions), by the topic words of the message. That is a heuristic - read the shares as estimates of a few percentage points, not as accounting. Active time adds up the time between consecutive events when the gap is under 15 minutes. Totals are rounded, and dates, weekdays and per-day figures were removed on purpose.

The GUI rule. Python written for a GUI feature counts as GUI: when a turn changes the web page and Python files, the Python part is booked as GUI if the author's request talks more about the GUI than about program logic (judged by its wording). The reason: the split should show how long the foundation took compared with the effort of polishing the operation of the tool. Counting by edited file only would give 23 % logic and 23 % GUI instead of 16 % and 30 % - the rule moves about seven percentage points from logic to GUI; the other areas do not change.

The hypothesis figures. "First end-to-end measurement" is the first tool result in the log that shows timing numbers (time to first token and tokens per second); "first edit of the web page" is the first change to the page file; both are given as a share of the active time. The message types are keyword weights over the author's messages (German and English words), so they are approximate. "Unchanged lines" compares the lines of the first web release with the final version, area by area; blank and very short lines are ignored.