You will find the 🇫🇷 French version of this article here
Every developer has an idea of what beautiful code looks like.
Some want short methods, even if it means writing a lot of them. Others prefer long methods, as long as each one holds together. For some, a well-placed comment saves lives; for others, good code needs no comments at all. There are schools of thought. Robert C. Martin built a career on one of them with Clean Code, Kent Beck on another, and whole teams have been arguing about it for years.
One question rarely gets asked precisely: what style do LLMs have?
We know they have preferences. Ask GPT, Claude or Gemini to write code and patterns show up fast. But that is anecdotal. As far as I know, nobody had run a truly systematic study of an LLM’s stylistic signature. How many methods on average? How many comments kept? How many identifiers renamed?
Three researchers from Saarland University, Norman Peitek, Julia Hess and Sven Apel, have just published a paper on arXiv that answers it: From Restructuring to Stabilization: A Large-Scale Experiment on Iterative Code Readability Refactoring with Large Language Models. The idea is simple. Take 230 Java snippets, ask GPT-5.1 five times in a row to refactor them for readability, and watch what happens.
To me, it is one of the most useful results of the past few months for anyone running LLMs in production. What matters is not that GPT-5.1 refactors. It is that it always refactors in a recognizable direction: its own.
The experimental protocol
Source: Peitek, Hess & Apel, arXiv:2602.21833, 2026.
The authors took 230 Java files from the GitHub repository TheAlgorithms/Java, an educational collection of classic algorithm implementations. The choice makes sense: idiomatic code, consistent conventions, and an MIT license, so anyone can reproduce the study. They kept only files between 50 and 200 lines, with at least 50% actual code rather than comments.
From each file, they built three variants:
Then, for each variant, they asked GPT-5.1 (temperature 0, so deterministic) to refactor the code five times in a row, with three prompt wordings:
PromptGeneral: “Refactor this code for improved readability.”PromptMeaning: “Refactor this code for improved readability, especially with respect to identifier naming.”PromptComments: “Refactor this code for improved readability, especially with respect to comments.”That makes 230 snippets × 3 variants × 3 prompts × 5 iterations, or 10,350 snippets produced by the model. A fine-grained diff tool, DiffParser, then sorts every changed line into one of five buckets: rename, syntax only, comment change, code change, or mixed.
A decent-sized study, and above all a very well instrumented one.
Share of unchanged lines at each iteration
Original variant, PromptGeneral. Source: Peitek et al., 2026.
The process runs in two distinct phases, and it is reproducible.
On the first refactoring (v0 → v1), GPT-5.1 touches a lot of code. Only 45% of the lines stay the same. It renames, splits methods, deletes comments and adds others, reformats. This is a real restructuring.
The second pass (v1 → v2) already looks different. 76% of the lines stay put, and the changes become local and surgical.
From the third pass on, the code stabilizes: 86%, 89%, then 92% of the lines are unchanged between successive versions. The model has found its “final” version and only tinkers at the edges.
Convergence, version by version
With each iteration, the LLM changes less of the code, until its style settles.
The first useful lesson: an LLM refactoring does not converge quickly. There is a real restructuring phase that takes one or two passes. Refactor once and compare, and you see a lot of changes. Ask again and you still see a lot of changes, but of a different kind. If you are automating assisted refactoring, that distinction changes everything.
Share of unchanged lines: the three variants
Share of unchanged lines per transition. All three lines end up in the same zone (about 89 to 92%). Source: Peitek et al., 2026.
This is where the paper’s most counter-intuitive result comes in.
To check that the convergence is real and not an artifact of the starting code, the authors measured what happens to the Meaningless and NoComment variants. Remember: Meaningless is unreadable on purpose, and NoComment has lost all its comments.
If the model only polished whatever code it was given, the three variants would still differ after five passes. Meaningless would keep uninformative names, NoComment would have few comments. That would make sense.
That is not what happens at all.
After five iterations, the three variants end up looking very much alike. Take method count: all three start at 3.1 methods on average (same underlying structure) and all end around 6 methods. Lines of code: all three start at 56 and end around 73. Inline comments: Original and Meaningless start at 1.3, and every variant converges to 0.2, which means they have all but disappeared.
The LLM carries an internal picture of what “readable code” is. The convergence proves empirically that this picture exists, and it barely depends on the code you start from.
The LLM does not just polish your code while respecting its style. It pushes the code toward its own style. And that target is stable enough that three very different inputs come out as three very similar outputs.
Handing refactoring to an LLM is not stylistically neutral. You implicitly adopt the model’s style.
GPT-5.1's stylistic signature after 5 iterations
Averages over 230 Java snippets, Original variant, PromptGeneral. Source: Peitek et al., 2026.
Across the 230 snippets refactored five times, here are the measured trends:
int x = 0; // counter kind) are almost wiped out, from 1.3 to 0.2 on average.Counter, Result or Helper.Short and numerous methods, no inline comments, PEP-8-style spacing, long descriptive names: put another way, and stretching the point a little, in this experiment GPT-5.1 behaves like a disciple of Clean Code. It is reasonable to suspect that this kind of literature shaped the patterns it learned, though nobody can prove it.
What matters here is how sharp the pattern is: reproducible and measurable across 230 different pieces of code. That is exactly what makes it a sensitive topic. Plenty of teams do not see themselves in Clean Code, and still get that style back from their LLM.
Similarity between successive versions: code that was already clean
Original variant. Even on clean code, GPT-5.1 keeps changing about 10% of the lines on every pass. Source: Peitek et al., 2026.
On the Original variant, code that is already clean and idiomatic, you would expect the model to leave things alone. It doesn’t. Similarity between successive versions creeps up to 0.90 by the end, but never reaches the ceiling. The model does not stop on its own.
The authors call this a tendency to over-refactor: ask for a refactoring and the model refactors, even when there is nothing left to do. If you run this kind of pipeline at scale, that is a concrete problem. Every change to code that didn’t need it is a chance for a bug, noise in git blame, and friction in review.
The fix is well known and rarely applied: explicit stopping criteria. Don’t say “refactor”, say “refactor if needed, otherwise return the code unchanged”. Even then, nothing is guaranteed.
An identifier oscillating under PromptMeaning
Representative example. The model alternates between two equivalent names across iterations, with no net improvement. Source: Peitek et al., 2026.
With PromptMeaning, the prompt that explicitly targets naming, something unexpected happens: names oscillate. The model picks a name in v1, changes it in v2, and goes back to the first one in v3. It doesn’t happen every time, but often enough to be documented as a structural risk.
The likely explanation: without outside guidance, such as a team convention or a domain glossary, the model hesitates between several equivalent names. The context shifts slightly with each pass, because other lines have moved, and the decision tips the other way. A senior developer would settle it once and stick to it. The LLM can change its mind.
If you automate this flow, treat it as a warning sign. Successive diffs fill up with changes that improve nothing, the kind of noise that clutters reviews and wears down trust in the tool.
The authors tested three prompts (PromptGeneral, PromptMeaning, PromptComments) to see whether the wording changes the refactoring dynamics.
The answer is mixed. Targeted prompts do change the kind of changes: PromptMeaning produces more renames, PromptComments more comment edits. But the overall dynamics stay the same whatever the wording: restructuring then stabilization, convergence toward the same stylistic ideal, residual over-refactoring.
The reassuring part: the prompt still has a local effect, so you can steer renames or comments. The worrying part: it is not enough to change the underlying stylistic trajectory. Ask nicely, insist, add detail, and the model will still drift back to its idea of beautiful code.
We already measure complexity. We already measure coupling. But we still do a poor job of measuring the structural stylistic consistency of a codebase. And that is exactly where LLMs will leave their mark.
A project’s maintainability also depends on the internal stylistic consistency of its codebase. A codebase where every method is long but consistent is easier to maintain than one that mixes Clean Code files with “200-line utility function” files. A developer’s brain gets used to a style. That is what lets you move around quickly, guess where things live, and spot the unusual.
Bringing an LLM into the refactoring flow of an existing codebase means risking fragmenting that internal style. The files it touches migrate toward its target. The others stay with the team’s historical ideal. The codebase turns into a patchwork, five-line methods here and fifty-line methods there, and the classic metrics show none of it. Average cyclomatic complexity can stay the same. Coupling may not move. But consistency erodes, silently.
That is the real gap in our tools. PhpMetrics, AST Metrics, linters and formatters were built to quantify maintainability, but through absolute metrics (complexity, coupling) or surface ones (formatting). Structural stylistic dispersion, meaning the variance in methods per file, method length, naming conventions and comment density, remains largely unmeasured.
Four things stand out.
An LLM has a style, and you should document it. If your team refactors with GPT-5.1, it is drifting toward Clean Code. Use Claude and the style is different. Same with Qwen or Mistral. Before delegating refactoring, get a sense of the model’s stylistic signature, and decide whether it fits your codebase.
Explicit stopping criteria. Over-refactoring is not a bug, it is the model’s default behavior. If you want it to stop when there is nothing left to do, put it in the prompt: Only refactor if the code needs it. If you judge it acceptable already, return it unchanged. Even then, check downstream that the diffs actually improve something.
Explicitly protect the comments that matter. The model systematically strips inline comments, and sometimes that is a real loss. A comment that explains why (“we take this branch first because it is the most frequent one in production”) carries information the code cannot. If your LLM pipeline deletes those, the codebase gets poorer.
Watch the naming. Name oscillation is real and noisy. Refactor the same piece of code several times with an LLM without asking it to reuse existing names, and you will watch the same file rename its variables on every pass, with no net gain.
Beyond these technical points lies a broader question: who decides the style of your codebase? For decades, human teams did, through their reviews, their internal guides, their arguments. Now, unless you pay attention, the LLM you use decides it implicitly, along with the biases of a training corpus whose makeup is anything but neutral.
Whether that reference is good or bad is another debate. The point is that it is a default decision, made for you and without you, that deserves to be made consciously.
The paper has several limitations, which the authors state openly.
Only one model. It is GPT-5.1 and nothing else. Claude, Qwen and Mistral probably have different signatures. A cross-model comparison is the natural next step, and very likely what comes next.
Only one language, Java. On Python, PHP or Go, the model’s favorite patterns may differ. Conventions vary by language, and the model adapts to them. The exact figures (twice as many methods, 26% more lines of code) do not carry over directly.
An academic corpus. The snippets from TheAlgorithms/Java are educational, short and self-contained. Behavior on real production code, spread across files, with dependencies and side effects, is not tested here. The broad trends probably hold, but the exact numbers are not guaranteed.
So the question is not whether GPT-5.1’s style is good or bad. What matters is that it exists.
When a team brings an LLM into its development flow, it also brings in an outside stylistic preference. You can accept it. You can measure it. You can constrain it. What you want to avoid is living with it without noticing.
Support my work
Open-source sponsoring
PHPMetrics, AST Metrics and the rest are freely available. A little help on GitHub keeps them going.
Sponsor me on GitHub
Audit & Training
Available on Malt
Want an outside look at your codebase? I help engineering teams with architecture, testing and delivery, at the pace of your own projects.
Find me on Malt© Jean-François Lépine, 2010 - 2026 · RSS feed