A Frustrating AI Code Smell: Every Problem Decomposes to a dataclass With Four Reasons

Did you know that all messy raw data and analysis can be cleanly mapped to a taxonomy of enums with between three and six reasons?

Apparently, this is what AI believes all problems can be reduced to, and it is one of the most annoying AI coding smells that I run into all the time.

Here’s a concrete example from my agent-session analysis work. I want to mine coding-session traces for skill invocations, find the common failures, and fix them.

When I ask AI to do this, a very common approach is that it will look at a few examples of the traces, vastly overfit to those examples, and come up with a clean dataclass taxonomy of four failure reasons.

from dataclasses import dataclass
from typing import Literal

@dataclass
class SkillFailure:
    reason: Literal[
        "did_not_activate_for_github_url",
        "used_bash_instead_of_read",
        "missed_nested_instructions",
        "poor_outcome",
    ]

In this case: skill did not activate, poor outcome, hyper-specific reason for only this relevant skill, and so on for three to six very arbitrary and heuristic-y reasons.

The problem is that the AI is jumping to set a data model. It’s turning messy into structured in a way that doesn’t make sense and is overfit.

Just like with most projects and things in life, if the data model is wrong, everything downstream is going to suffer.

@dataclass
class FallOfRomeResearchFailure(SkillFailure):
    missed_evidence: Literal[
        "visigoths_crossed_danube_in_376",
        "constantinople_survived_until_1453",
    ]
    inferred_root_cause: Literal[
        "barbarians",
        "lead_pipes",
        "christianity",
    ]
    recommended_fix: Literal[
        "retry",
        "add_regex_for_visigoths",
    ]

yeah that should cover all the bases

This is particularly problematic because the system creates a lossy abstraction. Raw evidence becomes structured data and things get lost.

This can lead to two common failure modes that I see.

One, overall recall gets worse. The raw data is getting filtered out, which means things won’t map to those reasons and will just be tossed aside.

The other is overall precision of analysis, in other words, whether it’s right or not. Messy or unclear cases sometimes get forced into that taxonomy, which means it’s just going to be bad.

The issues compound. The deeper this is ingrained into the system or tool, the worse it’s going to get over time.

AI reaches for this constantly because it struggles with turning ambiguous into deterministic. It wants and needs that structure for code. But that’s not how real life works.

It’s fine for databases to have a clean data model. But for anything messier, getting this part right is particularly hard, and AI is particularly bad at getting it right the first time.

This leads to debt, both cognitive and code-wise.

This is another reason dogfooding is the new code review: I need to actually use the code that I allow AI to write.

That means reviewing the plan, reviewing the design, guiding the agent, and applying thoughtfulness where it compounds downstream.

As a lightweight way to combat this, I’ve added it to my skill for AI anti-patterns and code smells and integrated it as a check during the planning phase, as well as during code review.

Whenever a heuristic becomes authoritative, that’s a code smell. It needs to earn that authority before the system relies on the core structure.

Eventually, I think this is a sharp edge that will get ironed out. Or maybe not, because life is messy and all problems in life don’t resolve cleanly to a dataclass with four reasons.

Until then: humans think, agents execute, and dogfood consistently.

Don’t become a human Enter key who approves everything that looks plausible.