set CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1

One Line

You use Claude Code. One session. One context window. One worker grinding through your codebase in order.

Maybe you use subagents. Good. Subagents go do a focused thing and report back… like good and useful interns.

This post is about one line:

set CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1
claude

That line turns on agent teams.
And agent teams are not interns.
They are a team.

What Agent Teams Are

An agent team has four parts:

• A team lead. Your main Claude Code session. It spawns teammates, builds the task list, and synthesizes results. In my experience, the Claude Code session replaces the orchestrator in a harness. (Assuming you’ve used – or even read about – agentic harnesses…)
• Teammates. Each one is a full, independent Claude Code session with its own context window. Not a helper inside your session. A whole other Claude.
• A shared task list. Tasks are pending, in progress, or completed. Tasks can depend on other tasks. A blocked task cannot be claimed until its dependencies finish. Teammates self-claim the next unblocked task when they finish one, and claiming uses file locking so two teammates don’t grab the same work.
• A mailbox. Teammates message each other directly. By name. No routing everything through the lead.

That last part is the difference that matters.

You can also talk to any teammate yourself, without going through the lead.

Turning It On

The feature is experimental and off by default. Update Claude Code first (claude --version; agent teams need v2.1.32 or later, and newer is better).

Then pick your flavor.

Windows Command Prompt (this window only):

set CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1
claude

PowerShell (this window only):

$env:CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS = "1"
claude

bash / zsh:

export CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1
claude

Every session in one repo – put it in <repo>\.claude\settings.local.json

Note: Claude wrote “on Windows, %USERPROFILE%\.claude\settings.json“.
I disagree and use “on Windows, <repo>\.claude\settings.local.json” instead.
I can hear some of you thinking,

“Why <repo>\.claude\settings.local.json, Andy?”

That’s an excellent question. I’m so glad you asked!

%USERPROFILE%\.claude\settings.json would modify settings for every Claude Code session your user account runs on your / my image (laptop, VM, container, etc.). Adding a “.claude” subdirectory to your repo, and then adding a settings.local.json file to that subdirectory, will override settings from broader scopes – %USERPROFILE%\.claude\settings.json, which is the broadest scope for a user on an image, included.

Claude Code wrote this settings.local.json for me at first. (I later moved the flag into a launcher – more on that below.):

{
  "env": {
    "CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS": "1"
  }
}

The global (USER) settings file is tempting. Set it once, forget it. But it turns teams on for every session, and that has a cost I’ll get to in a minute. settings.local.json is scoped to one repo. Your other projects never see it. That turns out to be a feature.

A Note for Windows Folks

Agent teams have two display modes.

• In-process (the default). All teammates run inside your main terminal. They show up in an agent panel below the prompt. Up and down arrows select a teammate. Enter opens its transcript so you can message it. Escape clears the selection, but while you’re viewing a teammate, it interrupts that teammate’s work. Ctrl+T toggles the task list. x stops a selected teammate. Works in any terminal.
• Split panes. Each teammate gets its own pane. Requires tmux or iTerm2. Not supported in Windows Terminal or the VS Code integrated terminal. Durnit!

So on Windows, in-process it is. That’s fine. The panel is good.

You’ve Seen This Before

Here’s the part that makes me all giggly inside:

• A lead that hands out work.
• Workers that each run in their own isolated context.
• A task list with dependencies, so step C waits on steps A and B. (i.e. orchestration, baby)
• Quality gates that can refuse to let a task be marked complete.

If you’ve built an SSIS framework, you recognize these patterns:

• It’s a controller package.
• It’s child packages.
• It’s precedence constraints and execution order stored as metadata.
• It’s the recon (reconciliation) gate that won’t let the load proceed until the validation passes.

Claude Code even ships the gates. Hooks named TeammateIdle, TaskCreated, and TaskCompleted run at those moments, and exiting with code 2 sends feedback and keeps the work from moving forward.

That’s a precedence constraint with an opinion.

One real difference:
SSIS child packages don’t argue with each other.
Teammates do.

And that turns out to be the point. (Read more of my thoughts about playing nice with others.)

A First Team for Data Engineers

The documentation recommends starting with research and review, not code changes. That’s good advice. Parallel writes are where teams get into trouble.

So point a team at a folder of SSIS packages (they’re XML; Claude reads them just fine) and try something like this:

Spawn three teammates to review the SSIS packages in this folder.
Name them errors, performance, and deployment.
- errors: error handling, event handlers, logging
- performance: data flow design, lookups, blocking transformations
- deployment: parameters, connection managers, environment configuration
Have them challenge each other's findings before reporting.
Do not change any files.

Naming the teammates matters. You can address them by name later (“ask performance to look at Load_FactSales.dtsx again”).

The “challenge each other” line matters more. One reviewer finds one plausible problem and stops. Three reviewers trying to disprove each other find the problem that survives.

How I (I Mean, Claude Code) Set Mine Up

Confession first. My first attempt was the bash one-liner, CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1 claude, typed into a Windows command prompt. cmd told me it had never heard of that command. My second attempt was claude --teammates ux,backend,adversary. There is no --teammates option. Claude Code told me so, more politely than cmd.

You don’t list teammates on the command line. You turn the feature on, then ask for teammates in plain language.

Once I stopped guessing, I (I mean, Claude Code) set up my framework repo in three tiers. One rule drives all three: the expensive thing is opt-in.

Tier 1: The Everyday Session

Most work doesn’t need a team. It needs one capable session that doesn’t burn tokens. The local repo’s settings JSON file: <repo>\.claude\settings.local.json:

{
  "model": "sonnet",
  "advisorModel": "opus",
  "permissions": {
    "allow": [
      "Bash(git status)",
      "Bash(git diff:*)",
      "Bash(dotnet build:*)",
      "Bash(dotnet test:*)"
    ]
  }
}

Sonnet does the routine work. Opus typically weighs in as an advisor at decision points: before committing to an approach, on recurring failures, before declaring a task done. Pre-approving build and test commands stops the permission prompts for those commands during long runs. See Configure permissions for more information.

A word about those allow rules. They match command text, not intent.

Bash(dotnet build:*) approves dotnet build with any arguments. It does not approve dotnet.exe build. It does not approve dotnet build && something-else either, because Claude Code checks every part of a chained command, and every part has to match a rule.

And on Windows there’s a wrinkle. Bash rules cover the Bash tool. Claude Code also has a PowerShell tool, and PowerShell commands need their own rules: PowerShell(dotnet build *).

How do you know which one you need? Watch. When a command prompts, the prompt shows the tool and the exact command. That tells you which rule is missing.

Notice what’s missing. The agent teams flag is not in this file. That’s on purpose.

Tier 2: Specialist Subagents

Reusable roles live in <repo>\.claude\agents\, one markdown file each. I have three:

• adversary (Opus, read-only plus Bash). A hostile reviewer. Assumes the change is wrong until proven otherwise. Stored in <repo>\.claude\agents\adversary.md.
• verifier (Sonnet, Read and Bash only). Runs the build and tests. Reports exact pass/fail output. No interpretation. Stored in <repo>\.claude\agents\verifier.md.
• scout (Haiku, read-only). “Find every place X is referenced.” Cheap lookups.
• Stored in <repo>\.claude\agents\scout.md.

Here’s the adversary:

---
name: adversary
description: Use before any PR, merge, or release. Attacks the change: security, failure paths, contract violations, claims not backed by evidence.
tools: Read, Grep, Glob, Bash
model: opus
---
You are a hostile reviewer for this framework. Assume the change is wrong
until proven otherwise. Check: failure paths actually report failure; recorded
status matches observable reality; inputs crossing a platform boundary are
validated; nothing is marked done that wasn't run. Report findings by severity.
Do not edit files.

As subagents, they report a summary back to my session. Cheap. And the same files work as teammate roles when I do want a team.

One thing a ‘guide’ I found online got wrong: don’t hand-write anything in %USERPROFILE%\.claude\teams or %USERPROFILE%\.claude\tasks. That’s runtime state. Claude Code generates it and overwrites your edits.

Roles go in <repo>\.claude\agents\.

Tier 3: Teams, On Purpose

Here’s the cost I promised. Claude names subagents on its own so it can message them later. With the agent teams flag on, a named subagent launches as a teammate. So with the flag in any settings file, teams can form when you didn’t ask for one. Every teammate is a full session. You pay for each.

So the flag lives in a launcher. team.cmd in the repo root turns teams on and makes Opus the lead:

@echo off
setlocal
set CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1
claude --model opus %*

Plain claude gets me Tier 1, no teams. team gets me teams with an Opus lead. Then I ask for a team:

Create an agent team with three teammates.
Name them ux, backend, and adversary; use the adversary agent type for adversary.
ux focuses on the user experience of [the thing you're working on].
backend focuses on implementation and architecture.
adversary tries to break or disprove whatever the other two propose.
Use Sonnet for ux and backend.

Teammates don’t see my conversation with the lead. The spawn prompt is all the task context they get, so I put the task in it.

Where teams earn their keep in my repo: platform adapters live in separate files, so one teammate per adapter can build in parallel without overwriting each other. And design debates, where backend and adversary argue over an execution-model decision.

Guardrails

• Keep CLAUDE.md short. Every teammate and subagent loads it. Every extra line gets paid for once per agent. Build commands, conventions, and the “adversary review before PR” rule go there. Everything else goes in docs Claude reads when it needs them.
• Hooks, not rules, where it matters. Use a TaskCompleted hook that runs the tests and exits with code 2 on failure. The task cannot be marked complete while tests fail.

A CLAUDE.md rule asks.
A hook enforces.
Data engineers know the difference between a convention and a constraint.

Best Practices (and Things That Will Bite You)

It’s experimental. Experimental means some of this:

• Tokens. Every teammate is a separate Claude with its own context window. Cost scales with team size. Start with three to five teammates. Three focused teammates beat five scattered ones.
• Same-file edits. Two teammates editing one file means overwrites. Give each teammate its own files.
• Teams you didn’t ask for. With CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS set to 1, a subagent Claude names on its own launches as a teammate. If you want plain subagents back, set CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS to 0.
• Interactive only. Run claude -p (headless, including Agent SDK sessions) and you get subagents, not teammates.
• Resume doesn’t bring them back. /resume and /rewind don’t restore in-process teammates. The lead may try to message teammates that no longer exist. Tell it to spawn new ones.
• One team per session. No nested teams. The lead is the lead for life.
• Permission prompts land on the lead. Pre-approve the common stuff in your permission settings or you’ll be clicking a lot.
• The lead gets impatient. Sometimes it starts doing the work itself instead of waiting. Tell it: “Wait for your teammates to complete their tasks before proceeding.” Sometimes it declares victory early. Tell it to keep going.

I have managed people. This list is familiar.

The Verdict

Software engineering spent years teaching AI to work alone. Now it’s teaching AI to work as a team: a lead, a task list, dependencies, gates, and a way to say “I think you’re wrong.”

Data engineering called that orchestration. We’ve been running it in production for twenty years.

One line to try it.

Andy :{>

This feature is experimental and changing fast.
Details current as of the date written (10 Oct 2026).
The source of truth is the Claude Code agent teams documentation.

Last Thursday the pipeline execution instance reported green at 3:00 AM and red by 9:00 AM. Nobody had touched the code.

I can hear some of you thinking, “If the output changed, Andy, then something changed.”
Well, you’re not wrong.

Maybe the source data changed. Maybe a reference value changed. Maybe an upstream team quietly changed the meaning of a column. This is where data engineering testing gets tricksy: The code can remain unchanged while the thing being tested moves underneath it.

…and it’s not necessarily a gap in your testing discipline. It may be a sign that the vocabulary does not quite fit the work.

What software engineering named

Software engineering named:

• Unit
• Integration
• Regression
• Fixture
• Mock
• Stub
• Coverage
• Red, green, refactor.

Every one of those words is a boundary. Each word tells you what’s inside and what’s outside the test, what you control and what you are pretending to control. Software engineering spent years refining testing, and the arguments were productive because of a shared vocabulary.
Everyone knew what the other person meant.

Data engineering borrowed the runners:

• pytest
• assertion libraries
• CI gates
• green checkmarks

All of it good, most of it free, none of it wrong to take. The tools (mostly) transferred. The vocabulary did not transfer as cleanly.

I have been thinking about this for a while.

In 2007, I co-authored Professional Software Testing with Visual Studio 2005 Team System and wrote the chapter titled “Testing the Database.” Since I hold data integration / engineering is software development, I maintain this applies when testing data engineering.

Almost twenty years later, I have learned that the principles sometimes transfer more cleanly than the vocabulary.

Where the words break

Regression – in software engineering, regression testing often asks whether a change to the code, configuration, or dependencies changed existing behavior. In data engineering, the output can move even when the pipeline code does not. The input changed. A reference value changed. An upstream system changed. Same symptom, but a different root cause. In my experience, changes in input, reference data, and upstream systems account for a surprising amount of what actually goes wrong.

Flaky – in software, a flaky test is a broken test. Quarantine it, fix it, or delete it. In data engineering, a test that passed Tuesday and failed Wednesday may be the most honest thing in the building. The test did not change. The world did. Calling that test flaky blames the witness.

Coverage – software engineering can measure line coverage, branch coverage, function coverage, and more. None of those measures is perfect, but at least the terms describe something countable. What does coverage mean for a pipeline? Rows? Deviations from previous data profiling results? The branches of a business domain nobody ever fully enumerated? Because data coverage is rarely defined, it is rarely measured. We fall back to reporting coverage of the code, even when the greater risk lives in the data.

The gate – a software test often acts as a gate during a build or deployment. A data test may need to act as a gate every night, forever, against inputs owned by people who do not report to you. That changes the conversation. A failed build can prevent a release. A failed data test at 3:00 AM may prevent the CFO from receiving a report at 9:00 AM. Someone has to decide whether to stop the load, continue with a warning, use yesterday’s data, or wake up the owner of the source system. That is more than a test result. It is an operational decision. We often file this under operations or observability. That can leave ownership unclear. Sometimes it ends up belonging to the wrong person. Worse, sometimes, it belongs to no one.

You already know this part

• You ran it in dev and eyeballed the row count.
• You compared it to last month. Last month was also unverified.
• The check that matters is a WHERE clause someone added in 2019. With no comment. Still load-bearing.
• The pipeline execution displays green on the Enterprise ETL dashboard and you still query the table before you tell anyone it finished.

That last one is the tell.
Green is not trust.
If it were, you would not look.

Words for what you are already doing

One way to improve this conversation is to name the checks data engineers are already performing:

1. Contract Assertion – what must be true of the input before the run is allowed to proceed. Shape, nullability, domain, arrival. This is a test of the world rather than of your code. Software tests can sometimes mock the world away. Data engineering cannot because the world is the subject.
2. Invariant Check – what must be true of every run regardless of what arrived. Totals tie. Keys do not duplicate. No orphan references. You do not need to know today’s input to write one.
3. Reference Run – frozen input, known output. The closest thing to determinism genuinely available to you, and the one place software’s testing vocabulary transfers cleanly. It is a smaller place than anyone wants it to be.
4. Reconciliation – the output agrees with an independent computation of the same fact. This is not a test of your code. It is a test of your claim.
5. Drift Detection – the input moved and the code did not. This is often what data engineers mean by “regression”: The code remained still while the world moved.

These are not necessarily new terms, and I do not pretend they are the only terms. They are one way to distinguish five different questions that are too often compressed into the word “test.”

Name the check in the pipeline. Name it in the alert. Name it in the runbook.

At 3:00 AM, “Contract Assertion failed: CustomerStatus domain changed” is more useful than “Data quality test failed.”

Future you will thank you.

The verdict

Software engineering has spent decades developing language for asking whether a change broke something. Data engineering has borrowed much of that language, but we have not always named the surprises that begin outside our code. A passing software test tells us something about a particular version of the code under particular conditions.

A passing test is never the whole truth.
In data engineering, the timestamp is part of the result.

[also posted at Engineer of Data Substack]

tl;dr – Sign up for SSIS Premium, use the coupon code “alblog” and save 10%!

You didn’t learn SSIS in a classroom.

You learned it because someone left and the packages stayed. You learned it at 11:00 PM, reading a Control Flow that a contractor built in 2014, working out why the deployment succeeded and the load didn’t.

You inherited it.

This is the reality for many people running production data integration right now.

If you maintain an SSIS estate, some of this will feel familiar:

• Your conference has forty sessions on the platform you’re migrating toward and none on the platform that ran payroll last night
• A line in the training budget says “modernization,” so the thing carrying production gets nothing
• You know which package fails on the third Tuesday of each month, and you know it because it wakes you up on the third Tuesday of each month
• The best documentation for your data engineering estate is the margin notes in your own copy of a book
• New people don’t get onboarded – well, at least not well. They get apprenticed in the gaps between incidents
• You stopped asking for SSIS training somewhere around 2019, because you assumed the answer was a raised eyebrow

None of this is a you problem.
It’s a supply problem.
And it’s systemic.

The teaching followed the roadmap

Software engineering solved this years ago. Nobody blinks when a shop pays for deep training on a framework that shipped fifteen years back, because somebody is still running it, and running it well is a skill worth buying. There’s a whole economy around maintaining things that work.

Data integration mostly didn’t solve this. We decided teaching attention should follow the vendor roadmap instead of the production estate. And now we act surprised when the people holding said estate together have nowhere to go.

There are few advantages to living through 51 years of software development. One advantage? I have watched this cycle three or four times now:

A platform gets declared “over.”
The teaching dries up.
The platform keeps running for another decade (or two).

And the people who understand it get quietly more valuable and quietly more tired. The tired part is the problem. Not the valuable part.

About the recordings

The SSIS material at ssis.training was recorded a few years ago.

I could bury that. I opt to surface it because it’s part of my argument.

Go open a package you shipped last month. Now open one from 2020. Same control flow. Same data flow. Same catalog. Same project deployment model. Same parameters, same environments, same references. The extension moved to a newer Visual Studio and SQL Server rolled forward.

The surface you actually work on did not move.

This is what a stable platform looks like. This is why your estate is still running. This is also why SSIS training holds its value.

If the recordings had aged out, SSIS would have changed.

SSIS didn’t change that much at all.

(On the Enterprise Data & Analytics team, “SSIS is dying” has been a running joke for years. Every time we hit a bug, somebody says it, and we laugh at our silly selves. The joke keeps working because the premise keeps not happening…)

So here’s where it lives

Visit – or have your boss visit – ssis.training to learn more about 35 hours of training I wrote based on my experience.

Instead of a migration pitch wearing a training costume, my training covers SSIS design patterns, automation, framework-think, SSIS catalog and lifecycle management – things that took me an embarrassing number of production incidents to work out the first time, arranged so you don’t have to repeat the incidents.

Inherited knowledge has holes in it by definition. You know the three patterns you were handed. You may not know the six you weren’t, or why the person before you chose the way they did, or what a deliberately designed version of your estate would look like. That Microsoft demo may have left you with the impression that Monoliths are good (even though that wasn’t their intention). You may never have heard anyone say (out loud), “Data engineering is software development.”

That gap is worth closing while the estate remains yours to shape.

I can hear some of you thinking, “Why might I want to close that gap, Andy?” That’s a fantastic question. I’m glad you asked! There’s a chance you’ll be migrating eventually – many of you are doing so now – and it’s simpler to migrate from an estate you understand than from one you’re afraid to touch.

We never stopped running it.
We should probably start teaching it again.

:{>

P.S. If you’re managing SSIS catalogs across lifecycle tiers today, SSIS Catalog Compare compares, scripts, and deploys catalog contents between tiers. If you’re orchestrating SSIS at enterprise scale, look at the SSIS Framework. The training and the tooling teach the same patterns, which is not an accident.

You know which server is Dev. You have known for years. It is the one with the instance name that almost matches the naming convention, the one that was supposed to be temporary, the one everybody just knows.

Until the new developer starts. Or the instance moves. Or you are three windows deep in a deployment at 4:45 on a Friday and two connection strings differ by four characters, and one of them is Production.

If you manage data integration across environments, some of this will feel familiar:

• Connection details live in your head, in OneNote, in a wiki page last updated two contract cycles ago
• Onboarding a developer means forwarding a list of server names and hoping it is current
• The same instance serves development, deployment, and administration, and nothing in the tooling tells you which hat you are wearing
• A server migration means updating the same connection string in every tool, and finding out later which one you missed
• “Which database is that again?” is a question someone asks every week

None of this is a you problem. It is a vocabulary problem, and it is systemic.

Software engineering solved this years ago. Application developers stopped hardcoding server names decades back. They connect to named environments – Dev, Test, Staging, Prod – and configuration resolves the name to infrastructure. The name carries the intent. The infrastructure is an implementation detail.

Data integration mostly didn’t. We borrowed the tools and skipped the pattern. Our tools still ask for a server and a database, and we still answer with trivia.

I have written before that you can’t govern what you can’t name. Connections are the clearest case I know. “d2\Dev” tells you where you are connecting. “DILM Dev” tells you why. One is an address. The other is a word in a working vocabulary – and vocabulary is the first rung on the ladder to governance and trust.

So we built the missing piece.

Lifecycle Management Enterprise Tiers (LM Enterprise Tiers, or LMET) is a free utility for DILM Suite and DELM Suite. It manages an enterprise-wide database of labels – short, descriptive names for your lifecycle management tier connections. “EDW Dev.” “Sales Prod.” “SCC Test.” Define the label once, and the label becomes the connection.

A few things follow from that:

One registry, every tool. SSIS Framework Manager, SSIS Catalog Compare, and the DILM Deployment Utility all accept LM labels and use them to connect. When a server moves, you update the registry once. Every tool follows.

Labels carry function, not just location. You can define the same connection string under different labels – one for development, one for deployment, one for administration. The purpose of the connection is visible before anyone connects. And when those functions later separate onto different instances, as they do in maturing enterprises, the labels are already in place. The migration is a registry edit, not a re-education.

Tiers become promotable. Because labels identify tiers, moving artifacts from Dev to Test to Prod is a movement between names, not between strings that look nearly identical. The 4:45-on-a-Friday failure mode gets structurally harder to hit.

New people learn words, not topology. “Connect to DILM Dev” is a first-day instruction. A spreadsheet of instance names is a liability.

LM Enterprise Tiers is a free download. No signup required. It is available now on the LM Enterprise Tiers product page.

Name the connection. The server name was never the point.

:{>

P.S. LM Enterprise Tiers is free and always will be. The tools that consume its labels are how DILM Suite pays the bills. If you are managing SSIS catalogs across lifecycle tiers today, SSIS Catalog Compare is the place to start – it compares, scripts, and deploys SSIS catalog contents between the tiers LMET names. If you are orchestrating SSIS at enterprise scale, look at the SSIS Framework.

We Borrowed the Tools, Not the Patterns

A Pattern Language for Orchestration

You Can’t Govern What You Can’t Name

The Words for the Work

As always, I welcome your feedback.

Enjoy!

:{>

[originally posted at Engineer of Data Substack]

I have spent a chunk of a career in the part of data that has no name.

Not the dashboards. Not the models everyone lines up to demo. The layer underneath – the part that moves the data, in the right order, at the right time, and puts it back together when something tears at 2:00 AM.

The plumbing.
The glue.
The work that is invisible when it goes right and catastrophic when it goes wrong – and either way, nobody has a word for it.

I have built that layer for over twenty years.

I have written the same restart logic more times than I can count, a little different each time, on teams that never knew they were re-solving a problem the team down the hall had already solved. I have stood in rooms and explained how our orchestration worked and watched four smart people nod at four different pictures.

I spent this June trying to give that work its words.

It came out in three pieces, and I want to tell you what they were really about, because I am not sure I knew when I started. In “We borrowed the tools, not the patterns,” I wrote about how we took software engineering’s tooling and skipped its vocabulary. In “A pattern language for orchestration,” I named some of the patterns we had been building blind. In “You can’t govern what you can’t name”, the names turned out to be the thing governance had been waiting on the whole time.

Three weeks.
One argument.
I couldn’t see the shape of it until it was done.

Here is the shape

They were not three topics. They were three rungs:

1. Vocabulary;
2. Then governance; and
3. Then trust

Each one stands on the one beneath it. You cannot govern what you cannot name, so governance stands on vocabulary. The business does not extend trust to what it cannot govern, so trust stands on governance. Pull out the bottom rung and the whole ladder is just pipelines that run – until the day someone asks you to prove it.

This is the distance I have helped teams cross for twenty years.

The distance between a script or package or pipeline that runs and an estate you can govern. It turns out that distance has a length, and the length is exactly one vocabulary.

I watched software engineering cross it a long time ago. Not because I, or it, was smarter. Because at some point it stopped treating its recurring work as a thousand private solutions and started treating it as a discipline – a discipline of the named, the structured, the shareable, the teachable, the answerable.

Data orchestration never quite had that.

We have tools.
We have heroics.
We have a great deal of undocumented competence walking around in three people’s heads.

What we never quite had is the discipline.

I think it is time we named it

So name it.

Not with a new word. This whole series has been one lesson: the work was there all along, and only the names were missing. The discipline is no different. We do not need to coin it. We need to claim it.

Data engineering.
Used the way software engineering uses its name – not as a job title, but as a discipline.

Named, structured, governable, answerable. The practice of treating the movement and assembly of data as a craft that can account for itself, instead of a thousand private solutions held together by competence and memory.

Not better SSIS.
Not better Databricks or Data Factory.
Not a tidy habit.
It is what software engineering became when it grew up, finally reaching the field that moves the data.

That is the name. It has been on my own masthead for years – Engineer of Data – and I am only now making it mean what it should.

I spent a month giving the work its words; words it took me a career to find.

The building starts where the naming ends.

[originally posted at Engineer of Data Substack]

Last week, I wrote:

Observable Run – every execution leaves a durable, queryable record of what ran, with what, and how it ended.

I called it a pattern.
It is also a door.

Behind it is the question every data team eventually gets asked – by an auditor, a regulator, a new VP, or a 2:00 AM incident:

Can you prove it?

You can describe what your pipelines do. You built them. They run, every night, correctly enough to ship.

But describing is not proving.

• The auditor asks for evidence and you assemble it by hand, after the fact, hoping the logs reach back far enough.
• The VP asks whether last quarter ran clean, and the honest answer is “probably.”
• You can say what a pipeline does. You can’t show what it did – on that day, with those inputs, under whose authority.
• Two environments drift apart, and no one can say when, or who changed what.

None of that is carelessness.

I have been the person opening folders in that meeting, assembling proof by hand while someone waited.

You are not ungoverned because you were sloppy.
You are ungoverned because the work was never named.

Strip the theater off governance and it is one plain capacity: the ability to answer questions about your system with evidence. What ran. With what. Who authorized it. Whether it did what you intended. Whether you can show it.

You cannot answer a single one of those about a thing that has no name.

A nameless estate is ungoverned by default. Not because anyone chose not to govern it. Because governance has nothing to hold.

Software engineering made its systems answerable years ago – provenance, audit trails, change control, signed releases. It could, because it named and structured the work first. Data engineering borrowed the tools and skipped the names. So our estates resist the very questions the business has started to ask.

And it is asking.

Governance was never really about paperwork. It is about ownership, operational consistency, and the trust that rides on both. Every “I think so” in answer to “can you prove it” is a quiet withdrawal from that trust. This is where data programs gain or lose their standing – not in the demo, but in the audit, the incident, the quarter that has to reconcile.

So the move is not to write more documentation about what should happen.
The move is to bind governance to what did happen.

Naming is never the destination.
It’s the precondition.

A name makes a thing inspectable. Inspectable makes it provable. Provable is governance.

You can’t govern what you can’t name.
Governance you cannot prove is governance you cannot trust.

So when they ask you to prove it – and they will – you can answer with a story, or you can answer with the record.

One of those is governance.

P.S. I shepherd SSIS Catalog Compare because this principle demanded a tool – one that inspects an estate and proves what’s deployed against what should be. The tool is downstream of the principle. The principle is the post.

[originally posted at Engineer of Data Substack]

Last week, I wrote:

Stateful problems need their own names. We never wrote them.

You’ve written idempotent loads, restart logic, dependency chains, promotion configs.
By hand.
Correctly enough to ship.

You already build every one of these.

You’re not missing the knowledge. You’re missing the words. Missing words have a cost:

• You explained your restart approach in a standup and watched three people nod at three different mental models.
• You reviewed a pipeline, knew something was missing, and had no word for what.
• You onboarded someone by walking them through “how we do it here,” because none of it was in a book they could read.
• You solved the same problem on two teams and called it two different things, so nobody noticed it was the same problem.

The Gang of Four applied Christopher Alexander’s definition of “patterns” to define software design patterns. I led a team of authors to write two editions of SSIS Design Patterns and, in so doing, applied the form to data engineering while leaving out Alexander’s metaphysics.

A pattern is not a blueprint.
Alexander’s whole claim was that a pattern lets you solve the same problem a thousand times and never once the same way. So the implementation was never part of it.

Naming the pattern is the work.
Hence, the words.

A starting vocabulary

Idempotent Write – a step may run more than once (retry, restart, backfill); the result must match running it once.
Nameless cost: every team reinvents “safe to re-run” and gets it subtly wrong.

Checkpointed Restart – a long run fails partway; you resume from a known position, not from zero and not by hand.
Nameless cost: restart means rerun-everything or surgery.

Declared Dependency – order stated as structure, not implied by schedule time or folder layout or memory.
Nameless cost: “it works because A usually finishes before B.”

Promotable Configuration – the same artifact moves dev to test to prod unchanged, because environment lives outside the logic.
Nameless cost: “we edit the package when we deploy.”

Parameterized Execution – one artifact serves many inputs instead of forty near-identical copies.
Nameless cost: copy-paste sprawl nobody dares refactor.

Observable Run – every execution leaves a durable, queryable record of what ran, with what, and how it ended.
Nameless cost: every post-mortem is archaeology.

This is not a list.
This is merely the beginning of a vocabulary, not a closed set.
Languages grow.

And like words in any language, the patterns listed presuppose each other. Checkpointed Restart presupposes Idempotent Write. Both presuppose Observable Run – you can’t resume or trust what you can’t see. Declared Dependency presupposes Observable Run – you sequence on what actually happened.

I left Alexander’s metaphysics out of the SSIS Design Patterns books. I’m not sure I would again. He didn’t name patterns for reuse. He named them to make places more alive, easier to live in. A pattern language for orchestration isn’t only about tidiness. It’s about making the work humane to inhabit, for the next engineer and for you at 2:00 AM.

A pattern you can’t name is a solution you have to rediscover. Name it once, and the rediscovery stops.

Software engineering didn’t pull ahead because it got smarter.
It pulled ahead because it got a vocabulary.
Your move.

[originally posted at Engineer of Data Substack]

Your orchestration runs. It Just Works every night. You built something real and that is not nothing. The ache isn’t that it’s broken – it’s that it lives in three people’s heads and a folder of conventions nobody wrote down.

• You wrote the retry logic from scratch in three different projects, a little different each time.
• A new hire can’t safely touch the orchestration for weeks, because none of it has names they’d find in a book.
• Two pipelines solve the same sequencing problem two incompatible ways, and both are “correct.”
• “How do you handle restartability?” gets answered with a story instead of a term.

There are few advantages to living through 51 years of software development. One advantage? I’ve seen the long game. Software engineering stood in this exact same spot once. Slinging code felt so good (it still feels good). After trial, after error, it still makes me giggly inside to see the code not only succeed, but do what I wanted. We, Programmers, (re-) solved the same structural problems until a small group named the recurring solutions.

Christopher Alexander’s pattern language first.
The Gang of Four.

It wasn’t merely a technical breakthrough. It was shared vocabulary. Naming converted reinvention into reuse. Design patterns made the work reviewable, testable, teachable, comparable.

Data engineering adopted software’s tooling: source control, workflow management, packaging. We bolted on CI/CD. We never adopted the vocabulary.

We borrowed the tools, not the patterns.

The gaps show up where concerns converge:

• retries
• idempotency
• restartability
• sequencing
• code promotion
• observability

Software addressed these concerns in a software-centric way.
That way had a trick: push the state to the edges, keep the moving parts stateless.
A stateless component is trivial to retry. Nothing to corrupt, nothing to resume.

Data can’t take that deal. The state is the whole point.

So when we borrowed their tools – the excellent tools of software engineering – we inherited an impedance mismatch that dogged our efforts in a manner difficult to quantify because it was barely visible. Data engineering looked and felt like software development because data engineering *is* software development.

Statelessness was never on the table for data engineers.
Not a chasm we failed to cross; a door that was never open.

Which means borrowing their patterns was never enough.
Stateful problems need their own names. We never wrote them.

Data forces us to deal with heterogeneous sources, schema drift, late-arriving data, fragile environments – with a toolset that never got its vocabulary layer.

Software engineering got thirty years and a library of canonical books.
Data integration got tools and a shrug.

Software engineering solved the naming problem years ago, and data engineering mostly didn’t.
That is the gap.
Not you. Not your team.

Your orchestration isn’t unmaintainable because your team fell short.
It’s unmaintainable because the work was never given a name.
Patterns are how a craft stops starting over.

If you’ve rebuilt the same retry logic for the third time, you already understood this.
You just never had the word for it.
You’re not behind.

The field is.

[ADO NET Source [41]] Error: ADO NET Source has failed to acquire the connection {<ConnectionManagerGUID>} with the following error message: “Could not load file or assembly ‘Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=<PublicKeyToken>’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)”.

Back in the day, I would search for an answer using a search engine to locate forum posts or blogs by my fellow SSIS data engineers. But this fine morning, I checked the system time in the Windows 11 system tray and noted that it’s 2026 (already, Can you believe it? Time flies, I tell ya…).

I fired up a Claude Code session instead

At the time of this writing (12 Jun 2026), Anthropic has recently released Fable 5 – touting it as the first “Mythos-class model” – and is allowing users to tinker around with the model until 22 Jun 2026 (I have an Anthropic Max 20 subscription, so I’m not sure if all users get to tinker. Max 20 subscribers get to tinker, at least).

I opened cmd on a Dev virtual machine – a VM running Windows Server 2022, SQL Server 2025, and Visual Studio 2026 Insiders¹ – navigated to the local repo, and shared the error with Claude Code.

CC’s first pass identified an issue:

This is a classic SSIS assembly-binding problem: the package’s ADO.NET connection manager references Microsoft.Data.SqlClient 6.0.0.0, but the version installed/registered on the machine doesn’t match. Let me look at the package and what’s actually installed on this machine.

Fair enough. The model did a little exploring using PowerShell to check some registry settings and adjusted some SSIS connection manager settings, and then claimed all was well. The model wasn’t wrong. the original error was solved, but we had a new error (it had that new error smell):

[ADO NET Source [41]] Error: ADO NET Source has failed to acquire the connection
{CC2302DB-AC2E-4F3C-851D-A1D81BCAE0E0} with the following error message: “An error occurred while sending the request.”.

I was a little surprised when the first “fix” applied by Fable 5 didn’t solve the entire issue in one shot, but I may have had high expectations. Claude Code and I endeavored to persevere.

I confess, I found this impressive: Fable 5 next used PowerShell to fire up a network trace. It found my server was attempting to connect to the Azure SQL DB using TLS 1.0. In June 2026, Azure AD endpoints flat refuse to accept TLS 1.0 connections, with good reason. the recommend fix was:

The fix: Set Microsoft’s recommended strong-crypto registry values, which make all .NET Framework apps (including the SSIS hosts) use the OS default of TLS 1.2+. This needs an elevated shell, which I don’t have — please run this in PowerShell started as Administrator:

Set-ItemProperty ‘HKLM:\SOFTWARE\Microsoft\.NETFramework\v4.0.30319’ -Name SchUseStrongCrypto -Value 1 -Type DWord
Set-ItemProperty ‘HKLM:\SOFTWARE\Microsoft\.NETFramework\v4.0.30319’ -Name SystemDefaultTlsVersions -Value 1 -Type DWord
Set-ItemProperty ‘HKLM:\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319’ -Name SchUseStrongCrypto -Value 1 -Type DWord
Set-ItemProperty ‘HKLM:\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319’ -Name SystemDefaultTlsVersions -Value 1 -Type DWord

This is Microsoft’s documented TLS best practice for .NET Framework, it’s machine-wide, and no reboot is needed — though restart Visual Studio so DtsDebugHost picks it up. The only theoretical downside is if some other .NET Framework app on this server depends on TLS 1.0 to reach an ancient endpoint, which is unlikely and would already be on borrowed time.

I took note of “…an ancient endpoint…”. Was that attitude, Fable 5??

Like a good HIL (Human In the Loop), I did what I was told. And, it worked!

Conclusion

Comparing the amount of time it took to write this post to the amount of time it took to implement the fix using Claude Code and the Fable 5 model, it took more time to write this post.

Notes

¹You can get my AI-Assisted Data Engineering Development Playbook at the link for $49. It’s included in a paid subscription to my Engineer of Data Substack newsletter, which is less expensive.