Long before machines, the people whose work could alter a life were held to something quiet and demanding: a standard of care. The clinician, the counsel, the trustee — each owed the person in front of them attention, competence, and honesty, whether or not it was convenient. The standard existed for the one with the least power in the exchange. It was never a courtesy. It was the price of being trusted to decide at all.
That obligation was attached to the decision, not to the hands that happened to make it. And so when the deciding moves — from a person to a model that screens, scores, ranks, flags, or forecasts — the obligation does not stay behind on the desk it left. It travels with the decision. It lands wherever the decision now lives. And it is the same duty whether the call denies a mortgage or merely routes a support ticket, ranks one résumé above another, or settles whose post gets seen — the stakes change; the obligation does not.
The trouble is that, in the move, the duty often lands on no one. The person who built the model says they only built a tool. The person who deployed it says they only followed the output. The output says nothing at all. And the human on the other side — turned down, flagged, ranked lower, quietly filtered out — is left with a decision that shaped their life and a door with no one behind it.
A machine can make the decision. It cannot assume the duty that comes with it. Care, when a machine is deciding, has to be designed — it does not happen on its own.
i.The duty did not get smaller. It got harder to see.
It is tempting to believe that automating a decision dilutes the responsibility for it — that a thousand decisions a second cannot each carry the weight one careful decision did. The opposite is true. Scale does not soften a duty; it multiplies the number of people to whom it is owed. A standard of care that applied to one applicant in a room now applies to every applicant the system never meets. The obligation is exactly as large as before, paid out across far more lives.
What changes is only its visibility. A human decision could be questioned in the moment, to a face. An automated one is made in advance, at a distance, behind a system the affected person cannot see. The care did not leave. It went quiet. And a duty that has gone quiet is the easiest kind to neglect without noticing.
ii.What the duty actually asks — four things.
If that duty is to mean anything once a machine holds the decision, it has to be said plainly, in terms a system can be held to. Four obligations carry over intact from the older, human standard — the same four a team writes its own Automation Style Guide around:
- iCompetence.first, do your best
The decision must meet a real standard — sound, current, fit for the person it concerns — not merely whatever the model returned. A system that decides worse than a careful person is not a tool. It is a hazard that was chosen. - iiCandor.informed consent
The person has a right to know that a machine decided, and to understand the grounds in terms they can act on. Care that hides itself from the cared-for is not care. "Proprietary" is not an answer one is owed. - iiiRecourse.the second opinion
Every automated decision needs a door back to a human with the authority to look again, and to overturn. A decision no one can revisit has not been governed. It has only been imposed. - ivNon-abandonment.continuity of presence
One does not deploy and walk away. The duty continues for as long as the system decides — watching for the person it begins to harm, and reaching them before the harm compounds.
iii.Keep a seat for the person it lands on.
In every one of those rooms — engineering, product, risk, legal — there is one person who is never at the table: the one the decision is actually about. The applicant behind the filter, the borrower behind the score, the patient behind the flag. They can't see the system, they didn't agree to it, and most of the time they can't afford to argue with it. Each team is looking at its own screen. No one is looking at them.
Writing it down together is, in the end, just pulling up a chair for that person — saying out loud what your systems owe them, before the systems settle it by accident. It doesn't slow the work down. It's what lets a team move fast and still recognize itself in what it shipped.
iv.We all care. The lane is what hides it.
Here is the belief I built Soma on. Inside the institutions that build these systems, almost no one is careless. The engineer cares whether the model is sound. The designer cares about the person moving through the screen. The risk officer cares about who gets hurt, and the lawyer about whether it can be defended to them later. Each one cares, genuinely — and each cares in the shape of their own work. The very focus that makes a person good in their lane is what narrows their care to the edge of it.
A human decision used to gather those edges into a single pair of hands. An automated one does not. It is assembled — a threshold set in one room, a default chosen in another, a policy written in a third — and it ships without anyone having held the whole of it at once. Automation does not respect lanes. It stitches them together and hands the result to a person who experiences it as one act, done to them. Care that stays inside separate lanes never reaches that person as care. It arrives as a decision no one quite owns.
The task is not to install care in people who lack it. It is to give the people who already have it a shared language — and then to hold themselves to it, together.
Care felt privately, in four rooms, does nothing for the person on the other side. Care said out loud — in words all four rooms agree to, and write down once — becomes something a system can actually carry. That is the quiet move this whole standard turns on: not more conscience, but a way for the conscience already in the room to speak in one voice, and keep speaking as the work moves on without it.