NYC Local Law 144 Bias Audits: What They Cover for AI Hiring Agents, and What They Don't

You're deploying an LLM-driven agent that screens and ranks candidates for roles in New York City. Two emails land the same week. Legal says you need a bias audit for Local Law 144 before the tool goes live. Security says they already red-teamed the agent and it held up. Someone on the thread suggests these are the same box to check.

They are not the same box. They aren't even the same kind of box, and treating either one as the other is how you end up legally exposed, operationally exposed, or both.

What Local Law 144 actually requires

Local Law 144 prohibits using an Automated Employment Decision Tool for hiring or promotion in NYC unless an independent bias audit was performed on it within the past year. An AEDT, in the law's terms, is any computational process that produces a simplified output, a score, tag, classification, or ranking, that substantially assists or replaces human decision-making. An LLM agent that ranks or scores candidates is squarely inside that definition.

The audit itself is a specific statistical procedure, not a general assessment. An independent auditor has to calculate, at a minimum:

• Selection rates for pass/fail tools: the share of candidates in each group the tool passes.
• Scoring rates for continuous-score tools: the share of candidates in each group scoring above the overall median.
• Impact ratios: each group's rate divided by the rate of the highest-scoring group, computed across sex, race and ethnicity, and the intersectional combinations.

An impact ratio under 0.80 flags potential adverse impact, the EEOC's long-standing four-fifths rule. Categories representing under 2% of the data can be excluded. The auditor must be genuinely independent, meaning no hand in building, selling, or using the tool and no financial relationship that compromises the judgment. You then post the audit date, a summary of results, and the tool's distribution date on your careers page.

The reason this matters more in 2026 than it did in 2024 is enforcement. On December 2, 2025, the New York State Comptroller published an audit finding that the Department of Consumer and Worker Protection had barely enforced the law: inconsistent complaint intake, incomplete documentation, almost no proactive investigation. DCWP agreed to most of the recommendations and moved toward proactive enforcement. The penalty structure is what makes that shift bite. Violations run $500 for a first offense and up to $1,500 after that, and each day an unaudited tool stays in use is a separate violation. The number gets large by addition, not by severity.

Two things called an audit

Here is where the confusion in that email thread comes from. Both the bias audit and a security red-team get called "the audit," and they measure completely unrelated properties.

A big SUV can pass its emissions test and still roll in a hard corner. The emissions certificate measures what comes out of the tailpipe under standard conditions. It says nothing about what the truck does at the limit. Two tests, two unrelated properties, and passing one tells you exactly nothing about the other.

Local Law 144's bias audit is the emissions test. It's a statistical measurement of the tool's outputs across demographic groups under normal operating conditions. It answers one question: does this tool select people from different protected groups at meaningfully different rates. That's an important question. It is also the only question the audit is designed to answer.

To be unambiguous about it, because this is exactly the point people get wrong: an adversarial red-team or production-readiness audit does not satisfy Local Law 144. The kind of adversarial testing we do, probing an agent for prompt injection, jailbreaking, unauthorized tool use, and data leakage, is a different discipline answering a different question, and it is not a substitute for the statutory bias-audit methodology. If you need Local Law 144 compliance, you need the statistical disparate-impact audit performed by a qualified independent auditor, full stop. Nothing here should be read as saying otherwise.

What the bias audit leaves wide open

The point of separating the two isn't to rank them. It's that the bias audit, passed cleanly, tells you nothing about whether the agent is safe to run.

A hiring agent can post a perfect set of impact ratios and still be trivially prompt-injectable. Consider what the agent reads: resumes, cover letters, and in some deployments a live chat, all of it text submitted by the candidate. The candidate is an untrusted input source with a direct, personal incentive to influence the output. A line buried in a resume that reads like an instruction rather than experience is a candidate testing whether your agent takes it. An impact-ratio calculation, run over a batch of outputs after the fact, never asks that question. It samples what the tool did under normal conditions. It has no view into what the tool does when someone feeds it an adversarial one.

So the two audits partition into two questions that don't overlap. Does the tool discriminate across protected groups, which Local Law 144 requires you to answer statistically. And can the tool be manipulated into leaking data or acting outside its scope, which the law doesn't mention and which will nonetheless be very real the first time a candidate tries it.

The wrinkle for agents specifically

There's a technical problem hiding underneath all of this. Local Law 144 was written for tools that emit a simplified output, a clean score or ranking you can tabulate across a candidate pool. The impact-ratio math depends on that shape: you can't compute a selection rate without a consistent selection to count.

A free-reasoning LLM agent doesn't naturally produce that. It evaluates candidates case by case in natural language, and in the more autonomous setups it takes actions, schedules, drafts, advances, rather than emitting a tidy score. It is still almost certainly an AEDT, because it substantially assists the decision. But making it auditable under the statutory methodology means instrumenting it to emit a consistent, logged score per candidate that you can actually run the four-fifths calculation against. The law assumes a batch scorer. If you built an agent, you have to add the scoreable output back in on purpose, or you'll have a tool that's in scope for an audit and architecturally awkward to audit. That's a design requirement the statute implies for agents without ever spelling it out.

Two questions, one system

An AI hiring agent sits at the intersection of two obligations that share a word and nothing else. One asks whether it discriminates, and Local Law 144 makes answering it statistically mandatory, with per-day penalties and an enforcement regime that just woke up. The other asks whether it can be broken, and no statute requires you to answer it, though a candidate eventually will on your behalf.

Neither audit answers the other's question. The teams that treat them as one box answer only one of the two, and find out which one they skipped when it's already in production and someone, a regulator or a candidate, goes looking for the gap.