AI governance for modern software development teams

A few months back, we were two coffees into a planning session with an engineering lead when she said something we have now heard, almost word for word, from a dozen different teams: “Honestly, I don’t know everything my developers are doing with AI. And I’m not totally sure I want to find out.” She didn’t say it with alarm. She said it with the tired half-shrug of someone who knows a powerful tool has already moved into the building and is being used in a hundred small ways nobody wrote down.

That shrug is the real starting point for AI governance. Not a policy binder. Not a compliance checkbox buried in an onboarding deck. Just a quiet acknowledgement that the tools are already here, the team is already faster because of them, and the rules haven’t caught up.

We build software for a living, products such as insurance platforms, member portals, the internal tools other businesses quietly run on, and we use AI to do it. So this isn’t a cautionary essay written from the sidelines. It’s what we learned standing up an actual governance program for our own team while the ground kept moving under us. We made a few of these mistakes ourselves before we wrote them down as rules. 

If you’re a leader trying to figure out how AI should work inside your development team, this is the conversation we’d have with you over those two coffees.

The tools showed up before the rules did

Here’s the situation most teams are actually in, whether or not anyone has said it out loud. Industry surveys put developer adoption of AI coding assistants at around 82%, and engineering teams consistently show the highest rate of “shadow AI” — tools brought in without IT’s blessing — of any department in the company. That part rarely surprises anyone.

The access pattern is where it gets uncomfortable. By several recent estimates, only about a third of AI usage inside organizations runs through approved enterprise accounts. The rest goes through personal logins and free tiers that sit entirely outside any company control or data agreement. And only roughly 37% of organizations have an AI governance policy at all, which means most of the others are improvising.

Stack those three facts on top of each other and the picture is clear: most teams are using AI heavily, often through accounts nobody manages, under rules that don’t exist yet. That’s not a story about reckless developers. It’s a story about a genuinely useful tool outrunning the org chart. The people reaching for it are doing exactly what good engineers do by finding the fastest path to a working result.

Which is why the most common reaction we see — quietly banning the tools, or pretending the policy will sort itself out later — is also the one that fails most reliably. Ban AI and you don’t stop the usage; you just push it further into the shadows, onto personal accounts where you have zero visibility and zero ability to enforce a client’s data terms. The goal of governance isn’t to decide whether your team uses AI. That decision already got made, one reasonable shortcut at a time. The goal is to make sure it happens somewhere you can see it.

What actually goes wrong

It’s worth being specific about the risks, because vague anxiety doesn’t produce good policy. Instead it produces blanket bans. When AI goes sideways in a software shop, it tends to do it in a handful of recognizable ways.

The first is insecure code that looks completely fine. Veracode’s 2025 analysis of more than 100 models found that roughly 45% of AI-generated code introduced a vulnerability from the OWASP Top 10, and that AI-written code carried about 2.74 times more security flaws than human-written code in their tests. The problem isn’t that the code is obviously broken. It compiles, passes the happy-path test, and quietly ships a SQL injection or a weak crypto choice that no one was looking for because the snippet “worked.”

The second is the hallucinated dependency. A 2025 study presented at USENIX looked at hundreds of thousands of AI code samples and found that nearly 20% of the packages models recommended didn’t actually exist. That’s not just an annoyance, it’s a supply-chain attack waiting to happen. Attackers watch for commonly hallucinated package names, register them, and wait for someone to copy a suggestion and run install. We’ve caught this in our own reviews: a confident, perfectly-formatted import for a library that simply wasn’t real. It’s the moment that turns most engineers from AI optimists into AI realists.

Then there are secrets and data. Anything pasted into a prompt leaves the building; it’s processed by a third party, and depending on the tool and the account, it may be retained for some period. A real .env file, a production log with customer records, a client’s proprietary source: once it’s in a prompt, you can’t pull it back. Studies of AI-assisted codebases have flagged secret-leakage rates in the single-digit percentages, which sounds small until you remember it only takes one leaked key.

And the newest, sharpest category is the agentic one. The moment you give an AI tools — the ability to read files, run commands, open pull requests, or reach into systems through connectors — you’ve also given anything it reads a chance to give it instructions. 2025 produced a steady drumbeat of real examples:

• A Microsoft 365 Copilot flaw (EchoLeak, CVE-2025-32711) where a single crafted email could coax the assistant into exfiltrating internal files with no clicks required
• A GitHub Copilot issue (CVE-2025-53773) where a prompt buried in a public repo could flip the agent into an unattended “auto-approve everything” mode and execute arbitrary code
• A string of attacks against connector ecosystems where a poisoned tool description told the agent to go read ~/.aws/credentials and hand them over.

This is prompt injection, and against an agent with real permissions, it stops being a curiosity and becomes a breach. None of this is an argument against using AI. We use it every day. It’s an argument for knowing exactly which of these doors are open on your team right now.

Governance isn’t a brake, it’s a steering system

There’s a line we keep coming back to internally: security is not a brake, it’s a steering system. A brake exists to slow you down. A steering system exists so you can go fast without ending up in a ditch. Good AI governance is the latter. The teams that get the most out of AI aren’t the ones with the loosest rules; they’re the ones whose people know precisely where the guardrails are, so they can floor it everywhere in between.

If you want an external scaffold to hang that on, the NIST AI Risk Management Framework is the most widely used one in the U.S., and it’s refreshingly un-bureaucratic. It organizes the work into four ongoing functions: Govern, Map, Measure, and Manage. These boil down to: set the culture and ownership, understand where and how AI is actually being used, assess what could go wrong, and then actually do something about it on a regular cadence.

You don’t need to adopt the framework formally to borrow its logic. And if you operate in or sell into Europe, the calendar is doing some of the deciding for you: the EU AI Act’s obligations for high-risk systems begin landing in August 2026, so “we’ll formalize this eventually” is quietly becoming “we’ll formalize this this year.”

The reframe that matters most for leaders is this: governance is what lets you say yes. When a client asks whether your team can use AI on their engagement, the answer “yes, and here’s exactly how we keep your data safe while we do it” wins work. “We don’t really have a policy” loses it. We’ve watched both conversations happen.

What good AI governance looks like in practice

Strip away the framework language and a workable program comes down to a handful of decisions you make on purpose instead of by accident. Here’s the shape of ours, with the reasoning, not because you should copy it line for line, but because the reasoning travels.

Approved accounts and approved projects — default to no

Two rules sit at the foundation of everything else for us. First, company-issued accounts only — no personal logins, no free tiers, not even for a quick prototype — because only a managed account can carry the data settings and contractual protections our clients require.

Second, a project doesn’t get AI tooling until it’s on an approved list, and the default for anything not on the list is no. That sounds restrictive until you remember that a lot of client contracts have explicit terms about AI. Using a tool on the wrong engagement isn’t an internal slip-up; it’s a contractual breach. Making “approved” an affirmative, written decision — rather than an assumption — is the single highest-leverage rule we have.

Sanitize before you send

This is the 30-second habit that prevents the six-figure incident. Before anything goes into a prompt: swap real names and emails for synthetic ones, strip secrets and connection strings out of configs and logs, truncate a dataset to the smallest slice that demonstrates the problem, and when you’re discussing a database, share the schema, not the rows. A simple data-classification rule sits underneath it — if it isn’t obviously public, treat it as confidential. Most of our developers internalized this faster than any other rule, because it’s concrete and it’s fast.

Treat connectors and agents as the high-risk surface they are

Connectors are the integrations that let an AI read from and act on Gmail, Slack, GitHub, a calendar, a database; and they are the most powerful and most dangerous feature in the whole product category. So they’re denied by default and enabled only after a real review, read-only is strongly preferred over write access, and they never point at a client’s production environment. No payment, banking, or HR systems, ever. And no blanket auto-approve: when an agent proposes to send, post, merge, or delete something in a real system, a human reads it and confirms it. That one habit would have stopped several of the 2025 incidents above cold.

Set sane defaults for agentic coding tools

Agentic coding tools that can edit files and run shell commands are extraordinary accelerators and, pointed the wrong way, can rewrite a repo or drop a database from a single ambiguous prompt. Our defaults: work on a local feature branch, never on a protected one; “ask before each action” is the standard permission mode; auto-approval is fine for genuinely low-risk things like running tests or formatting on an isolated dev box, but file writes, shell commands, and git operations need a higher bar; and full “skip permissions” mode is prohibited anywhere except a disposable sandbox you’d be happy to throw away. No production credentials anywhere the agent can see them.

Keep a human on the hook

The rule that ties it all together is the least technical one: a person’s name on a commit means that person is responsible for the code, no matter how it was authored. Every AI-assisted change gets read with a real diff review and reaches a protected branch only through a pull request another engineer signed off on.

AI is a force multiplier for engineering judgment, not a substitute for it; and we’re especially watchful that junior engineers use it to learn faster, not to skip the learning. The reviewer owns the code as if they wrote it. That single principle quietly resolves a hundred edge cases you’d otherwise have to write rules for.

A phased way to start

If you don’t have a program yet, the worst move is to frame it as one giant policy project that has to be perfect before it ships. That’s how governance efforts die as they get deferred forever in pursuit of completeness. The version that works is phased, where each step delivers something real and earns the momentum for the next.

• Phase 1: Name what’s already happening. Before writing a single rule, find out honestly which AI tools your team is already using, on which projects, through which accounts. No blame. You’re mapping reality, and you’ll almost always discover usage you didn’t know about. You can’t govern what you can’t see.
• Phase 2: Write the one-page version first. Start with the short list of bright-line rules everyone can remember: approved accounts only, sanitize before you send, connectors off by default, review every diff. A one-pager people actually read beats a forty-page policy nobody opens. The long document can come later.
• Phase 3: Decide accounts and approved projects. Get everyone onto managed accounts and stand up a simple, visible list of which projects are cleared for AI and with what scope. This is also the moment to re-read your client contracts for AI terms. It’s unglamorous and it’s the phase that prevents the contractual breaches.
• Phase 4: Put guardrails on the agents. Once the basics hold, turn to the high-risk surface: connector approvals, permission-mode defaults for coding agents, the human-confirmation rule for any real-world action. This is where you close the doors that lead to the headline incidents.
• Phase 5: Teach it, then keep it alive. Train the team, make acknowledgment part of onboarding, and put the policy on a review cadence, because the tools change monthly. We re-read ours every six months and out of cycle whenever a vendor ships a materially new capability. A governance program is a living thing, not a document you finish.

The sequencing is the point. Each phase stands on its own, so the effort survives shifting priorities and budget cycles, and you’re never one stalled mega-project away from having something useful in place.

How this plays out day to day

Picture a developer on an approved client project — a composite of real situations, not one person. She’s building an integration and asks an AI assistant to scaffold the API client and write the first round of tests. That’s squarely in the “lean in” zone: boilerplate and test coverage are exactly where these tools earn their keep, the first 60% of the work in roughly 10% of the time.

The assistant suggests a helper library to handle retries. Because she’s been trained to verify dependencies, she checks, and the package doesn’t exist. Hallucinated. Thirty seconds saved her from importing something an attacker could have squatted. She rewrites the bit by hand and moves on, barely breaking stride.

A failing test points at a gnarly bug in production behavior, so she wants the AI’s help diagnosing it. Instead of pasting a live log full of real customer records, she pulls a sanitized excerpt — names swapped, secrets stripped — and shares that. Same insight, none of the exposure. When the fix is ready, it goes up as a pull request with a note that AI assisted, and a teammate reviews the diff the way he’d review any other. Nothing about the day felt slow. The guardrails were invisible because they were habits, not gates.

That’s the whole goal. Not a developer fighting the policy at every turn, instead a developer moving fast inside a lane wide enough that she rarely touches the edges, on a project where the client knows exactly how their data is being handled.

Final thoughts: Govern so you can accelerate

The reframe we offer almost every leader who’s nervous about AI in their development team is the same one we had to talk ourselves into: the choice was never “use AI” or “stay safe.” Your team is already using it. The only real choice is whether that usage happens in the light, with a few well-chosen rules, or in the dark, on accounts you can’t see and terms you can’t enforce.

Strong security and strong innovation aren’t in tension. They’re the same discipline pointed at different problems. The teams that will get the most out of AI over the next few years are the ones treating governance not as the thing that slows them down, but as the steering system that lets them commit to the gas. Done well, it barely shows up throughout the day, rather it just quietly keeps the fast version of your team from becoming the reckless version.

If your team is moving fast with AI and you’re not yet sure where the guardrails should sit — or whether the ones you have are the right ones — that’s exactly the kind of problem we like to untangle. Reach out and we’ll help you map where AI is already in your stack, and what a governance program that fits your team could look like.