AI in government services: what US and Canadian rules now require for procurement, disclosure and accountability

Public bodies are adopting AI quickly. US federal agencies reported 3,611 AI use cases in their 2025 inventory, more than double the 1,757 reported a year earlier, and 445 of them were classed as "high-impact" ⁴. In Canada, the federal government’s first public AI register, launched in November 2025, lists more than 400 systems across 42 institutions ⁸.

Most of these tools are bought, not built. That puts government lawyers, procurement officers and privacy staff at the center of AI adoption. They write the contract, they decide what the public must be told, and they answer for the decision when something goes wrong. This explainer sets out what the main US and Canadian rules now require, where they differ, and what a legal team can do this quarter.

The business problem: AI is bought faster than it can be checked

In government, an AI tool is rarely just software. It may answer a resident’s question, sort a benefit application or summarize a case file for an officer. A wrong answer can become an unlawful decision.

New York City shows how this plays out. The city launched an AI chatbot for business owners in fall 2023. In 2024, testing by THE CITY and The Markup found it gave answers that would lead to illegal behavior, such as telling businesses they could take a cut of workers’ tips, refuse cash, or turn away tenants with housing vouchers. In January 2026, the new mayor called it "functionally unusable" and said it was costing about half a million dollars. By early February the city had taken it down ¹³.

The buying side is also stretched. The US Government Accountability Office (GAO) reviewed 13 AI acquisitions at four federal agencies and found officials had trouble getting technical experts to evaluate vendor proposals and found AI costs hard to understand. None of the four agencies systematically recorded lessons learned, so good contract terms on data rights and testing were not being reused ⁵.

What US federal rules now require

Two memos from the Office of Management and Budget (OMB), both issued April 3, 2025, set the federal baseline.

Use rules (M-25-21). For "high-impact" AI, meaning AI whose output is the main basis for decisions with significant effects on people’s rights or safety, agencies must test before deployment, complete an AI impact assessment, and give affected people a timely human review and a chance to appeal. Agencies had 365 days to document these practices, and if a high-impact system cannot be brought into compliance, the agency must stop using it ¹.

Buying rules (M-25-22). For contracts awarded under solicitations issued 180 days or more after the memo, and for options exercised after that point, agencies must include specific terms ². The most important for legal teams:

• Government data. Contracts must permanently prohibit vendors from using non-public agency data, and the outputs produced from it, to train publicly or commercially available AI, unless the agency explicitly consents.
• Lock-in protection. Terms should cover knowledge transfer, data and model portability, rights to code and models produced under the contract, and clear licensing and pricing.
• Testing access. Agencies must be able to evaluate performance on a regular basis, for example quarterly, using their own test data that the vendor cannot see. Contracts must not stop the agency from internally disclosing how the vendor tests.
• Notice of change. Agencies should consider requiring vendors to notify them before adding new AI features, and should decide whether contractors must disclose their own use of AI while doing the work.

Model neutrality terms (M-26-04). A December 11, 2025 memo implementing Executive Order 14319 requires every new federal solicitation for a large language model (LLM), the type of AI behind chatbots, to include contract terms on two "Unbiased AI Principles": truth-seeking and ideological neutrality. Agencies had to update procurement policies by March 11, 2026, including a way for staff to report outputs that break the principles. Existing LLM contracts should be modified where practicable, at the latest before an option is exercised ³.

What Canada’s rules require

Canada’s federal rules focus less on the contract and more on the decision.

The Treasury Board’s Directive on Automated Decision-Making applies to any automated system in production that makes, or helps make, an administrative decision about a client ⁶. Before such a system goes live, the department must complete and publish an Algorithmic Impact Assessment (AIA), a questionnaire of 65 risk and 41 mitigation questions that places the system in one of four impact levels ⁷. The directive then requires:

• Notice before the decision, in plain language, through every service channel in use.
• A meaningful explanation after it of how and why the decision was made.
• Bias testing of data and models before production, and scheduled monitoring after.
• Recourse that is timely, effective and easy to access.
• Human decisions at higher risk. At impact levels III and IV, the final decision must be made by a person ⁶.

One clause matters directly for contracts. If a system uses proprietary software, the department must keep the right to access, test and monitor it, including for an audit, investigation or judicial proceeding, and to let outside parties review it ⁶. The directive was updated on June 24, 2025, and systems in place before then have until June 24, 2026 to meet the new requirements ⁶.

On the buying side, Public Services and Procurement Canada runs an AI Source List of 145 pre-qualified suppliers for requirements up to $37.5 million, used by departments including Immigration, Refugees and Citizenship Canada and Employment and Social Development Canada ⁹. The new AI register adds public visibility: each entry states the system’s purpose and whether it was built in-house or by a vendor ⁸.

Provinces, states and cities are adding their own rules

Rules below the national level are growing, and they often reach further into public disclosure.

Ontario. Since December 1, 2024, the Responsible Use of Artificial Intelligence Directive has applied to all Ontario ministries and provincial agencies. When the public interacts directly with an AI service, or AI is involved in a decision that affects a person, the government must publicly disclose that use and give people an accessible way to ask about it. Ministries report their use cases each year, a list is published, and the directive calls for "clear ongoing human oversight" and warns against automation bias, the habit of trusting a machine’s output without checking it ¹⁰.

Texas. Since January 1, 2026, a Texas governmental agency that offers an AI system to interact with consumers must tell them they are dealing with AI, before or at the time of the interaction, even if it seems obvious. The notice must be clear, conspicuous and in plain language, with no manipulative design. The same law bars governments from using AI for social scoring ¹¹.

The direction across jurisdictions is consistent: tell people when AI is involved, keep a record of where it is used, and keep a person accountable for decisions that matter.

The benefits, and the limits

The case for adoption is real. In Pennsylvania’s year-long pilot of ChatGPT Enterprise, 175 employees across 14 state agencies reported saving about 95 minutes a day on writing, research and summaries, while spending about 35 minutes a day using the tool ¹².

The limits are just as clear.

• Most benefit figures are self-reported. Pennsylvania’s numbers come from exit surveys, and only 136 participants gave direct feedback. Users also reported trouble getting accurate citations ¹².
• Rules apply unevenly. The US federal memos cover executive agencies, not states or cities. Canada’s directive covers administrative decisions about clients, not every internal tool. Ontario’s rules cover ministries and provincial agencies, not municipalities.
• Federal priorities shift. US AI policy has changed with each administration, and M-26-04 is set to expire two years after issue unless renewed ³. Contracts outlast memos, so good terms matter more than any single policy.
• Accountability cannot be outsourced. When New York’s chatbot gave bad legal advice, the city, not the vendor, carried the public cost ¹³.

Where human oversight stays essential

AI can draft, sort and summarize. People must still own the steps that carry legal risk.

• Final decisions on rights and benefits. Both countries’ rules require a human review path for high-impact or high-risk decisions ¹⁶.
• Public-facing advice on the law. Any tool that tells residents or businesses what the rules are needs regular testing against the actual rules, and a named owner who can pull it.
• Contract changes. New model versions and new features should trigger review before release, not after a complaint ².
• The record. Keep the impact assessment, test results, notices and human overrides together, so an auditor or court can see what the system did and who approved it.

Practical next steps for government legal teams

1. Build an inventory. List every AI tool in use or under contract, including AI features added to existing software, and flag any that touch decisions about people.
2. Write a standard AI clause set. Cover data use and training bans, portability and exit, testing access, audit rights, change notices and contractor disclosure of AI use. Start from the federal US and Canadian terms above.
3. Tie risk level to approval. Use an impact assessment, such as Canada’s AIA or the US high-impact test, to decide who signs off and how much human review is needed.
4. Draft the public notice early. Write the plain-language disclosure and the explanation of how a decision was made before launch, not after a complaint.
5. Test public-facing tools against the law. Run a fixed set of real resident questions each month and log wrong answers.
6. Record lessons from every purchase. GAO’s finding is a warning for every level of government: if contract lessons are not written down, they are lost ⁵.

Sources

1. US Office of Management and Budget (April 3, 2025). M-25-21: Accelerating Federal Use of AI through Innovation, Governance, and Public Trust.
2. US Office of Management and Budget (April 3, 2025). M-25-22: Driving Efficient Acquisition of Artificial Intelligence in Government.
3. US Office of Management and Budget (December 11, 2025). M-26-04: Increasing Public Trust in Artificial Intelligence Through Unbiased AI Principles.
4. Nextgov/FCW, Alexandra Kelley (April 16, 2026). Agencies report over 3,000 AI use cases in 2025. Reporting on the OMB 2025 federal AI use case inventory.
5. US Government Accountability Office (April 13, 2026). Artificial Intelligence Acquisitions, GAO-26-107859. Review of 13 acquisitions at DOD, DHS, GSA and VA.
6. Treasury Board of Canada Secretariat (modified June 24, 2025). Directive on Automated Decision-Making.
7. Government of Canada (2026). Algorithmic Impact Assessment tool.
8. Treasury Board of Canada Secretariat (November 28, 2025). Canada launches first register of AI uses in federal government.
9. Public Services and Procurement Canada (modified October 1, 2025). Artificial intelligence source list.
10. Government of Ontario (in effect December 1, 2024, updated January 7, 2026). Responsible Use of Artificial Intelligence Directive.
11. Texas Legislature (signed June 22, 2025, effective January 1, 2026). HB 149, Texas Responsible Artificial Intelligence Governance Act, enrolled text.
12. Route Fifty (March 24, 2025). Pennsylvania governor says AI has ’real promise’ as he unveils pilot program results. Self-reported exit surveys, 136 of 175 participants gave feedback.
13. THE CITY and The Markup, Colin Lecher and Katie Honan (January 30, 2026, updated February 4, 2026). Mamdani to kill the NYC AI chatbot we caught telling businesses to break the law.