In the age of artificial intelligence, traditional methods of software procurement are no longer fit for purpose. Traditionally considered a business function—securing the best software at the best prices—procurement teams now make decisions about highly complex technology systems that can shape an organization’s governance posture, regulatory exposure, security, and operational resiliency.
Organizations interact with AI in many forms, from standalone software to features built into existing platforms. Regardless of how AI enters an organization, its presence immediately raises governance questions. Who owns the data? How was the system trained? Who is responsible for errors? Most procurement teams lack the technical expertise to analyze this complexity and review systems before making procurement decisions, creating governance risks from the start.
Compounding these complexities is the power asymmetry between procurement teams and AI vendors. A small number of dominant AI service providers can set terms of use and implement changes top-down without negotiating with customers. Urging executives to embrace AI at the organizational level only intensifies this pressure, leaving procurement officials navigating legacy processes without the benefit of AI-specific training or guidance.
Together, these factors have created new limits to governance. Procurement has transformed from a business function to one of the most important – and least mature – functions.Components of AI governance in organizations.
Emerging risks of purchasing artificial intelligence
Unlike traditional software procurement, AI procurement imposes governance risks that continue to evolve after the contract is signed. AI supply chains include models, infrastructure providers, APIs, and application layers, making it difficult to determine who is responsible for what happens if things go wrong. Even if responsibility is established at the time of purchase, ongoing updates and feature releases can complicate accountability structures and introduce new risks after contracts are signed.
The pace of technological and regulatory change is also shortening procurement and contract review cycles. The upside is that shorter courses provide flexibility; The downside is that it introduces ongoing procurement and governance requirements that organizations must manage on an ongoing basis.
These challenges are exacerbated by market concentration. Representing OpenAI, Anthropic, and Google collectively 88 percent of LLM use is in organizationsleaving buyers vulnerable to top-down changes in pricing, product features and contractual terms. Furthermore, the nature of AI drives “lock-in” to individual providers as models improve by interacting with user data and workflows. Switching providers can therefore become operationally disruptive, expensive and technically difficult.
Data management is one of the most important Underestimated risk areas In Artificial Intelligence Procurement. Basic due diligence questions surrounding data collection, storage, and model training are often left unanswered at the point of contract. As a result, organizations may inadvertently expose confidential information, private business data, or client records to external model training operations.
AI also presents intellectual property risks; Models trained on data from the web may produce output that includes copyrighted or unauthorized material, exposing organizations to legal and reputational risks. Other intellectual property considerations, such as ownership of AI outputs and metadata, should be addressed proactively during procurement.
Agent AI is the next frontier for purchasing risk. Able to traverse multiple platforms and data sets independently, these systems present risks of a radically different magnitude. Increasingly, sellers Excluding major damages related to artificial intelligence One of the liability clauses within AI contracts is the agent, leaving buyers exposed. As AI agents grow in sophistication and popularity, all of these concerns – responsibility, accountability, dependency, and data management – must be continually addressed and renegotiated.
Four pillars of responsible AI procurement
Because AI procurement presents unique risks not associated with traditional software, organizations need procurement frameworks specifically designed for AI. Based on our experience consulting organizations on responsible AI adoption, we recommend building such frameworks on four pillars.
First, teams’ skills should be improved. Procurement teams must be familiar with AI, safety and compliance functions. This does not mean that every procurement officer must become a technical expert, but they need to have sufficient understanding to evaluate governance implications.
Organizations must support this through certification programs, cross-functional procurement models, and close collaboration between procurement, legal, compliance, cybersecurity, and technical teams. Training the broader workforce in the responsible use of AI It’s also important, especially as employees increasingly adopt AI tools independently.
Emerging vendor engagement models, where procurement staff collaborate with AI vendors Forward deployment engineers To customize tools according to operational needs, ensuring a deeper understanding of the technical elements. While this approach can improve results, organizations also need to invest time in internal change management, governance reviews, and process design.
Second, customize purchases according to the type of AI system to be acquired. Procurement teams encounter a range of AI products, including AI-enabled tools, AI-enabled features, and core models. Each category introduces new governance, compliance and operational risks that require tailored procurement approaches.
For AI-powered products, procurement teams should focus on the appropriate use case, data sovereignty, and hosting arrangements. Consider a negative example: When Workday released a AI-powered applicant tracking systemhas been described as a ready-to-use tool for HR teams. However, the producer violated the Age Discrimination in Employment Act by favoring applicants under 40 years of age. Organizations that adopted the tool without adequate procurement auditing risked compliance under labor law.
When vendors introduce AI features into existing software products, procurement teams must renegotiate contracts that lack AI-specific provisions. This is an increasingly common situation and presents complex governance and compliance risks. For example, when GitHub updated its training requirements in early 2026organizations with lower-level subscriptions found that their own data was being used to train AI models. Such situations undermine organizations’ privacy, data protection, and security controls.
When purchasing a basic model or platform, organizations should focus on technical capabilities and strategic implications. For example, the United Kingdom’s National Health Service recently faced criticism over its procurement process It allowed the US company Palantir to access specific patient data While developing the unified data platform. The platform aims to deploy AI across patient records to improve efficiency. However, the procurement approach has undermined public confidence in the NHS and its services.
Third, AI procurement must be grounded in emerging governance frameworks, standards and regulatory requirements. Even in jurisdictions where AI regulation remains underdeveloped, alignment with the broader regulatory system allows organizations to demonstrate that their use of AI is safe, responsible, and trustworthy. It also creates defensible evidence that governance obligations have been taken into account prior to publication.
Standards like ISO 42001 For artificial intelligence management systems and ISO 23894 AI risk management can help organizations establish documented governance processes and create auditable evidence trails from the beginning. Other mechanisms, e.g IEEE Standards and conformity assessment and IAPP AI Governance Vendor Reportscan also support procurement due diligence.
Finally, procurement must be integrated into ongoing AI governance processes. When an organization purchases an AI system, it makes governance choices about data sovereignty, liability, regulatory compliance, and long-term vendor dependency. These choices require ongoing review with governance stakeholders as models evolve, regulations change, and vendors update products or terms of service.
Organizations must therefore integrate procurement directly into broader AI governance structures through frequent audits, compliance reviews, performance evaluations, and cross-functional oversight processes. Where possible, organizations should pursue shorter contracts and procurement cycles to avoid making long-term commitments in the face of rapidly changing technology.
AI governance starts at purchase
For many organizations, procurement remains one of the less mature dimensions of AI governance. However, purchasing decisions increasingly determine how data is managed, where accountability lies, which vendors gain influence over operations, and how organizations remain agile as AI systems evolve.
As AI adoption accelerates, procurement can no longer operate as a purely business function. Every AI contract now includes decisions on governance, risk, compliance, security and strategic dependency. It has become a governance mechanism, which will play a crucial role in whether enterprise AI systems are deployed responsibly, compliantly and effectively.
Amelia Williams He is the Chief Research Impact Officer at Triliteral Research and has experience communicating science at the intersection of emerging technologies, environmental issues, ethics, and policy. At Tritraler, you support the development and implementation of research projects alongside policy, media and industry engagement.
