Clifford Chance

Generative AI could augment or streamline many internal processes such as desktop research, generating meeting notes and calendar scheduling, assisting software development, creating first drafts of presentations, papers, emails and marketing materials, and much more.  Generative AI can also be used in connection with customer-facing products, services and support – potentially revolutionising certain interactions, client offerings and business models.

Whether these use cases appear on the board agenda iteratively and organically or as part of a project to proactively investigate generative AI opportunities, organisations will need to ensure that proposals to use generative AI are raised to appropriate levels for key stakeholder interrogation, support and oversight.

Some generative AI tools are freely available online – either as stand-alone tools or as products that can integrate into a chain of tools that are provided by multiple developers. Although early adoption and experimentation with generative AI is key to realising its potential, if your business does not guide or restrict the use of these tools, they could potentially be used by your personnel in unanticipated and undesirable ways.

Your suppliers may also be incorporating generative AI in the products and services they provide to your organisation, which could result in your business unknowingly using or relying on such AI, or your business and customer data being shared with third-party generative AI developers. Do you have a process for identifying AI in your supply chain, as well as associated decision-making, contracting and ongoing monitoring processes for receiving AI-assisted services and products?

The regulatory framework that applies to generative AI is complex and multilayered. Existing laws include privacy, cyber and operational resilience, intellectual property, antitrust, employment, product safety, content moderation, environment, human rights and consumer protection, as well as sector-specific or technology-targeting legislation. These will sit alongside new AI-specific laws and guidance as the capabilities of generative AI continue to develop and regulators across the world explore what AI-specific legislation looks like. For example, the EU, China, Canada, India and various US states are proposing and enacting AI-specific laws, while the US Blueprint for an AI Bill of Rights, the UK's AI policy paper and Singapore's Model Artificial Intelligence Governance Framework all set out principles for responsible AI use and governance.

In the EU, the AI Act is being negotiated and will sit alongside the proposed AI Liability Directive (as well as other laws such as the Revised Product Liability Directive and the Digital Services Act) in regulating how certain AI can be placed on the market, put into service and used and who is liable for harm caused by the use of AI. The current text of the EU AI Act specifically covers generative AI, by bringing 'general purpose AI systems', those which have a wide range of possible use cases (intended and unintended by their developers) in scope.

In the US, the Department of Commerce's National Telecommunications and Information Administration published a request for comment on how to achieve "trustworthy AI", which sits within an existing administration policy framework (including the White House Office of Science and Technology Policy Blueprint for an AI Bill of Rights and the National Institute of Standards and Technology's AI Risk Management Framework) and alongside a host of State-level AI-specific legislation and bills.  In addition, Senate Majority Leader Chuck Schumer has announced an early-stage legislative proposal aimed at advancing and regulating American AI technology.

At the same time, China is working hard to show leadership both on AI investment, home-grown technology and regulation – addressing specific issues such as deep-fakes whilst seeking to minimise social disruption. For example, the Regulations on the Administration of Deep Synthesis of Internet Information Services focus on ‘deep fake’-type use cases as well as generative AI-based chat services. China has also issued for public consultation its draft measures on the administration of generative AI services. These targeted measures sit alongside important regional approaches, notably in Shanghai and Shenzhen.

At the international level, G7 leaders recently announced the development of tools for trustworthy AI through multi-stakeholder international organisations through the 'Hiroshima AI process' by the end of the year.

As the laws governing AI evolve, definitions such as 'AI system', 'AI user', 'AI provider' and 'AI-generated content' are being created and negotiated. Some of these definitions may be broadly drafted and could capture companies that have not previously considered themselves to be AI providers or users. Organisations will need to understand the countries and manner in which they intend to roll out the use of generative AI, as well as the scope of potentially relevant laws, in order to identify the laws applicable to their procurement and use of generative AI.

Data must be processed in compliance with any ownership rights, legal requirements, contractual terms and company policies. Some of the key areas for legal risk management – privacy, intellectual property (IP) infringement, and other legal and commercial restrictions on data use – are discussed below.

Privacy: Where AI uses personal data protected by privacy laws, its developers and users will have to comply with relevant requirements. For example, the range of obligations under the EU's General Data Protection Law (GDPR) for developers and users of generative AI include having an appropriate legal basis for data processing, providing notices and adhering to privacy principles (such as transparency, fairness, purpose limitation, accuracy and data minimisation). Data subjects would also need to be able to exercise their rights, such as rights to deletion and rectification of their personal data.  More broadly, AI system developers and users would also need to consider how requirements for 'privacy by design and default' and appropriate security measures should be (or have been) approached.

The application of privacy laws to generative AI is new and untested. Data accuracy, rectification and deletion could be technically challenging in relation to generative AI which has 'learnt' incorrect information (even if that information is later deleted from, or corrected in, the data used in training the AI) and which is capable of 'hallucinations' (incorrectly generated answers).  Other open points include whether 'legitimate interests' can be relied upon for processing of data scraped from the internet in this context, how compatibility of processing will be seen in this context, and whether exemptions to privacy notice requirements will apply.  Early application of privacy law to generative AI developers is being seen in the activity of European data protection authorities (DPAs), such as the interim order on OpenAI's ChatGPT issued by Italy's Garante, which temporarily restricted ChatGPT in Italy until such time that OpenAI was able to provide it sufficient assurances relating to the services' compliance with GDPR.

While users have not been in the enforcement spotlight, they should ensure they apply their privacy compliance frameworks to generative AI use and procurement – including understanding whether use of the AI system involves any restricted transfers of personal data and communicating to personnel any restrictions regarding submissions of personal data when using generative AI.

IP infringement: IP risks and challenges can arise from the data and materials used to train generative AI, as well as outputs produced by the models. For creators of generative AI models (and, in some cases, their users) there are risks of IP infringements in relation to the training data used. For example, where datasets are derived from scraping publicly accessible materials online without a licence from copyright owners, or where datasets licensed for one specific use are re-purposed for training an AI model, the training process will typically make copies of works that are unauthorised by the rightsholder. Whether those copies constitute infringements of copyright will often depend on the scope of any applicable data mining or fair use exceptions (which vary significantly between countries), and the way in which the training process stores or deletes the source materials. Generative AI users should also be aware of the risk that the output generated in response to a user prompt could infringe copyright (and ancillary laws) and that this risk is heightened where small training data sets are used - if a given output is too similar to specific input materials. This risk is heightened where small training data sets are used, although some AI tools implement filtering systems aimed at this very risk, which prevent the models from returning outputs which are too similar to any specific inputs. As users of generative AI may often be reliant on the AI developer's controls in how it trained the system, exploration of these aspects should form part of a company's review of a generative AI system.

Contractual restrictions and confidential information: There could be other restrictions on a company's ability to share data with a generative AI system – for example, contractual restrictions on how a client's or supplier's information may be used or shared with third parties, such as usage restrictions in contracts with content providers. There may also be legal restrictions regarding certain data such as material non-public information and price-sensitive information. A company may also wish to keep confidential certain of its own data (including proprietary information such as software code, product designs or trade secrets). As information provided while using generative AI is likely to be accessible to its developers, inputting such data to that system without appropriate consents and licenses could breach legal or contractual restrictions or a company's own protective policies. Unless otherwise specifically agreed with the AI provider, there is also a risk that the information submitted may form part of the system's iterative development and a risk that this could inform (or form part of) answers given by that system to other users in the future. A company may have concerns regarding such AI systems learning from the company's data, even if that data is not sensitive, where competitors may also use a version of the system that has "learnt" from the company's use. 

Organisations that will roll out generative AI systems, or otherwise allow their personnel to use such AI, must consider the uses for which they will permit AI-generated output to be used, how they will educate staff regarding the limitations of the relevant AI system, and the policies or guidance to be implemented to ensure the output is quality checked and used appropriately.

Accuracy, completeness and bias: The textual responses of generative AI systems are based on patterns in the "corpora" (bodies) of data they have been trained on, and fundamentally operate by predicting the next word in a particular context. As with other AI, generative AI is dependent on the quality of its training data, and therefore susceptible to the introduction of errors and bias through the training and development process; particular vulnerabilities typically exist in the early stages of model building. The outputs of a generative AI system that has been trained using vast amounts of data of unknown provenance that is available on the internet could reflect the societal biases that are evidenced (and in some cases magnified) online, as well as make use of inaccurate or out-of-date information published on the internet. Some of the complex neural-network based techniques used in generative AI can make it difficult to understand how an answer has been created, which can lead to challenges in identifying inaccurate or biased responses. Generative AI can also produce 'hallucinations': responses which appear authoritative but are factually untrue and might not be traceable back to the real-world, factual data. Even citations generated by the AI can be inaccurate – increasing the importance of robust human-in-the-loop governance and auditing processes, particularly for high-risk use cases.

Organisations using AI will have a range of legal obligations regarding equality, diversity and fair treatment, as well as ethical and reputational imperatives. The accuracy and completeness of an AI system's output may also be important, with the degree of importance varying depending on the use for which the output will be used and the level of human review, expertise and judgement that will be applied. In some cases, accuracy will be operationally, commercially or reputationally critical, or legally required.

Before using generative AI in business processes, organisations should consider whether generative AI is the appropriate tool for the relevant task. Where use of generative AI is appropriate, organisations should identify the guard-rails they will put in place to ensure their staff understand the limitations and potential risks regarding inaccuracy and bias in AI output, the organisation's restrictions regarding the purposes for which such AI output may be used (and what the consequences of breaching these would be), and its requirements regarding an effective layer of human quality control. Factors such as cost will also have a role to play here, with the cost of generative AI system based searches currently far outweighing the cost of using, for instance, internet search engines.

IP protection: As AI becomes more sophisticated and 'creative', key legal issues include the concepts of "inventorship", "authorship" and "patentability" of products created by or using AI. There can be uncertainty regarding ownership of AI-generated or AI-assisted output; and whether IP rights can subsist in an output at all. If there is no IP protection, there may be nothing to stop widespread copying of the outputs of generative AI. The availability of IP protection is significantly affected by the nature of the output, the degree of human involvement and the jurisdictions involved. Can an AI system be an author or an inventor under IP laws? What about AI-assisted works? Jurisprudence is evolving – and, in some cases, with diverging approaches between jurisdictions. In general, AI-assisted creations will more readily attract IP protection than works that are wholly or substantially created by generative AI without human input. Organisations will need to consider whether generative AI might be used to create something (such as software code, a written paper or a product design) which the organisation wishes to protect under an IP regime, and how it will go about maximising the availability of those protections through the manner in which it guides internal AI use.

Employee concerns: Where generative AI output will streamline or alter processes and impact and/or remove the roles of certain personnel within an organisation, careful thought will need to be given regarding how the roll out and associated communications are managed. In some cases, individual employees and/or employee representatives (including works councils) will need to be informed and consulted.

Generative AI systems may be processing legally or commercially sensitive data and may be deployed in the context of regulated or operationally critical processes, with varying degrees of human involvement. As with other software, cyber-security and operational resilience requirements and considerations will apply to the use and procurement of generative AI systems.

More broadly, where the use of generative AI involves sharing data to a publicly available AI system or to a private instance on a third-party cloud-based platform (rather than being available on-premises) your company will want to understand and assess the risks of message interception and other cyber-attacks and any security measures applied by the supplier. As part of any AI procurement your company would also need to understand its responsibilities regarding system use and configuration, the supplier's business continuity plan and how the unavailability of that platform would affect your business.

In addition to requirements for appropriate security measures under privacy laws where personal data is processed, organisations will need to consider cyber, business continuity and operational resilience or broader governance requirements in relation to certain sectors and products. Such requirements are particularly important where AI systems are relied on for operationally critical, regulated or customer-facing processes, especially as it may not be immediately obvious when the operation of an AI system has been hijacked.

Some sectors, such as the financial services sector, may also have overarching governance and oversight frameworks under which cyber-security and operational resilience considerations may apply to certain uses of generative AI.

Will data entered on the AI system be protected, and will the operation of the system be robust? To what degree will your personnel rely on the use of that AI, and are contingencies needed in the event it becomes unavailable (for a temporary period, or permanently)? Before putting any generative AI into operation, organisations need to consider how they will identify system issues or failures, whether they need to have backup options and remediation plans in place, and whether stakeholders need to be informed in advance of any likelihood of failure, the responsible team, and the best course of action to take in case of any issue.

Many of the laws and regulatory principles referenced above (see section 2 above) include requirements regarding governance, oversight and documentation. For example, the EU GDPR includes a principle of accountability alongside a number of specific requirements for assessments and record-keeping, and the draft texts of the EU's AI Act include a number of requirements relating to risk-management systems, oversight, audit and record-keeping. In addition, sector-specific frameworks for governance and oversight can affect what 'responsible' AI use and governance means in certain contexts. For example, financial services regulators in many jurisdictions have rules and guidelines on oversight and governance in areas such as retail banking products, outsourcing arrangements, model risk management, operational risk management and the responsibilities of senior management. Additionally, laws that apply to specific types of technology, such as facial recognition software, online recommender technology or autonomous driving systems, will impact how AI should be deployed and governed in respect of those technologies.

Organisations will need to consider where AI sits within their governance and risk-management frameworks and how those frameworks may need to be tailored or expanded to address generative AI.

For many organisations, existing governance frameworks, including policies on advanced analytics innovation, data governance and IT risk management, could be a helpful starting point for governance of generative AI systems. Organisations could also produce a set of AI principles and map them to the existing risk frameworks. Alternatively, or in addition, some organisations may wish to create an overarching AI governance framework that applies (in a risk-adjusted manner) across use cases, and/or specific policies targeting particular uses – such as a policy for use of a specified generative AI system. In some cases, internal guidelines may be more appropriate. Consideration will need to be given to (i) the interplay between existing policies, (ii) regulatory expectations, (iii) how the approach chosen affects the ease with which the intended policy audience will understand the requirements and restrictions, (iv) company culture, and (v) the organisation's ability to implement supporting processes as well as monitor and enforce compliance.

Processes that exist in other contexts regarding procurement, development, implementation, testing and ongoing monitoring of IT systems should be reviewed, adapted and applied as necessary across the roll-out and use lifecycle of a generative AI system. This adaptive governance would need to be sensitive to differences between types of AI systems in order to apply effectively to the changing technology landscape. Organisations should also review how their related processes, including for training, record keeping and audit, would be applied in this context to support any policies, principles and guidelines.

Organisations will need to determine who will be responsible for AI oversight and decision making at senior levels, whether the organisation would benefit from establishing any AI-specific governance structures (such as an internal AI council, board or committee) and what the lines of accountability will be.

Some regulators may require designated senior management responsibility for oversight of AI technology in the context of wider senior management responsibility regimes. Even where not expressly required by a law or regulator, many organisations will find it beneficial to designate an appropriate senior individual or group of individuals with the relevant skill set and knowledge to consider, challenge, steer and approve AI-related decisions.

Consideration should also be given to establishing clear and appropriate accountability lines throughout the company up to senior management, and having in place people with the right skills, expertise, experience and information to support and advise. Recruitment, talent pipeline management and staff training will be aspects to consider in planning for effective AI risk management.

Contracts for AI procurement, development or investment form part of the wider governance framework mitigating AI risk. Contracts for the procurement or use of a generative AI system require careful review to understand and, as far as possible, negotiate appropriate terms to address AI-specific risks in the allocation of rights, responsibilities and liability. Such contracts can look very different from a standard contract for a traditional piece of software. Each implementation of AI needs to be evaluated on a case-by-case basis, considering the proposed uses for the system and how it will interact with other systems.

With non-deterministic systems, performance will vary over time. Questions therefore arise as to how to: define performance criteria; monitor performance; allocate liability; and define remedies for poor outcomes or any damage caused by the system. For private instances of an AI system that can be fine-tuned based on an organisation's data, other questions to consider include the rights relating to a privately 'trained' instance of the AI system, where responsibility lies for various aspects of legal and regulatory compliance in the system's operation, and how the system will be supported, monitored and audited.

In some cases, these AI systems may be made available only under supplier-friendly terms, which will require companies to understand the risks they are accepting regarding things such as system availability and updates, supplier data access, cyber-security protection and possible IP infringement – with little by way of precedent in terms of market practice or positions. Further, where generative AI products are integrated into a chain of tools provided by a number of suppliers, there will be multiple applicable contractual terms.

Although the degree to which an organisation can negotiate a contract for a generative AI system with the supplier will vary depending on the parties involved and type of procurement or use, the contractual framework will need to be understood by relevant stakeholders as part of an organisation's risk management and governance process for generative AI.