Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A. Lemley, Percy Liang &ndash; Stanford University, USA
Foundation Models and Fair Use
Existing foundation models are trained on copyrighted material. Deploying these models can pose both legal and ethical risks when data creators fail to receive appropriate attribution or compensation. In the United States and several other countries, copyrighted content may be used to build foundation models without incurring liability due to the fair use doctrine. However, there is a caveat: If the model produces output that is similar to copyrighted data, particularly in scenarios that affect the market of that data, fair use may no longer apply to the output of the model. In this work, we emphasize that fair use is not guaranteed, and additional work may be necessary to keep model development and deployment squarely in the realm of fair use. First, we survey the potential risks of developing and deploying foundation models based on copyrighted content. We review relevant U.S. case law, drawing parallels to existing and potential applications for generating text, source code, and visual art. Experiments confirm that popular foundation models can generate content considerably similar to copyrighted material. Second, we discuss technical mitigations that can help foundation models stay in line with fair use. We argue that more research is needed to align mitigation strategies with the current state of the law. Lastly, we suggest that the law and technical mitigations should co-evolve. For example, coupled with other policy mechanisms, the law could more explicitly consider safe harbors when strong technical tools are used to mitigate infringement harms. This co-evolution may help strike a balance between intellectual property and innovation, which speaks to the original goal of fair use. But we emphasize that the strategies we describe here are not a panacea and more work is needed to develop policies that address the potential harms of foundation models.
Implied Licenses and Common Crawl. On the other hand, many creators voluntarily post their works on the internet with permissions for web crawling. It is well-established that merely posting something on the internet does not waive the intellectual property interest in the work, but many data creators use an industry-standard “robots.txt{"} file to affirmatively to include their website and data in caches and search indexes. In Field v. Google, Inc. (D. Nev. 2006) a district court held that Google could cache web content that did not disallow scraping via robots.txt, suggesting that there was an implied license and thus the use was not infringement. This license only extended to caching in that case, which does not necessarily reflect the uses of foundation models we discuss throughout this work, so it is unlikely to cover all the use cases we describe here. And the bounds of the uses covered by the robots.txt file are untested in court.21 While the issue of whether the implied license extends to foundation model training has not been resolved in litigation, it is possible that an outcome like Field v. Google, Inc. (D. Nev. 2006) would extend to some foundation model uses—in particular, for building a cached dataset and training a model.¶ It is worth noting that the use of a robots.txt header or other opt-out mechanism has implications for fair use also. Datasets and models like C4 (Raffel et al., 2019) and LAION-400M (Schuhmann, 2021), rely on CommonCrawl data which is crawled only if users explicitly allow it through their robots.txt file. CommonCrawl is able to host a snapshot of the internet largely because of fair use arguments. As the organization’s director argues, there is a transformation into a different—not easily human-readable—format, the organization does not take a snapshot of entire webpages, and the use itself is transformative (from actively presenting content to caching content) and for the public benefit (Leetaru, 2017). In Field v. Google, Inc. (D. Nev. 2006), respect for the robots.txt file also was considered in the fair use assessment with the court noting that Google in good faith followed industry standards that would prevent caching (respecting disallowing crawling via a robots.txt). It is possible, then, that providing an opt-out mechanism for data creators and respecting the robots.txt opt-out mechanism will be taken into account in assessing a fair use argument, as it was in Field v. Google, Inc. (D. Nev. 2006).²²¶ [...] Furthermore, if web-crawled data is used, restricting it to data that respects robots.txt opt-outs can make a fair use argument more tractable, though not guaranteed. As we noted before, in Field v. Google, Inc. (D. Nev. 2006), respect for the robots.txt file was considered in the fair use assessment with the court because it gave the plaintiff opportunity to opt out. This is likely why many webcrawl-based models rely on the CommonCrawl dataset as a source. Its webcrawl automatically respects robots.txt opt-outs and does not crawl every webpage in full. It is possible then that future fair use assessments could consider respecting the robots.txt opt-out—or implementing other opt-out mechanisms—favorably, as was the case in Field v. Google, Inc. (D. Nev. 2006). Conversely, ignoring a robots.txt opt-out could negatively impact a fair use assessment. However, Kapoor & Narayanan (2023) have argued that there are structural critiques of opt-out mechanisms beyond the current state of the law.¶