Responsible Use of Generative Artificial Intelligence in Research

Researchers are to consider these principles as they explore the use of GenAI in developing research processes and products, while striving to maintain—and even improve upon— research integrity and standards relevant to their disciplines. This guidance also applies to research that is part of generating visual, compositional or performance-based creative works.

Three key duties of data protection, verification, and transparency frame appropriate use of GenAI in research. These duties are derived from the Cornell Statement on Generative AI in Academic Research. Beyond the short list above, researchers may also be obligated to follow the policies of external entities regarding GenAI, including funding agencies, publishers, suppliers, or other institutions where collaborating researchers reside. For instance, NIH and NSF prohibit the use of GenAI in peer review as a breach of confidentiality. Learn more about each of the three duties and principles below.

Principle #1: Protect data and information

Many popular GenAI tools collect and store information from users as part of their learning process. While some GenAI services offer a “no training use” policy as part of enterprise agreements, by default, researchers should first assume that any data or information entered into web-based GenAI could become part of its training data and thus be recoverable by third parties from future iterations of the model. Terms for the tool’s use of information entered by the user should first be verified by the researcher in advance: do not assume that data entered is private, that output is already in public domain, nor that one can share information that is private, export-controlled, sensitive, or proprietary.

align with existing University policies

All members of the University community (faculty, staff, students and affiliates) wishing to enter institutional research data or other sources of information into GenAI must ensure that doing so will not breach any University policies, including:

• Code of conduct
• HIPAA policy
• Student records (FERPA) policy
• HR policy on confidentiality
• IP policy
• Data stewardship policy: Proposed policy is currently in review for adoption by the University of Rochester Board of Trustees. Link to be updated as available.
• Information technology policy: Attention should be paid to the privacy components of this policy

As a general guide, only data classified as low risk by the University Data Security Classification Policy should be considered suitable for entry to GenAI. The use of GenAI with medium or high-risk data (especially identifiable human data) should only be contemplated when the relevant tool has undergone appropriate internal review.

Principle #2: Verify

Researchers have a duty to verify the output of GenAI systems to ensure that they have not introduced inaccuracies or biases into the research process. GenAI can produce extremely convincing text, imagery, audio and video (and increasingly other types of data); but GenAI tools may not stay faithful to some ground truth or fairly represent the world.

Use of GenAI carries real risks regarding the reproducibility and reliability of research. This is particularly true in the context of research involving cultural practices or human diversity, where the risk of bias is high.

Carefully scrutinize outputs

The following are best practices for instances when the use of GenAI requires a higher degree of scrutiny:

• Inquiry as the basis of forming research questions: verify the provenance of your resulting question(s) as the output of GenAI stems from others’ content put in. To what degree have you invented the focus of the research question or hypotheses to test?
• Literature review: verify to ensure that critical claims (i.e. effect size, statistical significance) from individual articles or sub-literatures are accurately reported. Do not rely on GenAI to represent the range of works available on a topic.
• Output of analysis and code: in concordance with disciplinary standards, verify reproducibility of results, and ensure authorial control of your original content. Also, verify to ensure that generated code is human interpretable and performs the intended function.
• Interpretation is an act of authorship: consider your discipline’s standards when GenAI is used to develop interpretations, and be aware that you are responsible for the integrity of the interpretation (e.g., how theory connects to the data, limitations for discussion, etc.).
• Attribution: verify to ensure that data sources and citations exist, and are correctly attributed. The text produced by GenAI tools may not include any attribution of authorship, and if attribution is included, it may attribute incorrectly. Content produced by GenAI tools is often based on copyrighted, web-scraped or publicly available material. In all cases, the author using the GenAI tool is responsible for correctly attributing prior work. Authors who fail to do this risk plagiarizing prior work, and/or infringing copyright.
• Research involving human meaning or difference: verify to ensure that outputs which report on or represent human diversity (incl. biological, social and humanistic research) are accurate.
• Composing and editing or revising products for dissemination: verify to ensure data summaries, or the information or knowledge generated (i.e., in creative works derived from the research inquiry) – including quotations, citations, author meaning or descriptions of methods – have not been altered.

Principle #3: Be transparent

Consistent with disciplinary and publishing norms, the use of GenAI should be disclosed/identified at the time of proposing, reporting, publishing or when engaging in other dissemination activities. In reporting such use, the following existing policies should be adhered to:

• Research Misconduct Policy
• Intellectual Property Policy

While norms around the reporting of GenAI use are evolving, we urge researchers to err on the side of acknowledging and reporting all use of GenAI tools in the research process, including use in idea development, assistance with composition, code assistance, or generation of presentations and figures.

For instance, NSF currently recommends attribution in the context of proposing research: “Proposers are encouraged to indicate in the project description the extent to which, if any, generative AI technology was used and how it was used to develop their proposal.”

If GenAI is used to conduct research that will result in a creative work with a registered copyright (e.g., exhibit, multimedia, music, script, book, dance, sculpture or other visual art form), then extra care should be taken. While the norms around authorship are evolving, if the final work contains more than de minimis AI-generated content as defined by your field, the nature and use of GenAI should be disclosed, and the parts of the work created by the author and by the GenAI should be distinguished and described.

Moreover, researchers should adequately acknowledge the use of GenAI in the research design. GenAI poses significant risks to the reproducibility of research processes, because of their inherently stochastic nature. The specific GenAI tool should be documented along with the parameters used (including prompts, context windows, and version numbers), for instance, when using GenAI to generate synthetic datasets. Use of GenAI systems where the model is freely available can aid reproducibility. Protecting provenance – that is, knowledge of the “ownership” history—or the contextual narrative of how information has been created through research—also maintains value in and authenticity of research.