ChatGPT in Research and Education: Continuing the Conversation

GPT stands for generative pretrained transformer, a natural language generation model that can learn and fine-tune outputs based on labeled prompt data, ranking of responses, and proximal policy optimization. Just two months after its launch, ChatGPT was estimated to have reached 100 million monthly active users, making it the fastest-growing consumer application in history.¹ But when the novelty wears off, there will be a need to encourage responsible and effective use if the tool is going to become an accepted part of the research process.

Several articles recently published in ACS journals have explored the topic, particularly how GPT-based AI might disrupt research in specific fields of chemistry as well as in education and publishing. One recent study published in the Journal of Chemical Information and Modeling sought to answer a few questions about how well large language models (LLMs) can understand and answer chemistry-specific questions. After conducting five different tests with the ChatGPT model, the researchers found that, without using any ‘tricks’ to help the system, the accuracy ranged from 25% to 100%. Typically, questions on popular subjects (“What is the water solubility of [polymer]?”) were easily answered, but there was low accuracy for very specific topics (“What is the SMILES representation of [compound name]?”). The AI particularly struggled with compound names, even for very small molecules such as alkanes of just two or three carbon atoms.²

Another study published in the Journal of Chemical Education found that ChatGPT as a learning tool is prone to conceptual errors in many of its chemistry answers and explanations—such as equating the speed of electromagnetic radiation with its energy—as well as errors related to periodic properties, ionization energy, and effective nuclear charge. This demonstrates that, while ChatGPT is very capable when it comes to interpreting chemical symbolism and communicating in ways chemistry students will understand, it is a below-average chemistry student itself and should not be relied upon for general chemistry instruction. However, ChatGPT could be used for students to demonstrate their understanding of certain topics by identifying and correcting the chatbot’s mistakes.³ 

A similar experiment conducted at the University of Hertfordshire, United Kingdom, also tested the chemistry knowledge of ChatGPT, and the researchers found it could generate responses to questions that focused on knowledge and understanding with “describe” and “discuss” commands.⁴ However, for queries focused on application of knowledge and interpretation with non-text information, the technology reached a limitation. The work was undertaken in the context of the pharmaceutical science program, a general applied chemistry discipline. Assessments from students’ Year 1 and 2 module exams were fed into ChatGPT, and responses were marked in line with the original exam mark scheme. Sadly for AI, it did not pass the exams in either study, and cannot claim a degree in chemistry any time soon. The overall grades on the Year 1 and 2 pharmaceutical science papers were 34.1% and 18.3%, which did not meet the pass criteria,⁴ and its general chemistry problem-solving score was 44%, a value well below the class average of 69%.³

Two new viewpoints published in Environmental Science & Technology aim to demonstrate how GPT can be used for research in the environmental sciences. The authors argue that key benefits could be in improving writing quality, identifying themes, retrieving information, and streamlining workflows. ChatGPT is also able to support with coding, debugging, and syntax explanation.⁵ Chief among the potential pitfalls and challenges is the fear that AI could generate false or fabricated information, especially in smaller academic fields where there is a lack of training data. Another controversial point is that this kind of reliance on AI removes human wisdom and judgment from the research process, which may ultimately lead to a reduction in academic integrity as well as a decline in skills and abilities.^5,6 From an environmental perspective, the authors argue the possibility of direct and indirect environmental impacts, including energy consumption, resource use, and carbon dioxide production.

Turning again to chemistry education, several recent articles have investigated how ChatGPT may be used in academia to support both educators and students. It seems likely that students will increasingly turn to AI for help with homework and for information and guidance on a wide range of topics. One possible place where AI could be useful is in helping students with calculations and writing skills—potentially for the conclusion section of written laboratory reports—but as we have already seen, it can struggle with compounds and chemical analysis. To use ChatGPT effectively, students need to understand the chemical principle they want an answer for, effectively ask the appropriate question, and then analyze the AI output; therefore, it may soon be necessary to provide students with tips and best practices for how to effectively use AI in their courses. But for educators who do not want their students to use AI for writing, there are a few GPT detectors already available to sniff out bot-written text.⁷

Big data is a basic need for AI, and at present, language bots are not capable of understanding new information outside their training set, generating insights, or conducting deep analysis. This limits their use in scientific research and writing, but the technology will evolve.^8,9 In the face of this new technology, ACS and other publishers have stated that AI tools do not qualify for authorship, and use of AI for text or image generation should be disclosed in the manuscript. Early recommendations around best-practice use suggest thinking of AI-generated text as a very early draft—a springboard for creativity—rather than a shortcut to a finished piece. Text should also be checked for plagiarism, and all citations verified by hand. Ultimately, current perspectives seem to agree that AI tools are adequate for regurgitating conventional wisdom but not for identifying or generating unique outcomes.⁹

References

1. Yu, K. ChatGPT sets record for fastest-growing user base - analyst note. Reuters 2023.
2. Casto Nacimiento, C. M. and Pimintel, A. S. Do Large Language Models Understand Chemistry? A Conversation with ChatGPT. J. Chem. Inf. Model. 2023, 63, 6, 1649–1655.
3. Clark, T. M. Investigating the Use of an Artificial Intelligence Chatbot with General Chemistry Exam Questions. J. Chem. Educ. 2023, 100, 5, 1905–1916.
4. Fergus, S. et al. Evaluating Academic Answers Generated Using ChatGPT. J. Chem. Educ. 2023, 100, 4, 1672–1675.
5. Zhu, J.-J. et al. ChatGPT and Environmental Research. Environ. Sci. Technol. 2023, Articles ASAP.
6. Rillig, M. C. et al. Risks and Benefits of Large Language Models for the Environment. Environ. Sci. Technol. 2023, 57, 9, 3464–3466.
7. Humphry, T. and Fuller, A. L. Potential ChatGPT Use in Undergraduate Chemistry Laboratories. J. Chem. Educ. 2023, 100, 4, 1434–1436.
8. Luo, Y. Chemistry in the Era of Artificial Intelligence. Precis. Chem. 2023, 1, 2, 127–128.
9. Buriak, J. M. et al. Best Practices for Using AI When Writing Scientific Manuscripts. ACS Nano 2023, 17, 5, 4091–4093.