I. Introduction

Large language models (LLMs) are multi-billion parameter transformer neural networks 1 that are trained on enormous collections of documents (a 'corpus') without supervision or labels. 2 LLMs can perform multiple tasks like classifying natural language, translating text, and document search. Perhaps the most remarkable task of LLMs is to complete an input string of text; via this mechanism (called causal language modeling), LLMs can write unit tests, document function, write code from a doc string, answer questions, and complete stoichiometric equations. 3,4 We previously discussed the outlook of LLMs in chemistry. 5 In the few months since then, LLMs have been both developed for specic chemistry problems 6,7 and general LLMs have been applied in chemistry. 8,9 On Nov 30, 2022, OpenAI released an interactive interface to an LLM termed ChatGPT (ref. 10) which substantially increased interest in this area as well as use by scientists for coding and writing tasks. An open question for LLMs such as GPT-3, 3 T5, 11 or GPT-neo (ref. 12) that are trained on very large and varied textual data is if they can be applied in domains like chemistry, which have specialized language and knowledge. In our initial work, 5 we found that relationships between SMILES and natural language is possible with GPT-3. SMILES is the standard method of representing molecules as strings. 13 It is even possible to loosely edit structures via natural language (see Fig. 6). 14,15 However, the extent to which LLMs can be directly applied in chemistry in the broad context of research and teaching is unexplored. The large amount of specic domain knowledge required to solve chemistry problems may limit applicability of general LLMs. For example, recent work has found that knowledge of the periodic table of elements requires very high parameter counts. 4 Recent comparisons of LLMs that generate code can be found in ref. 16. Here, we focus our study on whether LLMs that generate code 17 can be applied to chemistry tasks of a computational nature (both computational chemistry problems, and general tasks which can be expressed as simple computer programs, such as ranking elements by ionic radius). Most LLMs that generate computer code are causal decoder-only models [17][18][19] -a user provides a sequence of text (called the prompt) and it proposes a continuation of the text (the completion). 20 There are LLMs trained on code that can inll or match encoder/decoder natural language to code like Code-BERT, 21 but they are typically used for embedding code for tasks like classication, document retrieval, or translating code to natural language. Because it is not reasonable to use encoderdecoder or encoder-only models to generate code or answer questions with open-ended length, this paper explores solely decoder-only causal language models.

Evaluating LLMs' knowledge of chemistry should be distinguished from capability to reason or understand. LLMs can make compelling completions, but are incapable of reasoning and demonstrate supercial understanding. 22,23 Our goal is to evaluate LLMs' ability to correlate natural language, equations, code, and heuristics of chemistry.

II. Methods

We have compiled a categorized set of chemistry and related example prompts for benchmarking code-generating LLMs in a public repository. 24 To generate these problems, we rst decided upon a list of categories of chemistry and chemical engineering knowledge, listed in Table 1, and set a goal of having at least 10 examples in each category for our initial database of problems. Members of our research groups (the authors of this paper), who we consider to have sufficient expertise in these areas due to formal schooling, research, and teaching experience, contributed the prompts and reference solutions for these categories.

The examples in this table span a range of topics that we consider common questions across chemistry elds. There is some representation of computational chemistry research topics (categories corresponding to performing chemical simulations (sim), analyzing molecular dynamics simulations (md), chemical informatics (cheminf), and some quantum mechanics (qm)), but this constitutes less than half of the initial prompts created by us. The rest correspond to typical questions that one might encounter in general chemistry (genchem), biochemistry (bio), physical chemistry (thermodynamics, quantum mechanics, and spectroscopy), and in laboratory classes (plotting and statistics).

Within this set of topics, some examples were labeled as only expert evaluable, where automated evaluation is infeasible or insufficient (e.g. plotting). The total number of examples is 84, of which 25 were expert evaluable, and the accuracy is 75% for the best performing model.

There is a strong correlation between the model parameter count and accuracy, 25 so we focus only on the largest models with more than 1B parameters. The architectures of models are all decoder-only like GPT-3 (ref. 3) with the ability to insert completions, 26 (except when noted). The rst model is a GPT-3 12B ne-tuned on code (Codex) abbreviated as "cushman". It is known as code-cushman-001 in the OpenAI API. 27 This is modied from the original one in Austin et al. 17 somewhat and is described as "a stronger, multilingual version of the Codex 12B model.". 28 We also used code-davinci-002, abbreviated as "davinci". This model is part of the category of "GPT-3.5" models that are derived from GPT-3. 29 The number of parameters in davinci-class models is not public information, but may match the 175B parameters of the model described in the GPT-3.5 paper. 30 We also considered the recent text-davinci-003 model which is derived from code-davinci-002 with a reinforcement-learning adaption from human user feedback 30 although this model became available only aer human evaluation (below) was complete, so our analysis is reported only on automated evaluations. This model is denoted as 'davinci3' here. Finally, from publicly available information we know that ChatGPT is based on a slightly modied version of GPT-3.5, and so we expect its performance to be comparable to that of the model; however, it does not have an API that would allow us to systematically probe any differences in our study. One example use of ChatGPT is given in the ESI. †

We also study two "incoder" models from Fried et al. 18 trained on code only. We chose incoder because it is able to inll code in addition to completing code prompts, which gives a more direct comparison, and it has generally good performance. Lastly, we consider the 'codegen' model, 31 which is another decoder-only model trained on a similar dataset to 'incoder'. It was not trained for inlling, because it was Table 1 The number of prompts by topic and best accuracy achievable in this work. "Expert" is the number within a topic that must be evaluated by an expert. We used the "copyright" context for incoder-6B, "authority" for codegen-16B, and "insert" for davinci and T = 0.2 (best for all models). Accuracies are averaged (macro-averaging) across top-k sampling (we consider correct if valid prompt appeared in top-k results). Expert accuracies are macro-averaged across topics/ prompts designed for back-and-forth code synthesis with natural language. Although it is not exactly analogous to the other models, it is one of very few competitive models that can generate working code, and so we include it here for comparison. 31 Recent benchmarks show that davinci is the best or nearly the best for general programming tasks. 16,32 Incoder was used as implemented in HuggingFace transformers. 33 To avoid library changes since 2021 inuencing the accuracy, our evaluations are performed using the python version and packages from June 2021. The chosen date was based on the reported training range from ref. 32 and comes before the training time in ref. 18.

Topic

When developing example prompts and solutions, the prompts were tested and modied using davinci. Some prompt engineering was inevitable through this process. 3,34,35 However, prompts were not designed to get a correct answer and some prompts (e.g., two atom harmonic oscillator) were never correctly completed. We do emphasize that the reported accuracy is not what one would expect of the rst prompt constructed on-the-y for a given problem. Rather, they are constructed to answer "how much chemistry do these LLMs know?" These gures should not be construed as upper bounds either, as recent work on prompt engineering shows that multiple steps (sometimes known as using "scratchpads") 19 or eliciting multiple steps can further improve accuracy. 29 Following Chen et al., 32 a prompt completion is accurate if the code functions correctly, not if it matches a reference implementation. Most examples have both a prompt and unit tests. The accuracy of expert evaluable prompts for which there are no unit tests is not reported, unless specied. Five completions were generated via top-k sampling 36 and multiple temperatures at T = 0.05, 0.2, 0.5 (somax scaling). We explored nucleus sampling, 37 but found it to be no different compared to adjusting the temperature for balancing the diversity and correctness of completions. We chose k = 5 for all models, except for incoder-6B where GPU memory limitations prevented sampling more than k = 1. Thus, these results may be slightly inated since accuracy is reported on only a most likely output. Error bars in all plots are 95% condence intervals generated from bootstrap resampling across top-k.

Expert evaluation was performed on k = 3 outputs of davinci (T = 0.2, "insert" context) and accessed through a web interface. 38 Each example contains a link to a custom Google form which could be used to evaluate that example, with results saved in a spreadsheet. The multiple choice questions in the form were: "Is this question: Easy; Medium; Hard", "Is the solution: Perfect; Correct but not perfect; Runs and is almost correct; Does not run but is almost correct; Is far from correct". There was also a box for extra comments. This evaluation did breakout more detailed information like alignment between the prompt and completion or hazards of completion, similar to that recently proposed by Khlaaf. 39 The full set of evaluations, with personally identiable information (student emails) removed, is available as a comma separated value (CSV) le in the ESI. † To make a numerical evaluation of this data as shown in Fig. 3, we assigned scores from 1-5 with 5 being the best ("Perfect") and 1 being the worst ("Is far from correct"). To compute an overall accuracy as reported in Table 1, we assigned "Perfect", and "Correct but not perfect" a value of 1.0, and all others 0.0, and then computed the mean score for each prompt separately. It should be noted that each assessor had a different level of expertise on each topic, as well as a different level of python programming experience, although we feel all were sufficiently expert to evaluate each prompt with sufficient authority.

III. Results

A. Example problems

To illustrate the kinds of tasks and impressive (if not always correct) results produced by LLMs, we show the output for one 'sim' category task in Fig. 1. To standardize our tasks, each task is phrased as a function to be lled in, as in the top box. This prompt includes a rst line which loads the numerical python (numpy 40 ) library, which gives additional 'context' (see below). The rest of the information for the LLM is contained in two places, the names of the variables given as inputs 'n_steps', 'T', 'k', and a comment string which says what the function does/ should do. In this case, the function should perform Metropolis Monte Carlo for a harmonic potential. Implicit in the instruction by the creator is that k represents the spring constant, and so this code should produce samples from the energy function UðxÞ ¼ 1 2 kðx À x 0 Þ 2 , with x 0 = 0 since it was not specied as an input, and also that reduced units are used, such that Boltzmann's constant k B = 1.0. We can see that-with quite minimal instruction-the code in the output is correct except for an error in the line indicated with a '*'; in this line, the position of the particle is completely resampled from scratch on the range [-1,1). This code would actually be ne if the system were constrained to be within a box of length 2, and in the limit of k [ 1 it will also appear to give correct results. The inset shows the equivalent line in two other outputs of the model, both of which are acceptable; one displaces the position by a Gaussian random number with m = 0 and s 2 = 1, and the second chooses a new position from a Gaussian with the mean centered at the current position and s 2 = 1. Note that neither of these is optimized for the choice of (k,T), as s 2 = 1 may be too large or too small to be efficient, depending on the spring constant and temperature. Finally, in one of the other two outputs for this example (available in the ESI † or on the result website), k is interpreted as Boltzmann's constant, and the harmonic system is given a spring constant of 1.0 implicitly; this is a reasonable inference of the model. It illustrates how the author must be careful about what is implicit in their prompt and what is stated explicitly (e.g. here, that T is the temperature).

Fig. 2 shows an additional example to highlight how the davinci-codex model internally contains knowledge of chemistry topics (in this case, general chemistry pertaining to phase equilibrium). The output shows that the model "knows" the relevant rearrangement of the Claussius-Clapeyron equation, and returns the appropriate result, assuming that the heat of vaporization ('Hvap') was given in joules mol -1 . One gure in the ESI † shows that we can use ChatGPT to solve the same problem, either by asking it to ll in the "[insert]" text with the correct solution, or by describing the problem conversationally. When an API for ChatGPT is available, we would expect the performance in the former mode to be very similar to that of the underlying GPT-3.5 model.

B. Expert evaluations

Davinci, the best performing model, does have broad knowledge of equations and common calculations across multiple domains of chemistry. Table 1 gives the overall accuracy across the topics, models, and expert evaluable topics. Both models can correctly answer prompts across a range of topics, with davinci performing the best. About 30 percentage points of accuracy are from prompt engineering, which is discussed further below.

On average, the accuracy for human evaluable topics is lower, reecting their increased difficulty. These prompts include tasks like writing an input le for NWChem, 41 implementing a Monte Carlo simulation of a harmonic oscillator (Fig. 1), and generating a complex multi-panel plot. Fig. 3 shows a breakdown of difficulty from the individual evaluations. There is a balance of easy and hard prompts in the dataset, as judged by experts. Our primary result here is that the accuracy of the model is negatively correlated with perceived prompt difficulty, as might be expected but did not necessarily have to be the case. We did not perform any randomization or controls; each evaluator was able to see all prompts and all outputs, and so we acknowledge that scores could be biased by factors such as the order of the prompts on the website, and the order that results for a given prompt were presented on the website. In the rest of this article, we focus only on prompts whose correctness can be evaluated by comparison with an expected solution in an automated fashion.

C. How to improve performance

There is a large accuracy gain when using basic prompt engineering strategies. Fig. 4 shows the effect of different "contexts" on accuracy across models. A context here is code prepended before all prompts, or all prompts within a topic. The contexts are given both in the ESI † and our accompanying code. "Custom" includes two pieces: some imports related to the topic (e.g., rdkit 42 for cheminf) and a single example to teach the model how to indicate the end of a prompt completion. The imports are not just to prevent errors due to failure to include relevant libraries-they inuence the completions and give context. For example, a "structure" aer importing rdkit means a bonded arrangement of atoms; in contrast, a structure aer importing openmm 43 (a molecular dynamics simulation code) would implicitly mean a 3D arrangement of atoms, e.g. obtained from a PDB le.

The completion example is a one line statement (e.g., printing the version number of an imported package) with a comment above and #end below. This causes the LLM to end completions with #end. We tried to ad hoc look for certain keywords such as new function defs, returns, or comments as completion ends, but these heuristics were oen violated. The completion example is signicant for the Cushman model, which can only perform completions but not insertions. For the davinci and incoder models, we can replace this with the "insert" contexts which have the same imports but use a model capability to inll at a special insert token (as in Fig. 1). Avoiding our completion example in the context seems to be insignicant for davincni, but important for incoder.

LLMs seem to be very susceptible to conditioning contexts, like adding the word "very" many times to improve a completion 44 or stating that the code "has no bugs". We explored this in our benchmarks in two ways. We tried inserting copyright notices and found, as shown in Fig. 4 and 5 that it does signicantly improve accuracy at higher temperatures. This makes intuitive sense; lowering the temperature makes the LLM choose more likely completions and a copyright notice would more oen be included with standard/quality code, thus giving a similar effect to lowering the temperature. The best performing model/temperature combination was not improved because it already had a low temperature. We also tried inserting the statement "This is written by an expert Python programmer" as suggested by Austin, 45 and saw slightly less improvement. A similar recent work has found context or specic phrases (e.g., "let's think step by step") that elicit chainof-thought outputs which can give large accuracy improvements. 29,46 Fried et al. 18 and Wei et al. 35 have recently explored using metadata, including popularity of code, as a mechanism to condition completions, so that we do not need to use ad hoc prompt engineering. Interestingly, the results from davinci3 show that the improvements to the NLP model through human feedback removed some of the observed sensitivity 30 to prompt engineering on our examples.

Aside from contexts, there are a few strategies to ensure that a prompt aligns the intent of a user with the completion. If the prompt contains programming mistakes or spelling mistakes, then the completion will be of similar quality. So a correctly spelled and intelligible prompt is necessary. The LLM tries to agree with each word in the prompt. If a prompt is a function declaration and uses the phrase "compute the moment", the model will probably not return the value. Thus, the word "return" should be used. If a package is imported in the prompt, the model will try to make use of it. This can lead to problems if many packages are imported -it can be unexpected as to which packages the model will use, or if the model thinks it must use all of them.

A major source of the errors in some of the categories such as 'md' is the improper use of functions from a package such as mdtraj, in particular, improper knowledge of how many and what type of values are returned by that function; this could be a simple error or due to training on an earlier version of the module; these results may be able to be improved in the future by 'ne tuning' the LLM on examples from a particular package that is frequently used in one's work, or by adding additional context.

D. Molecular structures

Our goal is to evaluate how much chemistry LLMs know. Besides evaluating tasks that can be expressed as programs, we also explored whether LLMs can connect natural language directly with molecular structures. We tested both InstructGPT 30 and davinci in these examples, but found InstructGPT to work better. Neither could convert from molecular SMILES to the name of the molecule, as demonstrated with 0% accuracy on 100 random molecules from pubchem 47 when we tried a SMILES length of less than 60 characters (relatively small/simple molecules). The attempt from InstructGPT is shown in the ESI. † InstructGPT was able to convert a sentence describing a molecule into SMILES, as shown with examples in Fig. 6. InstructGPT is able to connect functional groups from SMILES to natural language. The molecules are not exact matches, but there is some correlation (e.g., oxygen near a ring for phenol and amine). It is also able to correlate molecular properties like lipophilicity with SMILES. InstructGPT rarely generates invalid SMILES; only the rst molecule in Fig. 6 had a single invalid character (see the ESI † for SMILES). It appears that InstructGPT or other LLMs could be trained/ne-tuned on the connection between natural language and chemical structures. Recently, specic models that can translate between molecular structure and natural language have also been trained from scratch. 48

E. Discussion

Davinci seems to not reason well about computational chemistry. If we prompt davinci to use a "highly accurate singlepoint" quantum calculation in pyscf, 49 it will frequently use relativistic Hartree-Fock regardless of the property being computed because it has memorized that "relativistic" is associated with accurate. Another example is in the "force constant" prompt which is meant to compute the force constant for a twoatom harmonic oscillator with different masses given a wavelength. Perhaps because this is an unusual variant of a common question (converting between the force constant and wavelength), davinci always fails on this question and is unable to rearrange the equation to take a harmonic mean of masses.

Davinci may also hallucinate functions that do not exist. If a difficult prompt is given, for example "return the residual dipole couplings given a SMILES string," the model will simply try to use a non-existent method MolToRDC. As reported previously, 22 LLMs are not able to perform chemical reasoning when completing prompts.

We would like to anecdotally note that the LLMs could perform many of the benchmark problems if the natural language was in Chinese, German, or Spanish. We did not explore this in depth, but a few example prompts written in Mandarin can be found in the ESI. † The use of LLMs with prompts that are not in English may be a valuable tool for lowering the barrier for employing computational tools for those who are not native English speakers, and who therefore may have a harder time interpreting documentation and programming forums.

IV. Conclusions

LLMs are now easily available via tools like tabnine, 50 copilot, 51 or ChatGPT. 52 We have found high accuracy on chemistry questions, and it is inevitable that students and researchers will begin using these tools. From our results, high accuracy should be expected with reasonable prompts. We emphasize that our results only give lower bounds on the chemistry knowledge in these models, since they cover only the specic topics so far included in our database, and further prompt engineering or other strategies for evaluating this knowledge besides python function writing could elicit even better results. Tricks like inserting copyright notices at the top of a source le seems to be another way to improve accuracy, although netuning with human feedback mitigates this effect, 30 as seen in davinci3. We found that humans are able to gauge accuracy for easy to medium prompts, but care should be taken if using completions of difficult prompts. The seeming ability to always generate syntactically valid code means LLMs oen produce something, but it is up to the user to assess it. We also found somewhat unexpected capabilities like generating molecules from natural language and accurate completions with non-English prompts. For a broader discussion of what impact this will have on education, we refer interested readers to our earlier perspective article. 5