The world faces an enormous number of problems in the coming decades on scales of complexity never-before-seen, in areas such as climate change, healthcare, and pandemics. To address these issues, we need to discover inventive scientific solutions which are scalable, flexible, and inexpensive. Broadly speaking, many of these problems will require molecular solutions from the chemistry domain, such as developing new drugs (e.g. kinase inhibitors (Ferguson and Gray, 2018)), materials (e.g. organic photovoltaics (Kippelen and Brédas, 2009)), and chemical processes (Zhong et al., 2023). These solutions exist in extremely large search spaces, which makes AI tools a necessity.
Language-molecule models have emerged as an exciting direction for molecular discovery and understanding (Edwards et al., 2021;Zeng et al., 2022;Edwards et al., 2022;Su et al., 2022;Liu et al., 2022;Xu et al., 2023;Christofidellis et al., 2023;Liu et al., 2023b;Luo et al., 2023;Zhao et al., 2023c;Seidl et al., 2023). However, training these models is challenging due to the scarcity of molecule-language pair datasets. At this point, datasets have been released which are 1) small and scraped from existing databases (Edwards et al., 2021;Zeng et al., 2023;Liu et al., 2023a,c;Pei et al., 2023), 2) large but noisy and constructed by performing entity linking on the scientific literature (Zeng et al., 2022;Su et al., 2022), and 3) templatebased built on prediction datasets (Zhao et al., 2023a;Fang et al., 2023). Approaches utilizing pseudo-data have also been attempted (Chen et al., 2023a). These approaches have helped remedy the problem of data scarcity in this domain; however, these approaches frequently ignore key benefits of natural language: 1) compositionality, 2) abstraction, and 3) functionality (Zhang et al., 2023). To this end, for the Language + Molecules Workshop at ACL 2024, we release L+M-24, which we construct to test these three goals, particularly compositionality, using recently released data sources (Zhao et al., 2023b;Kosonocky et al., 2023;Wishart et al., 2023). L+M-24 is divided into four categories with important applications in the small-molecule domain: 1) Biomedical, 2) Light and Electricity, 3) Human Interaction and Organoleptics, and 4) Agriculture and Industry. Improving understanding of these applications can have important implications in problems such as drug discovery, climate issues, more efficient and green industrial processes, and improved food production.
The dataset is primarily intended for language↔molecule translation, which consists of two tasks: generating 1) a caption given a molecule and 2) a molecule given a description.
Overall, we focused on four primary categories of importance: 1) Biomedical, 2) Light and Electricity, 3) Human Interaction and Organoleptics, and 4) Agriculture and Industry. These categories and three properties from each are displayed in Table 1. The biomedical category is focused on drug properties, functions, and interaction with proteins.
Light and electricity is focused on the ability for a molecule to produce or absorb light or electricity.
Human interaction and organoleptics focuses on the effect and experience molecules cause in humans.
Agriculture and industry focuses on molecules used in industrial processes and food production.
Based on our data sources (below), the properties we have selected already encode a large degree of functionality, enhanced by our manual curation. Further, since these properties are generally short phrases indicating functionality, they are also abstract and apply to many molecules (e.g., "insecticide"). For compositionality, we explicitly select certain pairs of properties which we hold out of the dataset. For example, a molecule may share two properties which are desirable together (e.g., low toxicity and fungicidal). L+M-24 will help to evaluate whether model's can generalize to unseen compositions of properties.
We constructed our dataset using three different databases. We will first describe the process we used to extract information from each, followed by our overall strategy for adding hierarchy into the dataset. We want to deeply thank the authors of these resources for making them publicly available for the community.
We used properties extracted from PubChem (Kim et al., 2016(Kim et al., , 2019) ) as described in (Zhao et al., 2023c). Properties from this approach include odor, taste, and decomposition. We note these properties consist of molecule-specific descriptions, which the other data sources do not provide.
Here, we used functional properties extracted from patent literature by Kosonocky et al. (2023). This allowed us to capture molecules from the patent literature in addition to the scientific literature. Here,
Biomedical anti neoplastic glaucoma treatment capillarigenic Light and Electricity photoelectric conversion photopolymerization dielectric Human Interaction pungent bitter nephrotoxic Agriculture and Industry herbicide emulsifier carcinogen Table 1: Example properties in the dataset. Antineoplastic drugs are used to treat cancer. Glaucoma is a group of eye diseases. Capillarigenic means producing or causing capillaries. Pungent means having a strong taste or smell. Nephrotoxic is toxicity in the kidneys. Photoelectric conversion is the conversion of light into electricity. Photopolymerization is the process through which monomers are linked together through a photochemical reaction. A dielectric is a poor conductor of electricity but can be polarized. A herbidicde is toxic to plants. An emulsifier stabilizes an emulsion. A carcinogen is an agent capable of causing cancer. The full property list and number of occurrences is available in the online data repository.
we started with CheF prefinal_v3 2 . We created a set of properties from both CheF's property summarizations and from the ChatGPT summarization source. For the summarization source, we also applied the WordNet lemmatizer (Bird et al., 2009) for deduplication. After obtaining a list of properties, we removed properties pertaining to less than 100 molecules. We then kept properties falling into the categories of "X-icide", "anti-X", "X treatment", "X modulators", "X inhibitors", "X agonists", "X antagonists", "light", and "electricity." We manually removed uninformative labels which were too broad or didn't describe enough function. Further, we manually corrected errors in label naming and duplication.
In addition to CheF, we also take advantage of another new chemical function data resource: Chem-FOnt (Wishart et al., 2023). From this datasource, we collect three categories: health effect relations, organoleptic effect relations, and role relations.
"alzheimer's treatment" "bace1 inhibitor"
"anti viral agent" "mitogen" "anti carcinogenic" "lipoxygenase inhibitor" "fungicide" "anti oxidant "
The molecule is both a alzheimer's treatment and a bace1 inhibitor.
The molecule is a mitogen and lipoxygenase inhibitor, belonging to the anti oxidant class, and is characterized as anti viral agent, anti carcinogenic, and fungicide.
Figure 1: Example descriptions created for molecules from the training set.
We utilize GPT-4 (OpenAI, 2023) to generate specific templates for each combinations of molecular properties. Specifically, we manually write six templates: "The molecule is a <0>.
For all properties in L+M-24, we first assigned them to possible templates based on their category or by individual consideration. Certain properties (e.g., polymerization, decomposition) were expressed in sentence format, so we did not use templates. Given a molecule with n properties, we first looked for a template that had the correct slots (e.g., <0>, <2>, and <2>) for its properties.
When we found possible templates, we picked one at random and used it to generate a sentence for the molecule's properties. If there were no matching templates, we split the properties into two separate equal-sized groups and tried with each group. We return the concatenation of the two sentence templates as the molecule description. Note this process can repeat multiple times.
We note that we are also releasing a version of the dataset with 5 captions for each molecule. In this case, we split group sizes at random. Further, we split sentences apart 50% of the time (even when there were matching templates) to increase caption diversity.
Duplicate molecules are merged using RDKit (Landrum, 2021) and molecules which cannot be processed are removed. We split the data by first examining property combinations. 20% of combinations are witheld into the evaluation set. From molecules in the remaining 80%, we keep 80% for training and put 20% in evaluation. The evaluation set is split into two tasks: molecule captioning and molecule generation. For each task, only one modality will be released prior to the shared task results.
The training set consists of 160,492 moleculedescription pairs. For the evaluation set, both molecule generation and captioning contain 21,839 pairs. Further, special splits are released for the training set which allow for validation using the training data. They are constructed using the same procedure as the official evaluation dataset.
Overall, we adopt the evaluation metrics proposed by Edwards et al. (2022). However, we include invalid molecules in the calculations of FTS metrics (setting the score to zero for invalid molecules). We also add a uniqueness metric to the generated molecules for held-out combinations of properties (Polykovskiy et al., 2020). Further, we also look at property-specific precision, recall, and F-1 scores. These scores are calculated by matching tokenized names in the generated captions. These scores are Table 3: Property-specific molecule captioning results on the validation split of L+M-24. Table 6: Molecule generation results on the subset of held-out combinations from the validation split of L+M-24 (2107 data points).
further aggregated across specific properties (e.g., inhibitors, X-icides, etc.) and the four broad categories. Aggregations are performed by averaging scores (i.e., macro-F1). We further compute these scores specifically for held-out combinations of properties.
MolT5 models (Edwards et al., 2022) were finetuned for 20 epochs on the "split_train" data split and evaluated on the "split_valid", both of which are available online. Huggingface's transformers (Wolf et al., 2019) was used for finetuning with a learning rate of 2e-5 and weight decay of 0.01. A batch size of 128 was used for small and base models, and a batch size of 48 for large models. Further, Meditron-7B (Chen et al., 2023b) was finetuned for 5 epochs with a context length of 930, 2e-6 learning rate, and batch size of 8/16 (molecule/caption generation). Models are released online. Results for captioning are reported in Tables 2, 3 and 4. Tables 5, and 6 shows results for molecule generation.
Overall, the dataset proves to be fairly challenging for these naively finetuned models. On captioning, Meditron-7B achieves a maximum overall F-1 score of 16.81 for property identification (Table 3). However, overall it has a much higher precision than recall, indicating the model only labels
The molecule is a luminescent member of the organic light-emitting class.
The molecule is both a platelet aggregation inhibitor and a cell adhesion inhibitor.
The molecule is a muscarinic agonist that impacts pain treatment and is both alzheimer's treatment and anxiety treatment.
The molecule is a jak2 inhibitor and is cancer treatment.
The molecule is both a anti psychotic and a nmda antagonist.
The molecule is a factor ixa inhibitor, a factor xa inhibitor, and anti thrombotic.
The molecule is a flavoring agent and a nutrient, as well as nutty and green. a molecule with a certain property when having higher confidence. Certain classes of molecules, such as X-icides, are never identified (Table 4). Other classes, such as toxins or electricity, show emergent behavior as model size scales. Interestingly, the models appear to be fairly capable at linking molecules to certain diseases or cancers. We find that, likely due to poor performance on individual properties, only the largest model succeeds on predicting held-out combos, and with poor results. Additionally, we find that the Text2Mol metric, as trained on ChEBI-20, shows poor domain transfer to L+M-24.
The models are able to capture a number of useful properties, such as electroluminescence, diabetes treatment, non-alcoholic fatty liver disease, and emulsifiers. In some cases, the model captures important characteristics about the molecule but uses differing language. This poses a challenge for our evaluation metrics. For example, a molecule identified in the ground truth as an anti tumor agent is identified as being a cancer treatment by the model. In particular, the models appear to struggle with rarer properties, which are common in our dataset formulation and in the chemical domain as a whole. They also struggle with identifying molecule-protein interactions (e.g., "monoamine reuptake inhibitor"), although Meditron shows a large performance jump.
For the molecule generation task, we also find the dataset to be challenging. We show results generated by different models on never-before-seen property combinations in Figures 2 and 3. We believe the difficulty is for two reasons. First, common property combinations may have structurally very different molecules which exhibit those properties, making evaluation difficult. Second, the model may not grasp rare properties well. Overall, this results in the naively finetuned models producing similar outputs to many different prompts. Further, as expected, performance falls on unseen property combinations and larger models prove
The molecule is a cytokine inhibitor, protein kinase inhibitor, pdk1 inhibitor, mk2 inhibitor.
The molecule is a aldose reductase inhibitor, a cox2 inhibitor, and anti inflammatory.
The molecule has Odor-like bitter almonds. When heated to decomposition it emits toxic fumes of nitrogen oxides and cyanides.
The molecule is a 5lipoxygenase inhibitor that impacts inflammatory disease treatment.
The molecule is alzheimer's treatment and it impacts epilepsy treatment.
The molecule is a renin inhibitor, hiv protease inhibitor, hypertension treatment, betasecretase inhibitor.
It has effects on both low voltage and luminous efficiency. more effective (Table 6).
Overall, this dataset proves to be quite challenging. We find that some specific properties in particular are challenging for the model. This may be because the model understands these properties, but is unwilling to use them in its descriptions due to the training procedure. This limitation may be addressed with more sophisticated decoding algorithms or by better finetuning methods. Future work will also likely benefit from incorporating other modalities, such as proteins, to provide better understanding to the model for some property types. Notably, certain properties display what may be emergent behavior; scaling training data or model size may yield non-linear improvements.
In this dataset, we focus on composition, abstraction, and function. Future work may also wish to integrate other recent trends: instruction-following and dialogue (Fang et al., 2023;Cao et al., 2023;Zeng et al., 2023;Zhao et al., 2024;Zhang et al., 2024;Yu et al., 2024), tool use (Boiko et al., 2023;Bran et al., 2023), additional molecule representations (e.g., 3D (Tang et al., 2023)), additional modalities (Xu et al., 2023), or molecule editing (Su et al., 2022). Further, we note the need for improved evaluation metrics, especially in the case of molecule generation for function where there may be many possible outputs. Specific methods for improving compositionality may be another fruitful avenue for research (Yellinek et al., 2023). It may also be interesting to use molecule-language instruction-following models within larger search frameworks, such as ChemReasoner (Sprueill et al., 2023(Sprueill et al., , 2024)).
In this manuscript, we describe the process for creating the L+M-24 dataset. L+M-24 is designed to focus on three key benefits of natural language in molecule design: compositionality, functionality, and abstraction. It is the featured shared task at the First Language + Molecules Workshop at ACL 2024.
• Prompts: You are an expert in the chemical domain whose task is to create templates to describe the properties of molecules. You will be challenged with a list of different cases. Each case willl have a list of **templates**, and a **question**. Each template will describe certain properties. Your goal is to generate a new template in a sentence based on all the previous templates.
•
Here, we give a brief description of properties in the dataset.
The dataset, finetuned baseline, and evaluation code are released publicly at github.com/language-plus-molecules/LPM-
24-Dataset through HuggingFace.
To convert these properties to natural language, we follow a template-based procedure using GPT-4 (OpenAI, 2023) generated compositional templates.2 obtained via personal communication.