An official website of the United States government
The 2023 update to the Artificial Intelligence Patent Dataset (AIPD) extends the original AIPD to all United States Patent and Trademark Office (USPTO) patent documents (i.e., patents and pre-grant publications, or PGPubs) published through 2023, while incorporating an improved patent landscaping methodology to identify AI within patents and PGPubs. This new approach substitutes BERT for Patents for the Word2Vec embeddings used previously, and uses active learning to incorporate additional training data closer to the “decision boundary” between AI and not-AI to help improve predictions. We show that this new approach achieves substantially better performance than the original methodology on a set of patent documents where the two methods disagreed—on this set, the AIPD 2023 achieved precision of 68.18 percent and recall of 78.95 percent, while the original AIPD achieved 50 percent and 21.05 percent, respectively. To help researchers, practitioners, and policy-makers better understand the determinants and impacts of AI invention, we have made the AIPD 2023 publicly available on the USPTO’s economic research web page.
more »
« less
Predicting Institution Outcomes for Inter Partes Review (IPR) Proceedings at the United States Patent Trial & Appeal Board by Deep Learning of Patent Owner Preliminary Response Briefs
A key challenge for artificial intelligence in the legal field is to determine from the text of a party’s litigation brief whether, and why, it will succeed or fail. This paper shows a proof-of-concept test case from the United States: predicting outcomes of post-grant inter partes review (IPR) proceedings for invalidating patents. The objectives are to compare decision-tree and deep learning methods, validate interpretability methods, and demonstrate outcome prediction based on party briefs. Specifically, this study compares and validates two distinct approaches: (1) representing documents with term frequency inverse document frequency (TF-IDF), training XGBoost gradient-boosted decision-tree models, and using SHAP for interpretation. (2) Deep learning of document text in context, using convolutional neural networks (CNN) with attention, and comparing LIME and attention visualization for interpretability. The methods are validated on the task of automatically determining case outcomes from unstructured written decision opinions, and then used to predict trial institution or denial based on the patent owner’s preliminary response brief. The results show how interpretable deep learning architecture classifies successful/unsuccessful response briefs on temporally separated training and test sets. More accurate prediction remains challenging, likely due to the fact-specific, technical nature of patent cases and changes in applicable law and jurisprudence over time.
more »
« less
- PAR ID:
- 10355435
- Date Published:
- Journal Name:
- Applied Sciences
- Volume:
- 12
- Issue:
- 7
- ISSN:
- 2076-3417
- Page Range / eLocation ID:
- 3656
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Patents are key strategic resources which enable firms to appropriate innovation returns and prevent rival imitation. Patent examiners – individuals who may be subject to various sources of bias – play a central role in determining which inventions are awarded patent rights. Using a novel dataset, we explore if one increasingly prevalent source of bias – political ideology – manifests in examiner decision-making. Reassuringly, our analysis suggests that the political ideology of patent examiners is largely unrelated to patent office outcomes. However, we do find evidence suggesting politically active conservative-leaning examiners are more likely to grant patents relative to politically active liberal-leaning examiners, but only for patent applications where there is ambiguity regarding what constitutes patentable subject matter and hence examiners have greater discretion.more » « less
-
Patent landscaping is the process of identifying all patents related to a particular technological area, and is important for assessing various aspects of the intellectual property context. Traditionally, constructing patent landscapes is intensely laborious and expensive, and the rapid expansion of patenting activity in recent decades has driven an increasing need for efficient and effective automated patent landscaping approaches. In particular, it is critical that we be able to construct patent landscapes using a minimal number of labeled examples, as labeling patents for a narrow technology area requires highly specialized (and hence expensive) technical knowledge. We present an automated neural patent landscaping system that demonstrates significantly improved performance on difficult examples (0.69 on ‘hard’ examples, versus 0.6 for previously reported systems), and also significant improvements with much less training data (overall 0.75 on as few as 24 examples). Furthermore, in evaluating such automated landscaping systems, acquiring good data is challenge; we demonstrate a higher-quality training data generation procedure by merging (Abood and Feltenberger Artif Intell Law 26:103–125 2018) “seed/anti-seed” approach with active learning to collect difficult labeled examples near the decision boundary. Using this procedure we created a new dataset of labeled AI patents for training and testing. As in prior work we compare our approach with a number of baseline systems, and we release our code and data for others to build upon “(Code and data may be downloaded from https://doi.org/10.34703/gzx1-9v95/QDLKVWCode and data are released under the Creative Commons NC-BY 4.0 license at https://creativecommons.org/licenses/by-nc/4.0/)”.more » « less
-
Conversational recommender systems (CRS) dynamically obtain the users' preferences via multi-turn questions and answers. The existing CRS solutions are widely dominated by deep reinforcement learning algorithms. However, deep reinforcement learning methods are often criticized for lacking interpretability and requiring a large amount of training data to perform.In this paper, we explore a simpler alternative and propose a decision tree based solution to CRS. The underlying challenge in CRS is that the same item can be described differently by different users. We show that decision trees are sufficient to characterize the interactions between users and items, and solve the key challenges in multi-turn CRS: namely which questions to ask, how to rank the candidate items, when to recommend, and how to handle user's negative feedback on the recommendations. Firstly, the training of decision trees enables us to find questions which effectively narrow down the search space. Secondly, by learning embeddings for each item and tree nodes, the candidate items can be ranked based on their similarity to the conversation context encoded by the tree nodes. Thirdly, the diversity of items associated with each tree node allows us to develop an early stopping strategy to decide when to make recommendations. Fourthly, when the user rejects a recommendation, we adaptively choose the next decision tree to improve subsequent questions and recommendations. Extensive experiments on three publicly available benchmark CRS datasets show that our approach provides significant improvement to the state of the art CRS methods.more » « less
-
Though the field of engineering has experienced significant changes over the last several decades, many graduate programs have not made any substantive changes in their curriculum. This is particularly important given that data show that over sixty percent of new doctorate program graduates do not go into academic research [1]. Recognizing the critical need for change, the National Academies of Sciences, Engineering, and Medicine [2] made recommendations for graduate STEM education programs. The intent was to examine how graduate STEM education can focus on evidence-based practices which better respond to the needs of students and broader society. The Committee on Revitalizing Graduate STEM Education identified key competencies for educational systems so that they are dynamic in addressing current needs of students while anticipating future contexts in STEM graduate education. These competencies were the framework for this research which employed curriculum analysis methods to the PAtENT (Pathways to Entrepreneurship), an alternate pathway to the doctorate in engineering at this University. The curriculum analysis included the two components of the Academies’ recommendations: 1) Develop scientific and technological literacy and conduct original research and 2) Develop leadership, communication, and professional competencies. The research used a dimensional core curriculum analysis [3 - 4] to analyze program information including documents, artifacts, and other data related to coursework, original research, student classroom experiences as well as laboratories and fieldwork. The descriptive content analysis used a systematic process to allow for identifying attributes within documents and data in order to align identified components to program activities and structures. Coding for the curriculum analysis used an inductive, thematic and descriptive approach in aligning program components and activities to ten elements listed for the two components in the Academies’ recommendations. Document analysis identified curriculum expectations and program outcomes that were tagged to the elements in the recommendations. The goal of this research was to identify PAtENT program activities and features that best addressed a particular element. Procedures followed key processes from curriculum study methodology including identifying desired outcomes, determining what content and activities contributed to those outcomes, and identifying experiences developed to result in those intended outcomes [5 - 7]. This systematic process identified attributes and components of PAtENT program features that aligned to the ten elements.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.3390/app12073656
