 NSFPAR ID:
 10378403
 Date Published:
 Journal Name:
 Proceedings of the VLDB Endowment
 Volume:
 15
 Issue:
 13
 ISSN:
 21508097
 Page Range / eLocation ID:
 3937  3949
 Format(s):
 Medium: X
 Sponsoring Org:
 National Science Foundation
More Like this

In this work, we formulate and solve a new type of approximate nearest neighbor search (ANNS) problems called ANNS after linear transformation (ALT). In ANNSALT, we search for the vector (in a dataset) that, after being linearly transformed by a userspecified query matrix, is closest to a query vector. It is a very general mother problem in the sense that a wide range of baby ANNS problems that have important applications in databases and machine learning can be reduced to and solved as ANNSALT, or its dual that we call ANNSALTD. We propose a novel and computationally efficient solution, called ONe Index for All Kernels (ONIAK), to ANNSALT and all its baby problems when the data dimension d is not too large (say d β€ 200). In ONIAK, a universal index is built, once and for all, for answering all future ANNSALT queries that can have distinct query matrices. We show by experiments that, when d is not too large, ONIAK has better query performance than linear scan on the mother problem (of ANNSALT), and has query performances comparable to those of the stateoftheart solutions on the baby problems. However, the algorithmic technique behind this universal index approach suffers from a socalled dimension blowup problem that can make the indexing time prohibitively long for a large dataset. We propose a novel algorithmic technique, called fast GOE quadratic form (FGoeQF), that completely solves the (prohibitively long indexing time) fallout of the dimension blowup problem. We also propose a JohnsonLindenstrauss transform (JLT) based ANNSALT (and ANNSALTD) solution that significantly outperforms any competitor when d is large.more » « less

Effective vector representation models, e.g., word2vec and node2vec, embed realworld objects such as images and documents in high dimensional vector space. In the meanwhile, the objects are often associated with attributes such as timestamps and prices. Many scenarios need to jointly query the vector representations of the objects together with their attributes. These queries can be formalized as rangefiltering approximate nearest neighbor search (ANNS) queries. Specifically, given a collection of data vectors, each associated with an attribute value whose domain has a total order. The rangefiltering ANNS consists of a query range and a query vector. It finds the approximate nearest neighbors of the query vector among all the data vectors whose attribute values fall in the query range. Existing approaches suffer from a rapidly degrading query performance when the query range width shifts. The query performance can be optimized by a solution that builds an ANNS index for every possible query range; however, the index time and index size become prohibitive  the number of query ranges is quadratic to the number n of data vectors. To overcome these challenges, for the query range contains all attribute values smaller than a userprovided threshold, we design a structure called the segment graph whose index time and size are the same as a single ANNS index, yet can losslessly compress the n ANNS indexes, reducing the indexing cost by a factor of Ξ©(n). To handle general range queries, we propose a 2D segment graph with averagecase index size O(n log n) to compress n segment graphs, breaking the quadratic barrier. Extensive experiments conducted on realworld datasets show that our proposed structures outperformed existing methods significantly; our index also exhibits superior scalability.

Given a collection of vectors, the approximate Knearestneighbor graph (KGraph for short) connects every vector to its approximate Knearestneighbors (KNN for short). KGraph plays an important role in high dimensional data visualization, semantic search, manifold learning, and machine learning. The vectors are typically vector representations of realworld objects (e.g., images and documents), which often come with a few structured attributes, such as timestamps and locations. In this paper, we study the allrange approximate Knearestneighbor graph (ARKGraph) problem. Specifically, given a collection of vectors, each associated with a numerical search key (e.g., a timestamp), we aim to build an index that takes a search key range as the query and returns the KGraph of vectors whose search keys are within the query range. ARKGraph can facilitate interactive high dimensional data visualization, data mining, etc. A key challenge of this problem is the huge index size. This is because, given n vectors, a bruteforce index stores a KGraph for every search key range, which results in O (K n 3 ) index size as there are O ( n 2 ) search key ranges and each KGraph takes O (K n ) space. We observe that the KNN of a vector in nearby ranges are often the same, which can be grouped together to save space. Based on this observation, we propose a series of novel techniques that reduce the index size significantly to just O (K n log n ) in the average case. Furthermore, we develop an efficient indexing algorithm that constructs the optimized ARKGraph index directly without exhaustively calculating the distance between every pair of vectors. To process a query, for each vector in the query range, we only need O (log log n + K log K) to restore its KNN in the query range from the optimized ARKGraph index. We conducted extensive experiments on realworld datasets. Experimental results show that our optimized ARKGraph index achieved a small index size, low query latency, and good scalability. Specifically, our approach was 1000x faster than the baseline method that builds a KGraph for all the vectors in the query range onthefly.more » « less

We study the framework of universal dynamic regret minimization with strongly convex losses. We answer an open problem in Baby and Wang 2021 by showing that in a proper learning setup, Strongly Adaptive algorithms can achieve the near optimal dynamic regret of πΜ (π1/3π1/3TV[π’1:π]2/3β¨π) against any comparator sequence π’1,β¦,π’π simultaneously, where π is the time horizon and TV[π’1:π] is the Total Variation of comparator. These results are facilitated by exploiting a number of new structures imposed by the KKT conditions that were not considered in Baby and Wang 2021 which also lead to other improvements over their results such as: (a) handling nonsmooth losses and (b) improving the dimension dependence on regret. Further, we also derive near optimal dynamic regret rates for the special case of proper online learning with expconcave losses and an πΏβ constrained decision set.more » « less

We study model selection in linear bandits, where the learner must adapt to the dimension (denoted by πβ ) of the smallest hypothesis class containing the true linear model while balancing exploration and exploitation. Previous papers provide various guarantees for this model selection problem, but have limitations; i.e., the analysis requires favorable conditions that allow for inexpensive statistical testing to locate the right hypothesis class or are based on the idea of βcorrallingβ multiple base algorithms, which often performs relatively poorly in practice. These works also mainly focus on upper bounds. In this paper, we establish the first lower bound for the model selection problem. Our lower bound implies that, even with a fixed action set, adaptation to the unknown dimension πβ comes at a cost: There is no algorithm that can achieve the regret bound πΛ(πβπβΎβΎβΎβΎβ) simultaneously for all values of πβ . We propose Pareto optimal algorithms that match the lower bound. Empirical evaluations show that our algorithm enjoys superior performance compared to existing ones.more » « less