Search for: All records

Artificial intelligence (AI) provides versatile capabilities in applications such as image classification and voice recognition that are most useful in edge or mobile computing settings. Shrinking these sophisticated algorithms into small form factors with minimal computing resources and power budgets requires innovation at several layers of abstraction: software, algorithmic, architectural, circuit, and device-level innovations. However, improvements to system efficiency may impact robustness and vice-versa. Therefore, a co-design framework is often necessary to customize a system for its given application. A system that prioritizes efficiency might use circuit-level innovations that introduce process variations or signal noise into the system, which may use software-level redundancy in order to compensate. In this tutorial, we will first examine various methods of improving efficiency and robustness in edge AI and their tradeoffs at each level of abstraction.Then, we will outline co-design techniques for designing efficient and robust edge AI systems, using federated learning as a specific example to illustrate the effectiveness of co-design. 

Approximate nearest neighbor search (ANNS) is a key retrieval technique for vector database and many data center applications, such as person re-identification and recommendation systems. It is also fundamental to retrieval augmented generation (RAG) for large language models (LLM) now. Among all the ANNS algorithms, graph-traversal-based ANNS achieves the highest recall rate. However, as the size of dataset increases, the graph may require hundreds of gigabytes of memory, exceeding the main memory capacity of a single workstation node. Although we can do partitioning and use solid-state drive (SSD) as the backing storage, the limited SSD I/O bandwidth severely degrades the performance of the system. To address this challenge, we present NDSEARCh, a hardware-software co-designed near-data processing (NDP) solution for ANNS processing. NDSeARCH consists of a novel in-storage computing architecture, namely, SEARSSD, that supports the ANNS kernels and leverages logic unit (LUN)-level parallelism inside the NAND flash chips. NDSEARCH also includes a processing model that is customized for NDP and cooperates with SearSSD. The processing model enables us to apply a two-level scheduling to improve the data locality and exploit the internal bandwidth in NDSearch, and a speculative searching mechanism to further accelerate the ANNS workload. Our results show that NDSEARCH improves the throughput by up to 31.7×,14.6×,7.4×, and 2.9× over CPU, GPU, a state-of-the-art SmartSSD-only design, and DeepStore, respectively. NDSEARCH also achieves two orders-of-magnitude higher energy efficiency than CPU and GPU. 

Theoretical bounds are commonly used to assess the limitations of photonic design. Here we introduce a more active way to use theoretical bounds, integrating them into part of the design process and identifying optimal system parameters that maximize the efficiency limit itself. As an example, we consider wide-field-of-view high-numerical-aperture metalenses, which can be used for high-resolution imaging in microscopy and endoscopy, but no existing design has achieved a high efficiency. By choosing aperture sizes to maximize an efficiency bound, setting the thickness according to a thickness bound, and then performing inverse design, we come up with high-numerical-aperture (NA=0.9) metalens designs with, to our knowledge, record-high 98% transmission efficiency and 92% Strehl ratio across all incident angles within a 60° field of view, reaching the maximized bound. This maximizing-efficiency-limit approach applies to any multi-channel system and can help a wide range of optical devices reach their highest possible performance. 

The rapidly advancing capabilities in nanophotonic design are enabling complex functionalities limited mainly by physical bounds. The efficiency of transmission is a major consideration, but its ultimate limit remains unknown for most systems. This study introduces a matrix formalism that puts a fundamental bound on the channel‐averaged transmission efficiency of any passive multi‐channel optical system based only on energy conservation and the desired functionality, independent of the interior structure and material composition. Applying this formalism to diffraction‐limited nonlocal metalenses with a wide field of view shows that the transmission efficiency must decrease with the numerical aperture for the commonly adopted designs with equal entrance and output aperture diameters. It also shows that reducing the size of the entrance aperture can raise the efficiency bound. This study reveals a fundamental limit on the transmission efficiency as well as provides guidance for the design of high‐efficiency multi‐channel optical systems. 

Recommendation systems have been widely embedded into many Internet services. For example, Meta’s deep learning recommendation model (DLRM) shows high predictive accuracy of click-through rate in processing large-scale embedding tables. The SparseLengthSum (SLS) kernel of the DLRM dominates the inference time of the DLRM due to intensive irregular memory accesses to the embedding vectors. Some prior works directly adopt near-data processing (NDP) solutions to obtain higher memory bandwidth to accelerate SLS. However, their inferior memory hierarchy induces a low performance-cost ratio and fails to fully exploit the data locality. Although some software-managed cache policies were proposed to improve the cache hit rate, the incurred cache miss penalty is unacceptable considering the high overheads of executing the corresponding programs and the communication between the host and the accelerator. To address the issues aforementioned, we proposeEMS-i, an efficient memory system design that integrates Solid State Drive (SSD) into the memory hierarchy using Compute Express Link (CXL) for recommendation system inference. We specialize the caching mechanism according to the characteristics of various DLRM workloads and propose a novel prefetching mechanism to further improve the performance. In addition, we delicately design the inference kernel and develop a customized mapping scheme for SLS operation, considering the multi-level parallelism in SLS and the data locality within a batch of queries. Compared to the state-of-the-art NDP solutions,EMS-iachieves up to 10.9× speedup over RecSSD and the performance comparable to RecNMP with 72% energy savings.EMS-ialso saves up to 8.7× and 6.6 × memory cost w.r.t. RecSSD and RecNMP, respectively.