KV caching is a memory-for-computation acceleration technique for LLM inference, enabling faster decoding by reusing intermediate hidden states (Waddington et al., 2013). However, during inference, each decoded token must attend to the full KV cache, causing cache reading latency to grow significantly with context length. For example, in LLaMA 7B (Touvron et al., 2023), a 32ktoken KV cache requires 16GB of memory and over 11ms to read-accounting for more than half of the total inference time (Tang et al., 2024).
To address this issue, previous works have proposed to selectively retain only a subset of KV cache entries based on token importance. Among them, one line of work-known as KV † While existing retrieval methods suffer from coarse granularity, Fier achieves higher accuracy through fine-grained token-level retrieval, and preserves selection efficiency by quantization.
eviction (Fig. 1(b))-focuses on retaining fixedposition tokens that typically receive higher attention weights, such as the initial tokens (due to the attention sink phenomenon) and the most recent tokens (Xiao et al., 2023;Zhang et al., 2023;Li et al., 2024;Liu et al., 2023). However, these approaches overlook the dynamic nature of KV criticality, i.e., tokens that are evicted earlier may become important in the future. Their inability to recall evicted tokens often results in degraded performance in multi-round QA or long conversation applications. Motivated by this, another line of work-KV retrieval (Fig. 1(c))-has been proposed to dynamically recall tokens that are most relevant to the current query during generation (Tang et al., 2024;Chen et al., 2024a). Despite achieving better performance, KV retrieval requires frequent estimation of the token importance for every new query, introducing additional computational overhead. To mitigate this, existing methods perform page-level retrieval, where all tokens that belong to a certain page of the KV cache are retrieved (or not re-trieved) simultaneously to avoid computing full attention scores, leading to a coarser granularity of retrieval.
However, we observe that in long-context tasks, information relevant to the current query may be scattered throughout the entire input, i.e., important tokens are often sparsely distributed across the KV cache (shown in Fig. 2). Consequently, page-level retrieval inevitably leads to imprecise selection: retrieved pages may include irrelevant tokens, while evicted pages may exclude critical ones, thereby affecting the model performance.
In this paper, we aim to address the retrieval inaccuracy while preserving selection efficiency. Specifically, we find that quantizing the key cache to as low as 1-bit has minimal impact on the accuracy of Top-k selection in token importance estimation (shown in Fig. 3). Despite quantization truncates large values, important tokens are still preserved in the Top-k after computing quantized attention. Based on this insight, we propose Fine-Grained and Efficient KV cache Retrieval (Fier), a 1-bit quantization-based KV retrieval method. As shown in Fig. 1(d), Fier enables more accurate recovery of important tokens and reduces the selection cost. This leads to improved model performance under the same cache budget.
We evaluate Fier across PG19 (Rae et al., 2019), LongBench (Bai et al., 2023), and the passkey retrieval benchmarks (Peng et al., 2023). The results demonstrate the effectiveness of Fier in both generative and retrieval-focused settings. Experiments show that Fier achieves performance comparable to using the full KV cache while requiring only 11% of the cache budget, and consistently outperforms existing KV eviction and retrieval baselines. Additionally, Fier achieves 1.2× to 1.5× decoding speedup across different context lengths on a single RTX 4090 GPU. In summary, we make the following contributions in this work:
• We observe the sparse distribution of important tokens within the KV cache in longcontext scenarios, highlighting the necessity of fine-grained retrieval.
• We propose Fier, a KV retrieval method built on 1-bit key quantization, which enables efficient and accurate token-level retrieval.
• We conduct comprehensive evaluations of Fier across diverse long-context tasks and model architectures, demonstrating its superior performance and efficiency.
Long-Context LLMs. Large Language Models (LLMs) have transformed the landscape of natural language processing, largely due to their strong ability to deal with long context. Their context length capacity has increased dramatically-from 4k to 128k (Grattafiori et al., 2024), 1M (Yang et al., 2025), and even 10M (Team et al., 2024) tokens. This expansion unlocks a range of advanced capabilities, including o1 long-range reasoning (Guo et al., 2025;OpenAI, 2024), incontext learning (Li et al., 2025), and multimodal intelligence (Weng et al., 2024). Fier aims to improve the inference efficiency of long-context LLMs by exploiting the sparsity of the KV cache. KV Cache Eviction. Previous work identified the sparsity of attention matrices, showing that retaining only a small fraction of tokens is sufficient for the performance. For example, Xiao et al. (2023) propose to retain the first few tokens based on the "attention sink" phenomenon. H2O (Zhang et al., 2023) retains a limited set of KV cache by selecting tokens with the highest cumulative attention scores. SnapKV (Li et al., 2024) selects clustered historical tokens along with a localized window of recent tokens. However, these approaches ignore the fact that tokens evicted can become important in the future. Fier addresses this via query-specific KV retrieval, enabling dynamic reassessment of token importance at each decoding step. KV Cache Retrieval. KV retrieval methods, including our proposed Fier, dynamically select tokens relevant to the current query. However, existing approaches like Quest (Tang et al., 2024), ArkVale (Chen et al., 2024a), PQCache (Zhang et al., 2025), SparQattn (Ribar et al., 2023) and MagicPIG (Chen et al., 2024b) retrieve at the page or cluster level or compute partial tokens' criticality for efficiency, overlooking the sparse distribution of important tokens. In this paper, we propose a fine-grained, token-level retrieval strategy that maintains efficiency while improving accuracy. This design better captures critical information missed by existing methods. KV Cache Quantization. Another related line of work is KV quantization (Liu et al., 2024;Dong et al., 2024), which compresses the cache by performing self-attention in low-bit space. The objec-tive of KV quantization is to minimize global reconstruction error. In contrast, Fier achieves cache compression by retaining only a subset of the full KV and adopts a relaxed quantization objective focused on preserving Top-ranked tokens, enabling the use of extremely low bit-widths.
In an autoregressive LLM, the inference process typically consists of two stages: the prefill stage and the decoding stage.
Prefill Stage. Given an input context X ∈ R lprompt×d , the model computes the initial key and value representations, denoted as K 0 ∈ R lprompt×d and V 0 ∈ R lprompt×d , which together form the initial KV cache. Decoding Stage. At each step, a new query q ∈ R 1×d is generated and the corresponding key k and value v are appended to the initial cache (K 0 , V 0 ) to form the current KV cache:
The attention output is then computed as:
The major overhead of decoding comes from computation of attention score s, which reflects the importance of KV token k j to the query. At each step, the current query must attend to the entire KV cache. This cost becomes higher in longcontext tasks. To address this, previous works have demonstrated attention sparsity, observing that initial tokens (attention sink) and recent tokens (locality) tend to receive higher attention scores. Based on this, they retain a fixed subset of tokens, denoted as (K ′ , V ′ ) ∈ R n×d at these positions for all queries, where n is cache budget. However, subsequent studies show that token criticality varies across different queries. As a result, query-specific token selection is necessary, where token importance needs to be recomputed for each query. To improve importance estimation efficiency, existing works tend to perform selection at a coarser, page-level granularity. For example, Quest (Tang et al., 2024) partitions the key cache K ∈ R l×d into l L pages (L is typically 16 or 32). For each page P, it extracts maximum and minimum vectors k max P and k min P ∈ R 1×d , per-forms point-wise multiplication with q,
and takes the maximum across both the hidden dimension and the two vectors to obtain the page importance score.
Pages with the highest importance scores are then selected for self-attention computation.
During the selection, Quest loads only 2 K vectors per page. Assuming K is stored in float16, this results in a key cache load ratio of:
It is clear that larger page sizes reduce importance estimation costs, but lead to coarser granularity by grouping more tokens together. There exists a trade-off between efficiency and accuracy.
To improve upon existing page-level KV retrieval methods, we make the following two observations on the token importance estimation process. OB1: Important Token Sparsity Makes Page Retrieval Inaccurate. To understand the trade-off of page granularity, we visualize both the highestattended tokens and those selected under pagelevel partitioning by mapping them back to the original context. As shown in Fig. 2, queries Q1 and Q2 attend to different regions of the context, which aligns with prior findings on the dynamic nature of token criticality. Moreover, tokens with high attention scores are sparsely distributed across the context, and we observe that pages 7, 16, and 54 each contain a mixture of both important and unimportant tokens. This overlap makes inaccurate retrieval, which pinpoints the necessity of fine-grained retrieval strategies that identify important information at the token level, rather than relying on coarse-grained page grouping.
Motivated by this, we aim to design a retrieval strategy that operates at fine-grained token level while incurring minimal additional computation overhead. Notably, quantization enables low-bit computation, achieving high efficiency while still allowing every token to participate in the criticality estimation. We begin by validating this insight through the following observation. OB2: Quantization has Minimal Impact on Top-k Selection, even at 1-bit. Quantizing KV cache values to a lower precision significantly reduces the cost of data movement and attention computation. Let K ∈ R l×d denote the original key cache, where k i ∈ R 1×d is the key vector corresponding to the i-th token, quantization converts each k i to its dequantized counterpart ki as
where s K i , z K i ∈ R 1×d are the per-channel calibrated scaling and bias vectors specific to the ith key vector. Previous KV quantization methods (Zhang et al., 2024) aim to optimize these factors through the calibration process to minimize the impact of quantization on the computed attention score. This objective can be formulated as an ℓ 2 loss:
However, quantization introduces perturbations on the values, drawing computation results away from intended. The impact of quantization is more severe if outliers exist in the distribution. Previous methods like Kivi (Liu et al., 2024), equipped with advanced channel-wise grouping and rescaling scheme, cannot quantize the KV cache below 2 bit while retaining model performance.
Meanwhile, we observe that quantizing the key cache to low bits has significantly less impact on the token importance estimation than retaining the attention scores. Here we explore an extreme case by quantizing K to just 1-bit. Using the fullprecision attention scores as ground truth, we evaluate whether Top-k token selection can still be accurately recovered under such an aggressive quantization setting. Specifically, we feed a long input context (14k tokens) into LlaMA during the prefill stage, and compute attention scores under both full-precision and 1-bit quantized K using the same query. As shown in Fig. 3, despite the fact that low-bit quantization truncates large values and distorts the overall distribution, the Top-k tokens remain largely unchanged. This suggests that token criticality is still well captured, even un-der extreme quantization.
To understand the reason, we analyze the quantization objective implied by the goal of token importance estimation. For importance estimation, we aim to maintain the ranking of the Top-k tokens rather than preserving all attention scores precisely. Assume m is the minimum margin between the attention scores of Top-k and non-Topk tokens in the full-precision setting. To preserve this ranking after quantization, it is sufficient to ensure that the attention score of each token deviates from its full-precision counterpart by at most m/2. This leads to the following hinge objective:
Compared to the ℓ 2 loss, the hinge loss imposes a relaxed objective that prioritizes preserving the relative ranking of Top-k tokens (Fig. 4). More importantly, outlier tokens that lead to large attention scores enjoy larger margins under the hinge loss, making their quantization errors less impactful. This enables quantization with 1-bit and larger group sizes g while still maintaining Top-k accuracy.
Easy to minimize Hard to minimize
Motivated by previous observations, we propose Fier, a token-level retrieval method based on 1-bit linear quantization. Fier compresses the key cache into 1-bit using a simple round-to-nearest (RTN) quantizer. Given a query q, approximate attention scores are computed efficiently using quantized keys. Based on these scores, Fier selects the Top-k tokens and performs full-precision selfattention over the selected subset. We summarize the workflow of Fier in Algorithm 1.
Beyond retrieval accuracy, the efficiency of the selection and decoding stage is also critical for prac-Algorithm 1 Fier: Token-Level KV Retrieval via 1-Bit RTN Quantization 1: Input: Query q, full-precision (K, V), group size g 2: Output: Attention output o 3: // Step 1: Quantize K to 1-bit 4: Partition K into groups of size g along each channel 5: For each group, compute the scaling factors (s, z) and broadcast them to construct s
Step 2: Compute Approximate Attention Scores 9: s = q • K⊤ 10: // Step 3: Select Top-k Tokens 11: Sq = Top-k(s) 12: // Step 4: Compute Real Attention on Selected Tokens 13:
tical deployment. We measure the efficiency using the key cache access ratio (CAR), defined as the fraction of the key cache accessed throughout the pipeline. Specifically, we compare both retrievaland eviction-based methods under the same top-k setting. In the decoding phase, all methods (Fier, Quest, and H2O) access k tokens, so their CAR is identical. The key difference lies in the selection phase. For K stored in float16, we quantize it to 1-bit with group size g. Note that in addition to the 1-bit K Q , each group also needs to store a pair of (s, z) in float16. Therefore, the CAR of Fier will be calculated as:
which decreases with a larger group size. Recall that Quest has a load ratio of 2/L. For fairness, we set g = 32, which matches the load ratio of 1/8 with page size L = 16 as implemented in the Quest baseline. In contrast, H2O maintains a dynamic pool of size k/l during the selection stage. Therefore, when k = l/10, the CAR of the three methods can be summarized in Tab. 1. While retrieval does introduce additional overhead during the selection compared to eviction, its goal is to improve performance with only a modest CAR increase. Datasets. We evaluate Fier on the language modeling task PG19 (Rae et al., 2019). To assess its performance on long-context QA, we further conduct experiments on LongBench (Bai et al., 2023) using six representative datasets: Narra-tiveQA, HotpotQA, Qasper, TriviaQA, GovReport, and MultifieldQA. The detailed information about the six datasets is in Appendix A. We evaluate Fier on the passkey retrieval task (Peng et al., 2023) to assess its ability to model long-range dependencies. We also compare responses from Fier and Quest enabled chatbots in Appendix E.
We apply our method to three open-sourced models:
LLaMA-3-8B-Instruct (Grattafiori et al., 2024), LongChat-v1.5-7B-32k (Li et al., 2023), and Mistral-7B-Instruct (Jiang et al., 2023). Following the same setup as in Quest, neither Fier nor the baseline methods are applied to the first two layers of the model. We evaluate the performance of each method under varying KV cache budgets. Baselines. We thoroughly compare the performance of Fier and Quest (Tang et al., 2024) across various benchmarks. Note that in all performance evaluation, we set the page size of Quest to 16 and the grouping size of Fier to 32 for a fair comparison. We also compare Fier with four KV eviction baselines: H2O (Zhang et al., 2023), StreamingLLM (Xiao et al., 2023), SnapKV (Li et al., 2024) andTOVA (Oren et al., 2024). The results in the experiments are either taken from the original paper or obtained by running open-source code. More implementation details can be found in Appendix B.
More Accurate Retrieval of Important Tokens. In Fig. 6, we visualize the positions of Top-64 tokens selected by Quest with different page sizes and by Fier-1bit-g32, all mapped back onto the full KV cache. We then compute the recall rate, defined as the overlap between the retrieved tokens and those selected using the full attention score. The experiment is conducted on LLaMA.
We observe that Quest with either small or large page sizes tends to retain unimportant tokens and evict important ones, due to its coarse-grained page-level retrieval. In contrast, Fier performs token-level retrieval through low-bit quantized attention computation, resulting in a significantly higher recall rate and better alignment with the full attention distribution.
We begin by evaluating language modeling perplexity on PG19, a benchmark consisting of 100 books with an average length of 70k tokens. We evaluate three different models by feeding them texts of varying lengths and compare the results against both Full KV and Quest (Fig. 5). Note that both Fier and Quest are evaluated under the same KV cache budget of 4096 tokens. Fier achieves performance close to that of Full KV and significantly outperforms Quest on both the LLaMA and Mistral models.
We evaluate on the LongBench benchmark using LLaMA-3-8B-Instruct across a diverse set of long-context tasks, including single-document QA (NarrativeQA, Qasper, MultiFieldQA), multidocument QA (HotpotQA), summarization (Gov-Report), and few-shot learning (TriviaQA). We also compare Fier with H2O, StreamingLLM, SnapKV, and Quest under varying KV cache budget settings. In addition, we perform evaluations using the Mistral-7B and LongChat-7B models to verify the generality of our method across different model architectures.
As shown in Fig. 7, Fier consistently achieves superior performance compared to all baselines across six long-context datasets under various KV cache budgets. Overall, Fier surpasses all baselines at cache budgets of 512, 1024, and 2048 tokens, and achieves comparable performance to full KV using only 1k tokens, which is only 11% of the full cache. This suggests that Fier benefits from accurately retrieving critical tokens, enabling efficient use of limited cache resources without compromising model quality. Similar results are observed on the other two models. As shown in Tab. 2, Fier shows consistent improvements on both single-document and multi-document tasks.
We further evaluate Fier's ability to handle longdistance dependencies. Specifically, we employ the passkey retrieval task (Peng et al., 2023) as our benchmark. This task assesses whether the model can retrieve a simple passkey from a large amount of irrelevant content. Following the setup in Tang et al. (2024), we use a context length of 10k tokens for evaluation with both LongChat-7B and a smaller model, LLaMA3-1B. KV eviction methods perform poorly due to their inability to recall discarded tokens, while Quest provides noticeable improvements over them. Fier, however, achieves even higher retrieval accuracy, performing well under extremely low budgets of just 32 64 tokens, especially improving the longcontext processing capability of smaller models (shown in Tab. 3).
Token Granularity vs. Quantized Attention.
To understand whether Fier's performance gain primarily stems from its fine-grained token-level selection (as opposed to page-level) or from the use of quantized attention as an importance metric, we conduct an ablation study on LLaMA-3-8B-Instruct, isolating these two factors. Specifically, we reduce the page size in Quest to approximate finer granularity and compare performance. In addition, we apply the averaged quantized attention score as the page-level importance metric under the same page size, and evaluate its effect on Quest. As shown in Tab. 4, using smaller page sizes in Quest leads to improved per- formance. However, it also increases the cache load ratio. Additionally, incorporating quantized attention for scoring further enhances its effectiveness. Notably, Fier can be viewed as quantized attention with a page size of 1, achieving the best overall results. These results suggest that Fier benefits from both the token-level granularity and the use of quantized attention as a lightweight yet effective importance estimator. Fier w/ Different Group Sizes. We also investigate how the group size used during key quantization affects Fier's performance. We find that as the group size increases, the cache load ratio decreases, but this comes at the cost of reduced performance. Nevertheless, Fier consistently outperforms Quest under the same cache load ratio.
Inference Efficiency. In Fig. 8, we present the decoding latency of 256 tokens on LLaMA-2-7B under different prefill context lengths. To ensure a fair comparison with Full KV, we include both the time spent on computing quantized attention and the time required to recall the selected Topk tokens. We implement the group-wise quantization kernel using Triton, and employ the Top-k CUDA operator to efficiently perform Top-k token recall. Fier's efficiency gain is mainly attributed to the speedup in the self-attention computation, as it restricts attention to only a small subset of the KV cache. This acceleration becomes more pronounced as the context length increases; for instance, at a context length of 32k tokens, Fier achieves over 1.5× decoding speedup. We also provide a detailed latency breakdown comparison of Fier and Quest in Appendix D.
We present Fier, a fine-grained and efficient KV cache retrieval algorithm that selects important tokens using 1-bit quantization. By involving all tokens in the computation, Fier enables token-level criticality estimation, leading to improved recall rate and model performance. Extensive experiments across various tasks and model architectures show that Fier consistently outperforms existing methods. Notably, Fier matches full cache performance using only 11% of the KV budget and achieves a 1.2×-1.5× decoding speedup.
Model Scale. Due to limited computational resources, our experiments are restricted to models up to 8B parameters. Evaluating Fier on larger models (e.g., 13B, 70B) may reveal further insights into its scalability and effectiveness. System Optimization. Our current implementation uses Triton to develop low-bit operators for quantized attention. While Triton offers flexibility and ease of development, it does not match the low-level optimization potential of custom CUDA kernels, potentially limiting the achievable inference speedup. Compatibility with GQA. Fier is not yet integrated with grouped-query attention (GQA) (Ainslie et al., 2023). This is because token pruning and grouped-query attention are orthogonal in principle: GQA reduces the number of KV heads, while token pruning reduces the number of tokens.
Exploring their compatibility remains an important direction for future work.
Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. 2024. Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization. Advances in Neural Information Processing Systems, 37:3304-3331. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, and 1 others. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34710.
We use a subset of LongBench (Bai et al., 2023) for long-context QA evaluation. Tab. 5 shows the statistics and evaluation metrics used in the experiments.
For experiments on LongBench (Bai et al., 2023) and PG19 (Rae et al., 2019), we use three models: LLaMA-3-8B-Instruct (Grattafiori et al., 2024), LongChat-v1.5-7B-32k (Li et al., 2023), and Mistral-7B-Instruct (Jiang et al., 2023), with the maximum input length uniformly set to 32k tokens to ensure a fair comparison. All inference runs are conducted on NVIDIA A6000 GPUs. During the self-attention phase, we utilize FlashAttention (Dao, 2023) for acceleration, except for H2O (Zhang et al., 2023), which requires computation of historical attention scores and therefore cannot benefit from FlashAttention. Except for H2O (Zhang et al., 2023) and Quest (Tang et al., 2024), which are run using their publicly released implementations, all other baselines are run using the unified KV Cache-Factory framework (Cai et al., 2024). Experiments on observations and decoding latency are conducted separately on NVIDIA RTX 4090 GPUs.
Different from the methods discussed in the main text, MiKV (Yang et al., 2024) adopts a hybrid strategy: during decoding, it stores top tokens in high precision while retaining the remaining tokens in a low-bit format. The main distinction between Fier and MiKV is that MiKV maintains the entire KV cache in mixed precision, whereas Fier only preserves top tokens during decoding. To ensure fairness, we evaluated Fier on the MMLU dataset (Hendrycks et al., 2020) using the same cache size as reported for MiKV. Note that the cache size of Fier accounts for the CAR in the selection phase, which is not explicitly stated in the MiKV. As shown in Tab. 6, Fier demonstrates better performance compared to MiKV under the same cache budget. We provide a detailed latency breakdown between the pipelines of Fier and Quest in Tab. 7. The primary difference lies in the importance estimation stage: Fier uses an extreme 1-bit quantized attention, while Quest adopts page-level attention. In the top-k recall stage: Quest retrieves pages, while Fier directly retrieves top-k tokens. The final topk self-attention is identical for both methods. In Tab. 7, we compare the average per-layer latency for each token, as well as the end-to-end generation latency. With the same k, Fier has higher latency than Quest due to its fine-grained top-k sorting in the recall stage. However, Fier achieves better performance. By reducing k , Fier can achieve lower latency than Quest while still maintaining better performance. Overall, Fier offers a better trade-off between latency and performance compared to Quest.
To better illustrate the practical differences between the two retrieval methods, we deploy a LLaMA-3-8B-Instruct chatbot using Fier and Quest, respectively. Given the same long context and user question, we present the corresponding responses from each chatbot for a qualitative comparison (Fig. 9). In the first example, which asks about the orders in which Mufti-e-Azam-e-Hind received Khilafat, the Fier-enabled chatbot