Simulia Abaqus Helmet Impact

Howard Yenp Tianyu Gaop Minmin Houi Ke Dingi

Daniel Fleischeri Peter Izsaki Moshe Wasserblati Danqi Chenp

Abstract

Many benchmarks exist for evaluating long-context language models (LCLMs), yet developers often rely on synthetic tasks such as needle-in-a-haystack (NIAH) or an arbitrary subset of tasks.

However, It Remains Unclear Whether These

benchmarks reflect the diverse downstream applications of LCLMs, and such inconsistencies further complicate model comparison.

We Investigate The Un-

derlying reasons behind these practices and find that existing benchmarks of- ten provide noisy signals due to limited coverage of applications, insufficient context lengths, unreliable metrics, and incompatibility with base models.

In

this work, we introduce HELMET (How to Evaluate Long-context Models Effectively and Thoroughly), a comprehensive benchmark encompassing seven diverse, application-centric categories. We also address several issues in previous benchmarks by adding controllable lengths up to 128K tokens, model-based eval- uation for reliable metrics, and few-shot prompting for robustly evaluating base models. Consequently, we demonstrate that HELMET offers more reliable and consistent rankings of frontier LCLMs. Through a comprehensive study of 59 LCLMs, we find that (1) synthetic tasks like NIAH do not reliably predict down- stream performance; (2) the diverse categories in HELMET exhibit distinct trends and low correlations with each other; and (3) while most LCLMs achieve perfect NIAH scores, open-source models significantly lag behind closed ones when tasks require full-context reasoning or following complex instructions—the gap widens as length increases. Finally, we recommend using our RAG tasks for fast model development, as they are easy to run and better predict other downstream perfor- mance; ultimately, we advocate for a holistic evaluation across diverse tasks.1

Most Lclms Evaluate On Synthetic

(Syn.) tasks. ♭: base models. ∞: ∞BENCH.

✗

Xiong et al.

Ing (Chen Et Al., 2023; Xiong Et Al., 2023;

Peng et al., 2024; Fu et al., 2024) still rely ei-

2024; Team Et Al., 2024C; Openai, 2023; Team

1Our data and code are available at https://github.com/princeton-nlp/HELMET.

<

<

<

Figure 1: Long-context benchmark results of frontier LCLMs (Llama-3.1 8B/70B, GPT-4o-mini, GPT-4o-08-06, and Gemini-1.5 Flash/Pro) at 128K input length. NIAH is saturated for almost all models; RULER (Hsieh et al., 2024) and ∞BENCH (Zhang et al., 2024b) show unexpected trends for Llama-3.1 (Dubey et al., 2024). In contrast, HELMET demonstrates more consistent rankings of these frontier models.

et al., 2024a) also mostly report NIAH, sometimes with arbitrary subsets of other datasets, as shown in Table 1. Such inconsistencies complicate comparisons between different models. It is also un- clear whether synthetic NIAH tasks or the few chosen datasets offer a holistic picture of models’ long-context abilities and their performance in real-world applications.

Why don’t model developers agree on these evaluations? We take a closer look and find that existing benchmarks suffer from many critical design flaws, including: • Insufficient coverage of downstream tasks: Existing benchmarks either focus on synthetic tasks (Hsieh et al., 2024) or include only simple question answering (Zhang et al., 2024b). Other works study particular aspects of LCLMs, such as summarization (Chang et al., 2024), in-context learning (Li et al., 2024c), and retrieval-augmented generation (RAG; Lee et al., 2024), but they do not provide a holistic evaluation of LCLMs.

• Inadequate lengths: Most natural language datasets in existing benchmarks (Shaham et al., 2023; An et al., 2024; Table 4) are too short to effectively test frontier long-context abilities (usually ≥128K).

• Unreliable metrics: For commonly used long-document QA and summarization tasks, most existing benchmarks still rely on metrics like ROUGE (Lin, 2004), which are often noisy and unreliable (Goyal et al., 2023; Deutsch et al., 2022; Chang et al., 2024).

• Incompatibility with base models: Many LCLM developments focus on base models without instruction tuning, but most existing benchmarks require models to be instruction-tuned—hence developers can only rely on synthetic tasks or perplexity.

Consequently, existing benchmarks are either not applicable to long-context works (inadequate lengths or incompatibility) or provide highly noisy signals (insufficient coverage or unreliable met- rics). We summarize the shortcomings of existing benchmarks in Table 2. For the three benchmarks that support a 128K2 context length (NIAH, RULER, and ∞BENCH), we use them to evaluate six frontier models and report the numbers in Figure 1. We see that NIAH does not reflect differences across models; RULER and ∞BENCH show unexpected trends—on RULER, Gemini Flash outper- forms Gemini Pro; on ∞BENCH, the 70B Llama model underperforms compared to the 8B one. This raises concerns about the reliability of the benchmarks and how they can differentiate long-context models, which likely contributes to the lack of use of these benchmarks in model development.

To address these challenges, we present HELMET (How to Evaluate Long-context Models Effectively and Thoroughly). We curate a diverse set of application-centric long-context tasks across seven categories. Beyond widely adopted categories like synthetic recall, long-document question answering (QA), and summarization, we also add many-shot in-context learning (ICL), retrieval- augmented generation (RAG), passage re-ranking, and generation with citations (Gao et al., 2023).

We further address the shortcomings of existing benchmarks: (1) we ensure all datasets support input lengths of 128K tokens and are easily extendable to longer contexts (§2.1); (2) we introduce reference-based model evaluation for QA and summarization and show that it significantly improves over n-gram overlap metrics (§2.2); (3) we refine the prompts and in-context demonstrations used for all tasks, reducing evaluation noise caused by different output formats and allowing base models to 2Throughout the paper, we use binary prefixes K= 210.

Published As A Conference Paper At Iclr 2025

Table 2: Comparison of long-context benchmarks: ZeroSCROLLS (Shaham et al., 2023), Long- Bench (Bai et al., 2024), L-Eval (An et al., 2024), RULER (Hsieh et al., 2024), ∞BENCH (Zhang et al., 2024b), and our HELMET. L: input tokens.

†: All Datasets Have L < 128K Except One

dataset. ‡: L-Eval uses LLMs to compute reference-free, pairwise win-rates; we design reference- based model evaluation for specific tasks.

Robust

Eval.

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

be evaluated robustly across most categories (§2.3). Together, HELMET enables a holistic evalua- tion of long-context capabilities and provides more reliable signals for model development. Figure 1 demonstrates that HELMET can clearly differentiate models with varying capabilities and reflect comparisons consistent with human perception. More discussion on HELMET’s improvements over previous benchmarks and direct comparisons can be found in §A.

To understand the progress of LCLMs and how different long-context capabilities correlate with one another, we evaluate a comprehensive list of 59 LCLMs of various architectures, scales, and training approaches. Our analysis reveals that (1) synthetic tasks poorly indicate ownstream performance (§3.1), (2) different categories in HELMET show distinct trends (§3.2), and (3) open-source models significantly lag behind closed ones on tasks that require reasoning over long contexts or following complex instructions—the gap further widens as context length increases (§3.3). Finally, we find that the RAG category strikes a good balance between ease of use, stronger correlation with downstream tasks, and compatibility with base models. Ultimately, it is imperative to evaluate LCLMs across a diverse spectrum of categories. We hope that our insights provide a more effective way to evaluate LCLMs for future model development and benchmarking.

Our Benchmark: Helmet

In this work, we seek to overcome the shortcomings of existing benchmarks by meeting the fol- lowing desiderata: (1) diverse coverage across different tasks and capabilities of LCLMs, (2) con- trollable context lengths that support more than 128K input tokens, and (3) reliable evaluation for both base and instruction-tuned models. In this section, we describe the datasets used in HELMET and how they improve upon existing evaluation benchmarks in terms of settings and metrics. An overview of HELMET is shown in Table 3.

Realistic And Diverse Long-Context Applications

Retrieval-augmented generation (RAG). We use open-domain question answering (ODQA)— which requires retrieving from a knowledge corpus and then generating correct answers (Chen et al., 2017)—as a representation of retrieval-augmented generation (RAG) applications. We utilize Natu- ral Questions (NQ; Kwiatkowski et al., 2019), TriviaQA (TQA; Joshi et al., 2017), HotpotQA (Yang et al., 2018), and PopQA (Mallen et al., 2023). We use the gold passage (the passage with the answer) from Petroni et al. (2021), or otherwise select any passage that contains the answer.

Given an input length L, we first determine the number of passages k that can fit within L tokens, then retrieve k passages3 from the corpus4 that do not contain the answer as distractors. This differs from previous works that randomly sample passages from the corpus (Lee et al., 2024) and is more 3We use Alibaba-NLP/gte-large-en-v1.5 for retrieval (Zhang et al., 2024a).

4We use Wikipedia 2019-8-01 dump, split into 100-word passages (Petroni et al., 2021).

Published As A Conference Paper At Iclr 2025

Table 3: Overview of evaluation datasets. We select datasets that cover various important long- context capabilities. SubEM: substring exact match.

Subem

Retrieve the needle (a number) within noisy needles

Retrieve Multiple Values For One Needle (Key)

realistic and challenging. For NQ, TQA, and PopQA, we take the top k −1 distractors and insert the gold passage at six evenly distributed positions following Liu et al. (2023). For HotpotQA, which requires two gold passages, we combine them and the top k −2 distractors and randomly shuffle them into three permutations. We use substring exact match (SubEM; whether the answer is included in the output), following previous work (Asai et al., 2024a). See §B.1 for more details.

Generation with citations (Cite). We leverage ALCE (Gao et al., 2023) to evaluate LCLMs on a realistic application of answering questions while providing correct attributions (Bohnet et al., 2022). Given multi-faceted questions and relevant passages, models are required to generate a long- text answer and cite supporting passage IDs at the end of each sentence. This tests models’ ability to utilize the passages in the context and also to follow the instructions about citation formats. We use the ASQA (Stelmakh et al., 2022) and QAMPARI (Rubin et al., 2022) subsets from ALCE. For an input length L, we first determine the number of passages k, and use the top k retrieved passages from Wikipedia as contexts. The model’s outputs are evaluated on correctness and citation quality, and we report the average across all metrics. See §B.2 for more details.

Passage re-ranking (Re-rank). Re-ranking retrieved passages based on their relevance to the query is an important application of LCLMs (Sun et al., 2023). The task requires the model to retrieve relevant information, compare, and reason over different parts of the contexts. We use the MS MARCO dataset (Bajaj et al., 2018), where each instance contains a query and passages retrieved by BM25 (Robertson & Zaragoza, 2009) from the Internet. Each passage has annotations of a relevance label—perfect, highly relevant, or not relevant. We determine the number of passages k from the input length L, and randomly sample k passages with balanced labels for each test query.

The model is prompted with the query and the shuffled k passages and is instructed to output the Many-shot in-context learning (ICL). In-context learning (ICL) is a key ability that enables LLMs to adapt to new tasks on the fly (Brown et al., 2020). Recent studies (Ratner et al., 2023; Xu et al., 2024; Li et al., 2024c; Bertsch et al., 2024) explore performing many-shot ICL (with thousands of examples) with LCLMs. Following Bertsch et al. (2024), we focus on datasets with large la- bel spaces: TREC-coarse, TREC-fine (Li & Roth, 2002), BANKING77 (Casanueva et al., 2020), CLINC150 (Larson et al., 2019), and NLU (Liu et al., 2019). We adjust the number of shots to con- trol the input length L, and the number of examples in each class is balanced. We report accuracy on the test set.

Published As A Conference Paper At Iclr 2025

One difference from previous works is that we map original natural language labels (e.g., location) into numbered labels (i.e., 0, 1) to test how well a model can learn new tasks instead of relying on its pre-trained priors (Wei et al., 2023; Pan et al., 2023; Min et al., 2022). More details are in §B.4.

Table 4: Dataset lengths.

5M

Long-document question answering (LongQA). We use Narra- tiveQA (Koˇcisk´y et al., 2018) and the English book QA and multiple choice (MC) subsets from ∞BENCH (Zhang et al., 2024b) for evaluat- ing long-document QA. We select those tasks for their abundant con- text lengths (Table 4). We truncate the document from the end based on L. We use ROUGE F1 for ∞BENCH QA (answers are mostly en- tity names) and accuracy for ∞BENCH MC. For NarrativeQA, where the answers can be long text and open-ended, we design and use a model-based evaluation (§2.2).

Summarization (Summ).

Summarization Tests Lclms’ Ability To

synthesize information across the contexts. We choose Multi-LexSum (legal document summarization) and the English summarization task from ∞BENCH (novel summarization) for their extensive lengths (Ta- ble 4). We truncate the document from the end based on the evalu- ation length L. We use our model-based evaluation (§2.2) for both datasets instead of the commonly used ROUGE, as it better reflects human judgment.

Synthetic recall. Synthetic recall tasks, such as needle-in-a-haystack (NIAH), stress test models’ ability to recall relevant information (the “needle”) from long contexts. They have gained popularity for being easy to use (as they can test any arbitrary length) and easy to control (can placing the “needle” at any position). For this category, we select multiple synthetic recall tasks from RULER (an extended version of NIAH; Hsieh et al., 2024) and also add a JSON KV retrieval task (Liu et al., 2023), which we find more challenging. We intentionally select the synthetic tasks that correlate well with application-driven tasks; in-depth discussions are in §3.1. Following previous works, we report the percentage of the ground truth answers that are substrings in the generation (SubEM).

Refer to §B.5 for more details.

Reliable Evaluation Metrics

Existing long-context benchmarks (Zhang et al., 2024b; Shaham et al., 2023) largely rely on n-gram overlap metrics like ROUGE (Lin, 2004), which have been shown to correlate poorly with human judgment for tasks with long outputs, such as summarization (Goyal et al., 2023; Deutsch et al., 2022; Krishna et al., 2023). L-Eval (An et al., 2024) uses LLMs to score reference-free “win rates,” which neglect the available answer annotations and always require evaluating model pairs. Instead, we design a reference-based model evaluation method for long-document QA and summarization that is more reliable and easy to use.

Question answering. In NarrativeQA, we prompt GPT-4o5 with the question, the ground truth, and the model output to check for fluency and correctness. The fluency score is either 0 (incoherent or repetitive) or 1 (fluent), and the correctness score takes on the value of 0 (incorrect), 1 (partly correct), 2 (correct but not fully relevant), and 3 (correct and relevant). We take the product of the two as the final score, normalizing it to a range of [0, 100].

Summarization. Following previous works (Kamoi et al., 2023; Zhang & Bansal, 2021), we first decompose the gold summary into atomic claims and use GPT-4o to check if each claim is supported by the generation (recall) and if each sentence in the generation is supported by the reference sum- mary (precision). We then compute the F1 score from the recall and precision scores. Additionally, we ask GPT-4o to evaluate fluency (0 or 1) and take its product with the F1 score as the final score.

In each step, we prompt GPT-4o with handwritten examples.

Multi-Lexsum

Figure 2: Comparison between ROUGE-L F1 and our model-based evaluation metric on summa- rization tasks. Our metric shows more consistent trends: it reflects the performance gain on GPT-4o with increased input length, while ROUGE remains almost the same; our metric also clearly differ- entiates models while ROUGE shows little distinction.

Empirically, our reference-based model evaluation reflects more consistent trends, as shown in Fig- ure 2: (1) Llama-3.1-8B-Inst achieves similar ROUGE scores to GPT-4o, while our evaluation re- veals a significant gap. (2) Our metric better identifies incoherent generations and shows lower performance for models with smaller context windows, such as the Mistral models. (3) Our metric exhibits a substantially more positive trend for GPT-4o as input length increases, whereas ROUGE- L remains within a 2-point absolute difference. We further validate the model-based evaluation through human studies, which suggest our new metrics strongly correlate with human judgments: for example, on ∞BENCH Sum, our metric reaches a human-model agreement of Cohen’s κ = 0.91 for summary precision and κ = 0.76 for recall. More details on the human studies are in §B.

Robust Prompting And Controlled Evaluation Settings

Robust prompting reduces noise and enables evaluation on base models. Many long-context benchmarks require models to follow instructions and only support evaluating instruction-tuned models (Zhang et al., 2024b; Shaham et al., 2023). However, many model developments do not incorporate instruction tuning (Chen et al., 2023; Fu et al., 2024), leaving these models reliant on perplexity-based evaluation or synthetic tasks. To support long-context research efforts, we design our benchmark so that at least a subset of the datasets accommodates evaluating base models.

Existing benchmarks mostly use zero-shot prompting (Shaham et al., 2023; Zhang et al., 2024b), which leads to inconsistent output formats, especially for base models. For example, the model may output a long answer in RAG when a short answer is required. We add two-shot demonstrations in the prompt for all tasks to address this problem.6 For long-document QA and summarization, we replace the original document with a placeholder phrase to reduce the number of input tokens in the ICL example. As shown in Table 8, both base and instruction-tuned models significantly benefit from the demonstrations.

Furthermore, we employ the length-instruction-enhanced evaluation from L-Eval for long- generation tasks (i.e., summarization), which has been shown to have substantially more consistent and reliable evaluations (An et al., 2024). As a result, we find that our reproduction of previous datasets, such as ∞BENCH QA tasks, better reflects the capabilities of LCLMs, as shown in Table 7.

The use of demonstrations and improved instructions more accurately depicts how models perform in real applications. Controlled input length and difficulty. An important dimension to consider when evaluating LCLMs is the input length L, as longer inputs can provide more information while challenging the model’s ability to process distracting contexts. As we discussed in §2.1, we can control the input length L for each task by either adjusting the number of retrieved passages, the number of demon- strations, or truncating the document text to fit within the specified lengths. This allows us to study model performance at or beyond the length of current frontier LCLMs (≥128K).

6Except for ICL (the number of shots varies) and RULER (we follow the original formatting).

Published As A Conference Paper At Iclr 2025

As shown in Figure 1 and 2, HELMET provides rankings more consistent with human perception of model performance. The diverse tasks, controllable lengths, and robust evaluation metrics and prompting enable a thorough examination of LCLMs across multiple dimensions.

Analysis

We evaluate 59 LCLMs with HELMET. To our best knowledge, this is the most thorough and controlled comparison of long-context models on diverse applications. These models cover closed- source models, such as GPT-4, Claude, and Gemini, as well as open-source model families, such as Llama (Dubey et al., 2024), Mistral (Jiang et al., 2023), Phi (Abdin et al., 2024), and Qwen (Qwen et al., 2025). We also consider models that use different architectures—full-attention transformers (Vaswani et al., 2017), sliding-window attention (Beltagy et al., 2020), and hybrid models with SSM modules (Dao & Gu, 2024). We also benchmark position extrapolation models such as YaRN (Peng et al., 2024) and LongRoPE (Ding et al., 2024). We list all the models evaluated in Table 15. We evaluate each model at input lengths: L ∈{8K, 16K, 32K, 64K, 128K}, where L is the number of Llama-2 tokens (Touvron et al., 2023), and use greedy decoding for all models to ensure consistency.

We randomly sample 100 to 600 examples from each dataset; more details are in §D.

3.1

SIMPLE SYNTHETIC TASKS ARE POOR PREDICTORS OF REAL-WORLD PERFORMANCE

Summ

Avg.

At 128K Input Length, Calculated Across

35 instruction-tuned models. Many model developers rely on simple synthetic tasks, such as NIAH, for evaluating long-context language mod- els, but it is unclear if these tasks accurately represent real-world performance. To this end, we calculate Spear- man’s rank correlation ρ between synthetic and real- world tasks for 35 instruction-tuned models. First, Fig- ure 3 shows that none of the synthetic tasks achieves an average correlation higher than 0.8. We also make the following observations.

Not all synthetic tasks are created equal. The original NIAH, which places a needle in the middle of unrelated essays and asks the model to retrieve it, exhibits weak cor- relation with real-world tasks: all correlations are ≤0.8.

Similarly, the popular RULER average score—which in- cludes not only NIAH variants but also synthetic aggre- gation, multi-hop tracing, and QA—does not yield strong correlations (all < 0.85).

We take a closer look at different RULER tasks and find that harder recall-type tasks are more reflective of real-world categories—for example, RULER MK, which places distracting needles around the target needle. Despite the overall low correlation, we believe these tasks can still serve as a useful sanity check during model development. We compile several such RULER tasks, along with JSON KV, to form the HELMET synthetic recall set (more discussions in §E.1).

Tasks with noisier, more distracting contexts better differentiate models. To understand why synthetic tasks exhibit weak correlation with real-world tasks, we plot the performance of different models on NIAH, RULER MK (one of our recall tasks), and HotpotQA (one of our RAG tasks) in Figure 4. We use ∞BENCH QA as a representative real-world task. We find that most models achieve either perfect or near-zero performance on the original NIAH, leaving few data points in the middle and resulting in poor separability between models. In contrast, RULER MK, which introduces more distracting contexts, better distributes model performance between 0% and 100%, leading to clearer differentiation.

RAG is a better proxy for real-world tasks. Finally, we find that RAG datasets, such as HotpotQA, consistently achieve higher correlation with other real-world tasks. Figure 4 also shows that Hot- potQA exhibits an almost linear relationship with the QA dataset. Similar to synthetic tasks, RAG

P=2.9E-12

Figure 4: Distribution of instruction-tuned models’ performance on ∞BENCH QA with respect to NIAH, RULER MK, and HotpotQA. tasks are easy to control and assess models’ recall abilities. However, since all passages are retrieved and relevant to the query, RAG contexts are more distracting and therefore harder to saturate.

Diverse Lclm Applications Call For Diverse Evaluation

In long-context language modeling, realistic tasks are often only used in isolation (Karpinska et al., 2024; Li et al., 2024c; Dubey et al., 2024), which limits the understanding of LCLMs in a broader context. In this work, we cross-examine model performance over a wide range of real tasks, and find that different categories do not consistently correlate with each other, as shown in Figure 5.

Figure 5: Spearman Rank Correlation Between

different categories at L =128K.

Trieval Component. However, The Added Complexity

of generating citations in ALCE results in lower cor-

Relation With Other Categories.Naturally, Rag And

passage re-ranking moderately correlate due to the

Shared Retrieval Component As Shown In Figure 9,

generating correct answers and producing valid ci- tations are not strongly correlated, suggesting that instruction following and recalling facts within long contexts are distinct capabilities.

Tations And In-Context Learning—Do Not Correlate

well with other categories. Intuitively, summariza- tion tests for the model’s ability to aggregate infor- mation across the entire input, while ICL evaluates its ability to learn new tasks from many examples.

Such capabilities are orthogonal to recall facts in long contexts. Such capabilities are orthogonal to recalling facts in long contexts. Therefore, model developers should evaluate across these distinct axes to form a more holistic understanding of a model’s capabilities (see additional analysis in §E.2).

Model Performance Across Tasks And Lengths

We show the performance of instruction-tuned models on HELMET at five different lengths in Figure 6, and the full results are illustrated in Figure 10. We analyze the model performance across two critical dimensions of long-context language modeling: task complexity and input length.

Open-source models lag behind closed-source models on complex tasks. First, we consider the performance of frontier LCLMs at the longest input length of 128K tokens. We find that the closed- source models—notably GPT-4o-08 and Gemini-1.5-Pro—stand out as the strongest LCLMs. Other than ICL, the closed-source models outperform the open-source models on all tasks. The gap is relatively small on synthetic recall and LongQA, where the task is to retrieve information from the context. There is a stark contrast in the generation with citations and re-ranking performance, where the closed-source models are 30 to 40 absolute points better than the best open-source models.

49.4

Avg. Figure 6: Results of HELMET. All models are instruction-tuned and have a claimed context win- dow of 128K tokens or more.

Performance degradation with longer inputs is category-dependent.

Most Frontier Models

largely retain performance on recall and RAG with longer inputs; however, even the best models experience significant degradation as context length increases on tasks like re-ranking and genera- tion with citations. As illustrated in Figure 7, performance degradation at longer lengths becomes more pronounced as task complexity increases from left to right. On generation with citations, open-source models completely collapse at 128K, while GPT-4o remains relatively stable. This underscores the importance of evaluating models on more complex long-context applications.

No clear winner across all categories. As we observe from the previous sections, the different cate- gories do not always correlate with each other. This is evident in the varying top-performing models across categories: for instance, GPT-4o excels in recall and generation with citations, while Gem- ini performs better in passage re-ranking and long-document QA. Furthermore, many open-source models outperform closed-source models on ICL, potentially because heavy instruction tuning neg- atively impacts ICL. We provide qualitative examples in Table 19. Thus, evaluating models across multiple axes is essential. In the appendix, we also present additional analysis, such as the perfor- mance of positional extrapolation methods (§E.3), the lost-in-the-middle phenomenon (§E.4), and the performance of Claude (§E.6).

Cite

Figure 7: Results of selected instruct models on various lengths and increasing complexity of tasks. Notably, Qwen2 relies on position extrapolation, while other open-models are trained at or greater than 128K context window.

Related Works

Long-context language models. Frontier models such as GPT-4 (OpenAI, 2023), Gemini (Team et al., 2024b), and Claude claim to have expanded their context window beyond 100K tokens. In the open-source community, there are also efforts to train models with longer input lengths (Dubey et al., 2024; Fu et al., 2024; AI et al., 2024; Gao et al., 2024), explore position extrapolation techniques (Peng et al., 2024; Chen et al., 2023; Ding et al., 2024), and experiment with efficient architectures (Beltagy et al., 2020; Bertsch et al., 2023; Gu & Dao, 2024; Dao & Gu, 2024; Yen et al., 2024; Lieber et al., 2024, inter alia).

Synthetic tasks. Synthetic tasks are often used to evaluate LCLMs since they can be procedurally generated, enabling arbitrarily long input lengths and controlled “needle” placement (Tay et al., 2021; Liu et al., 2023). In particular, Needle-in-a-Haystack (NIAH; Kamradt, 2024) inserts a “nee- dle” at specific depths of a long essay (i.e., the haystack) and asks the model to recall the fact. Recent works have expanded upon it and designed new procedures to test different aspects of LCLMs (Hsieh et al., 2024; Li et al., 2024b; Levy et al., 2024; Arora et al., 2023; Laban et al., 2024; Goldman et al., 2024, inter alia). However, they do not study how results on synthetic tasks transfer to real ap- plications. In contrast, we evaluate both synthetic and downstream tasks and investigate how they correlate with each other.

Long-context benchmarks. As discussed in the main paper, existing benchmarks either are limit to relatively short context lengths (Shaham et al., 2022; 2023; Li et al., 2024a; An et al., 2024; Bai et al., 2024; Dong et al., 2024; Kwan et al., 2024) or lack rigorous evaluation methods (Zhang et al., 2024b; Yuan et al., 2024). Many works instead focus on specific domains, such as question answering (Wang et al., 2024b; Karpinska et al., 2024; Wang et al., 2024a;c), in-context learning (Li et al., 2024c; Bertsch et al., 2024; Xu et al., 2024; Anil et al., 2024; Agarwal et al., 2024), summarization (Chang et al., 2024; Kim et al., 2024; Shen et al., 2022), or RAG (Lee et al., 2024).

In this work, we construct a comprehensive benchmark that tests across diverse downstream tasks at long input lengths and also present a unified comparison and analysis across 59 LCLMs.

Conclusion

In this work, we first identify the shortcomings of long-context evaluation settings and existing benchmarks—over-reliance on synthetic tasks, limited coverage of realistic applications, and un- reliable metrics among others. We seek to address these issues by constructing HELMET, an application-centric benchmark with diverse domains and reliable evaluation settings.

We Then

present a comprehensive evaluation of 59 frontier LCLMs across multiple dimensions, including different tasks, input lengths, and model types. Our analysis shows that synthetic tasks poorly pre- dict downstream performance and that different categories exhibit distinct capabilities and trends.

Thus, evaluating models on a diverse set of real-world tasks is essential. Furthermore, open-source models still lag behind closed-source models on complex tasks at longer lengths. Finally, we hope that our benchmark and comprehensive evaluation serve as a valuable resource for future research in long-context language modeling.

Reproducibility Statement

We have publicly released all of our code and data to ensure reproducibility. Experimental settings are outlined in the main text (§3) and the appendix (§D), and we will provide additional details and instructions in the code repository. Furthermore, we plan on releasing the model generated outputs and statistics to enable further analysis and comparison.

Acknowledgments

We would like to acknowledge Mengzhou Xia, Howard Chen, Xi Ye, Yinghui He, Lucy He, Alexan- der Wettig, Sadhika Malladi, and other members of Princeton Language and Intelligence for their helpful feedback and discussion. We also thank Adithya Bhaskar and Joie Zhang for helping with validating the model-based evaluation. Tianyu Gao is supported by an IBM PhD Fellowship. This work is gratefully supported by an NSF CAREER award (IIS-2239290), a grant from Intel, and the Microsoft Accelerate Foundation Models Research (AFMR) grant program.

Related Journal Articles & DOI Links

Selected peer-reviewed publications relevant to 12 Lead ECG Acquisition. Click the DOI to access the full paper (may require institutional access).

Why Choose Us?

Bangalore guidance for robotics, Spectre and autonomous systems projects.