Large language models (LLMs) have shown remarkable general capabilities (Brown et al., 2020;Wei et al., 2022;Dubey et al., 2024) and have advanced multilingual machine translation (Xu et al., 2024a;Yang et al., 2023;Alves et al., 2024). For example, Aya-101 (Aryabumi et al., 2024) expands support to 101 languages and achieves strong performance in multilingual machine translation, while LLaMAX (Lu et al., 2024b) further pushes performance beyond 100 languages. The common practice behind these successes is the large-scale pretraining, which typically involves monolingual pretrainingfoot_0 , parallel pretraining, or both-followed by a small-scale, high-quality posttraining phase. However, as LLMs scale to more languages, they suffer from the issue of Curse of Multilinguality (CoM) (Conneau, 2019), which degrades the translation performance.
Understanding and mitigating CoM is not new in the MMT literature. In traditional MMT, existing research has identified critical factors such as resource imbalances, limited model capacity, linguistic similarity, and complex interactions between language pairs, particularly for low-resource languages (Arivazhagan et al., 2019;Aharoni et al., 2019;Shaham et al., 2023;Meng and Monz, 2024), and proposed solutions including language-specific modules (Fan et al., 2021;Zhao et al., 2024;Xu et al., 2023), vocabulary optimization (Han et al., 2024), data sampling techniques (Wang et al., 2020;Wang and Neubig, 2019;Lin et al., 2019), and continual learning approach (Liu et al., 2023). Based on these studies, recent LLM-based MMT research focuses on designing increasingly complex training pipelines and modular architectures. For instance, Xu et al. (2024b) proposed a five-stage training pipeline incorporating language-specific modules. However, existing analyses primarily focus on the encoder-decoder paradigm, while current LLM-based approaches heavily rely on scaling up model capacity and computational resources, making them prohibitively expensive. This raises a critical question: Is scaling up the training and computation budget truly necessary for high-quality MMT, or can a deeper understanding of CoM in LLM-based MMT provides a more efficient solution?
In this work, we systematically investigate linguistic conflicts and synergy during post-training phase. We conduct extensive experiments with different settings: across 5 to 50 languages, three pretrained LLMs -ALMA-7B-Pretrain, ALMA-13B-Pretrain and X-ALMA-13B-Pretrain, three distinct post-training strategies -multilingual training, group multilingual training, and separate training. We observe a consistent pattern: asymmetry in linguistic conflicts and synergy (Figure 2, Appendix B.1and B.2 ). For example, in multilingual training, XX→En translation directions experience significant linguistic conflicts, leading to performance degradation, whereas En→XX translations benefit from linguistic synergy, where XX denotes 49 different languages other than English. We further show this asymmetric phenomenon cannot be easily mitigated through existing training approaches, such as group multilingual training (Table 1). This finding illustrates the need to develop a direction-aware training strategy for optimal post-training.
Another key finding of our work is that a simple multilingual pre-training stage can be sufficient to equip foundation models with ideal multilingual capabilities, whereas the bottleneck lies in the post-training stage (dotted lines in Figure 2 (g-i)). Motivated by these findings, we propose a novel Direction-Aware Training (DAT) approach and build an efficient MMT starting from a relatively efficient base model, the X-ALMA-13B-Pretrain-utilizing only simple multilingual pretraining on 20 billion tokens. Our approach fully leverages the interactive characteristics of different language directions to reduce conflicts while maximizing synergy. We also present a scalable version of the approach, named DATM, which utilizes model merging to further enhance efficiency with only negligible performance degradation.
Through comprehensive evaluations on Flores-200 and WMT23 Benchmark, we demonstrate the effectiveness of our approach. Notably, as shown in Figure 1, compared to X-ALMA (Only SFT) (Xu et al., 2024b), our model X-ALMA-13B-DAT maintains comparable performance while having two advantages: 1) utilizing a simple and efficient training recipe -starting from base models with fewer pre-training tokens and employing a post-training stage. 2) parameter-efficient -we consume 1.7x fewer parameters compared to X-ALMA (Only SFT). These results demonstrate that simple pre-training combined with dedicated post-training can also achieve good multilingual performance.
In this section, we introduce the basic experimental settings used in Section 3 and Section 4.
We use the high-quality parallel dataset curated by Xu et al. (2024b), covering fifty languages across low-, medium-, and high-resource categories. Following (Xu et al., 2024b), these languages are grouped into eight linguistic groups based on linguistic similarity and a balanced number of languages. Details are provided in Section A in Appendix. The dataset primarily consists of samples from the Flores-200 development set and NTREX (Barrault et al., 2019). For languages in both Flores-200 and WMT'15-22, corresponding test sets are incorporated, yielding an average of 4K examples per language. For evaluation, we use Flores-200 and WMT23 benchmarks to assess performance.
We select three representative fully open multilingual LLMs for our study: ALMA-Pretrain (Xu et al., 2024a) (7B-13B parameters) and X-ALMA-Pretrain (Xu et al., 2024b) (13B parameters). The ALMA-Pretrain models were pre-trained on 12B or 20B tokens across six languages, while X-ALMA-Pretrain underwent continued pre-training on 20B tokens from 50 languages, both based on LLaMA-2. We exclude other state-of-the-art multilingual models for two key reasons: (1) their pre-trained checkpoints are unavailable, as in the case of Ayaseries (Aryabumi et al., 2024) and BigTrans (Yang et al., 2023); or (2) they exhibit suboptimal multilingual performance in certain languages as shown in Xu et al. (2024b); Cui et al. (2025), such as LLaMA-3 (Dubey et al., 2024).
Fine-tuning Strategies We employ three distinct training strategies for fine-tuning the models: Multilingual Training, Separate Training, and Group Multilingual Training. • Multilingual Training (Tang et al., 2020): This is typically achieved by mixing data from all languages and using it to fine-tune the model. The resulting model is a single model that possesses shared representations across all languages. • Group Multilingual Training (Xu et al., 2024b): We group the languages and then apply multilingual training within each group, resulting in multiple models, each for its respective languages. • Separate Training: Separate tuning involves training a distinct model for each language without considering linguistic synergies or conflicts. Training Configurations In this work, all models are trained with a learning rate of 2e-3 using an inverse square root scheduler, a weight decay of 0.01, and a warmup ratio of 0.01. The total batch size is set to 128. Fine-tuning is conducted for 1 epoch, with both max_new_tokens and max_source_length set to 512. Additionally, FP16 precision training is enabled to optimize performance and efficiency. All models are trained on 4 NVIDIA H100 with LoRA (Hu et al., 2022) as Xu et al. (2024a) has shown a negligible performance gap between LoRA tuning (Hu et al., 2022; Zhang et al., 2025) and full fine-tuning.
We set the number of beams to 5 and both max_new_tokens and max_source_tokens to 512. We evaluate performance mainly using COMET-22 (Rei et al., 2022) and SacreBLEU (Post, 2018).
3 The Phenomenon: Asymmetry in Linguistic Conflicts and Synergy
In this section, we investigate the phenomenon of Asymmetry in Linguistic Conflicts and Synergy in LLM-based MMT. We begin by illustrating the phenomenon (Section 3.1) and analyzing its distribution across two essential factors: language resources and groups (Section 3.2). Finally, we show how this phenomenon poses challenges to existing post-training strategies (Section 3.3).
We investigate linguistic conflicts and synergy during the post-training phase. To explore this, we utilize three foundation models, as mentioned in Section 2.2, to perform multilingual training with training datasets that include a range of languages, from 5 to 50, and evaluate the average performance on corresponding languages.
To quantify linguistic conflicts and synergy, we compare multilingual training with separate training, where each language pair is trained independently, eliminating cross-lingual interactions.
• Linguistic Conflicts: If multilingual training underperforms compared to separate training (i.e., COMET drop), conflicts dominate over synergy. • Linguistic Synergy: If multilingual training outperforms separate training, synergy dominates. • Intensity: the magnitude of the performance gap measures the strength of conflicts/synergy. Figure 2 displays the results. We can have the following observations: • Key Findings 1: Asymmetry in Linguistic Conflicts and Synergy. As shown in Figures 2 (a), (d), and (g), the average performance decreases with an increase in the number of languages, a phenomenon known as the CoM (Conneau, 2019; Xu et al., 2024b). However, by decomposing the average performance across all language directions into XX→En and En→XX, we uncover an intriguing asymmetry in the distribution of linguistic conflicts and synergies, as illustrated in Figures 2 (b), (c), (e), (f), (h), and (i). Specifically, in the XX→En direction, linguistic conflicts are more dominant, as shown by multilingual training consistently underperforming separate training. Conversely, in the En→XX direction, linguistic synergy is significant, with multilingual tuning consistently outperforming separate training. Furthermore, comparing different models reveals that increasing model capacity (e.g., from 7B to 13B) or incorporating more languages in the pre-training corpus can mitigate conflicts. However, a significant gap remains between separate and multilingual tuning, indicating that simply increasing model capacity and the ALMA-7B-Pretrain ALMA-13B-Pretrain X-ALMA-13B-Pretrain High Mid Low 0 5 10 15 20 (a) Resource Level ∆ COMET-22 1 2 3 4 5 6 7 8 0 5 10 15 20 (b) Linguistic Feature Figure 3: ∆ COMET-22 between separate training and multilingual training in XX → En translation, grouped by resource level and linguistic features. The magnitude of ∆ COMET-22 denotes the intensity of linguistic conflicts. work (Xu et al., 2024b) designed complex training regimens with up to five stages, including three pre-training and two post-training stages with language-specific group training, to address this issue. In contrast, our findings suggest there may be a more efficient way to tackle the CoM.
For example, we could start with a base model that only undergoes multilingual pretraining and then apply a dedicated post-training approach to achieve high-quality translation.
We further address a key question: Does Asymmetry in Conflicts and Synergies occur across all language pairs, or is it concentrated in specific pairs?
To answer this, we analyze its distribution across different language groups and resource levels. Figure 3 displays the results. We can have the following observations: • Asymmetry in linguistic conflicts is consistently observed across languages with varying resource levels and language groups, but its intensity is not uniformly distributed. • While increasing model capacity or pre-training data can help narrow the performance gap, consistent with findings in previous work (Arivazhagan et al., 2019;Aharoni et al., 2019;Shaham et al., 2023;Meng and Monz, 2024), a substantial gap of nearly 10 COMET-22 points still persists.
The asymmetry in linguistic conflicts and synergies may pose challenges for LLM-based
MMT, leading to suboptimal performance for existing posttraining approaches. Intuitively, translation directions where linguistic conflicts dominate may benefit from post-training strategies that minimize such conflicts. Conversely, translation directions where linguistic synergies prevail may require strategies that effectively enhance high-quality synergy. To see this, we fine-tune foundation models using three key approaches: multilingual training, group multilingual training, and separate training on 50 languages and compare their performance. Table 1 displays the experimental results on the Flores-200 test set. We observe the following: • Key Findings 3: The effectiveness of the existing training strategy exhibits an asymmetrical pattern.: In XX→En translations, separate training consistently achieves the best performance, followed by group multilingual training, while full multilingual training performs the worst. This result is expected, as linguistic conflict is prominent in these translation directions. En-De De-En En-Zh Zh-En ... ... En-Fr Fr-En Data Selection Base Model LoRA1 LoRA2 ... LoRAN + + + Task1 Task2 ... TaskN (a) Seperate/Multilingual/Group Training En-De De-En En-Zh Zh-En ... ... En-Fr Fr-En Group Selection Base Model LoRA1 LoRA2 ... LoRAN G + + + Task enxx 1 Task enxx 2 ... Task enxx NG Base Model LoRA2 LoRA1 ... LoRAN L + + + Group-wise Merged Model Table 3-4, with an adapter trained for each group. (b) Group-wise model merging: For XX→En translation, separate training is applied to each language pair. For En→XX translation, group training is applied, where different tasks share LoRA weights within language groups. By contrast, in En→XX translations, multilingual training or group multilingual training consistently outperforms separate training. This indicates that while linguistic conflicts dominate in the XX→En direction, the En→XX direction benefits from cross-linguistic knowledge transfer, leading to an enhanced translation quality. When model capacity is sufficiently large, the general pattern observed is: group multilingual training > multilingual training > separate training.
This highlights two things: 1) linguistic similarity benefits positive cross-linguistic transfer. 2) the widely adopted group multilingual training approach remains insufficient to address the challenges posed by the asymmetry. These findings underscore the critical impact of asymmetry in linguistic conflicts and synergy phenomenon on the effectiveness of existing training strategies, highlighting the need for novel training approaches to consider such an asymmetry to achieve optimal performance in both directions.
In this section, we show how to construct an efficient MMT system by leveraging the insights from Section 3, starting from a base model with simple multilingual pre-training.
As demonstrated in Section 3, linguistic conflicts and synergy exhibit asymmetry during the posttraining stage, posing significant challenges to multilingual translation. A widely adopted technique to mitigate conflicts and enhance synergy is language-specific group multilingual training (Fan et al., 2021;Zhao et al., 2024;Xu et al., 2023Xu et al., , 2024b). However, it still achieves sub-optimal performance. The state-of-the-art XALMA system (Xu et al., 2024b) achieves high-quality translations by employing eight large language-specific adapters within a MoE framework combined with group multilingual training. However, this approach incurs high computational and storage costs, as each adapter contains up to 15% of the base model's parameters, making large-scale deployment challenging. Additionally, XALMA requires a massive amount of tokens during pre-training, further increasing resource consumption. This raises an important question: Can we achieve comparable high translation quality in a more efficient manner?
Intuitively, we could develop a more efficient training approach for high-quality MMT by considering the asymmetry in linguistic conflicts and synergy. To this end, we propose a direction-aware training framework combined with model merging, which fully leverages the inherent asymmetry to enhance both performance and efficiency. Our approach primarily consists of two key components: 1) Direction-aware training strategies for efficiently and effectively mitigating linguistic conflicts and encouraging linguistic synergy and 2) Group-wise model merging for running efficiency.
As shown in Figure 4
Although the direction-aware training approach achieves promising performance, the number of LoRA weights increases linearly with the number of supported languages, posing challenges for deployment and inference, especially at large language scales. Model merging (Yadav et al., 2024;Zhang et al., 2023) provides a feasible solution to reduce the number of LoRA weights and improve efficiency. However, directly using model merge for efficient MMT is non-trial. In our preliminary experiments, we have two key observations: • Merging LoRA weights into one for each direction leads to performance degradation. Notably, Dang et al. (2024) find that model merging can improve performance, contrasting our findings. However, this discrepancy may arise because their comparison is against a weaker baseline, such as multilingual training, whereas we compare against the most vigorous baseline-separate training. • The degradation effect of model merging exhibits an asymmetric nature. The performance degradation per parameter in the En→XX direction is 6.86× greater than in the XX→En direction. A potential explanation is that linguistic syn-ergy plays a crucial role in En→XX directions, while model merging introduces low-quality linguistic synergy, leading to a performance drop. Therefore, a more dedicated design is needed to preserve performance as much as possible.
Motivated by these observations, we only apply model merging to XX→En directions in a groupwise manner. Specifically, we apply model merging to languages within each group, resulting in N G LoRA weights. We adopt the TIES (Yadav et al., 2024) for model merging. We also compare this approach with other methods such as DARE-TIES (Yu et al., 2024) and find no significant performance difference. With this approach, we can reduce the number of LoRA weights from O(N L ) to O(N G ), improving scalability while lead minimal performance degradation.
We evaluated our models using the Flores-200 test set for 50 languages and the WMT23 test sets for five languages (de, ru, uk, ja, zh). We provide more details in Appendix A. We select existing state-ofthe-art open multilingual MT system as baselines:
• Aya-101 (Üstün et al., 2024) Table 2 shows the results. We can have the following observations: • Both X-ALMA-13B-DAT and X-ALMA-13B-DATM can achieve high translation performance. Compared to previous multilingual LLMs, such as Aya-101, Aya-23-8B, and LLa-MAX, our approach consistently outperforms them across both benchmarks and translation directions. Moreover, compared to X-ALMA, our X-ALMA-13B-DAT achieves comparable performance in XX→En directions; however, in En→XX, a significant performance gap remains, up to 0.95 COMET-22 on average. • Our approach provides an efficient way to build effective MMT. Our model is built upon X-ALMA-13B-Pretrain with only 20 billion tokens of simple multilingual pre-training. Moreover, it utilizes multiple small LoRA weight compositions and achieves relatively high translation performance across all directions, which is consistent with previous work (Zheng et al., 2024a) 5 Related Work
Existing research has explored both understanding and addressing this issue in MMT, identifying critical factors such as resource imbalances, limited model capacity, and complex interactions between language pairs, particularly for low-resource languages (Arivazhagan et al., 2019;Aharoni et al., 2019;Shaham et al., 2023). Interestingly, studies have shown that while linguistic similarity enhances positive transfer, dissimilar languages can also act as regularizers, improving training stability (Meng and Monz, 2024). To address these challenges, proposed solutions in recent research include language-specific modules (e.g., adapters, sparse experts) to dynamically allocate capacity and reduce interference (Fan et al., 2021;Zhao et al., 2024;Xu et al., 2023), vocabulary optimization to better support new languages through improved token representations (Han et al., 2024), data sampling techniques to enhance representation for underrepresented languages (Wang et al., 2020;Wang and Neubig, 2019;Lin et al., 2019) and continual learning techniques (Liu et al., 2023). Notably, techniques, such as language-specific modules, have been integrated into LLM-based MMT systems, resulting in substantial improvements in multilingual performance (Xu et al., 2024b). In this work, we systematically investigate how posttraining in LLM-based MMT contributes to the CoM, providing a fine-grained analysis of its impact on linguistic conflicts and synergies.
Prior to the emergence of LLMs, neural machine translation (NMT) research predominantly focused on optimizing encoder-decoder architectures and attention mechanisms (Li et al., 2022;Zheng et al., 2024a,b;Li et al., 2024) or decoding strategy (Reheman et al., 2023;Mu et al., 2023). Recently, many efforts have been made to adapt LLMs for effective machine translation. A key approach is prompting, which enhances translation performance without additional training (He et al., 2024;Lu et al., 2024a). Beyond this, growing research focuses on fine-tuning open and smaller LLMs to achieve high translation quality while ensuring efficiency (Xu et al., 2024a;Yang et al., 2023;Alves et al., 2024;Aryabumi et al., 2024;Luo et al., 2025). Yang et al. (2023) propose a training pipeline that integrates monolingual pre-training to improve language modeling and parallel instruction finetuning for enhanced translation performance. Similarly, Xu et al. (2024a) emphasize the quality over quantity of parallel data, introducing a training recipe: (1) large-scale monolingual pre-training, followed by (2) small-scale, high-quality parallel fine-tuning. Further revisiting the role of parallel data, Guo et al. (2024) highlights its importance in the pre-training stage. Additionally, Xu et al. (2024c) underscore the necessity of alignment in post-training, proposing the CPO algorithm. More recently, with the need to scale models across more languages, Xu et al. (2024b) introduces languagespecific modules combined with group training to mitigate language conflicts. In this work, we focus on the post-training stage, which has been underexplored in previous studies, and propose a directionaware training approach with model merging to achieve efficient and effective MMT.
In this work, we systematically investigate linguistic conflicts and synergy during post-training in LLM-based MMT and identify a phenomenon we term asymmetry in linguistic conflicts and synergy. We provide an in-depth analysis of its distribution and challenges for LLM-based MMT. Based on these insights, we propose a direction-aware training approach combined with model merging to build an effective MMT system from X-ALMA-13B-Pretrain with only multilingual pre-training. Our approach highlights the importance of posttraining in LLM-based MMT and offers insights into building MMT resource-efficiently.
One limitation of this work is that our approach does not surpass state-of-the-art methods like X-ALMA in performance, particularly in En→XX directions, despite requiring less training cost and fewer model parameters. Second, while this work identifies a novel phenomenon and designs an efficient approach leveraging it, it does not provide a deeper or more rigorous analysis of why asymmetry in linguistic conflicts and synergy exists. We leave the analysis of the underlying mechanism of asymmetry in linguistic conflicts and synergy for future work.
Additionally, although this work conducts extensive experiments on fifty languages and three pre-trained models, further scaling is necessary to validate our findings on a broader scale, such as extending to over 100 languages. This would help push the boundaries of multilingual machine translation research, which we also leave for future work.
In this section, will discuss the detailed setup of our experiment, including the datasets.
A.1 Details of Dataset in Section 2.1 Following (Xu et al., 2024b), we present a classification of languages based on linguistic families, scripts, and resource availability in Tables 34. Fifty languages are grouped into eight distinct categories, primarily guided by linguistic similarity while considering a balanced distribution of languages across groups. Each group encompasses a mix of low-, medium-, and high-resource languages to ensure comprehensive multilingual coverage. Additionally, English is included in each group to facilitate English-centric translation and mitigate catastrophic forgetting. This structured grouping provides a well-rounded dataset for multilingual research, enabling robust language modeling and cross-lingual transfer learning.
We train the translation model on X-ALMA-Parallel-Data, a parallel dataset in (Xu et al., 2024b). The distribution of the parallel datasets for each language is illustrated in Figure 5.
The evaluation dataset primarily consists of samples from the Flores-200 development set and NTREX (Barrault et al., 2019). In our experiment, we follow the setting in (Xu et al., 2024b), where the translation sentences are sampled to contain 1012 sentences in each language pair. We also use WMT23 benchmarks to assess performance for evaluation. The distribution of WMT23 for each language is illustrated in Figure 6.
For languages in both Flores-200 and WMT'15-22, corresponding test sets are incorporated, yielding an average of 4K examples per language.
As shown in Figure 7, we observe a clear asymmetry in linguistic conflicts and synergy based on the SacreBLEU metric. This aligns with our main findings in the paper, where we used the COMET metric, further reinforcing the consistency of the observed phenomenon across different evaluation measures.
We further design another setting to validate the asymmetry in linguistic conflicts and synergy.
Experimental Setup We select anchor sets of varying sizes and perform post-training using training sets that include different numbers of languages but cover those anchor sets. We then observe the performance changes of these anchor sets. If the performance declines as more languages are included in the training set, this would indicate the presence of linguistic conflicts.
Results Table 5 displays the results. We can clearly observe that in the XX-En directions, the average performance of each anchor set consistently decreases as the number of languages increases. However, this phenomenon is not observed in the En-XX directions, where performance remains relatively stable. The findings are consistent with Section 3.
We observed an asymmetry in linguistic conflicts and synergies. A natural question arises: could this be due to using a low LoRA rank, which might limit learning capacity and, consequently, degrade performance? To address this concern, we selected the ALMA-13B-Pretrain model and trained it on 16 languages using different LoRA ranks, specifically 16 and 32. We then compared the performance of models with these LoRA ranks on the FLores-200 test sets. As shown in Table 6, increasing the LoRA rank did not yield performance improvements. Therefore, we conclude that the observed asymmetry is not attributed to using a low LoRA rank.
Language ISO-639-1 Script Family Subgroup Resource English Latin Indo-European Germanic High Group 1: Germanic Languages Afrikaans af Latin Indo-European Germanic Mid Danish da Latin Indo-European Germanic Mid Dutch nl Latin Indo-European Germanic High German de Latin Indo-European Germanic High Icelandic is Latin Indo-European Germanic Low Norwegian no Latin Indo-European Germanic Low Swedish sv Latin Indo-European Germanic High Group 2: Romance Languages Catalan ca Latin Indo-European Italic High Galician gl Latin Indo-European Italic Mid Italian it Latin Indo-European Italic High Portuguese pt Latin Indo-European Italic High Romanian ro Latin Indo-European Italic Mid Spanish es Latin Indo-European Italic High Group 3: Eastern and Southern Slavic Languages Bulgarian bg Cyrillic Indo-European Balto-Slavic Mid Macedonian mk Cyrillic Indo-European Balto-Slavic Low Russian ru Cyrillic Indo-European Balto-Slavic High Serbian sr Cyrillic Indo-European Balto-Slavic High Ukrainian uk Cyrillic Indo-European Balto-Slavic Mid Group 4: Southeast Asian Languages French fr Latin Indo-European Italic High Indonesian id Latin Austronesian Malayo-Polynesian Mid Malagasy mg Latin Austronesian Malayo-Polynesian Low Malay ms Latin Austronesian Malayo-Polynesian Mid Thai th Thai Tai-Kadai Kam-Tai Mid Vietnamese vi Latin Austronesian Vietic High d e -e n e n -c s e n -d e e n -h e e n -i s e n -j a e n -r u e n -u k e n -z h h e -e n i s -e n j a -e n r u -e n u k -e n z h -e n 0.0 0.5 1.0 1.5 2.0 #sentences…pairs…/…k
We also refer to this as multilingual pretraining, where data from all languages are mixed during the pretraining process.