Findings of EMNLP 2023

Data Pruning for Efficient Model Pruning in Neural Machine Translation

Abdul Hameed Azeemi, Ihsan Ayyub Qazi, and Agha Ali Raza

Lahore University of Management Sciences

Training-data pruning makes iterative movement pruning faster. Hard-to-learn translation examples preserve model quality while reducing convergence steps and fine-pruning time.

Paper ACL Anthology

Findings of the Association for Computational Linguistics

Abstract illustration of synchronized dataset and neural network pruning.

Introduction

Model pruning lowers memory use and inference cost after deployment, but producing a sparse model is expensive. Movement pruning repeatedly trains and prunes a model until it reaches the target sparsity, making the fine-pruning procedure itself a substantial computational burden.

We combine data pruning with movement pruning for neural machine translation. By identifying the training examples that matter most, we can fine-prune on a smaller dataset, converge in fewer steps, and retain translation quality.

Three-stage data and model pruning framework. — The full dataset is scored once, a data-pruning strategy constructs a compact subset, and movement pruning trains the sparse NMT model on that subset.

Prune the data before fine-pruning the model

The pipeline separates one-time data scoring from repeated fine-pruning runs, allowing the initial scoring cost to be amortized across sparsity levels and architectures.

1. Score the complete dataset

An unpruned pretrained language model is fine-tuned on the full parallel corpus. Per-example training cross-entropy is recorded as the data-pruning score.

2. Construct training subsets

Examples are ranked and selected with Top-K, Bottom-K, stratified, or random sampling. Top-K retains the highest-loss, hardest-to-learn sentence pairs.

3. Fine-prune on less data

A fresh model is trained on the selected subset while movement pruning gradually removes attention heads and dense-layer parameters to the target sparsity.

4. Reuse the ranking

The same data ranking can support multiple pruning runs. The paper also tests rankings computed with T5 when fine-pruning BART-base.

Evaluation

Experiments use WMT16 Romanian–English and Turkish–English with T5-small. Models are pruned to 10% and 50% sparsity while the training dataset is pruned by 20%, 40%, 60%, or 80%. Translation quality is measured with BLEU.

Turkish–English provides a particularly challenging setting because Turkish morphology produces a large vocabulary and the source and target languages use different word orders.

Dataset	Training pairs	Dev	Test
WMT16 En–Ro	610,320	1,999	1,999
WMT16 En–Tr	205,756	1,001	3,000

Results

Top-K selection consistently preserves the strongest BLEU as pruning becomes more aggressive, particularly at higher model sparsity. Retaining hard-to-learn examples is more effective than easy-example, random, or distribution-stratified selection.

For Romanian–English at 10% model sparsity, moving from 20% to 60% data pruning reduces convergence from 54,000 to 38,000 steps—a 29.6% reduction—with only a 2.22% relative BLEU decrease. On Turkish–English, fine-pruning with 40% of the data is 48.9% faster than using 80%.

BLEU across data pruning strategies and model sparsity levels. — Translation quality across data-pruning fractions and model sparsity. Top-K hard-example selection is the most robust as less training data remains.

Quality versus convergence

Data pruning exposes an explicit compute–quality frontier. Practitioners can choose a subset size from available training compute or work backward from an acceptable BLEU range. The one-time ranking cost is then amortized over subsequent fine-pruning runs.

BLEU versus convergence steps at several pruning fractions. — BLEU versus convergence steps for Romanian–English and Turkish–English at 10% model sparsity. Labels show the data-pruning fraction.

Conclusion

Data efficiency and model efficiency can be optimized together. Cross-entropy-based Top-K selection reduces movement-pruning cost while retaining difficult translation examples, and its rankings transfer across pruning configurations and model architectures.

Citation

@inproceedings{azeemi-etal-2023-data,
  title     = {Data Pruning for Efficient Model Pruning
               in Neural Machine Translation},
  author    = {Azeemi, Abdul Hameed and Qazi, Ihsan
               and Raza, Agha},
  booktitle = {Findings of EMNLP},
  pages     = {236--246},
  year      = {2023}
}