01 CONTEXT PRUNING
EntroPrefill
Rényi-Guided Context Pruning with Conditional Stability Guarantees for Retrieval-Augmented Generation
I study which parts of a long context can be removed while an LLM is still processing the prompt. EntroPrefill uses Rényi entropy to propose what to prune, with checks on the attention mass that is removed.
Technical notes
- I implemented a training-free context-pruning method that reduced time to first token (TTFT) by 43.1% in the reported experiments.
- The scoring uses Order-2 Rényi entropy on GQA-pooled attention, alongside conditional transformer perturbation bounds.
