March 7, 2025
Introduction to State Space Models as Natural Language Models
State Space Models (SSMs) use first-order differential equations to represent dynamic systems. The HiPPO framework provides a mathematical foundation for maintaining continuous representations of time-dependent data, enabling efficient approximation of long-range dependencies in sequence modeling. Discretization of continuous-time SSMs lays the groundwork for processing natural language and modeling long-range dependencies in a computationally efficient way. LSSL, S4, and S5 are increasingly sophisticated and efficient sequence-to-sequence state-space models that pave the way for viable SSM-based alternatives to transformer models. While transformer-based models are in the limelight of the NLP community, a quiet revolution in sequence modeling is underway. State Space Models