Language is a way to tell stories to other people. Mathematics is definitely limited as a language for quantitative reasoning, just as you say.
The spectra of mathematics can describe qualia, but in a different sense than the words of language. For example, blue and yellow are both colors that we commonly see in reflected light because of absorption by pigments. The math of those absorption spectra is straightforward and so math can represent colors with spectra as long as a white spectrum of light from a source illuminates the object. So I still argue that math can add and subtract blue and yellow absorption spectra and math can fully represent the parrot green spectrum in reflected light given a white light source illuminating an object.
However, the word "blue" also communicates more than the spectrum that math calls blue. The word "blue" communicates a lifetime of both singular and shared relational experiences with blue objects and so the word "blue" describes a blue object in relational ways far beyond what a math spectrum represents.