Rethinking Tokenization: How SuperBPE Breaks the Space Barrier
It just took questioning an arbitrary assumption (the Einstein way) to bring tokenization closer to the reality and overcome a years-long limitation in one of the fundamental layers of the NLP stack.
It just took questioning an arbitrary assumption (the Einstein way) to bring tokenization closer to the reality and overcome a years-long limitation in one of the fundamental layers of the NLP stack.