Clean tokenizer code


**Describe the solution you'd like**
The code written for the tokenizer is pretty sloppy and should be split up into more sub-components.
This could should be documented fully, better encapsulated, and (preferably) more functional