5 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
0x4261756D	10c6bcd483	Deduplicate more tokenizer work and avoid constructing strings	2023-06-07 02:44:37 +02:00
0x4261756D	317878dd3b	deeduplicate code	2023-06-06 23:56:38 +02:00
0x4261756D	a149609f03	Implement more states for the tokenizer	2023-06-06 20:38:40 +02:00
0x4261756D	21c5098cf7	Implement more tokenizer states + some last_index fixes	2023-06-06 19:18:27 +02:00
0x4261756D	294ecb7712	Initial commit, tokenizer stands (barely)	2023-06-06 02:16:29 +02:00