6 Commits

Author	SHA1	Message	Date
Marco Vinciguerra	47645d92ce	removed unused files	2024-10-12 09:41:02 +02:00
Marco Vinciguerra	a200f595de	refactoring of examples	2024-10-08 08:54:18 +02:00
Marco Vinciguerra	d0969feb7c	removed unused imports and comments + removed dead code	2024-09-23 09:25:13 +02:00
Tom Robinson	dc4a76b9c5	use semchunk by default as the other code is causing tokenizers to be called for every individual word which is very slow especially with the mistral tokenizer	2024-09-12 08:46:52 +01:00
Tom Robinson	da9726f738	updates to tokenization for #651 to implement for mistral and ollama	2024-09-12 08:28:30 +01:00
Marco Vinciguerra	1a7f21fbf3	feat: removed semchunk and used tikton	2024-09-10 14:03:52 +02:00