CodeCow/ollama
1
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-06-11 21:01:17 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
42e1d49fbe
ollama/tokenizer
History
Michael Yang f1373193dc
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
..
testdata move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
bytepairencoding_test.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
bytepairencoding.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
sentencepiece_test.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
sentencepiece.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
tokenizer.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
vocabulary_test.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
vocabulary.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
wordpiece_test.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
wordpiece.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00