Commit Graph

3 Commits

Author SHA1 Message Date
Marco Vinciguerra
a18471688f fix: removed tokenizer 2024-10-19 07:18:56 +02:00
yusefes
d291819be3 Fix tokenizer loading for GPT2
Fixes #752

Fix the issue with loading the tokenizer for 'gpt2'.

* **scrapegraphai/utils/tokenizer.py**
  - Add a check for `GPT2TokenizerFast` in the `num_tokens_calculus` function.
  - Import `GPT2TokenizerFast` from `transformers`.

* **scrapegraphai/utils/tokenizers/tokenizer_ollama.py**
  - Modify the `num_tokens_ollama` function to handle `GPT2TokenizerFast`.

* **tests/graphs/smart_scraper_ollama_test.py**
  - Add a test case to verify the tokenizer loading for `GPT2TokenizerFast`.

---

For more details, open the [Copilot Workspace session](https://copilot-workspace.githubnext.com/ScrapeGraphAI/Scrapegraph-ai/issues/752?shareId=XXXX-XXXX-XXXX-XXXX).
2024-10-17 16:34:13 +03:30
Tom Robinson
da9726f738 updates to tokenization for #651 to implement for mistral and ollama 2024-09-12 08:28:30 +01:00