Commit Graph

3 Commits

Author SHA1 Message Date
yusefes
d291819be3 Fix tokenizer loading for GPT2
Fixes #752

Fix the issue with loading the tokenizer for 'gpt2'.

* **scrapegraphai/utils/tokenizer.py**
  - Add a check for `GPT2TokenizerFast` in the `num_tokens_calculus` function.
  - Import `GPT2TokenizerFast` from `transformers`.

* **scrapegraphai/utils/tokenizers/tokenizer_ollama.py**
  - Modify the `num_tokens_ollama` function to handle `GPT2TokenizerFast`.

* **tests/graphs/smart_scraper_ollama_test.py**
  - Add a test case to verify the tokenizer loading for `GPT2TokenizerFast`.

---

For more details, open the [Copilot Workspace session](https://copilot-workspace.githubnext.com/ScrapeGraphAI/Scrapegraph-ai/issues/752?shareId=XXXX-XXXX-XXXX-XXXX).
2024-10-17 16:34:13 +03:30
Marco Vinciguerra
cec5537f2a add new tests
Some checks are pending
/ build (push) Waiting to run
Release / Build (push) Waiting to run
Release / Release (push) Blocked by required conditions
Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>
2024-08-11 17:10:55 +02:00
VinciGit00
a73496df9b refactoring of nodes 2024-04-24 21:13:27 +02:00