CodeCow/Scrapegraph-ai
1
0
Fork 0
mirror of https://github.com/VinciGit00/Scrapegraph-ai.git synced 2026-07-01 21:00:48 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
ec9ef2bcda
Scrapegraph-ai/tests
History
yusefes d291819be3 Fix tokenizer loading for GPT2 
Fixes #752

Fix the issue with loading the tokenizer for 'gpt2'.

* **scrapegraphai/utils/tokenizer.py**
  - Add a check for `GPT2TokenizerFast` in the `num_tokens_calculus` function.
  - Import `GPT2TokenizerFast` from `transformers`.

* **scrapegraphai/utils/tokenizers/tokenizer_ollama.py**
  - Modify the `num_tokens_ollama` function to handle `GPT2TokenizerFast`.

* **tests/graphs/smart_scraper_ollama_test.py**
  - Add a test case to verify the tokenizer loading for `GPT2TokenizerFast`.

---

For more details, open the [Copilot Workspace session](https://copilot-workspace.githubnext.com/ScrapeGraphAI/Scrapegraph-ai/issues/752?shareId=XXXX-XXXX-XXXX-XXXX).
2024-10-17 16:34:13 +03:30
..
graphs Fix tokenizer loading for GPT2 2024-10-17 16:34:13 +03:30
inputs refactoring of merging answers nodes 2024-06-13 21:41:54 +02:00
nodes feat: fix tests 2024-07-30 16:12:31 +02:00
utils fix:fix boto3 client copy 2024-09-14 11:15:31 +08:00
Readme.md refactoring of nodes 2024-04-24 21:13:27 +02:00

Readme.md

Test section

Regarding the tests for the folder graphs and nodes it was created a specific repo as a example (link of the repo). The test website is hosted here. Remember to activating Ollama and having installed the LLM on your pc

For running the tests run the command:

pytest