yusefes d291819be3 Fix tokenizer loading for GPT2 ... Fixes #752 Fix the issue with loading the tokenizer for 'gpt2'. * **scrapegraphai/utils/tokenizer.py** - Add a check for `GPT2TokenizerFast` in the `num_tokens_calculus` function. - Import `GPT2TokenizerFast` from `transformers`. * **scrapegraphai/utils/tokenizers/tokenizer_ollama.py** - Modify the `num_tokens_ollama` function to handle `GPT2TokenizerFast`. * **tests/graphs/smart_scraper_ollama_test.py** - Add a test case to verify the tokenizer loading for `GPT2TokenizerFast`. --- For more details, open the [Copilot Workspace session](https://copilot-workspace.githubnext.com/ScrapeGraphAI/Scrapegraph-ai/issues/752?shareId=XXXX-XXXX-XXXX-XXXX).		2024-10-17 16:34:13 +03:30
..
inputs	add csv scraper	2024-05-01 17:20:04 +02:00
.env.example	add compatibility for versions	2024-07-04 21:04:47 +02:00
abstract_graph_test.py	Merge pull request #630 from ScrapeGraphAI/595-rate-limit-error	2024-09-06 15:03:10 +02:00
code_generator_graph_openai_test.py	add tests	2024-10-09 11:19:09 +02:00
depth_search_graph_openai_test.py	add tests	2024-10-09 11:19:09 +02:00
scrape_plain_text_mistral_test.py	add new tests	2024-08-11 17:10:55 +02:00
scrape_xml_ollama_test.py	add new tests	2024-08-11 17:10:55 +02:00
screenshot_scraper_test.py	feat: refactoring of the code	2024-08-18 20:53:35 +02:00
script_generator_test.py	add new tests	2024-08-11 17:10:55 +02:00
search_graph_openai_test.py	add tests	2024-10-09 11:19:09 +02:00
search_link_ollama.py	add new tests	2024-08-11 17:10:55 +02:00
smart_scraper_ernie_test.py	add tests	2024-10-09 11:19:09 +02:00
smart_scraper_fireworks_test.py	add new tests	2024-08-11 17:10:55 +02:00
smart_scraper_ollama_test.py	Fix tokenizer loading for GPT2	2024-10-17 16:34:13 +03:30
smart_scraper_openai_test.py	add smart_scraper_openai_test	2024-06-17 11:17:08 +02:00
xml_scraper_openai_test.py	add tests	2024-10-09 11:19:09 +02:00