Fixes#752
Fix the issue with loading the tokenizer for 'gpt2'.
* **scrapegraphai/utils/tokenizer.py**
- Add a check for `GPT2TokenizerFast` in the `num_tokens_calculus` function.
- Import `GPT2TokenizerFast` from `transformers`.
* **scrapegraphai/utils/tokenizers/tokenizer_ollama.py**
- Modify the `num_tokens_ollama` function to handle `GPT2TokenizerFast`.
* **tests/graphs/smart_scraper_ollama_test.py**
- Add a test case to verify the tokenizer loading for `GPT2TokenizerFast`.
---
For more details, open the [Copilot Workspace session](https://copilot-workspace.githubnext.com/ScrapeGraphAI/Scrapegraph-ai/issues/752?shareId=XXXX-XXXX-XXXX-XXXX).
- Added pytest fixture to provide sample text from a file.
- Added pytest fixture to provide graph configuration.
- Implemented test_scraping_pipeline to test the execution of SmartScraperGraph.
- Added assertions to verify the result is not None and to check the expected structure of the result.
Contributed by @tejhande
- Added pytest fixture to provide sample text from a file.
- Added pytest fixture to provide graph configuration.
- Implemented test_scraping_pipeline to test the execution of SmartScraperGraph.
- Added assertions to verify the result is not None and to check the expected structure of the result.
Contributed by @tejhande
- Added pytest fixture to provide sample text from a file.
- Added pytest fixture to provide graph configuration.
- Implemented test_scraping_pipeline to test the execution of SmartScraperGraph.
- Added assertions to verify the result is not None and to check the expected structure of the result.
Contributed by @your-github-username
- Added pytest fixture to set up the RobotsNode with the initial state.
- Implemented test_robots_node to test the execution of RobotsNode.
- Used unittest.mock.patch to mock the execute method, ensuring faster and more reliable tests without actual network calls.
- Added assertions to verify the correctness of the result and ensure the execute method is called once with the correct arguments.
