PeriniM
5cdf0550fe
chore: made some libs optional
2025-01-06 03:42:45 +01:00
PeriniM
54c69a2b0b
chore: pandas package is now optional
2025-01-06 02:36:05 +01:00
Marcelo Busana
489adf7006
Add the AbstractGraph run_safe_async and its test
2024-12-21 09:53:56 -03:00
Marco Vinciguerra
ffa1067f0d
Merge pull request #756 from shenghongtw/pre/beta
...
The smart_scraper_multi_graph method is too expensive
2024-10-20 10:13:47 +02:00
roryhaung
28dda2b476
rename graph name
2024-10-18 03:14:08 +08:00
yusefes
d291819be3
Fix tokenizer loading for GPT2
...
Fixes #752
Fix the issue with loading the tokenizer for 'gpt2'.
* **scrapegraphai/utils/tokenizer.py**
- Add a check for `GPT2TokenizerFast` in the `num_tokens_calculus` function.
- Import `GPT2TokenizerFast` from `transformers`.
* **scrapegraphai/utils/tokenizers/tokenizer_ollama.py**
- Modify the `num_tokens_ollama` function to handle `GPT2TokenizerFast`.
* **tests/graphs/smart_scraper_ollama_test.py**
- Add a test case to verify the tokenizer loading for `GPT2TokenizerFast`.
---
For more details, open the [Copilot Workspace session](https://copilot-workspace.githubnext.com/ScrapeGraphAI/Scrapegraph-ai/issues/752?shareId=XXXX-XXXX-XXXX-XXXX ).
2024-10-17 16:34:13 +03:30
shenghong
2512262be8
Rename smart_scraper_multi_parse_merge_first_graph_test.py to smart_scraper_multi_parse_merge_first_graph_openai_test.py
2024-10-17 06:46:34 +08:00
roryhaung
464b8b04ea
test: Add smart_scraper_multi_parse_merge_first_graph test
2024-10-16 20:05:36 +08:00
roryhaung
cdb3c1100e
test: Add scrape_graph test
2024-10-16 20:05:03 +08:00
Marco Vinciguerra
7dba46a0e9
add tests
2024-10-09 11:19:09 +02:00
Marco Vinciguerra
5d1fe68dea
Merge branch 'pre/beta' into temp
2024-09-14 10:53:43 +02:00
smith peng
7e3f66e180
fix:fix boto3 client copy
2024-09-14 11:15:31 +08:00
Marco Vinciguerra
c382b9d126
Merge pull request #630 from ScrapeGraphAI/595-rate-limit-error
...
feat(AbstractGraph): add adjustable rate limit
2024-09-06 15:03:10 +02:00
Federico Aguzzi
2859fb72d6
feat(AbstractGraph): add adjustable rate limit
2024-09-05 16:31:33 +02:00
Federico Aguzzi
63a5d18486
fix(AbstractGraph): Bedrock init issues
...
Closes #633
2024-09-05 10:20:20 +02:00
Marco Vinciguerra
81af62d35f
Merge pull request #622 from LorenzoPaleari/pre/beta
...
ScrapeGraphAI/580-OmniScraperGraph-fix
2024-09-02 15:13:07 +02:00
Lorenzo Paleari
57337a0a8c
fix: Removed link_urls and img_ulrs from FetchNode output
2024-09-02 12:53:40 +02:00
Federico Aguzzi
398b2c556f
fix(Ollama): instance model from correct package
2024-09-02 12:28:36 +02:00
smith peng
553527a269
fix: fix pydantic object copy
2024-09-01 16:42:48 +08:00
smith peng
71b22d4880
feat: add deepcopy error
2024-08-31 17:55:14 +08:00
smith peng
36818b1fb3
feat:adjust uncopiable obj raise error and remove memo
2024-08-31 17:39:33 +08:00
smith peng
4f4d091b82
feat:add deepcopy tool
2024-08-31 11:30:20 +08:00
Federico Aguzzi
f7a85c266a
fix(models): better DeepSeek and OneApi integration
2024-08-28 12:51:50 +02:00
Federico Aguzzi
4f120e29c5
fix(AbstractGraph): model selection bug
2024-08-28 12:36:03 +02:00
Marco Vinciguerra
cf73883451
fix: bug for abstract graph
2024-08-27 18:05:34 +02:00
Federico Aguzzi
229d74d4bd
test(AbstractGraph): add AbstractGraph tests
2024-08-27 13:46:42 +02:00
Marco Vinciguerra
5eb3cff64f
feat: refactoring of the code
2024-08-18 20:53:35 +02:00
Marco Vinciguerra
9098725adf
Delete scrape_plain_text_llama3.1_test.py
2024-08-17 14:17:32 +02:00
Marco Vinciguerra
cec5537f2a
add new tests
...
/ build (push) Waiting to run
Release / Build (push) Waiting to run
Release / Release (push) Blocked by required conditions
Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>
2024-08-11 17:10:55 +02:00
Marco Vinciguerra
1db164e9e6
feat: fix tests
2024-07-30 16:12:31 +02:00
Marco Vinciguerra
602dd00209
feat: refactoring_to_md function
2024-07-16 12:39:48 +02:00
Marco Vinciguerra
da0b744443
add test
2024-07-15 20:46:22 +02:00
Marco Vinciguerra
ba782a6af9
add compatibility for versions
2024-07-04 21:04:47 +02:00
Federico Aguzzi
104d8692d6
Merge branch 'pre/beta' into support
/ build (push) Has been cancelled
2024-07-04 18:36:34 +02:00
Federico Aguzzi
720f18729b
Merge branch 'fireworks_integration' into support
2024-07-04 18:17:38 +02:00
Marco Vinciguerra
4b56604413
add examples + test
/ build (3.10) (push) Has been cancelled
2024-06-25 10:32:29 +02:00
Marco Vinciguerra
3a537eec6f
fix: add test
2024-06-23 17:54:09 +02:00
Marco Vinciguerra
cf9a3d1a2f
add test
2024-06-21 14:42:54 +02:00
Marco Vinciguerra
aa2160c108
feat: add research with bing + test function
/ build (3.10) (push) Has been cancelled
2024-06-18 21:28:29 +02:00
Marco Vinciguerra
073d226723
feat: add new search engine avaiability and new tests
2024-06-18 14:35:13 +02:00
Marco Vinciguerra
6a753f2803
add smart_scraper_openai_test
2024-06-17 11:17:08 +02:00
Marco Vinciguerra
2419003999
fix: fix robot node
/ build (3.10) (push) Waiting to run
Release / Build (push) Waiting to run
Release / Release (push) Blocked by required conditions
2024-06-16 14:04:36 +02:00
Marco Vinciguerra
c31706fdae
fixed tests
...
/ build (3.10) (push) Waiting to run
Release / Build (push) Waiting to run
Release / Release (push) Blocked by required conditions
Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>
2024-06-15 20:41:26 +02:00
Marco Vinciguerra
7a34562d50
refactoring of merging answers nodes
2024-06-13 21:41:54 +02:00
Marco Vinciguerra
49c7e0eaab
fix: test for fetch node
2024-06-13 11:04:59 +02:00
iamgodot
17dd936af7
test: fix tests for fetch node with proper mock&refactor
2024-06-12 22:45:43 -07:00
Tejas Amol Hande
40747c3e01
Merge branch 'main' into main
2024-06-10 13:36:44 +05:30
Tejas Amol Hande
c927145bd0
feat: Add tests for SmartScraperGraph using sample text and configuration fixtures (@tejhande)
...
- Added pytest fixture to provide sample text from a file.
- Added pytest fixture to provide graph configuration.
- Implemented test_scraping_pipeline to test the execution of SmartScraperGraph.
- Added assertions to verify the result is not None and to check the expected structure of the result.
Contributed by @tejhande
2024-06-10 13:33:05 +05:30
Tejas Amol Hande
c286b1649e
feat: Add tests for SmartScraperGraph using sample text and configuration fixtures (@tejhande)
...
- Added pytest fixture to provide sample text from a file.
- Added pytest fixture to provide graph configuration.
- Implemented test_scraping_pipeline to test the execution of SmartScraperGraph.
- Added assertions to verify the result is not None and to check the expected structure of the result.
Contributed by @tejhande
2024-06-10 13:27:31 +05:30
Tejas Amol Hande
08f1be682b
feat: Add tests for SmartScraperGraph using sample text and configuration fixtures (@tejhande)
...
- Added pytest fixture to provide sample text from a file.
- Added pytest fixture to provide graph configuration.
- Implemented test_scraping_pipeline to test the execution of SmartScraperGraph.
- Added assertions to verify the result is not None and to check the expected structure of the result.
Contributed by @your-github-username
2024-06-10 13:25:10 +05:30
