Commit Graph

1159 Commits

Author SHA1 Message Date
Marco Perini
6f994cef8c
Merge pull request #370 from VinciGit00/dev
Multiple Branches
2024-06-12 01:54:17 +02:00
Marco Perini
ab00f23d85 fix(node): fixed generate answer node pydantic schema 2024-06-12 01:40:49 +02:00
Marco Perini
15421eff70 feat(merge): add scriptcreatormulti, rag cache and semchunk 2024-06-12 01:23:01 +02:00
Marco Perini
650c3aaa60 docs(scriptcreator): enhance documentation 2024-06-12 01:16:50 +02:00
Marco Perini
a10b060409
Merge pull request #361 from VinciGit00/multi_scraper_implementation
Multi scraper implementation
2024-06-12 00:49:18 +02:00
Marco Perini
5d692bff9e feat(schema): merge scripts to follow pydantic schema 2024-06-12 00:48:08 +02:00
Marco Perini
893aaddc5c
Merge pull request #359 from VinciGit00/semchunk_integration
feat: add new chunking function
2024-06-11 23:16:00 +02:00
Marco Perini
589da1d695
Merge pull request #351 from VinciGit00/faiss_integration
Faiss integration
2024-06-11 23:15:23 +02:00
Marco Perini
edddb682d0 docs(cache): added cache_path param 2024-06-11 23:01:31 +02:00
Marco Perini
c881f64209 fix(cache): correctly pass the node arguments and logging 2024-06-11 22:56:09 +02:00
Marco Vinciguerra
fa951b4c8b
Merge pull request #368 from stevenmichaelthomas/wait-for-network-idle
Add the ability to specify load state
2024-06-11 19:59:23 +02:00
Steven Thomas
8f405ff87a Add the ability to specify load state 2024-06-11 11:22:39 -04:00
Marco Vinciguerra
81f89d88be
Update search_link_node.py 2024-06-10 16:14:24 +02:00
semantic-release-bot
e5bb5ae473 ci(release): 1.7.0-beta.2 [skip ci]
## [1.7.0-beta.2](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.7.0-beta.1...v1.7.0-beta.2) (2024-06-10)

### Features

* Add tests for RobotsNode and update test setup ([b0511ae](b0511aeaaa))
* Add tests for SmartScraperGraph using sample text and configuration fixtures ([@tejhande](https://github.com/tejhande)) ([c927145](c927145bd0))
* Add tests for SmartScraperGraph using sample text and configuration fixtures ([@tejhande](https://github.com/tejhande)) ([9e7038c](9e7038c596))
* Add tests for SmartScraperGraph using sample text and configuration fixtures ([@tejhande](https://github.com/tejhande)) ([c286b16](c286b1649e))
* Add tests for SmartScraperGraph using sample text and configuration fixtures ([@tejhande](https://github.com/tejhande)) ([08f1be6](08f1be682b))
2024-06-10 08:41:23 +00:00
Marco Vinciguerra
0748e10e3a
Merge pull request #365 from tejhande/main 2024-06-10 10:40:16 +02:00
Tejas Amol Hande
40747c3e01
Merge branch 'main' into main 2024-06-10 13:36:44 +05:30
Tejas Amol Hande
c927145bd0
feat: Add tests for SmartScraperGraph using sample text and configuration fixtures (@tejhande)
- Added pytest fixture to provide sample text from a file.
- Added pytest fixture to provide graph configuration.
- Implemented test_scraping_pipeline to test the execution of SmartScraperGraph.
- Added assertions to verify the result is not None and to check the expected structure of the result.

Contributed by @tejhande
2024-06-10 13:33:05 +05:30
Tejas Amol Hande
9e7038c596
feat: Add tests for SmartScraperGraph using sample text and configuration fixtures (@tejhande)
- Added pytest fixture to provide sample text from a file.
- Added pytest fixture to provide graph configuration.
- Implemented test_scraping_pipeline to test the execution of SmartScraperGraph.
- Added assertions to verify the result is not None and to check the expected structure of the result.

Contributed by @tejhande
2024-06-10 13:28:48 +05:30
Tejas Amol Hande
c286b1649e
feat: Add tests for SmartScraperGraph using sample text and configuration fixtures (@tejhande)
- Added pytest fixture to provide sample text from a file.
- Added pytest fixture to provide graph configuration.
- Implemented test_scraping_pipeline to test the execution of SmartScraperGraph.
- Added assertions to verify the result is not None and to check the expected structure of the result.

Contributed by @tejhande
2024-06-10 13:27:31 +05:30
Tejas Amol Hande
08f1be682b
feat: Add tests for SmartScraperGraph using sample text and configuration fixtures (@tejhande)
- Added pytest fixture to provide sample text from a file.
- Added pytest fixture to provide graph configuration.
- Implemented test_scraping_pipeline to test the execution of SmartScraperGraph.
- Added assertions to verify the result is not None and to check the expected structure of the result.

Contributed by @your-github-username
2024-06-10 13:25:10 +05:30
Tejas Amol Hande
b0511aeaaa
feat: Add tests for RobotsNode and update test setup
- Added pytest fixture to set up the RobotsNode with the initial state.
- Implemented test_robots_node to test the execution of RobotsNode.
- Used unittest.mock.patch to mock the execute method, ensuring faster and more reliable tests without actual network calls.
- Added assertions to verify the correctness of the result and ensure the execute method is called once with the correct arguments.
2024-06-10 13:21:56 +05:30
semantic-release-bot
84a74b2f79 ci(release): 1.7.0-beta.1 [skip ci]
## [1.7.0-beta.1](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.6.0...v1.7.0-beta.1) (2024-06-09)

### Features

* add csv scraper and xml scraper multi ([b408655](b4086550cc))
* **indexify-node:** add example ([5d1fbf8](5d1fbf806a))
* add forcing format as json ([5cfc101](5cfc10178a))
* add json as output ([5d20186](5d20186bf2))
* add json multiscraper ([5bda918](5bda918a39))
* add pdf scraper multi graph ([f5cbd80](f5cbd80c97))
* **pydantic:** added pydantic output schema ([376f758](376f758a76))
* **append_node:** append node to existing graph ([f8b08e0](f8b08e0b33))
* fix an if ([c8d556d](c8d556da4e))
* refactoring of abstract graph ([fff89f4](fff89f431f))
* refactoring of an in if ([244aada](244aada2de))
* removed a bug ([8de720d](8de720d379))
* removed rag node ([930f673](930f673747))
* **version:** update burr version ([cfa1336](cfa13368f4))

### Bug Fixes

* **schema:** fixed json output ([5c9843f](5c9843f141))
* oneapi model ([4fcb990](4fcb9902fe))
* typo in prompt ([4639f0c](4639f0cac5))

### Docs

* stylize badges in readme ([8696ade](8696adede7))

### CI

* **release:** 1.5.3-beta.1 [skip ci] ([6ea1d2c](6ea1d2c4d0))
* **release:** 1.5.3-beta.2 [skip ci] ([b57bcef](b57bcef5c1))
* **release:** 1.5.5-beta.1 [skip ci] ([38d138e](38d138e36f))
* **release:** 1.6.0-beta.1 [skip ci] ([1d217e4](1d217e4ae6))
* **release:** 1.6.0-beta.10 [skip ci] ([4d0d8fa](4d0d8fa453))
* **release:** 1.6.0-beta.11 [skip ci] ([3453ac0](3453ac01f5))
* **release:** 1.6.0-beta.2 [skip ci] ([ed1dc0b](ed1dc0be08))
* **release:** 1.6.0-beta.3 [skip ci] ([b70cb37](b70cb37c62))
* **release:** 1.6.0-beta.4 [skip ci] ([08a14ef](08a14efdd3))
* **release:** 1.6.0-beta.5 [skip ci] ([dde0c7e](dde0c7e27d))
* **release:** 1.6.0-beta.6 [skip ci] ([ac8e7c1](ac8e7c12fe))
* **release:** 1.6.0-beta.7 [skip ci] ([cab5f68](cab5f6828c))
* **release:** 1.6.0-beta.8 [skip ci] ([7a6f016](7a6f016f92))
* **release:** 1.6.0-beta.9 [skip ci] ([ca8aff8](ca8aff8d88))
2024-06-09 15:07:07 +00:00
Marco Vinciguerra
2f0568c83a
Merge pull request #364 from VinciGit00/all
allignment
2024-06-09 17:05:59 +02:00
Marco Vinciguerra
8bec47eac2
Merge branch 'pre/beta' into all 2024-06-09 17:05:51 +02:00
semantic-release-bot
3453ac01f5 ci(release): 1.6.0-beta.11 [skip ci]
## [1.6.0-beta.11](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.6.0-beta.10...v1.6.0-beta.11) (2024-06-09)

### Bug Fixes

* bug on generate_answer_node ([1d38ed1](1d38ed146a))
* getter ([67d83cf](67d83cff46))
* update openai tts class ([10672d6](10672d6ebb))

### Docs

* add Japanese README ([4559ab6](4559ab6db8))
* update japanese.md ([f0042a8](f0042a8e33))
* update README.md ([871e398](871e398a26))

### Test

* Enhance JSON scraping pipeline test ([d845a1b](d845a1ba7d))

### CI

* **release:** 1.5.5 [skip ci] ([3629215](36292150da))
* **release:** 1.5.6 [skip ci] ([49cdadf](49cdadf117))
* **release:** 1.5.7 [skip ci] ([c17daca](c17daca409))
2024-06-09 15:05:20 +00:00
Marco Vinciguerra
9326637cc3 Merge branch 'main' into pre/beta 2024-06-09 17:04:11 +02:00
semantic-release-bot
58086eef47 ci(release): 1.6.0 [skip ci]
## [1.6.0](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.5.7...v1.6.0) (2024-06-09)

### Features

* Add tests for RobotsNode and update test setup ([dedfa2e](dedfa2eaf0))

### Test

* Enhance JSON scraping pipeline test ([d845a1b](d845a1ba7d))
2024-06-09 15:00:59 +00:00
Marco Vinciguerra
e688480771
Merge pull request #362 from tejhande/patch-4
feat: Add tests for RobotsNode and update test setup
2024-06-09 16:59:50 +02:00
Marco Vinciguerra
2781c3c46b
Merge pull request #355 from tejhande/patch-3
Test ScriptCreatorGraph and print execution info
2024-06-09 16:57:36 +02:00
tejhande
13f8ca56b2 "Refactor SearchLinkNode test: simplify setup, add patching for execute method, and enhance assertions" 2024-06-09 19:48:33 +05:30
Tejas Amol Hande
dedfa2eaf0
feat: Add tests for RobotsNode and update test setup
- Added pytest fixture to set up the RobotsNode with the initial state.
- Implemented test_robots_node to test the execution of RobotsNode.
- Used unittest.mock.patch to mock the execute method, ensuring faster and more reliable tests without actual network calls.
- Added assertions to verify the correctness of the result and ensure the execute method is called once with the correct arguments.
2024-06-09 19:39:11 +05:30
Marco Vinciguerra
14d10117b7
Merge pull request #354 from tejhande/patch-2
Enhance tests for FetchNode with mocking
2024-06-09 15:50:27 +02:00
Marco Vinciguerra
bde02492c0 add examples 2024-06-09 15:26:56 +02:00
Marco Vinciguerra
fe8083fe48 Update pdf_scraper_graph_haiku.py
Some checks failed
/ build (3.10) (push) Has been cancelled
Release / Build (push) Has been cancelled
Release / Release (push) Has been cancelled
2024-06-09 10:02:29 +02:00
Marco Vinciguerra
5dc6165881 add example
Some checks failed
CodeQL / Analyze (python) (push) Has been cancelled
/ build (3.10) (push) Has been cancelled
Release / Build (push) Has been cancelled
Release / Release (push) Has been cancelled
2024-06-09 09:25:37 +02:00
Marco Vinciguerra
c14fb88fca add examples
Some checks failed
/ build (3.10) (push) Has been cancelled
2024-06-09 08:58:47 +02:00
Marco Vinciguerra
cb00c4fb17 changed model 2024-06-08 12:22:50 +02:00
Marco Vinciguerra
1981230e6f add multi scraper integration 2024-06-08 12:13:18 +02:00
semantic-release-bot
4d0d8fa453 ci(release): 1.6.0-beta.10 [skip ci]
## [1.6.0-beta.10](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.6.0-beta.9...v1.6.0-beta.10) (2024-06-08)

### Features

* **version:** update burr version ([cfa1336](cfa13368f4))

### Docs

* stylize badges in readme ([8696ade](8696adede7))
2024-06-08 10:11:47 +00:00
Marco Perini
8228225dff
Merge pull request #360 from VinciGit00/burr_integration
Burr integration
2024-06-08 12:10:34 +02:00
Marco Perini
cfa13368f4 feat(version): update burr version
Some checks failed
/ build (3.10) (push) Has been cancelled
2024-06-08 12:06:25 +02:00
Marco Perini
a6061cb3b5
Merge pull request #344 from DAGWorks-Inc/burr-spawning-apps
Integrates with Burr's Forking/spawning ability
2024-06-08 12:04:09 +02:00
Marco Vinciguerra
e1f045b280 feat: add new chunking function 2024-06-08 11:44:09 +02:00
Marco Vinciguerra
c78aa439af beautofy readmes
Some checks are pending
CodeQL / Analyze (python) (push) Waiting to run
/ build (3.10) (push) Waiting to run
Release / Build (push) Waiting to run
Release / Release (push) Blocked by required conditions
2024-06-08 11:20:06 +02:00
Marco Vinciguerra
2000baa185
Merge pull request #358 from iamgodot/docs
Some checks failed
/ build (3.10) (push) Has been cancelled
Release / Build (push) Has been cancelled
Release / Release (push) Has been cancelled
2024-06-08 08:04:09 +02:00
iamgodot
8696adede7 docs: stylize badges in readme 2024-06-07 16:25:07 -07:00
semantic-release-bot
ca8aff8d88 ci(release): 1.6.0-beta.9 [skip ci]
## [1.6.0-beta.9](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.6.0-beta.8...v1.6.0-beta.9) (2024-06-07)

### Features

* **indexify-node:** add example ([5d1fbf8](5d1fbf806a))

### Bug Fixes

* **schema:** fixed json output ([5c9843f](5c9843f141))
2024-06-07 18:28:38 +00:00
Marco Perini
f41a755519
Merge pull request #356 from VinciGit00/321-integration-with-indexify
fixed pydantic schema
2024-06-07 20:27:25 +02:00
Tejas Amol Hande
ff9df81e60
Test ScriptCreatorGraph and print execution info
This commit enhances the test suite for the ScriptCreatorGraph class by improving code readability, adding more informative assertions, and printing the prettified execution information.

Changes:
- Added more descriptive docstrings for better code documentation.
- Improved assertion messages to provide better debugging experience in case of failures.
- Added a line to print the prettified execution information using the `prettify_exec_info` function.
- Included a comment to remind developers to add additional assertions on the result or execution info if needed.
- Fixed a minor typo in the configuration dictionary (`beautifulsoup` instead of `beautifoulsoup`).

Benefits:
- Improved code readability and maintainability with better documentation.
- Enhanced debugging experience with more informative assertion messages.
- Easier analysis of the ScriptCreatorGraph execution by printing the prettified execution information.
- Reminder to add more assertions for comprehensive testing of the ScriptCreatorGraph.
- Corrected a minor typo to ensure consistency.

The test suite now provides a more user-friendly experience for developers working on the ScriptCreatorGraph class. The printed execution information will aid in debugging and understanding the graph's execution flow, while the improved assertions and documentation will make the test suite more robust and maintainable.
2024-06-07 23:17:58 +05:30
Tejas Amol Hande
320f13fa89
Enhance tests for FetchNode with mocking
This commit enhances the test suite for the FetchNode class by introducing mocking for the execute method using the unittest.mock module.

Changes:
- Imported the patch and MagicMock classes from unittest.mock.
- Decorated each test function with @patch('scrapegraphai.nodes.FetchNode.execute') to mock the execute method.
- Set the return_value of the mocked execute method to a MagicMock instance.
- Added assertions to check if the mocked execute method was called with the expected state dictionary.
- Updated the test functions to use the mocked execute method instead of the actual implementation.

Benefits:
- Improved test reliability by isolating the FetchNode class from external dependencies.
- Faster test execution since external resources (e.g., URLs, files) are not required.
- Better test coverage by testing the execute method's behavior with various input states.
- Increased maintainability by decoupling tests from the implementation details of the execute method.

The functionality of the FetchNode class remains unchanged, but the tests now use mocking to ensure the correct behavior of the execute method without relying on external resources or dependencies.
2024-06-07 23:11:32 +05:30