CodeCow/Scrapegraph-ai
1
0
Fork 0
mirror of https://github.com/VinciGit00/Scrapegraph-ai.git synced 2026-07-09 21:19:20 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,293 Commits 2 Branches 448 Tags 17 MiB
5faac9fb46
Commit Graph

2293 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Marco Vinciguerra
5faac9fb46
Merge pull request #784 from ScrapeGraphAI/pre/beta 
fixed json on generate answer
 2024-11-04 09:13:19 +01:00
semantic-release-bot
950e859b1b ci(release): 1.29.0-beta.1 [skip ci] 
## [1.29.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.28.0...v1.29.0-beta.1) (2024-11-04)

### Features

* Serper API integration for Google search ([c218546](c218546a3d))

### Bug Fixes

* resolved outparser issue ([e8cabfd](e8cabfd1ae))

### CI

* **release:** 1.28.0-beta.3 [skip ci] ([65d39bb](65d39bbaf0))
* **release:** 1.28.0-beta.4 [skip ci] ([b90bb00](b90bb00beb))
 2024-11-04 08:10:24 +00:00
Marco Vinciguerra
8d7fd08730
Merge branch 'main' into pre/beta 2024-11-04 09:08:37 +01:00
semantic-release-bot
b90bb00beb ci(release): 1.28.0-beta.4 [skip ci] 
## [1.28.0-beta.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.28.0-beta.3...v1.28.0-beta.4) (2024-11-03)

### Bug Fixes

* resolved outparser issue ([e8cabfd](e8cabfd1ae))
 2024-11-03 16:57:13 +00:00
Marco Vinciguerra
ec667d21a0
Merge pull request #783 from aziz-ullah-khan/pre/beta 
fix: resolved outparser issue
 2024-11-03 17:55:33 +01:00
aziz-ullah-khan
7666a91f4f cleaning 2024-11-03 20:28:50 +05:00
aziz-ullah-khan
e8cabfd1ae fix: resolved outparser issue 2024-11-03 20:23:29 +05:00
semantic-release-bot
65d39bbaf0 ci(release): 1.28.0-beta.3 [skip ci] 
## [1.28.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.28.0-beta.2...v1.28.0-beta.3) (2024-11-02)

### Features

* Serper API integration for Google search ([c218546](c218546a3d))
 2024-11-02 17:36:10 +00:00
Marco Vinciguerra
902699c8d2
Merge pull request #781 from aziz-ullah-khan/pre/beta 2024-11-02 18:34:51 +01:00
aziz-ullah-khan
c218546a3d feat: Serper API integration for Google search 2024-11-02 15:02:08 +05:00
semantic-release-bot
7b5010f949 ci(release): 1.28.0 [skip ci] 
## [1.28.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.27.0...v1.28.0) (2024-11-01)

### Features

* add new mistral models ([6914170](6914170890))
* refactoring of the base_graph ([12a6c18](12a6c18f6a))
* update generate answer ([7172b32](7172b32a0f))

### Bug Fixes

* **AbstractGraph:** manually select model tokens ([f79f399](f79f399ee0)), closes [#768](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/768)

### CI

* **release:** 1.27.0-beta.11 [skip ci] ([3b2cadc](3b2cadce1a))
* **release:** 1.27.0-beta.12 [skip ci] ([62369e3](62369e3e28))
* **release:** 1.27.0-beta.13 [skip ci] ([deed355](deed355551)), closes [#768](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/768)
* **release:** 1.28.0-beta.1 [skip ci] ([8cbe582](8cbe582ea9)), closes [#768](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/768) [#768](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/768)
* **release:** 1.28.0-beta.2 [skip ci] ([7e3598d](7e3598ddfa))
 2024-11-01 09:44:12 +00:00
Marco Vinciguerra
9f0ba35be6
Merge pull request #780 from ScrapeGraphAI/pre/beta 
Pre/beta
 2024-11-01 10:42:44 +01:00
Lorenzo Padoan
ea2ff5099b
Update funding.json
Some checks failed
CodeQL / Analyze (python) (push) Has been cancelled
Details
Release / Build (push) Has been cancelled
Details
Release / Release (push) Has been cancelled
Details
2024-10-31 10:36:11 +01:00
Lorenzo Padoan
a04be3eec5
Update funding.json 2024-10-31 10:35:34 +01:00
Lorenzo Padoan
5970594f58
Update funding.json 2024-10-31 10:32:26 +01:00
Lorenzo Padoan
463ade403b
Update funding.json 2024-10-31 10:31:50 +01:00
Lorenzo Padoan
59ad0a231f
Update funding.json 2024-10-31 10:30:29 +01:00
Lorenzo Padoan
a01ec4867d
Update funding.json 2024-10-31 10:23:05 +01:00
semantic-release-bot
7e3598ddfa ci(release): 1.28.0-beta.2 [skip ci] 
## [1.28.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.28.0-beta.1...v1.28.0-beta.2) (2024-10-31)

### Features

* update generate answer ([7172b32](7172b32a0f))
 2024-10-31 09:21:45 +00:00
Federico Aguzzi
76e2820ff1
Merge pull request #778 from ScrapeGraphAI/refactoring-generate-answer 
feat: update generate answer
 2024-10-31 10:20:17 +01:00
Marco Vinciguerra
7440f3735e
Merge pull request #779 from ScrapeGraphAI/main 
allignment
 2024-10-31 09:06:15 +01:00
Marco Vinciguerra
c2b68095c1 Update funding.json 2024-10-30 18:07:40 +01:00
Marco Vinciguerra
b75fbd26c5 add funding 2024-10-30 18:04:41 +01:00
Marco Vinciguerra
655ef28069 Update funding.json 2024-10-30 17:52:34 +01:00
Marco Vinciguerra
542751d35b Update funding.json 2024-10-30 14:23:11 +01:00
semantic-release-bot
8cbe582ea9 ci(release): 1.28.0-beta.1 [skip ci] 
## [1.28.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.27.0...v1.28.0-beta.1) (2024-10-30)

### Features

* add new mistral models ([6914170](6914170890))
* refactoring of the base_graph ([12a6c18](12a6c18f6a))

### Bug Fixes

* **AbstractGraph:** manually select model tokens ([f79f399](f79f399ee0)), closes [#768](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/768)

### CI

* **release:** 1.27.0-beta.11 [skip ci] ([3b2cadc](3b2cadce1a))
* **release:** 1.27.0-beta.12 [skip ci] ([62369e3](62369e3e28))
* **release:** 1.27.0-beta.13 [skip ci] ([deed355](deed355551)), closes [#768](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/768)
 2024-10-30 08:02:00 +00:00
Marco Vinciguerra
bb2373d7a2
Merge pull request #775 from U-C4N/main 
This commit focuses on optimizing the utility modules in the codebase…
 2024-10-30 09:00:33 +01:00
Marco Vinciguerra
09c967822d
Merge branch 'pre/beta' into main 2024-10-30 09:00:20 +01:00
Marco Vinciguerra
7172b32a0f feat: update generate answer 2024-10-30 08:59:13 +01:00
semantic-release-bot
deed355551 ci(release): 1.27.0-beta.13 [skip ci] 
## [1.27.0-beta.13](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.27.0-beta.12...v1.27.0-beta.13) (2024-10-29)

### Bug Fixes

* **AbstractGraph:** manually select model tokens ([f79f399](f79f399ee0)), closes [#768](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/768)
 2024-10-29 08:02:25 +00:00
Marco Vinciguerra
8ab726e9c1
Merge pull request #773 from ScrapeGraphAI/768-fix-model-tokens 
fix(AbstractGraph): manually select model tokens
 2024-10-29 09:00:46 +01:00
Umut CAN
827f7260ad This commit focuses on optimizing the utility modules in the codebase for better performance and maintainability. Key improvements include: - More efficient HTML processing with combined regex operations and optimized tag handling - Enhanced deep copy functionality with better type handling and optimized recursion - Refactored web search with improved error handling and modular helper functions The changes maintain all existing functionality while improving code quality, performance, and maintainability. Documentation and type hints have been enhanced throughout. 
Optimize utils modules for better performance and maintainability

- Improve HTML cleanup and minification:
  - Combine regex operations for better performance
  - Add better error handling for HTML processing
  - Optimize tag removal and attribute filtering

- Enhance deep copy functionality:
  - Add special case handling for primitive types
  - Improve type checking and error handling
  - Optimize recursive copying for collections

- Refactor web search functionality:
  - Add input validation and error handling
  - Split search logic into separate helper functions
  - Improve proxy handling and configuration
  - Add better timeout and error management
  - Optimize URL filtering and processing

Technical improvements:
- Better type hints and documentation
- More efficient data structures
- Improved error handling and validation
- Reduced code duplication
- Better separation of concerns

No breaking changes - all existing functionality maintained
 2024-10-28 22:40:32 +03:00
Federico Aguzzi
f79f399ee0 fix(AbstractGraph): manually select model tokens 
closes #768
 2024-10-28 14:51:39 +01:00
Marco Vinciguerra
2d91848b76 a
Some checks failed
CodeQL / Analyze (python) (push) Has been cancelled
Details
Release / Build (push) Has been cancelled
Details
Release / Release (push) Has been cancelled
Details
2024-10-28 14:16:47 +01:00
Marco Vinciguerra
15415eebbc Update funding.json 2024-10-28 14:15:24 +01:00
Marco Vinciguerra
bc19e898ae Update funding.json 2024-10-28 14:07:28 +01:00
Marco Vinciguerra
5ed28976d2 Update funding.json 2024-10-28 14:05:47 +01:00
Marco Vinciguerra
6418479f49 Update funding.json 2024-10-28 13:59:46 +01:00
Marco Vinciguerra
e97add5daf Update funding.json 2024-10-28 13:58:01 +01:00
Marco Vinciguerra
8a69fb5ccc Update funding.json 2024-10-28 13:55:23 +01:00
Marco Vinciguerra
300fd5ac5b Create funding.json 2024-10-28 13:38:56 +01:00
semantic-release-bot
62369e3e28 ci(release): 1.27.0-beta.12 [skip ci] 
## [1.27.0-beta.12](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.27.0-beta.11...v1.27.0-beta.12) (2024-10-28)

### Features

* refactoring of the base_graph ([12a6c18](12a6c18f6a))
 2024-10-28 08:59:32 +00:00
Marco Vinciguerra
12a6c18f6a feat: refactoring of the base_graph 2024-10-28 09:58:03 +01:00
semantic-release-bot
3b2cadce1a ci(release): 1.27.0-beta.11 [skip ci] 
## [1.27.0-beta.11](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.27.0-beta.10...v1.27.0-beta.11) (2024-10-27)

### Features

* add new mistral models ([6914170](6914170890))
 2024-10-27 09:23:55 +00:00
Marco Vinciguerra
58bb0dccfa
Merge pull request #769 from ScrapeGraphAI/add-mistral-models 
feat: add new mistral models
 2024-10-27 10:22:11 +01:00
Marco Vinciguerra
6914170890 feat: add new mistral models 2024-10-26 16:07:43 +02:00
Marco Vinciguerra
eb24da5a8d Update overview.rst
Some checks failed
CodeQL / Analyze (python) (push) Has been cancelled
Details
/ build (push) Has been cancelled
Details
Release / Build (push) Has been cancelled
Details
Release / Release (push) Has been cancelled
Details
2024-10-26 10:29:37 +02:00
Marco Vinciguerra
a7df68490e Merge branch 'main' of https://github.com/ScrapeGraphAI/Scrapegraph-ai 2024-10-26 10:27:55 +02:00
Marco Vinciguerra
849fe395da update doc 2024-10-26 10:27:53 +02:00
semantic-release-bot
3933d64601 ci(release): 1.27.0 [skip ci] 
## [1.27.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.7...v1.27.0) (2024-10-26)

### Features

* add conditional node structure to the smart_scraper_graph and implemented a structured way to check condition ([cacd9cd](cacd9cde00))
* add integration with scrape.do ([ae275ec](ae275ec5e8))
* add model integration gpt4 ([51c55eb](51c55eb3a2))
* implement ScrapeGraph class for only web scraping automation ([612c644](612c644623))
* Implement SmartScraperMultiParseMergeFirstGraph class that scrapes a list of URLs and merge the content first and finally generates answers to a given prompt. ([3e3e1b2](3e3e1b2f3a))
* refactoring of export functions ([0ea00c0](0ea00c078f))
* refactoring of get_probable_tags node ([f658092](f658092dff))
* refactoring of ScrapeGraph to SmartScraperLiteGraph ([52b6bf5](52b6bf5fb8))

### Bug Fixes

* fix export function ([c8a000f](c8a000f1d9))
* fix the example variable name ([69ff649](69ff649556))
* remove variable "max_result" not being used in the code ([e76a68a](e76a68a782))

### chore

* fix example ([9cd9a87](9cd9a874f9))

### Test

* Add scrape_graph test ([cdb3c11](cdb3c1100e))
* Add smart_scraper_multi_parse_merge_first_graph test ([464b8b0](464b8b04ea))

### CI

* **release:** 1.26.6-beta.1 [skip ci] ([e0fc457](e0fc457d1a))
* **release:** 1.27.0-beta.1 [skip ci] ([9266a36](9266a36b2e))
* **release:** 1.27.0-beta.10 [skip ci] ([eee131e](eee131e959))
* **release:** 1.27.0-beta.2 [skip ci] ([d84d295](d84d295389))
* **release:** 1.27.0-beta.3 [skip ci] ([f576afa](f576afaf0c))
* **release:** 1.27.0-beta.4 [skip ci] ([3d6bbcd](3d6bbcdaa3))
* **release:** 1.27.0-beta.5 [skip ci] ([5002c71](5002c713d5))
* **release:** 1.27.0-beta.6 [skip ci] ([94b9836](94b9836ef6))
* **release:** 1.27.0-beta.7 [skip ci] ([407f1ce](407f1ce4eb))
* **release:** 1.27.0-beta.8 [skip ci] ([4f1ed93](4f1ed939e6))
* **release:** 1.27.0-beta.9 [skip ci] ([fd57cc7](fd57cc7c12))
 2024-10-26 08:06:36 +00:00