CodeCow/Scrapegraph-ai
1
0
Fork 0
mirror of https://github.com/VinciGit00/Scrapegraph-ai.git synced 2026-07-09 21:19:20 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,279 Commits 2 Branches 448 Tags 17 MiB
e8cabfd1ae
Commit Graph

2279 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
aziz-ullah-khan
e8cabfd1ae fix: resolved outparser issue 2024-11-03 20:23:29 +05:00
semantic-release-bot
65d39bbaf0 ci(release): 1.28.0-beta.3 [skip ci] 
## [1.28.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.28.0-beta.2...v1.28.0-beta.3) (2024-11-02)

### Features

* Serper API integration for Google search ([c218546](c218546a3d))
 2024-11-02 17:36:10 +00:00
Marco Vinciguerra
902699c8d2
Merge pull request #781 from aziz-ullah-khan/pre/beta 2024-11-02 18:34:51 +01:00
aziz-ullah-khan
c218546a3d feat: Serper API integration for Google search 2024-11-02 15:02:08 +05:00
semantic-release-bot
7e3598ddfa ci(release): 1.28.0-beta.2 [skip ci] 
## [1.28.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.28.0-beta.1...v1.28.0-beta.2) (2024-10-31)

### Features

* update generate answer ([7172b32](7172b32a0f))
 2024-10-31 09:21:45 +00:00
Federico Aguzzi
76e2820ff1
Merge pull request #778 from ScrapeGraphAI/refactoring-generate-answer 
feat: update generate answer
 2024-10-31 10:20:17 +01:00
Marco Vinciguerra
7440f3735e
Merge pull request #779 from ScrapeGraphAI/main 
allignment
 2024-10-31 09:06:15 +01:00
Marco Vinciguerra
c2b68095c1 Update funding.json 2024-10-30 18:07:40 +01:00
Marco Vinciguerra
b75fbd26c5 add funding 2024-10-30 18:04:41 +01:00
Marco Vinciguerra
655ef28069 Update funding.json 2024-10-30 17:52:34 +01:00
Marco Vinciguerra
542751d35b Update funding.json 2024-10-30 14:23:11 +01:00
semantic-release-bot
8cbe582ea9 ci(release): 1.28.0-beta.1 [skip ci] 
## [1.28.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.27.0...v1.28.0-beta.1) (2024-10-30)

### Features

* add new mistral models ([6914170](6914170890))
* refactoring of the base_graph ([12a6c18](12a6c18f6a))

### Bug Fixes

* **AbstractGraph:** manually select model tokens ([f79f399](f79f399ee0)), closes [#768](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/768)

### CI

* **release:** 1.27.0-beta.11 [skip ci] ([3b2cadc](3b2cadce1a))
* **release:** 1.27.0-beta.12 [skip ci] ([62369e3](62369e3e28))
* **release:** 1.27.0-beta.13 [skip ci] ([deed355](deed355551)), closes [#768](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/768)
 2024-10-30 08:02:00 +00:00
Marco Vinciguerra
bb2373d7a2
Merge pull request #775 from U-C4N/main 
This commit focuses on optimizing the utility modules in the codebase…
 2024-10-30 09:00:33 +01:00
Marco Vinciguerra
09c967822d
Merge branch 'pre/beta' into main 2024-10-30 09:00:20 +01:00
Marco Vinciguerra
7172b32a0f feat: update generate answer 2024-10-30 08:59:13 +01:00
semantic-release-bot
deed355551 ci(release): 1.27.0-beta.13 [skip ci] 
## [1.27.0-beta.13](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.27.0-beta.12...v1.27.0-beta.13) (2024-10-29)

### Bug Fixes

* **AbstractGraph:** manually select model tokens ([f79f399](f79f399ee0)), closes [#768](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/768)
 2024-10-29 08:02:25 +00:00
Marco Vinciguerra
8ab726e9c1
Merge pull request #773 from ScrapeGraphAI/768-fix-model-tokens 
fix(AbstractGraph): manually select model tokens
 2024-10-29 09:00:46 +01:00
Umut CAN
827f7260ad This commit focuses on optimizing the utility modules in the codebase for better performance and maintainability. Key improvements include: - More efficient HTML processing with combined regex operations and optimized tag handling - Enhanced deep copy functionality with better type handling and optimized recursion - Refactored web search with improved error handling and modular helper functions The changes maintain all existing functionality while improving code quality, performance, and maintainability. Documentation and type hints have been enhanced throughout. 
Optimize utils modules for better performance and maintainability

- Improve HTML cleanup and minification:
  - Combine regex operations for better performance
  - Add better error handling for HTML processing
  - Optimize tag removal and attribute filtering

- Enhance deep copy functionality:
  - Add special case handling for primitive types
  - Improve type checking and error handling
  - Optimize recursive copying for collections

- Refactor web search functionality:
  - Add input validation and error handling
  - Split search logic into separate helper functions
  - Improve proxy handling and configuration
  - Add better timeout and error management
  - Optimize URL filtering and processing

Technical improvements:
- Better type hints and documentation
- More efficient data structures
- Improved error handling and validation
- Reduced code duplication
- Better separation of concerns

No breaking changes - all existing functionality maintained
 2024-10-28 22:40:32 +03:00
Federico Aguzzi
f79f399ee0 fix(AbstractGraph): manually select model tokens 
closes #768
 2024-10-28 14:51:39 +01:00
Marco Vinciguerra
2d91848b76 a
Some checks failed
CodeQL / Analyze (python) (push) Has been cancelled
Details
Release / Build (push) Has been cancelled
Details
Release / Release (push) Has been cancelled
Details
2024-10-28 14:16:47 +01:00
Marco Vinciguerra
15415eebbc Update funding.json 2024-10-28 14:15:24 +01:00
Marco Vinciguerra
bc19e898ae Update funding.json 2024-10-28 14:07:28 +01:00
Marco Vinciguerra
5ed28976d2 Update funding.json 2024-10-28 14:05:47 +01:00
Marco Vinciguerra
6418479f49 Update funding.json 2024-10-28 13:59:46 +01:00
Marco Vinciguerra
e97add5daf Update funding.json 2024-10-28 13:58:01 +01:00
Marco Vinciguerra
8a69fb5ccc Update funding.json 2024-10-28 13:55:23 +01:00
Marco Vinciguerra
300fd5ac5b Create funding.json 2024-10-28 13:38:56 +01:00
semantic-release-bot
62369e3e28 ci(release): 1.27.0-beta.12 [skip ci] 
## [1.27.0-beta.12](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.27.0-beta.11...v1.27.0-beta.12) (2024-10-28)

### Features

* refactoring of the base_graph ([12a6c18](12a6c18f6a))
 2024-10-28 08:59:32 +00:00
Marco Vinciguerra
12a6c18f6a feat: refactoring of the base_graph 2024-10-28 09:58:03 +01:00
semantic-release-bot
3b2cadce1a ci(release): 1.27.0-beta.11 [skip ci] 
## [1.27.0-beta.11](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.27.0-beta.10...v1.27.0-beta.11) (2024-10-27)

### Features

* add new mistral models ([6914170](6914170890))
 2024-10-27 09:23:55 +00:00
Marco Vinciguerra
58bb0dccfa
Merge pull request #769 from ScrapeGraphAI/add-mistral-models 
feat: add new mistral models
 2024-10-27 10:22:11 +01:00
Marco Vinciguerra
6914170890 feat: add new mistral models 2024-10-26 16:07:43 +02:00
Marco Vinciguerra
eb24da5a8d Update overview.rst
Some checks failed
CodeQL / Analyze (python) (push) Has been cancelled
Details
/ build (push) Has been cancelled
Details
Release / Build (push) Has been cancelled
Details
Release / Release (push) Has been cancelled
Details
2024-10-26 10:29:37 +02:00
Marco Vinciguerra
a7df68490e Merge branch 'main' of https://github.com/ScrapeGraphAI/Scrapegraph-ai 2024-10-26 10:27:55 +02:00
Marco Vinciguerra
849fe395da update doc 2024-10-26 10:27:53 +02:00
semantic-release-bot
3933d64601 ci(release): 1.27.0 [skip ci] 
## [1.27.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.7...v1.27.0) (2024-10-26)

### Features

* add conditional node structure to the smart_scraper_graph and implemented a structured way to check condition ([cacd9cd](cacd9cde00))
* add integration with scrape.do ([ae275ec](ae275ec5e8))
* add model integration gpt4 ([51c55eb](51c55eb3a2))
* implement ScrapeGraph class for only web scraping automation ([612c644](612c644623))
* Implement SmartScraperMultiParseMergeFirstGraph class that scrapes a list of URLs and merge the content first and finally generates answers to a given prompt. ([3e3e1b2](3e3e1b2f3a))
* refactoring of export functions ([0ea00c0](0ea00c078f))
* refactoring of get_probable_tags node ([f658092](f658092dff))
* refactoring of ScrapeGraph to SmartScraperLiteGraph ([52b6bf5](52b6bf5fb8))

### Bug Fixes

* fix export function ([c8a000f](c8a000f1d9))
* fix the example variable name ([69ff649](69ff649556))
* remove variable "max_result" not being used in the code ([e76a68a](e76a68a782))

### chore

* fix example ([9cd9a87](9cd9a874f9))

### Test

* Add scrape_graph test ([cdb3c11](cdb3c1100e))
* Add smart_scraper_multi_parse_merge_first_graph test ([464b8b0](464b8b04ea))

### CI

* **release:** 1.26.6-beta.1 [skip ci] ([e0fc457](e0fc457d1a))
* **release:** 1.27.0-beta.1 [skip ci] ([9266a36](9266a36b2e))
* **release:** 1.27.0-beta.10 [skip ci] ([eee131e](eee131e959))
* **release:** 1.27.0-beta.2 [skip ci] ([d84d295](d84d295389))
* **release:** 1.27.0-beta.3 [skip ci] ([f576afa](f576afaf0c))
* **release:** 1.27.0-beta.4 [skip ci] ([3d6bbcd](3d6bbcdaa3))
* **release:** 1.27.0-beta.5 [skip ci] ([5002c71](5002c713d5))
* **release:** 1.27.0-beta.6 [skip ci] ([94b9836](94b9836ef6))
* **release:** 1.27.0-beta.7 [skip ci] ([407f1ce](407f1ce4eb))
* **release:** 1.27.0-beta.8 [skip ci] ([4f1ed93](4f1ed939e6))
* **release:** 1.27.0-beta.9 [skip ci] ([fd57cc7](fd57cc7c12))
 2024-10-26 08:06:36 +00:00
Marco Vinciguerra
b7d5a20ae0
Merge pull request #764 from ScrapeGraphAI/pre/beta 2024-10-26 10:05:15 +02:00
semantic-release-bot
eee131e959 ci(release): 1.27.0-beta.10 [skip ci] 
## [1.27.0-beta.10](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.27.0-beta.9...v1.27.0-beta.10) (2024-10-25)

### Bug Fixes

* fix export function ([c8a000f](c8a000f1d9))
 2024-10-25 06:45:23 +00:00
Marco Vinciguerra
f9c1432342
Merge pull request #767 from ScrapeGraphAI/fix-export-function 2024-10-25 08:43:40 +02:00
semantic-release-bot
fd57cc7c12 ci(release): 1.27.0-beta.9 [skip ci] 
## [1.27.0-beta.9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.27.0-beta.8...v1.27.0-beta.9) (2024-10-24)

### Features

* add model integration gpt4 ([51c55eb](51c55eb3a2))
 2024-10-24 22:39:44 +00:00
Marco Vinciguerra
9e5e76abbb
Merge pull request #765 from ScrapeGraphAI/add-model-integration-for-images 
feat: add model integration gpt4
 2024-10-25 00:38:16 +02:00
Marco Vinciguerra
4cd5ef296e add docstring files
Some checks failed
CodeQL / Analyze (python) (push) Has been cancelled
Details
/ build (push) Has been cancelled
Details
Release / Build (push) Has been cancelled
Details
Release / Release (push) Has been cancelled
Details
2024-10-24 15:28:27 +02:00
Marco Vinciguerra
6179ab99a4 Update data_export.py 2024-10-24 15:20:36 +02:00
Marco Vinciguerra
c8a000f1d9 fix: fix export function 2024-10-24 10:11:36 +02:00
Marco Vinciguerra
51c55eb3a2 feat: add model integration gpt4 2024-10-24 09:10:51 +02:00
semantic-release-bot
4f1ed939e6 ci(release): 1.27.0-beta.8 [skip ci] 
## [1.27.0-beta.8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.27.0-beta.7...v1.27.0-beta.8) (2024-10-24)

### Bug Fixes

* removed tokenizer ([a184716](a18471688f))

### CI

* **release:** 1.26.7 [skip ci] ([ec9ef2b](ec9ef2bcda))
 2024-10-24 06:55:58 +00:00
Marco Vinciguerra
066e77dbe7
Merge branch 'main' into pre/beta 2024-10-24 08:54:17 +02:00
semantic-release-bot
407f1ce4eb ci(release): 1.27.0-beta.7 [skip ci] 
## [1.27.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.27.0-beta.6...v1.27.0-beta.7) (2024-10-24)

### Features

* refactoring of get_probable_tags node ([f658092](f658092dff))
 2024-10-24 06:45:14 +00:00
Marco Vinciguerra
a1bd05da10
Merge pull request #763 from ScrapeGraphAI/refactoring-get-probable-tags 
feat: refactoring of get_probable_tags node
 2024-10-24 08:43:49 +02:00
Marco Vinciguerra
f658092dff feat: refactoring of get_probable_tags node 2024-10-23 12:15:16 +02:00