CodeCow/Scrapegraph-ai
1
0
Fork 0
mirror of https://github.com/VinciGit00/Scrapegraph-ai.git synced 2026-07-12 21:01:56 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
783 Commits 2 Branches 448 Tags 17 MiB
4fd8a39d5b
Commit Graph

783 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Marco Perini
4fd8a39d5b Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta 2024-05-14 02:21:57 +02:00
Marco Perini
ced2bbcdc9 docs(concurrent): refactor theme and added benchmarck searchgraph 2024-05-14 02:21:46 +02:00
semantic-release-bot
fa4edb4703 ci(release): 0.11.0-beta.8 [skip ci] 
## [0.11.0-beta.8](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.7...v0.11.0-beta.8) (2024-05-13)

### Features

* **parallel-exeuction:** add asyncio event loop dispatcher with semaphore for parallel graph instances ([627cbee](627cbeeb20))
* **batchsize:** tested different batch sizes and systems ([a8d5e7d](a8d5e7db05))

### Bug Fixes

* **asyncio:** replaced deepcopy with copy due to serialization problems ([dedc733](dedc733047))
 2024-05-13 21:52:03 +00:00
Marco Perini
62a74a5737
Merge pull request #213 from DiTo97/feat/parallel-node-execution 
feat(parallel-exeuction): parallel asyncio event loop dispatcher
 2024-05-13 23:50:48 +02:00
Marco Perini
367dea5cbd
Merge branch 'pre/beta' into feat/parallel-node-execution 2024-05-13 23:50:40 +02:00
Marco Perini
a8d5e7db05 feat(batchsize): tested different batch sizes and systems 2024-05-13 23:49:48 +02:00
VinciGit00
c0d26d61d7 ad bedrocl 2024-05-13 20:17:03 +02:00
Marco Vinciguerra
28ab8da1fd
Merge pull request #229 from JGalego/feat/custom-aws-creds 
feat: Custom AWS credentials
 2024-05-13 20:14:28 +02:00
JGalego
859c5d5ba5 Refactored to include custom AWS client for bedrock; Added missing Anthropic class 2024-05-13 17:57:53 +01:00
Marco Perini
dedc733047 fix(asyncio): replaced deepcopy with copy due to serialization problems 2024-05-13 18:46:34 +02:00
Marco Vinciguerra
f3d44c09ae
Merge pull request #228 from JGalego/fix/bedrock-support 
fix: Amazon Bedrock Support
 2024-05-13 14:57:23 +02:00
semantic-release-bot
f0f73736f7 ci(release): 0.11.0-beta.7 [skip ci] 
## [0.11.0-beta.7](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.6...v0.11.0-beta.7) (2024-05-13)

### Bug Fixes

* bug for claude ([d0167de](d0167dee71))

### Docs

* **refactor:** changed example ([c7ec114](c7ec114274))
 2024-05-13 11:44:57 +00:00
VinciGit00
d0167dee71 fix: bug for claude 2024-05-13 13:43:42 +02:00
JGalego
39be38f292 Fixed anthropic/bedrock conflict; Removed duplicate class Claude; Updated BedrockEmbeddings class 2024-05-13 12:35:42 +01:00
VinciGit00
7c91f9f94c add examples for deepseek 2024-05-13 12:56:57 +02:00
Marco Perini
60ed80ff66 Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta 2024-05-13 12:28:11 +02:00
Marco Perini
c7ec114274 docs(refactor): changed example 2024-05-13 12:28:03 +02:00
semantic-release-bot
2724d3dd5f ci(release): 0.11.0-beta.6 [skip ci] 
## [0.11.0-beta.6](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.5...v0.11.0-beta.6) (2024-05-13)

### Bug Fixes

* **fetch-node:** removed isSoup from default ([0c15947](0c1594737f))
 2024-05-13 10:11:05 +00:00
Marco Perini
0c1594737f fix(fetch-node): removed isSoup from default 2024-05-13 12:09:55 +02:00
semantic-release-bot
353382b4d3 ci(release): 0.11.0-beta.5 [skip ci] 
## [0.11.0-beta.5](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.4...v0.11.0-beta.5) (2024-05-13)

### Features

* **webdriver-backend:** add dynamic import scripts from module and file ([db2234b](db2234bf5d))
* **proxy-rotation:** add parse (IP address) or search (from broker) functionality for proxy rotation ([2170131](217013181d))
* added proxy rotation ([0c36a7e](0c36a7ec1f))
* **safe-web-driver:** enchanced the original `AsyncChromiumLoader` web driver with proxy protection and flexible kwargs and backend ([768719c](768719cce8))

### Bug Fixes

* **pytest:** add dependency for mocking testing functions ([2f4fd45](2f4fd45700))
* **chromium-loader:** ensure it subclasses langchain's base loader ([b54d984](b54d984c13))
* **proxy-rotation:** removed duplicated arg and passed the loader_kwarhs correctly to the node ([1e9a564](1e9a564616))
* **proxy-rotation:** removed max_shape duplicate ([5d6d996](5d6d996e8f))

### Docs

* **refactor:** added proxy-rotation usage and refactor readthedocs ([e256b75](e256b758b2))
 2024-05-13 09:19:15 +00:00
Marco Perini
b8079f8759
Merge pull request #211 from DiTo97/fix/fetch-node-proxybroker 
feat: flexible web driver and proxy broker in fetch node
 2024-05-13 11:17:51 +02:00
Marco Perini
7e8acd8e6a
Merge branch 'pre/beta' into fix/fetch-node-proxybroker 2024-05-13 11:17:37 +02:00
Marco Perini
0c36a7ec1f feat: added proxy rotation 2024-05-13 11:04:56 +02:00
Marco Perini
e256b758b2 docs(refactor): added proxy-rotation usage and refactor readthedocs 2024-05-13 11:03:33 +02:00
Marco Perini
5d6d996e8f fix(proxy-rotation): removed max_shape duplicate 2024-05-13 07:26:43 +02:00
VinciGit00
30758b4b99 Create smart_scarper_deepseek.py 2024-05-12 20:27:35 +02:00
Marco Perini
1e9a564616 fix(proxy-rotation): removed duplicated arg and passed the loader_kwarhs correctly to the node 2024-05-12 18:39:03 +02:00
semantic-release-bot
4ccddda5eb ci(release): 0.11.0-beta.4 [skip ci] 
## [0.11.0-beta.4](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.3...v0.11.0-beta.4) (2024-05-12)

### Features

* add new prompt info ([e2350ed](e2350eda62))
 2024-05-12 16:10:12 +00:00
Marco Vinciguerra
f359d5cda4
Merge pull request #224 from VinciGit00/fixing-prompts 
feat: add new prompt info
 2024-05-12 18:08:54 +02:00
VinciGit00
e2350eda62 feat: add new prompt info 2024-05-12 11:14:30 +02:00
semantic-release-bot
106fb12531 ci(release): 0.11.0-beta.3 [skip ci] 
## [0.11.0-beta.3](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.2...v0.11.0-beta.3) (2024-05-12)

### Features

* add support for deepseek-chat ([156b67b](156b67b91e)), closes [#222](https://github.com/VinciGit00/Scrapegraph-ai/issues/222)

### Docs

* add diagram showing general structure/flow of the library ([13ae918](13ae9180ac))
* update overview diagram with more models ([b441b30](b441b30a5c))
 2024-05-12 07:14:00 +00:00
Marco Vinciguerra
e004c7c70e
Merge pull request #223 from f-aguzzi/pre/beta 
feat: add support for deepseek-chat
 2024-05-12 09:12:47 +02:00
Federico Aguzzi
156b67b91e feat: add support for deepseek-chat 
closes #222
 2024-05-12 01:00:11 +02:00
Marco Vinciguerra
3b9ec9b1f5
Merge pull request #220 from daniele-roncaglioni/102-library-overview-diagram 
docs: add diagram showing general structure/flow of the library
 2024-05-11 14:17:22 +02:00
roncaglionidaniele
b441b30a5c docs: update overview diagram with more models 2024-05-11 14:15:55 +02:00
Marco Vinciguerra
d8ed76ba50
Merge pull request #221 from mayurdb/deepScrape 
feat(n-level deep scrape): Modify SearchLinkNode to find out the relevant links from the webpage
 2024-05-11 13:54:20 +02:00
mayurdb
dd29c16cbe Merge branch 'deepScrape' of github.com:mayurdb/Scrapegraph-ai into deepScrape 2024-05-11 16:58:54 +05:30
mayurdb
9a67a26cd3 Update documentation 2024-05-11 16:57:22 +05:30
Mayur Bhosale
ea3b545573
Merge branch 'pre/beta' into deepScrape 2024-05-11 16:54:40 +05:30
mayurdb
8f1fbe7e10 minor changes 2024-05-11 16:52:36 +05:30
mayurdb
df271b6451 Add search link node that can find out relevant links in the webpage 2024-05-11 16:39:55 +05:30
roncaglionidaniele
13ae9180ac docs: add diagram showing general structure/flow of the library 2024-05-11 12:27:04 +02:00
Federico Minutoli
b54d984c13 fix(chromium-loader): ensure it subclasses langchain's base loader 2024-05-11 11:44:56 +02:00
VinciGit00
dc91719365 Update cleanup_html.py 2024-05-11 10:49:16 +02:00
Marco Vinciguerra
b752499fab
Merge pull request #217 from mayurdb/fetchLinkFix 
Fetch links in the page while parsing html
 2024-05-11 09:42:40 +02:00
Marco Vinciguerra
1fa77e579f
Merge pull request #215 from epage480/fix-GenerateScraperGraph 
Fix for GenerateScraperGraph
 2024-05-11 09:42:11 +02:00
mayurdb
300fd5d253 Fetch links in the page while parsing html 2024-05-11 09:46:51 +05:30
Eric Page
0683e78e78
Merge branch 'pre/beta' into fix-GenerateScraperGraph 2024-05-11 01:59:28 +02:00
Eric Page
24c3b0580c Removed nonfunctional RAG node from ScriptCreatorGraph 2024-05-11 01:38:40 +02:00
Eric Page
aac51ba290 Removed dead code, allows GenerateScraperNode to generate scraper with 
one chunk of context
 2024-05-11 01:34:51 +02:00