CodeCow/Scrapegraph-ai
1
0
Fork 0
mirror of https://github.com/VinciGit00/Scrapegraph-ai.git synced 2026-07-12 21:01:56 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
774 Commits 2 Branches 448 Tags 17 MiB
28ab8da1fd
Commit Graph

774 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Marco Vinciguerra
28ab8da1fd
Merge pull request #229 from JGalego/feat/custom-aws-creds 
feat: Custom AWS credentials
 2024-05-13 20:14:28 +02:00
JGalego
859c5d5ba5 Refactored to include custom AWS client for bedrock; Added missing Anthropic class 2024-05-13 17:57:53 +01:00
Marco Vinciguerra
f3d44c09ae
Merge pull request #228 from JGalego/fix/bedrock-support 
fix: Amazon Bedrock Support
 2024-05-13 14:57:23 +02:00
semantic-release-bot
f0f73736f7 ci(release): 0.11.0-beta.7 [skip ci] 
## [0.11.0-beta.7](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.6...v0.11.0-beta.7) (2024-05-13)

### Bug Fixes

* bug for claude ([d0167de](d0167dee71))

### Docs

* **refactor:** changed example ([c7ec114](c7ec114274))
 2024-05-13 11:44:57 +00:00
VinciGit00
d0167dee71 fix: bug for claude 2024-05-13 13:43:42 +02:00
JGalego
39be38f292 Fixed anthropic/bedrock conflict; Removed duplicate class Claude; Updated BedrockEmbeddings class 2024-05-13 12:35:42 +01:00
VinciGit00
7c91f9f94c add examples for deepseek 2024-05-13 12:56:57 +02:00
Marco Perini
60ed80ff66 Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta 2024-05-13 12:28:11 +02:00
Marco Perini
c7ec114274 docs(refactor): changed example 2024-05-13 12:28:03 +02:00
semantic-release-bot
2724d3dd5f ci(release): 0.11.0-beta.6 [skip ci] 
## [0.11.0-beta.6](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.5...v0.11.0-beta.6) (2024-05-13)

### Bug Fixes

* **fetch-node:** removed isSoup from default ([0c15947](0c1594737f))
 2024-05-13 10:11:05 +00:00
Marco Perini
0c1594737f fix(fetch-node): removed isSoup from default 2024-05-13 12:09:55 +02:00
semantic-release-bot
353382b4d3 ci(release): 0.11.0-beta.5 [skip ci] 
## [0.11.0-beta.5](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.4...v0.11.0-beta.5) (2024-05-13)

### Features

* **webdriver-backend:** add dynamic import scripts from module and file ([db2234b](db2234bf5d))
* **proxy-rotation:** add parse (IP address) or search (from broker) functionality for proxy rotation ([2170131](217013181d))
* added proxy rotation ([0c36a7e](0c36a7ec1f))
* **safe-web-driver:** enchanced the original `AsyncChromiumLoader` web driver with proxy protection and flexible kwargs and backend ([768719c](768719cce8))

### Bug Fixes

* **pytest:** add dependency for mocking testing functions ([2f4fd45](2f4fd45700))
* **chromium-loader:** ensure it subclasses langchain's base loader ([b54d984](b54d984c13))
* **proxy-rotation:** removed duplicated arg and passed the loader_kwarhs correctly to the node ([1e9a564](1e9a564616))
* **proxy-rotation:** removed max_shape duplicate ([5d6d996](5d6d996e8f))

### Docs

* **refactor:** added proxy-rotation usage and refactor readthedocs ([e256b75](e256b758b2))
 2024-05-13 09:19:15 +00:00
Marco Perini
b8079f8759
Merge pull request #211 from DiTo97/fix/fetch-node-proxybroker 
feat: flexible web driver and proxy broker in fetch node
 2024-05-13 11:17:51 +02:00
Marco Perini
7e8acd8e6a
Merge branch 'pre/beta' into fix/fetch-node-proxybroker 2024-05-13 11:17:37 +02:00
Marco Perini
0c36a7ec1f feat: added proxy rotation 2024-05-13 11:04:56 +02:00
Marco Perini
e256b758b2 docs(refactor): added proxy-rotation usage and refactor readthedocs 2024-05-13 11:03:33 +02:00
Marco Perini
5d6d996e8f fix(proxy-rotation): removed max_shape duplicate 2024-05-13 07:26:43 +02:00
VinciGit00
30758b4b99 Create smart_scarper_deepseek.py 2024-05-12 20:27:35 +02:00
Marco Perini
1e9a564616 fix(proxy-rotation): removed duplicated arg and passed the loader_kwarhs correctly to the node 2024-05-12 18:39:03 +02:00
semantic-release-bot
4ccddda5eb ci(release): 0.11.0-beta.4 [skip ci] 
## [0.11.0-beta.4](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.3...v0.11.0-beta.4) (2024-05-12)

### Features

* add new prompt info ([e2350ed](e2350eda62))
 2024-05-12 16:10:12 +00:00
Marco Vinciguerra
f359d5cda4
Merge pull request #224 from VinciGit00/fixing-prompts 
feat: add new prompt info
 2024-05-12 18:08:54 +02:00
VinciGit00
e2350eda62 feat: add new prompt info 2024-05-12 11:14:30 +02:00
semantic-release-bot
106fb12531 ci(release): 0.11.0-beta.3 [skip ci] 
## [0.11.0-beta.3](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.2...v0.11.0-beta.3) (2024-05-12)

### Features

* add support for deepseek-chat ([156b67b](156b67b91e)), closes [#222](https://github.com/VinciGit00/Scrapegraph-ai/issues/222)

### Docs

* add diagram showing general structure/flow of the library ([13ae918](13ae9180ac))
* update overview diagram with more models ([b441b30](b441b30a5c))
 2024-05-12 07:14:00 +00:00
Marco Vinciguerra
e004c7c70e
Merge pull request #223 from f-aguzzi/pre/beta 
feat: add support for deepseek-chat
 2024-05-12 09:12:47 +02:00
Federico Aguzzi
156b67b91e feat: add support for deepseek-chat 
closes #222
 2024-05-12 01:00:11 +02:00
Marco Vinciguerra
3b9ec9b1f5
Merge pull request #220 from daniele-roncaglioni/102-library-overview-diagram 
docs: add diagram showing general structure/flow of the library
 2024-05-11 14:17:22 +02:00
roncaglionidaniele
b441b30a5c docs: update overview diagram with more models 2024-05-11 14:15:55 +02:00
Marco Vinciguerra
d8ed76ba50
Merge pull request #221 from mayurdb/deepScrape 
feat(n-level deep scrape): Modify SearchLinkNode to find out the relevant links from the webpage
 2024-05-11 13:54:20 +02:00
mayurdb
dd29c16cbe Merge branch 'deepScrape' of github.com:mayurdb/Scrapegraph-ai into deepScrape 2024-05-11 16:58:54 +05:30
mayurdb
9a67a26cd3 Update documentation 2024-05-11 16:57:22 +05:30
Mayur Bhosale
ea3b545573
Merge branch 'pre/beta' into deepScrape 2024-05-11 16:54:40 +05:30
mayurdb
8f1fbe7e10 minor changes 2024-05-11 16:52:36 +05:30
mayurdb
df271b6451 Add search link node that can find out relevant links in the webpage 2024-05-11 16:39:55 +05:30
roncaglionidaniele
13ae9180ac docs: add diagram showing general structure/flow of the library 2024-05-11 12:27:04 +02:00
Federico Minutoli
b54d984c13 fix(chromium-loader): ensure it subclasses langchain's base loader 2024-05-11 11:44:56 +02:00
VinciGit00
dc91719365 Update cleanup_html.py 2024-05-11 10:49:16 +02:00
Marco Vinciguerra
b752499fab
Merge pull request #217 from mayurdb/fetchLinkFix 
Fetch links in the page while parsing html
 2024-05-11 09:42:40 +02:00
Marco Vinciguerra
1fa77e579f
Merge pull request #215 from epage480/fix-GenerateScraperGraph 
Fix for GenerateScraperGraph
 2024-05-11 09:42:11 +02:00
mayurdb
300fd5d253 Fetch links in the page while parsing html 2024-05-11 09:46:51 +05:30
Eric Page
0683e78e78
Merge branch 'pre/beta' into fix-GenerateScraperGraph 2024-05-11 01:59:28 +02:00
Eric Page
24c3b0580c Removed nonfunctional RAG node from ScriptCreatorGraph 2024-05-11 01:38:40 +02:00
Eric Page
aac51ba290 Removed dead code, allows GenerateScraperNode to generate scraper with 
one chunk of context
 2024-05-11 01:34:51 +02:00
Eric Page
40884747c7 Added parse_html option in parse_node 2024-05-11 00:32:01 +02:00
Eric Page
67d8fec6b3 Minor typo fix for clarity 2024-05-11 00:05:13 +02:00
Federico Minutoli
fc2aa3ac1c Merge branch 'pre/beta' of https://github.com/DiTo97/Scrapegraph-ai into fix/fetch-node-proxybroker 2024-05-10 21:20:40 +02:00
Federico Minutoli
768719cce8 feat(safe-web-driver): enchanced the original AsyncChromiumLoader web driver with proxy protection and flexible kwargs and backend 
the original class prevents passing kwargs down to the playwright backend, making some config unfeasible, including passing a proxy server to the web driver.

the new class has backward compatibility with the original, but 1) allows any kwarg to be passed down to the web driver, 2) allows specifying the web driver backend (only playwright is supported for now) in case more (e.g., selenium) will be supported in the future and 3) automatically fetches a suitable proxy if one is not passed already
 2024-05-10 21:13:38 +02:00
Federico Minutoli
217013181d feat(proxy-rotation): add parse (IP address) or search (from broker) functionality for proxy rotation 
the broker has been made fully configurable for anonymity level, admissible locations, scheme and max shape not to waste resources, unlike the original `free-proxy` package.

other options have been explored (e.g., `proxybroker`, `proxybroker2`) due to their built-in proxy server and rotation capabilities, but the former is no longer maintained, and the latter has issue with any python version outside of python 3.9
 2024-05-10 21:09:48 +02:00
Federico Minutoli
db2234bf5d feat(webdriver-backend): add dynamic import scripts from module and file 2024-05-10 21:06:05 +02:00
Federico Minutoli
2f4fd45700 fix(pytest): add dependency for mocking testing functions 2024-05-10 21:05:48 +02:00
semantic-release-bot
7ae50c035e ci(release): 0.11.0-beta.2 [skip ci] 
## [0.11.0-beta.2](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.1...v0.11.0-beta.2) (2024-05-10)

### Features

* revert fetch_node ([864aa91](864aa91326))
 2024-05-10 13:13:20 +00:00