Marco Vinciguerra
d8ed76ba50
Merge pull request #221 from mayurdb/deepScrape
...
feat(n-level deep scrape): Modify SearchLinkNode to find out the relevant links from the webpage
2024-05-11 13:54:20 +02:00
mayurdb
dd29c16cbe
Merge branch 'deepScrape' of github.com:mayurdb/Scrapegraph-ai into deepScrape
2024-05-11 16:58:54 +05:30
mayurdb
9a67a26cd3
Update documentation
2024-05-11 16:57:22 +05:30
Mayur Bhosale
ea3b545573
Merge branch 'pre/beta' into deepScrape
2024-05-11 16:54:40 +05:30
mayurdb
8f1fbe7e10
minor changes
2024-05-11 16:52:36 +05:30
mayurdb
df271b6451
Add search link node that can find out relevant links in the webpage
2024-05-11 16:39:55 +05:30
roncaglionidaniele
13ae9180ac
docs: add diagram showing general structure/flow of the library
2024-05-11 12:27:04 +02:00
VinciGit00
76b0e39db0
update tests
2024-05-11 12:23:50 +02:00
Federico Minutoli
b54d984c13
fix(chromium-loader): ensure it subclasses langchain's base loader
2024-05-11 11:44:56 +02:00
VinciGit00
dc91719365
Update cleanup_html.py
2024-05-11 10:49:16 +02:00
semantic-release-bot
d359814c4a
ci(release): 0.10.1 [skip ci]
...
## [0.10.1](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.10.0...v0.10.1 ) (2024-05-11)
### Bug Fixes
* crash asyncio due dependency version ([256377325637734474bf90f34bf90f32a8cc28d5acc28d5a64fae5655fae5655fdde198420c198420c505772e064772e064c55f8d8d71f8d8d71589
2024-05-11 08:17:32 +00:00
Lorenzo Padoan
2563773447
fix: crash asyncio due dependency version
2024-05-11 10:13:41 +02:00
VinciGit00
78f21741cf
Merge branch 'main' of https://github.com/VinciGit00/Scrapegraph-ai
2024-05-11 09:44:22 +02:00
VinciGit00
04a4d84b69
Update serp_api_logo.png
2024-05-11 09:44:20 +02:00
Marco Vinciguerra
b752499fab
Merge pull request #217 from mayurdb/fetchLinkFix
...
Fetch links in the page while parsing html
2024-05-11 09:42:40 +02:00
Marco Vinciguerra
1fa77e579f
Merge pull request #215 from epage480/fix-GenerateScraperGraph
...
Fix for GenerateScraperGraph
2024-05-11 09:42:11 +02:00
Marco Vinciguerra
20604bd7ac
Merge pull request #218 from skrawcz/burr
2024-05-11 09:19:21 +02:00
Stefan Krawczyk
f2bb1cc4f8
Fixes LC document deserialization
...
Depends on https://github.com/DAGWorks-Inc/burr/pull/175 .
2024-05-11 00:06:02 -07:00
mayurdb
300fd5d253
Fetch links in the page while parsing html
2024-05-11 09:46:51 +05:30
Eric Page
0683e78e78
Merge branch 'pre/beta' into fix-GenerateScraperGraph
2024-05-11 01:59:28 +02:00
Eric Page
24c3b0580c
Removed nonfunctional RAG node from ScriptCreatorGraph
2024-05-11 01:38:40 +02:00
Eric Page
aac51ba290
Removed dead code, allows GenerateScraperNode to generate scraper with
...
one chunk of context
2024-05-11 01:34:51 +02:00
Eric Page
40884747c7
Added parse_html option in parse_node
2024-05-11 00:32:01 +02:00
Federico Minutoli
627cbeeb20
feat(parallel-exeuction): add asyncio event loop dispatcher with semaphore for parallel graph instances
...
TODO: still untested
2024-05-11 00:13:27 +02:00
Eric Page
67d8fec6b3
Minor typo fix for clarity
2024-05-11 00:05:13 +02:00
Federico Minutoli
fc2aa3ac1c
Merge branch 'pre/beta' of https://github.com/DiTo97/Scrapegraph-ai into fix/fetch-node-proxybroker
2024-05-10 21:20:40 +02:00
Federico Minutoli
768719cce8
feat(safe-web-driver): enchanced the original AsyncChromiumLoader web driver with proxy protection and flexible kwargs and backend
...
the original class prevents passing kwargs down to the playwright backend, making some config unfeasible, including passing a proxy server to the web driver.
the new class has backward compatibility with the original, but 1) allows any kwarg to be passed down to the web driver, 2) allows specifying the web driver backend (only playwright is supported for now) in case more (e.g., selenium) will be supported in the future and 3) automatically fetches a suitable proxy if one is not passed already
2024-05-10 21:13:38 +02:00
Federico Minutoli
217013181d
feat(proxy-rotation): add parse (IP address) or search (from broker) functionality for proxy rotation
...
the broker has been made fully configurable for anonymity level, admissible locations, scheme and max shape not to waste resources, unlike the original `free-proxy` package.
other options have been explored (e.g., `proxybroker`, `proxybroker2`) due to their built-in proxy server and rotation capabilities, but the former is no longer maintained, and the latter has issue with any python version outside of python 3.9
2024-05-10 21:09:48 +02:00
Federico Minutoli
db2234bf5d
feat(webdriver-backend): add dynamic import scripts from module and file
2024-05-10 21:06:05 +02:00
Federico Minutoli
2f4fd45700
fix(pytest): add dependency for mocking testing functions
2024-05-10 21:05:48 +02:00
Marco Vinciguerra
0bcb0fb3bb
Merge pull request #210 from skrawcz/burr
...
Burr
2024-05-10 19:54:29 +02:00
Stefan Krawczyk
82afa0ef3d
Working smart scraper graph
...
Bugs found in tracker persistence:
- serializing inputs properly for tracker
- deserializing state from a previous found
2024-05-10 10:49:53 -07:00
Stefan Krawczyk
d94195f823
WIP
2024-05-10 09:48:46 -07:00
Stefan Krawczyk
be16fecb55
WIP
2024-05-10 09:48:44 -07:00
semantic-release-bot
7ae50c035e
ci(release): 0.11.0-beta.2 [skip ci]
...
## [0.11.0-beta.2](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.1...v0.11.0-beta.2 ) (2024-05-10)
### Features
* revert fetch_node ([864aa91864aa91326
2024-05-10 13:13:20 +00:00
Marco Perini
864aa91326
feat: revert fetch_node
2024-05-10 15:11:54 +02:00
Marco Perini
b8a8ebbceb
Merge pull request #207 from VinciGit00/docs
...
Docs main readme
2024-05-10 14:49:39 +02:00
Marco Perini
23b1e5fc07
Merge branch 'main' into docs
2024-05-10 14:49:33 +02:00
Marco Perini
b5c1a7b607
Merge pull request #206 from VinciGit00/revert-205-patch-1
...
Revert "docs: update instructions to use with LocalAI"
2024-05-10 14:38:16 +02:00
Marco Perini
86be41ec6c
Revert "docs: update instructions to use with LocalAI"
2024-05-10 14:37:10 +02:00
Marco Vinciguerra
a433399bae
Merge pull request #205 from mudler/patch-1
...
docs: update instructions to use with LocalAI
2024-05-10 14:28:31 +02:00
Ettore Di Giacinto
198420c505
docs: update instructions to use with LocalAI
2024-05-10 14:17:53 +02:00
Marco Vinciguerra
702e913193
Update README.md
2024-05-10 11:58:22 +02:00
semantic-release-bot
63c0dd9372
ci(release): 0.11.0-beta.1 [skip ci]
...
## [0.11.0-beta.1](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.10.0...v0.11.0-beta.1 ) (2024-05-10)
### Features
* Add support for passing pdf path as source ([f10f3b1f10f3b14384ed0fb84ed0fb89c30ab31c30ab31c3fdbf8ce3d5f8ce3d5916324e977324e977b85a37fbbca37fbbcbcfad32298ad32298e70548bff9548bff9d7728c9dce28c9dce7cb460d292460d292af2
2024-05-10 09:15:24 +00:00
Marco Vinciguerra
4e62689eaa
Merge pull request #203 from mayurdb/fetchNodeFix
...
fix: Augment the information getting fetched from a webpage
2024-05-10 11:14:01 +02:00
Marco Vinciguerra
99adc9799f
Merge branch 'pre/beta' into fetchNodeFix
2024-05-10 11:13:54 +02:00
Marco Perini
f8d8d71589
docs: updated sponsor logo
2024-05-10 10:42:10 +02:00
VinciGit00
67b8a14592
Update README.md
2024-05-10 10:21:10 +02:00
VinciGit00
7ee5078ce0
Merge branch 'main' of https://github.com/VinciGit00/Scrapegraph-ai
2024-05-10 10:19:56 +02:00
VinciGit00
82318b9fc8
add sponsor
2024-05-10 10:19:34 +02:00
