Marco Vinciguerra
602dd00209
feat: refactoring_to_md function
2024-07-16 12:39:48 +02:00
Federico Aguzzi
7ba2f6ae0b
chore: correct search engine name
/ build (push) Has been cancelled
2024-07-14 22:39:34 +02:00
Marco Vinciguerra
7e5789baa3
Update research_web.py
2024-07-14 19:31:22 +02:00
Federico Aguzzi
104d8692d6
Merge branch 'pre/beta' into support
/ build (push) Has been cancelled
2024-07-04 18:36:34 +02:00
Marco Vinciguerra
30ca15ca28
Merge branch 'md_scraper_integration' into integration_markdown
/ build (3.10) (push) Has been cancelled
2024-06-30 16:58:37 +02:00
Marco Vinciguerra
2804434a9e
feat: add integrations for markdown files
/ build (3.10) (push) Waiting to run
2024-06-29 13:35:39 +02:00
Marco Vinciguerra
3a537eec6f
fix: add test
2024-06-23 17:54:09 +02:00
Marco Vinciguerra
9917972c11
fixed request
/ build (3.10) (push) Waiting to run
2024-06-22 21:39:37 +02:00
Marco Vinciguerra
7af411aa99
add trigger
...
Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>
2024-06-21 13:36:27 +02:00
Marco Vinciguerra
5d6123847e
add new convert function
...
/ build (3.10) (push) Waiting to run
Co-Authored-By: Federico Minutoli <40361744+DiTo97@users.noreply.github.com>
2024-06-20 21:15:16 +02:00
Marco Vinciguerra
8bb560a489
add convert function
2024-06-19 20:17:45 +02:00
Marco Vinciguerra
aa2160c108
feat: add research with bing + test function
/ build (3.10) (push) Has been cancelled
2024-06-18 21:28:29 +02:00
Marco Vinciguerra
073d226723
feat: add new search engine avaiability and new tests
2024-06-18 14:35:13 +02:00
Marco Perini
283b61fafc
docs: better logging
2024-06-13 18:13:47 +02:00
Marco Vinciguerra
a6757aca57
Merge pull request #380 from Kshitij-Jande/main
...
Strip out the scheme from the proxy server address URI.
2024-06-13 13:23:26 +02:00
Kshitij Jande
2a9ab69f4b
Strip out the scheme from the server address URI
2024-06-13 12:18:41 +05:30
Marco Vinciguerra
5065aa00e1
Merge branch 'pre/beta' into supercoder-327
2024-06-12 12:12:15 +02:00
supercoder-dev
879c94a2b5
Update cleanup_html.py
2024-06-12 14:30:02 +05:30
seyf97
acece72c28
Update cleanup_html.py
...
Remove redundant lines in Links extraction
2024-06-04 13:49:00 +03:00
Marco Perini
8be27bad80
fix(3.9): python 3.9 logging fix
2024-05-31 22:32:20 +02:00
Marco Perini
5fb9115330
feat(version): python 3.12 is now supported 🚀
2024-05-26 08:49:27 +02:00
Marco Perini
5684578fab
fix(kg): removed unused nodes and utils
2024-05-25 00:44:14 +02:00
Marco Vinciguerra
b913b51cca
Merge branch 'logger-integration' into pre/beta
2024-05-24 12:39:14 +02:00
Federico Minutoli
d1394809d7
fix(logging): source code citation
2024-05-24 01:08:08 +02:00
Marco Perini
c75e6a06b1
feat(kg): working rag kg
2024-05-18 10:26:25 +02:00
Marco Perini
0196423bde
feat(knowledgegraph): add knowledge graph node
2024-05-17 23:41:44 +02:00
VinciGit00
cc5adefd29
fix: come back to the old version
2024-05-15 15:54:00 +02:00
VinciGit00
29d284e497
Merge branch 'main' into logger-integration
2024-05-15 15:28:20 +02:00
Marco Vinciguerra
cffcf80a75
Merge branch '88-blockscraper-implementation' into main
2024-05-15 15:20:12 +02:00
VinciGit00
e53766b16e
feat: add logger integration
2024-05-14 15:20:39 +02:00
Marco Perini
a296927624
feat(omni-scraper): working OmniScraperGraph with images
2024-05-14 13:46:49 +02:00
Marco Perini
7e8acd8e6a
Merge branch 'pre/beta' into fix/fetch-node-proxybroker
2024-05-13 11:17:37 +02:00
Marco Perini
5d6d996e8f
fix(proxy-rotation): removed max_shape duplicate
2024-05-13 07:26:43 +02:00
Marco Perini
1e9a564616
fix(proxy-rotation): removed duplicated arg and passed the loader_kwarhs correctly to the node
2024-05-12 18:39:03 +02:00
VinciGit00
dc91719365
Update cleanup_html.py
2024-05-11 10:49:16 +02:00
mayurdb
300fd5d253
Fetch links in the page while parsing html
2024-05-11 09:46:51 +05:30
Federico Minutoli
fc2aa3ac1c
Merge branch 'pre/beta' of https://github.com/DiTo97/Scrapegraph-ai into fix/fetch-node-proxybroker
2024-05-10 21:20:40 +02:00
Federico Minutoli
217013181d
feat(proxy-rotation): add parse (IP address) or search (from broker) functionality for proxy rotation
...
the broker has been made fully configurable for anonymity level, admissible locations, scheme and max shape not to waste resources, unlike the original `free-proxy` package.
other options have been explored (e.g., `proxybroker`, `proxybroker2`) due to their built-in proxy server and rotation capabilities, but the former is no longer maintained, and the latter has issue with any python version outside of python 3.9
2024-05-10 21:09:48 +02:00
Federico Minutoli
db2234bf5d
feat(webdriver-backend): add dynamic import scripts from module and file
2024-05-10 21:06:05 +02:00
Marco Perini
864aa91326
feat: revert fetch_node
2024-05-10 15:11:54 +02:00
mayurdb
f8ce3d5916
fix: Augment the information getting fetched from a webpage
2024-05-10 13:28:53 +05:30
Marco Vinciguerra
b326886250
Merge branch '88-blockscraper-implementation' into asdt
2024-05-07 13:27:30 +02:00
VinciGit00
aeb1acbf05
feat: refactoring search function
2024-05-03 21:06:09 +02:00
Perinim
cf038b33ea
docs: update utils docstrings
2024-05-01 12:35:12 +02:00
VinciGit00
a9b11e433a
fix: bug for calculate costs
2024-04-28 14:36:49 +02:00
VinciGit00
adbc08f27b
fix: robot node and proxyes
2024-04-27 19:07:37 +02:00
VinciGit00
b754dd909c
fix: changed proxy function
2024-04-27 14:40:50 +02:00
VinciGit00
f6077d1f98
feat: add new proxy rotation function
2024-04-27 13:31:53 +02:00
EURAC\marperini
e778d27169
added tree metadata
2024-04-26 15:17:44 +02:00
EURAC\marperini
dd99ac595e
structural and textual hashing
2024-04-25 11:56:41 +02:00
