159 Commits

Author	SHA1	Message	Date
smith peng	553527a269	fix: fix pydantic object copy	2024-09-01 16:42:48 +08:00
Marco Vinciguerra	a73573d87a	update version	2024-08-31 12:32:25 +02:00
smith peng	71b22d4880	feat: add deepcopy error	2024-08-31 17:55:14 +08:00
smith peng	36818b1fb3	feat:adjust uncopiable obj raise error and remove memo	2024-08-31 17:39:33 +08:00
smith peng	4f4d091b82	feat:add deepcopy tool	2024-08-31 11:30:20 +08:00
Marco Vinciguerra	388630c0ff	fix: screenshot scraper	2024-08-30 13:02:13 +02:00
Marco Vinciguerra	a0d21137b7	refactoring of folders	2024-08-30 12:17:21 +02:00
Marco Vinciguerra	f7ba1f30de	refactoring of the code	2024-08-23 11:33:22 +02:00
Federico Aguzzi	b48ee825ee	Merge branch 'pre/beta' into support_structured_output_shema_openai	2024-08-19 13:31:01 +02:00
Federico Aguzzi	683bf57d89	fix(ParseNode): leave room for LLM reply in context window	2024-08-19 11:33:09 +02:00
Marco Vinciguerra	8b8d8f09b7	refactoring of the code according to pylint style	2024-08-18 11:51:34 +02:00
Marco Vinciguerra	b115f9612f	refactoring of code and documentation	2024-08-16 09:01:27 +02:00
Marco Vinciguerra	8b2c266aff	refactoring of the code ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-08-10 17:44:35 +02:00
Marco Vinciguerra	886bab78e7	Merge branch 'main' into temp-1	2024-08-06 10:04:57 +02:00
Marco Vinciguerra	9355507a2d	feat: refactoring of the code	2024-08-02 12:00:00 +02:00
Marco Perini	5c08eea189	docs: prev version	2024-07-24 23:37:57 +02:00
Marco Vinciguerra	e08b30495d	boh	2024-07-24 14:33:55 +02:00
Marco Perini	1d41f6eafe	fix: md conversion	2024-07-23 19:27:18 +02:00
Marco Perini	5a45e9f2d8	fix: md conversion	2024-07-23 18:09:15 +02:00
Marco Perini	12b5eada6e	fix(md_conversion): add absolute links md, added missing dependency	2024-07-23 15:34:12 +02:00
Marco Vinciguerra	602dd00209	feat: refactoring_to_md function	2024-07-16 12:39:48 +02:00
Federico Aguzzi	7ba2f6ae0b	chore: correct search engine name	2024-07-14 22:39:34 +02:00
Marco Vinciguerra	7e5789baa3	Update research_web.py	2024-07-14 19:31:22 +02:00
Federico Aguzzi	104d8692d6	Merge branch 'pre/beta' into support	2024-07-04 18:36:34 +02:00
Marco Vinciguerra	30ca15ca28	Merge branch 'md_scraper_integration' into integration_markdown	2024-06-30 16:58:37 +02:00
Marco Vinciguerra	2804434a9e	feat: add integrations for markdown files	2024-06-29 13:35:39 +02:00
Marco Vinciguerra	3a537eec6f	fix: add test	2024-06-23 17:54:09 +02:00
Marco Vinciguerra	9917972c11	fixed request	2024-06-22 21:39:37 +02:00
Marco Vinciguerra	7af411aa99	add trigger ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-06-21 13:36:27 +02:00
Marco Vinciguerra	5d6123847e	add new convert function ... Co-Authored-By: Federico Minutoli <40361744+DiTo97@users.noreply.github.com>	2024-06-20 21:15:16 +02:00
Marco Vinciguerra	8bb560a489	add convert function	2024-06-19 20:17:45 +02:00
Marco Vinciguerra	aa2160c108	feat: add research with bing + test function	2024-06-18 21:28:29 +02:00
Marco Vinciguerra	073d226723	feat: add new search engine avaiability and new tests	2024-06-18 14:35:13 +02:00
Marco Perini	283b61fafc	docs: better logging	2024-06-13 18:13:47 +02:00
Marco Vinciguerra	a6757aca57	Merge pull request #380 from Kshitij-Jande/main ... Strip out the scheme from the proxy server address URI.	2024-06-13 13:23:26 +02:00
Kshitij Jande	2a9ab69f4b	Strip out the scheme from the server address URI	2024-06-13 12:18:41 +05:30
Marco Vinciguerra	5065aa00e1	Merge branch 'pre/beta' into supercoder-327	2024-06-12 12:12:15 +02:00
supercoder-dev	879c94a2b5	Update cleanup_html.py	2024-06-12 14:30:02 +05:30
seyf97	acece72c28	Update cleanup_html.py ... Remove redundant lines in Links extraction	2024-06-04 13:49:00 +03:00
Marco Perini	8be27bad80	fix(3.9): python 3.9 logging fix	2024-05-31 22:32:20 +02:00
Marco Perini	5fb9115330	feat(version): python 3.12 is now supported 🚀	2024-05-26 08:49:27 +02:00
Marco Perini	5684578fab	fix(kg): removed unused nodes and utils	2024-05-25 00:44:14 +02:00
Marco Vinciguerra	b913b51cca	Merge branch 'logger-integration' into pre/beta	2024-05-24 12:39:14 +02:00
Federico Minutoli	d1394809d7	fix(logging): source code citation	2024-05-24 01:08:08 +02:00
Marco Perini	c75e6a06b1	feat(kg): working rag kg	2024-05-18 10:26:25 +02:00
Marco Perini	0196423bde	feat(knowledgegraph): add knowledge graph node	2024-05-17 23:41:44 +02:00
VinciGit00	cc5adefd29	fix: come back to the old version	2024-05-15 15:54:00 +02:00
VinciGit00	29d284e497	Merge branch 'main' into logger-integration	2024-05-15 15:28:20 +02:00
Marco Vinciguerra	cffcf80a75	Merge branch '88-blockscraper-implementation' into main	2024-05-15 15:20:12 +02:00
VinciGit00	e53766b16e	feat: add logger integration	2024-05-14 15:20:39 +02:00
Marco Perini	a296927624	feat(omni-scraper): working OmniScraperGraph with images	2024-05-14 13:46:49 +02:00
Marco Perini	7e8acd8e6a	Merge branch 'pre/beta' into fix/fetch-node-proxybroker	2024-05-13 11:17:37 +02:00
Marco Perini	5d6d996e8f	fix(proxy-rotation): removed max_shape duplicate	2024-05-13 07:26:43 +02:00
Marco Perini	1e9a564616	fix(proxy-rotation): removed duplicated arg and passed the loader_kwarhs correctly to the node	2024-05-12 18:39:03 +02:00
VinciGit00	dc91719365	Update cleanup_html.py	2024-05-11 10:49:16 +02:00
mayurdb	300fd5d253	Fetch links in the page while parsing html	2024-05-11 09:46:51 +05:30
Federico Minutoli	fc2aa3ac1c	Merge branch 'pre/beta' of https://github.com/DiTo97/Scrapegraph-ai into fix/fetch-node-proxybroker	2024-05-10 21:20:40 +02:00
Federico Minutoli	217013181d	feat(proxy-rotation): add parse (IP address) or search (from broker) functionality for proxy rotation ... the broker has been made fully configurable for anonymity level, admissible locations, scheme and max shape not to waste resources, unlike the original `free-proxy` package. other options have been explored (e.g., `proxybroker`, `proxybroker2`) due to their built-in proxy server and rotation capabilities, but the former is no longer maintained, and the latter has issue with any python version outside of python 3.9	2024-05-10 21:09:48 +02:00
Federico Minutoli	db2234bf5d	feat(webdriver-backend): add dynamic import scripts from module and file	2024-05-10 21:06:05 +02:00
Marco Perini	864aa91326	feat: revert fetch_node	2024-05-10 15:11:54 +02:00
mayurdb	f8ce3d5916	fix: Augment the information getting fetched from a webpage	2024-05-10 13:28:53 +05:30
Marco Vinciguerra	b326886250	Merge branch '88-blockscraper-implementation' into asdt	2024-05-07 13:27:30 +02:00
VinciGit00	aeb1acbf05	feat: refactoring search function	2024-05-03 21:06:09 +02:00
Perinim	cf038b33ea	docs: update utils docstrings	2024-05-01 12:35:12 +02:00
VinciGit00	a9b11e433a	fix: bug for calculate costs	2024-04-28 14:36:49 +02:00
VinciGit00	adbc08f27b	fix: robot node and proxyes	2024-04-27 19:07:37 +02:00
VinciGit00	b754dd909c	fix: changed proxy function	2024-04-27 14:40:50 +02:00
VinciGit00	f6077d1f98	feat: add new proxy rotation function	2024-04-27 13:31:53 +02:00
EURAC\marperini	e778d27169	added tree metadata	2024-04-26 15:17:44 +02:00
EURAC\marperini	dd99ac595e	structural and textual hashing	2024-04-25 11:56:41 +02:00
EURAC\marperini	c5f9fcaabe	Merge branch 'asdt' of https://github.com/VinciGit00/Scrapegraph-ai into asdt	2024-04-25 11:06:54 +02:00
EURAC\marperini	c927f707e9	subtrees implementation	2024-04-25 11:06:05 +02:00
VinciGit00	f232717bf8	REFACTORING	2024-04-25 09:19:44 +02:00
EURAC\marperini	ae69e4b340	dom tree structure	2024-04-24 18:59:44 +02:00
VinciGit00	74d7ef614a	refactor of conbert to json function	2024-04-21 20:30:56 +02:00
Alok Saboo	6cbf992870	chore: Remove .csv extension from filename in convert_to_csv.py	2024-04-21 14:13:34 -04:00
VinciGit00	3f95801737	Merge branch 'main' of https://github.com/VinciGit00/Scrapegraph-ai	2024-04-19 10:36:40 +02:00
EURAC\marperini	134c94e0ed	added first asdt implementation	2024-04-17 17:12:59 +02:00
Andrea Rota	b0e446f014	feat: apply remove to the document before updating the state	2024-04-17 11:24:56 +02:00
Marco Vinciguerra	4703a0b94c	Update remover.py	2024-04-16 12:27:17 +02:00
VinciGit00	9661c77ebe	add minimizer function	2024-04-16 12:07:43 +02:00
VinciGit00	3640434f5c	add utils	2024-04-12 12:59:15 +02:00
VinciGit00	6dc4c6dec4	Merge branch 'main' into research_branch	2024-04-06 14:44:56 +02:00
EURAC\marperini	57a53bd6af	refactor examples	2024-04-06 13:22:42 +02:00
EURAC\marperini	c2709aa50c	working search graph	2024-04-06 13:01:04 +02:00
VinciGit00	4c3ea8f4ac	fixed image_to_tex_node and refactoring	2024-04-03 12:53:12 +02:00
VinciGit00	b53bfefeea	removed junk if	2024-04-02 12:57:27 +02:00
VinciGit00	471df48a22	add google search	2024-04-01 13:36:02 +02:00
VinciGit00	b8a3590235	add SearcInetnetNode	2024-03-28 13:14:39 +01:00
VinciGit00	fff1f9ee51	new function for searching	2024-03-28 12:41:45 +01:00
VinciGit00	a28ca993de	refactoring of examples folder	2024-03-27 21:41:21 +01:00
VinciGit00	672e9a5191	refactoring of research function	2024-03-27 21:29:10 +01:00
VinciGit00	97d0a6887f	add reserch web function	2024-03-27 21:22:04 +01:00
VinciGit00	587dfa5c68	refactoring of the saving functions	2024-03-26 14:15:36 +01:00
VinciGit00	1afb950751	refactoring of code, add example on the readme	2024-03-18 14:35:47 +01:00
Perinim	875a7cc4f0	refactored RagNode and GenerateAnswerNode	2024-03-17 16:48:58 +01:00
Perinim	3585cd81e5	refactor fetch node	2024-03-17 12:02:35 +01:00
Perinim	20d7b69008	refactored BaseNode and ParseNode	2024-03-17 11:40:35 +01:00
Perinim	538133dfff	added parse input string function	2024-03-13 19:34:51 +01:00
Perinim	9d31db2d35	working smart scraper example	2024-03-12 11:06:08 +01:00