1007 Commits

Author	SHA1	Message	Date
Federico Aguzzi	d29338b7c2	feat(GenerateAnswerNode): built-in structured output through LangChain ... Co-Authored-By: Marco Vinciguerra <88108002+VinciGit00@users.noreply.github.com>	2024-08-19 13:45:37 +02:00
Federico Aguzzi	b48ee825ee	Merge branch 'pre/beta' into support_structured_output_shema_openai	2024-08-19 13:31:01 +02:00
Federico Aguzzi	683bf57d89	fix(ParseNode): leave room for LLM reply in context window	2024-08-19 11:33:09 +02:00
Marco Vinciguerra	f60aa3acde	feat: add async call	2024-08-19 11:22:40 +02:00
ekinsenler	08e9d9d6a0	feat: Implemented a filter logic in search_link_node.py ... feat: Added dict entry for Llama3.1:8b	2024-08-19 11:05:31 +03:00
Marco Vinciguerra	0bf79b5926	Update generate_answer_from_image_node.py	2024-08-19 02:21:45 +02:00
Marco Vinciguerra	79fa3f6bd6	add if and cool stuff	2024-08-19 02:01:41 +02:00
Marco Vinciguerra	c72c077eb6	refactoring of the nodes	2024-08-19 01:15:45 +02:00
Marco Vinciguerra	8d6c0b7288	Update generate_answer_node.py	2024-08-18 22:01:39 +02:00
Marco Vinciguerra	103c21c86e	Update generate_answer_from_image_node.py	2024-08-18 20:54:35 +02:00
Marco Vinciguerra	5eb3cff64f	feat: refactoring of the code	2024-08-18 20:53:35 +02:00
Marco Vinciguerra	8e3d5deaaa	add screenshot scraper	2024-08-18 19:39:49 +02:00
Marco Vinciguerra	8b8d8f09b7	refactoring of the code according to pylint style	2024-08-18 11:51:34 +02:00
Marco Vinciguerra	ebdb74967d	add new model for gpt	2024-08-17 15:00:44 +02:00
Marco Vinciguerra	609316c8e7	Update abstract_graph.py	2024-08-17 14:39:39 +02:00
Marco Vinciguerra	391d051a5e	updated	2024-08-17 14:39:19 +02:00
Marco Vinciguerra	0aca28732b	feat: update model tokens dict	2024-08-17 14:34:06 +02:00
Marco Vinciguerra	c2c2e6582f	Update generate_answer_node.py	2024-08-17 13:30:44 +02:00
Marco Vinciguerra	f0587908e2	Merge branch 'pre/beta' of https://github.com/ScrapeGraphAI/Scrapegraph-ai into pre/beta	2024-08-17 12:21:03 +02:00
Marco Vinciguerra	1d7f30b65b	fix: browser-base integration	2024-08-17 12:21:00 +02:00
Federico Aguzzi	5eed0a7654	fix(FetchNode) add OpenAI optimization to GPT models on Azure ... closes #547	2024-08-17 10:33:54 +02:00
Marco Vinciguerra	faef3186f7	fix: model count	2024-08-16 17:38:55 +02:00
Marco Vinciguerra	5100368b5a	Merge branch 'pre/beta' of https://github.com/ScrapeGraphAI/Scrapegraph-ai into pre/beta	2024-08-16 09:01:30 +02:00
Marco Vinciguerra	b115f9612f	refactoring of code and documentation	2024-08-16 09:01:27 +02:00
Marco Vinciguerra	333f09a289	Merge pull request #552 from sandeepchittilla/550-add-support-gpt4o-variants ... Add new feature to support gpt-4o variant models with different pricing	2024-08-16 09:00:31 +02:00
Federico Aguzzi	ea72d7609c	Merge pull request #536 from ScrapeGraphAI/refactoring-of-abstract-graph ... Refactoring of abstract graph	2024-08-15 16:46:03 +02:00
sandeepchittilla	855144876d	feat: Add new feature to support gpt-4o variant models with different pricing	2024-08-15 14:59:48 +01:00
Marco Vinciguerra	203ee2c186	removed unused imports	2024-08-14 11:07:11 +02:00
Marco Vinciguerra	597f2acd07	fixed doc bugs and removed unused rag node	2024-08-14 10:59:53 +02:00
Marco Vinciguerra	c5aebd73d3	Merge pull request #541 from ScrapeGraphAI/496-not-match-gemini-embeddings-model	2024-08-13 20:54:57 +02:00
Marco Vinciguerra	9be44742d6	Update abstract_graph.py	2024-08-13 19:29:03 +02:00
Marco Vinciguerra	795c9325cc	Merge branch 'pre/beta' into refactoring-of-abstract-graph	2024-08-12 18:35:55 +02:00
Marco Vinciguerra	c105c26b2f	Update abstract_graph.py	2024-08-12 18:32:41 +02:00
Federico Aguzzi	cb6b35397e	fix(models_tokens): incorrect provider names	2024-08-12 10:52:13 +02:00
Federico Aguzzi	bae89e832e	Merge branch 'pre/beta' into 530-probable-bug	2024-08-12 10:37:50 +02:00
Federico Aguzzi	e6bedb6701	fix(AbstractGraph): pass kwargs to Ernie and Nvidia models ... Co-Authored-By: Marco Vinciguerra <88108002+VinciGit00@users.noreply.github.com>	2024-08-12 10:11:45 +02:00
Marco Vinciguerra	8cece1d31b	Merge branch 'pre/beta' into refactoring-of-abstract-graph	2024-08-11 19:20:26 +02:00
Marco Vinciguerra	cef2fdb420	Update abstract_graph.py ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-08-11 19:19:17 +02:00
Marco Vinciguerra	c77231c983	feat: update abstract graph ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-08-11 19:18:24 +02:00
Marco Vinciguerra	de1ec250ef	refactoring pyproject.toml ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-08-11 18:04:31 +02:00
Marco Vinciguerra	cec5537f2a	add new tests ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-08-11 17:10:55 +02:00
Matteo Vedovati	a3b7181f95	quick fix typo	2024-08-11 11:52:50 +02:00
Matteo Vedovati	734b740bf9	Template name refactoring	2024-08-11 11:46:01 +02:00
Matteo Vedovati	98779d193b	search_link_node prompts moved	2024-08-11 11:43:22 +02:00
Matteo Vedovati	ef966525c3	search_link_node prompt moved	2024-08-11 11:34:49 +02:00
Matteo Vedovati	86a4903e99	search_internet_node prompt moved	2024-08-11 11:31:51 +02:00
Matteo Vedovati	f455fcbc5f	robot_node prompt moved	2024-08-11 11:27:27 +02:00
Matteo Vedovati	9814b6dd85	fixed versioning errors	2024-08-11 10:04:15 +02:00
Matteo Vedovati	7c6210019e	Merge branch 'pre/beta' of https://github.com/vedovati-matteo/Scrapegraph-ai into pre/beta	2024-08-11 09:42:38 +02:00
Matteo Vedovati	f17cef94bb	fix: merge_anwser prompt import	2024-08-11 09:41:31 +02:00
Matteo Vedovati	9fa3df2390	Changed prompts import from helpers to prompts	2024-08-11 09:33:21 +02:00
Matteo Vedovati	3b5b24d6f8	Added new folder for prompts	2024-08-11 09:32:00 +02:00
Matteo Vedovati	4ca606cf7a	remove promps form helpers folders	2024-08-11 09:31:26 +02:00
Marco Vinciguerra	0cda020d48	Merge branch 'pre/beta' into temp	2024-08-10 17:45:52 +02:00
Marco Vinciguerra	8b2c266aff	refactoring of the code ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-08-10 17:44:35 +02:00
Marco Vinciguerra	778efd4c87	fix: conditional node	2024-08-10 12:21:09 +02:00
Marco Vinciguerra	c3f1520240	Update abstract_graph.py	2024-08-10 12:13:09 +02:00
Marco Vinciguerra	6c3b37ab00	feat: add refactoring of default temperature	2024-08-10 11:51:37 +02:00
Marco Vinciguerra	f01b55e89b	fix: fetch node	2024-08-10 11:41:51 +02:00
Marco Vinciguerra	ce00345495	fix: conditional node	2024-08-09 23:38:26 +02:00
Marco Vinciguerra	1272273499	fix: broken node	2024-08-09 17:35:43 +02:00
Marco Vinciguerra	157d59d5c0	Merge branch 'pre/beta' into temp	2024-08-09 10:28:59 +02:00
Federico Aguzzi	50edbcc7f8	fix(FetchNode): missing bracket syntax error	2024-08-09 09:37:51 +02:00
Marco Vinciguerra	0048ba1542	Merge branch 'pre/beta' of https://github.com/ScrapeGraphAI/Scrapegraph-ai into pre/beta	2024-08-08 12:09:39 +02:00
Marco Vinciguerra	71ae3845b4	Update generate_answer_node_prompts.py	2024-08-08 12:09:38 +02:00
Federico Aguzzi	5ec2de9e1a	fix(chunking): count tokens from words instead of characters ... closes #513	2024-08-08 10:50:52 +02:00
Marco Vinciguerra	7f03926432	Merge pull request #515 from vedovati-matteo/pre/beta ... fix: refactoring of merge_answer_node	2024-08-07 21:50:14 +02:00
Matteo Vedovati	898e5a7af5	fix: refactoring of merge_answer_node	2024-08-07 21:42:54 +02:00
Marco Vinciguerra	b6bff0be33	Merge branch 'pre/beta' into mistral-ai-integration	2024-08-07 21:24:18 +02:00
Marco Vinciguerra	a7db47ecad	refatoring	2024-08-07 21:23:42 +02:00
Federico Aguzzi	786af992f8	fix(AbstractGraph): LangChain warnings handling, Mistral tokens	2024-08-07 15:16:10 +02:00
Federico Aguzzi	07720b6e0c	fix(FetchNode): handling of missing browser_base key	2024-08-07 15:15:13 +02:00
Marco Vinciguerra	b52e4a390b	fix: generate answer node omni	2024-08-07 14:09:02 +02:00
Marco Vinciguerra	625ca9f22a	fix: generate answer node pdf has a bug	2024-08-07 14:06:53 +02:00
Matteo Vedovati	bfc6852b77	fix: refactoring of fetch_node adding comment	2024-08-07 13:02:02 +02:00
Matteo Vedovati	1ea2ad8e79	fix: refactoring of fetch_node qixed error	2024-08-07 12:04:54 +02:00
Matteo Vedovati	29ad140fa3	fix: refactoring of fetch_node	2024-08-07 11:56:10 +02:00
Federico Aguzzi	f8ad616e10	chore(mistral): create examples	2024-08-06 18:16:15 +02:00
Federico Aguzzi	17f2707313	feat: add mistral support	2024-08-06 17:51:50 +02:00
Federico Aguzzi	5e824327c3	chore(models_tokens): add mistral models	2024-08-06 17:22:39 +02:00
Marco Vinciguerra	82e63213ae	Merge branch 'pre/beta' of https://github.com/ScrapeGraphAI/Scrapegraph-ai into pre/beta	2024-08-06 17:07:35 +02:00
Marco Vinciguerra	37dd6e9ba9	update reqs ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-08-06 17:07:33 +02:00
Federico Aguzzi	bc9e70031f	Merge branch 'main' into pre/beta	2024-08-06 16:50:46 +02:00
Marco Vinciguerra	6e4d04450f	Update base_graph.py	2024-08-06 16:29:59 +02:00
Marco Vinciguerra	fa651d4cd9	feat: add grok integration	2024-08-06 15:17:49 +02:00
Marco Vinciguerra	0571b6da55	feat: update base_graph	2024-08-06 14:01:11 +02:00
Marco Vinciguerra	886bab78e7	Merge branch 'main' into temp-1	2024-08-06 10:04:57 +02:00
Marco Vinciguerra	66a29bc5cc	refactoring of the openai examples ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-08-05 20:54:53 +02:00
Marco Vinciguerra	1fb711ae51	Merge branch 'pre/beta' of https://github.com/ScrapeGraphAI/Scrapegraph-ai into pre/beta	2024-08-05 14:54:16 +02:00
Marco Vinciguerra	a3d7f1b71c	Update csv_scraper_graph.py	2024-08-05 14:54:14 +02:00
Marco Vinciguerra	ddb3135109	Merge pull request #503 from ScrapeGraphAI/497-abstract-graph-refactor ... fix(AbstractGraph): instantiation of Azure GPT models	2024-08-02 14:32:29 +02:00
Federico Aguzzi	ade28fca2c	fix(AbstractGraph): instantiation of Azure GPT models ... Closes #498	2024-08-02 12:15:43 +02:00
Marco Vinciguerra	9355507a2d	feat: refactoring of the code	2024-08-02 12:00:00 +02:00
Marco Vinciguerra	fe099f9cbc	Merge branch 'pre/beta' into browserbase_integration	2024-08-01 15:14:49 +02:00
Lorenzo Padoan	abd0ea13f9	Merge pull request #500 from ScrapeGraphAI/497-abstract-graph-refactor ... 497 abstract graph refactor	2024-08-01 14:51:36 +02:00
Marco Vinciguerra	53dc06a2cb	Update speech_graph.py	2024-08-01 14:48:19 +02:00
Marco Vinciguerra	9ac74de622	Update speech_graph.py	2024-08-01 14:47:49 +02:00
Marco Vinciguerra	0b4cfd6522	fix: abstract_graph and removed unused embeddings ... Co-Authored-By: Federico Aguzzi <62149513+f-aguzzi@users.noreply.github.com>	2024-08-01 14:38:50 +02:00
Marco Vinciguerra	be870a4316	Update fetch_node.py ... Co-Authored-By: Federico Minutoli <40361744+DiTo97@users.noreply.github.com>	2024-08-01 13:24:48 +02:00
Marco Vinciguerra	6d8e02cd62	Update browser_base.py ... Co-Authored-By: Federico Minutoli <40361744+DiTo97@users.noreply.github.com>	2024-08-01 13:24:32 +02:00
Marco Vinciguerra	968c69e217	fix: fixed bug on fetch_node ... Co-Authored-By: Federico Minutoli <40361744+DiTo97@users.noreply.github.com>	2024-08-01 13:23:54 +02:00
Marco Vinciguerra	e21d461710	push	2024-08-01 13:16:49 +02:00
Marco Vinciguerra	d03eedccd7	Update chromium.py	2024-08-01 13:05:13 +02:00
Marco Vinciguerra	65f9e3a24c	Delete browser_base.py	2024-08-01 12:58:35 +02:00
Marco Vinciguerra	4cd5986516	Merge branch 'pre/beta' into browserbase_integration	2024-08-01 12:52:36 +02:00
Marco Vinciguerra	5ecdbe715f	feat: add integration in the abstract grapgh	2024-08-01 12:51:18 +02:00
Federico Aguzzi	a94ebcde00	refactor: move embeddings code from AbstractGraph to RAGNode	2024-08-01 11:53:17 +02:00
Marco Vinciguerra	7076ab12d3	allignment	2024-08-01 11:31:27 +02:00
Marco Vinciguerra	4caed545e5	feat: intregration of firebase	2024-07-31 19:49:59 +02:00
Federico Aguzzi	bb73d916a1	refactor: reuse code for common interface models	2024-07-31 13:41:09 +02:00
Federico Aguzzi	b17756d934	style: enforce pylint styling	2024-07-31 13:07:19 +02:00
Federico Aguzzi	88710f1a7c	chore: remove unused import	2024-07-30 15:57:08 +02:00
Federico Aguzzi	bc2c9967d2	refactor: remove redundant wrappers for Ernie and Nvidia	2024-07-30 10:59:12 +02:00
Federico Aguzzi	9275486240	refactor: remove redundant LangChain wrappers	2024-07-29 21:59:39 +02:00
Marco Vinciguerra	5007167af1	removed unused models ... Co-Authored-By: Federico Aguzzi <62149513+f-aguzzi@users.noreply.github.com>	2024-07-29 17:59:16 +02:00
Marco Vinciguerra	f6b7c6a430	refactoring ... Co-Authored-By: Federico Aguzzi <62149513+f-aguzzi@users.noreply.github.com>	2024-07-29 17:56:32 +02:00
Federico Aguzzi	25066b2bc5	refactor: remove LangChain wrappers for Ollama	2024-07-29 17:49:21 +02:00
Federico Aguzzi	2c5f934f10	refactor: remove LangChain wrappers	2024-07-29 17:46:17 +02:00
Marco Vinciguerra	9e795f4e35	removed unused init files ... Co-Authored-By: Federico Aguzzi <62149513+f-aguzzi@users.noreply.github.com>	2024-07-29 17:35:10 +02:00
Federico Aguzzi	5e3eb6e43d	refactor(OpenAI): integrate new LangChain chat init	2024-07-29 11:26:20 +02:00
Federico Aguzzi	d177afb68b	refactor(Ollama): integrate new LangChain chat init	2024-07-29 11:24:34 +02:00
amosdinh	05cf9adaa7	Ollama: Use no json format when creating the search query	2024-07-26 12:10:14 +02:00
Marco Vinciguerra	5137b8aa5b	Update models_tokens.py	2024-07-26 10:18:08 +02:00
Federico Aguzzi	d745564ad1	Merge pull request #485 from ScrapeGraphAI/generate_answer_parallel ... merge: generate answer parallel	2024-07-25 11:49:43 +02:00
Marco Perini	5c08eea189	docs: prev version	2024-07-24 23:37:57 +02:00
Marco Vinciguerra	377d679eec	feat: pdate models_tokens.py	2024-07-24 15:17:48 +02:00
Marco Vinciguerra	e08b30495d	boh	2024-07-24 14:33:55 +02:00
Marco Vinciguerra	41ea3e0e27	add llama3.1 model	2024-07-24 13:38:27 +02:00
Federico Aguzzi	ddbfbe1205	Merge branch 'pre/beta' into generate_answer_parallel	2024-07-23 20:51:35 +02:00
Marco Perini	1d41f6eafe	fix: md conversion	2024-07-23 19:27:18 +02:00
Marco Perini	5a45e9f2d8	fix: md conversion	2024-07-23 18:09:15 +02:00
Marco Perini	49b6468fb1	Merge branch 'main' into pre/beta	2024-07-23 16:14:15 +02:00
Marco Perini	0ebc387446	Merge pull request #487 from ScrapeGraphAI/update-generate-answer-node ... Update generate answer node	2024-07-23 15:45:20 +02:00
Marco Perini	12b5eada6e	fix(md_conversion): add absolute links md, added missing dependency	2024-07-23 15:34:12 +02:00
Marco Vinciguerra	df1ecc0019	Update generate_answer_node.py	2024-07-23 13:06:59 +02:00
Marco Vinciguerra	cf2734392c	removed unused function	2024-07-23 13:05:50 +02:00
Marco Vinciguerra	8dd941e39e	update generate answer nodes ... Co-Authored-By: Federico Aguzzi <62149513+f-aguzzi@users.noreply.github.com>	2024-07-23 12:40:11 +02:00
Marco Vinciguerra	879a554a78	Update telemetry.py	2024-07-23 11:15:44 +02:00
Marco Vinciguerra	70911f83be	Update generate_answer_pdf_node.py	2024-07-23 11:07:35 +02:00
Marco Vinciguerra	0c4b2908d9	feat: add generate_answer node paralellization ... Co-Authored-By: Federico Minutoli <40361744+DiTo97@users.noreply.github.com>	2024-07-22 19:58:33 +02:00
Marco Vinciguerra	2ae19aee56	update function	2024-07-22 16:50:58 +02:00
Marco Vinciguerra	09256f7b11	fix: parse node	2024-07-22 16:15:38 +02:00
Marco Vinciguerra	71f894eee3	fix: parse_html node have a bug	2024-07-22 16:01:27 +02:00
Marco Vinciguerra	fc0dadb8f8	feat: add nvidia connection	2024-07-22 11:56:33 +02:00
Marco Vinciguerra	b49f9866fd	fixed telemetry version	2024-07-21 19:20:19 +02:00
Marco Vinciguerra	ce6be37fbc	fix: abstract_graph moel token bug	2024-07-21 18:37:15 +02:00
Marco Vinciguerra	7a3c21b58d	Update generate_answer_node.py	2024-07-20 20:19:16 +02:00
Marco Vinciguerra	6a9205920f	Merge branch 'pre/beta' into temp	2024-07-20 20:03:20 +02:00
Marco Perini	99dc8497d8	docs(gpt-4o-mini): added new gpt, fixed chromium lazy loading, ... add documentation and metrics	2024-07-20 20:02:26 +02:00
Marco Perini	61a51d4bce	Merge pull request #466 from ScrapeGraphAI/refactoring_convert_to_md ... refactoring of generate answer node	2024-07-20 19:40:57 +02:00
Marco Vinciguerra	19af78c7d8	Merge branch 'pre/beta' into main	2024-07-20 18:14:36 +02:00
DragonelRoland	4b8360658e	Update models_tokens.py ... added gpt4o mini to the models	2024-07-20 16:51:24 +02:00
DragonelRoland	e9ecf5f0dc	Update typo in conditional_node.py ... "thar" -> "that"	2024-07-20 15:06:52 +02:00
Marco Vinciguerra	3cf7089e3f	Merge branch 'pre/beta' into temp	2024-07-20 13:27:37 +02:00
Marco Vinciguerra	77777c898d	fix: add gpt o mini for azure	2024-07-20 13:20:16 +02:00
Marco Vinciguerra	03f4a3aa29	fix: azure models	2024-07-20 13:19:21 +02:00
Marco Vinciguerra	3a9d556e1b	Merge branch 'pre/beta' into add-openai-supported-model-gpt-4o-mini	2024-07-19 10:08:41 +02:00
Kernick	4f079cfc74	Fixed a typo in the model name	2024-07-18 21:06:16 +03:00
Kernick	b91020ce56	Update models_tokens.py with gpt-4o-mini ... OpenAI added a new available model - gpt-4o-mini. Adding this model to the supported models, so it would be possible to use it with the Scrapegraph-ai.	2024-07-18 21:03:55 +03:00
Federico Aguzzi	dfa759e834	style(models): fix module docstrings	2024-07-18 11:02:51 +02:00
Marco Vinciguerra	032a491605	Update parse_node.py	2024-07-17 23:06:05 +02:00
Marco Vinciguerra	07f1e23d23	fix: parse_node	2024-07-17 22:58:21 +02:00
Marco Vinciguerra	68f58cc4dd	refactoring of generate answer node	2024-07-17 22:41:49 +02:00
Marco Vinciguerra	602dd00209	feat: refactoring_to_md function	2024-07-16 12:39:48 +02:00
Federico Aguzzi	dccb893fdf	Merge pull request #456 from ScrapeGraphAI/refactoring-of-search_link_node ... fix: search link node	2024-07-15 23:06:03 +02:00
Marco Vinciguerra	830daee1f3	Update search_link_node.py	2024-07-15 20:47:09 +02:00
Marco Vinciguerra	57fdaf9e3a	create search_link_graph	2024-07-15 12:36:11 +02:00
Federico Aguzzi	789c16fb76	Merge pull request #458 from ScrapeGraphAI/searchngx_integration ... merge: update research_web.py	2024-07-14 22:55:52 +02:00
Federico Aguzzi	7ba2f6ae0b	chore: correct search engine name	2024-07-14 22:39:34 +02:00
Federico Aguzzi	fd1b7cb24a	chore: remove unused import	2024-07-14 22:36:43 +02:00
Marco Vinciguerra	7e5789baa3	Update research_web.py	2024-07-14 19:31:22 +02:00
Marco Vinciguerra	cf3ab5564a	fix: search link node	2024-07-14 16:49:29 +02:00
Marco Vinciguerra	a7249685cb	removed rag node	2024-07-14 10:24:38 +02:00
salamanderxing	4b65392cca	Resolves #447	2024-07-12 10:56:25 +02:00
Marco Vinciguerra	765b548c0e	Merge pull request #417 from ScrapeGraphAI/md_scraper_integration ... feat: add integrations for markdown files	2024-07-04 19:42:53 +02:00
Marco Vinciguerra	8a52914871	Merge pull request #429 from ScrapeGraphAI/421-default-prompt-template-customization ... 421 default prompt template customization	2024-07-04 19:42:35 +02:00
Federico Aguzzi	104d8692d6	Merge branch 'pre/beta' into support	2024-07-04 18:36:34 +02:00
Federico Aguzzi	1bbd25a63f	Merge pull request #407 from ScrapeGraphAI/404-split-unit-testing-from-src ... Merge #404 (split unit testing from src)	2024-07-04 18:33:45 +02:00
Federico Aguzzi	d77a622b05	Merge branch '423-add-vertex-ai-integration' into support	2024-07-04 18:23:22 +02:00
Federico Aguzzi	720f18729b	Merge branch 'fireworks_integration' into support	2024-07-04 18:17:38 +02:00
Federico Aguzzi	2ab7ddc95a	Merge pull request #405 from ScrapeGraphAI/integration_markdown ... Integration markdown	2024-07-04 18:13:12 +02:00
Marco Vinciguerra	3ee1743687	update prompts	2024-07-04 14:22:45 +02:00
Marco Vinciguerra	ed2af51501	update the chunk size	2024-07-02 12:03:08 +02:00
Marco Vinciguerra	3bf5f570a8	feat: add integration for infos	2024-07-01 21:19:16 +02:00
AmosDinh	3f9bf5b4d4	fixed typo	2024-07-01 13:42:10 +00:00
Marco Vinciguerra	119514bdfc	feat: add vertexai integration	2024-07-01 12:21:47 +02:00
Marco Vinciguerra	5fe694b6b4	feat: improve md prompt recognition	2024-06-30 18:10:00 +02:00
Marco Vinciguerra	30ca15ca28	Merge branch 'md_scraper_integration' into integration_markdown	2024-06-30 16:58:37 +02:00
Marco Vinciguerra	b6a78e926f	Merge branch 'main' into temp	2024-06-30 16:54:15 +02:00
Marco Vinciguerra	2804434a9e	feat: add integrations for markdown files	2024-06-29 13:35:39 +02:00
Marco Vinciguerra	9b45ebcdcf	modify fetch node with no cut mode	2024-06-28 14:38:36 +02:00
Marco Vinciguerra	f3cbbcee92	Merge pull request #412 from oviniciusfeitosa/patch-1 ... 🐛 Rename user_prompt parameter to prompt	2024-06-28 11:09:25 +02:00
Marco Vinciguerra	5d2e592a67	add new claude models	2024-06-28 10:53:40 +02:00
Marco Vinciguerra	4d936410cc	fix: add new claude model	2024-06-28 10:50:01 +02:00
Vinícius Feitosa da Silva	a3c43c9a99	🐛 Rename user_prompt parameter to prompt ... This adjustment makes the class consistent with the definition of the arguments.	2024-06-27 17:53:20 -03:00
Marco Vinciguerra	228a1de2be	add new force	2024-06-27 18:57:27 +02:00
Marco Vinciguerra	ff4ccb94a1	add claude 3.5 integration	2024-06-25 11:21:06 +02:00
Marco Vinciguerra	0dfa5bf07e	Merge branch 'pre/beta' into main	2024-06-25 10:47:15 +02:00
Maorsg	f75e0835fc	added a function to the search_graph class to allow user to return URLs considered in the search	2024-06-24 21:31:28 -04:00
Marco Vinciguerra	df0e310829	feat: add fireworks integration	2024-06-24 23:11:28 +02:00
Marco Vinciguerra	3a537eec6f	fix: add test	2024-06-23 17:54:09 +02:00
Djamel Feddad	2b4f4f12be	doc: Fixed a small typo in the warning message ... This PR fixes a small type in the warning message	2024-06-23 15:34:48 +08:00
Marco Vinciguerra	9917972c11	fixed request	2024-06-22 21:39:37 +02:00
Marco Vinciguerra	afd46ac77b	fixed generate_answer_node	2024-06-22 11:31:54 +02:00
Marco Vinciguerra	d1c3de777f	fixed a bug	2024-06-21 14:14:43 +02:00
Marco Vinciguerra	599512d2e5	fix: add new model for claude	2024-06-21 14:08:10 +02:00
Marco Vinciguerra	7af411aa99	add trigger ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-06-21 13:36:27 +02:00
Marco Vinciguerra	5d6123847e	add new convert function ... Co-Authored-By: Federico Minutoli <40361744+DiTo97@users.noreply.github.com>	2024-06-20 21:15:16 +02:00
Marco Vinciguerra	2f02830c81	refactoring of fetch node	2024-06-20 13:44:42 +02:00
Marco Vinciguerra	5664eb292b	Update generate_answer_node_prompts.py	2024-06-20 11:57:11 +02:00
Marco Vinciguerra	23bc6332d0	fixed a bug	2024-06-19 21:46:31 +02:00
Marco Vinciguerra	8bb560a489	add convert function	2024-06-19 20:17:45 +02:00
Marco Vinciguerra	88f9def69d	fix: reduced model tokens	2024-06-19 19:55:13 +02:00
Marco Vinciguerra	79a2f51c34	add new models tokens	2024-06-19 10:42:41 +02:00
Marco Vinciguerra	40b4b2d939	add new models tokens	2024-06-19 10:42:00 +02:00
Marco Perini	0cb46f5f30	Merge pull request #392 from VinciGit00/391-switch-between-search-engines ... feat: add new search engine avaiability and new tests	2024-06-18 23:46:28 +02:00
Marco Perini	c7870905e1	fix: total tokens and docs	2024-06-18 23:22:23 +02:00
Marco Vinciguerra	aa2160c108	feat: add research with bing + test function	2024-06-18 21:28:29 +02:00
Jason Vertrees	aedda44868	fix: updated for schema changes ... docs: updated for schema changes	2024-06-18 13:31:10 -05:00
Marco Vinciguerra	073d226723	feat: add new search engine avaiability and new tests	2024-06-18 14:35:13 +02:00
Marco Vinciguerra	1d0cbbc6d6	fix: add new embedding models	2024-06-18 09:13:15 +02:00
Marco Vinciguerra	48ff87f20a	Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta	2024-06-17 14:17:46 +02:00
Marco Vinciguerra	03ffebc52d	fix: add chinese embedding model	2024-06-17 14:17:44 +02:00
Marco Perini	080a318ff6	feat(telemetry): add telemetry module	2024-06-17 13:00:33 +02:00
Marco Vinciguerra	da93162b53	Merge branch 'pre/beta' into main	2024-06-17 11:02:56 +02:00
shubihu	d8d5cd267a	Update abstract_graph.py ... fix: Incorrect API Key Error with OpenAI Proxy	2024-06-17 16:26:33 +08:00
Marco Perini	4c8becc721	overwrite common params to affect nodes config	2024-06-16 15:19:40 +02:00
Marco Vinciguerra	2419003999	fix: fix robot node	2024-06-16 14:04:36 +02:00
Marco Vinciguerra	3d34ea65df	Merge pull request #384 from liaoliaojun/fix/merge ... fix: TypeError ‘Model provided by the configuration not supported’	2024-06-16 13:34:43 +02:00
Marco Vinciguerra	dcd216e345	fix: removed duplicate from ollama dictionary ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-06-15 19:51:20 +02:00
liaoliaojun	62b372b675	fix: shallow copy config of create_embedder	2024-06-15 17:53:13 +08:00
Marco Perini	09cb6e964e	refactor: add missing schemas and renamed files	2024-06-14 15:38:46 +02:00
Marco Perini	12f4386552	Merge branch 'pre/beta' into 349-problem-with-scrapegraphaigraphspdf_scraper_graphpy	2024-06-14 15:22:43 +02:00
Marco Perini	203de83405	fix(pdf): correctly read .pdf files	2024-06-14 15:20:30 +02:00
Marco Perini	91c5b5af43	fix(multi): updated multi pdf scraper with schema	2024-06-14 14:59:12 +02:00
Marco Vinciguerra	7a34562d50	refactoring of merging answers nodes	2024-06-13 21:41:54 +02:00
Marco Perini	283b61fafc	docs: better logging	2024-06-13 18:13:47 +02:00
Marco Vinciguerra	a6757aca57	Merge pull request #380 from Kshitij-Jande/main ... Strip out the scheme from the proxy server address URI.	2024-06-13 13:23:26 +02:00
Kshitij Jande	2a9ab69f4b	Strip out the scheme from the server address URI	2024-06-13 12:18:41 +05:30
Marco Vinciguerra	1705046cc7	Update pdf_scraper_graph.py	2024-06-12 18:00:12 +02:00
Marco Vinciguerra	dc1340e302	Update generate_answer_pdf_node.py	2024-06-12 15:47:04 +02:00
Marco Vinciguerra	e45f159a31	enhanced performance and readibility	2024-06-12 14:59:10 +02:00
Marco Vinciguerra	58a257f05b	update model tokens	2024-06-12 12:41:58 +02:00
Marco Vinciguerra	e6c7940a57	feat: add Parse_Node	2024-06-12 12:29:14 +02:00
Marco Vinciguerra	5065aa00e1	Merge branch 'pre/beta' into supercoder-327	2024-06-12 12:12:15 +02:00
supercoder-dev	d0e300af72	Update fetch_node.py	2024-06-12 14:32:01 +05:30
supercoder-dev	879c94a2b5	Update cleanup_html.py	2024-06-12 14:30:02 +05:30
supercoder-dev	828bdeedce	Update smart_scraper_graph.py	2024-06-12 14:28:33 +05:30
Marco Perini	6b4cdf92b8	fix: common params	2024-06-12 03:06:21 +02:00
Marco Perini	ab00f23d85	fix(node): fixed generate answer node pydantic schema	2024-06-12 01:40:49 +02:00
Marco Perini	a10b060409	Merge pull request #361 from VinciGit00/multi_scraper_implementation ... Multi scraper implementation	2024-06-12 00:49:18 +02:00
Marco Perini	5d692bff9e	feat(schema): merge scripts to follow pydantic schema	2024-06-12 00:48:08 +02:00
Marco Perini	893aaddc5c	Merge pull request #359 from VinciGit00/semchunk_integration ... feat: add new chunking function	2024-06-11 23:16:00 +02:00
Marco Perini	589da1d695	Merge pull request #351 from VinciGit00/faiss_integration ... Faiss integration	2024-06-11 23:15:23 +02:00
Marco Perini	c881f64209	fix(cache): correctly pass the node arguments and logging	2024-06-11 22:56:09 +02:00
Marco Vinciguerra	fa951b4c8b	Merge pull request #368 from stevenmichaelthomas/wait-for-network-idle ... Add the ability to specify load state	2024-06-11 19:59:23 +02:00
Steven Thomas	8f405ff87a	Add the ability to specify load state	2024-06-11 11:22:39 -04:00
Marco Vinciguerra	81f89d88be	Update search_link_node.py	2024-06-10 16:14:24 +02:00
Marco Vinciguerra	9326637cc3	Merge branch 'main' into pre/beta	2024-06-09 17:04:11 +02:00
Marco Vinciguerra	1981230e6f	add multi scraper integration	2024-06-08 12:13:18 +02:00
Marco Perini	8228225dff	Merge pull request #360 from VinciGit00/burr_integration ... Burr integration	2024-06-08 12:10:34 +02:00
Marco Vinciguerra	e1f045b280	feat: add new chunking function	2024-06-08 11:44:09 +02:00
Marco Vinciguerra	543b48764a	add default folder for the cache	2024-06-07 09:47:21 +02:00
Marco Vinciguerra	d79036149a	feat: add caching	2024-06-06 21:35:52 +02:00
Marco Vinciguerra	10672d6ebb	fix: update openai tts class	2024-06-06 10:00:58 +02:00
Marco Perini	5d1fbf806a	feat(indexify-node): add example	2024-06-05 18:45:37 +02:00
Tin Do	d772453220	Refactor model_name attribute access in llm_model in robots_node.py ... - Changed the access of model_name from dictionary-style to attribute-style in llm_model to comply with langchain BaseChatModel. - Updated the conditional and split operations accordingly.	2024-06-05 11:00:33 -04:00
Marco Perini	5c9843f141	fix(schema): fixed json output	2024-06-05 15:07:21 +02:00
duke147	1a404e358d	Merge remote-tracking branch 'upstream/main' into ernie	2024-06-05 19:14:44 +08:00
duke147	2b2b910a37	support ernie	2024-06-05 19:02:34 +08:00
Marco Vinciguerra	2ef6d67a9e	Merge pull request #346 from duke147/ernie ... support ernie	2024-06-05 12:28:43 +02:00
Marco Vinciguerra	67d83cff46	fix: getter	2024-06-05 12:02:00 +02:00
duke147	4e16c9a81d	support ernie	2024-06-05 17:37:35 +08:00
Marco Vinciguerra	450fde601e	add get functions on the dictionary	2024-06-05 09:28:24 +02:00
Marco Vinciguerra	5d20186bf2	feat: add json as output	2024-06-05 09:20:20 +02:00
Marco Vinciguerra	1d38ed146a	fix: bug on generate_answer_node	2024-06-05 09:17:29 +02:00
Marco Vinciguerra	74fd530914	Merge branch 'pre/beta' into 332-pydantic-schema-validation	2024-06-05 09:05:19 +02:00
elijahbenizzy	e96b7018b6	Integrates with Burr's Forking/spawning ability ... Requires this PR: https://github.com/DAGWorks-Inc/burr/pull/225	2024-06-04 21:17:52 -07:00
Marco Perini	f8b08e0b33	feat(append_node): append node to existing graph	2024-06-04 23:34:43 +02:00
Marco Perini	376f758a76	feat(pydantic): added pydantic output schema	2024-06-04 23:07:49 +02:00
Marco Vinciguerra	fff89f431f	feat: refactoring of abstract graph	2024-06-04 19:41:11 +02:00
Marco Vinciguerra	58cd523bc5	Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta	2024-06-04 18:41:46 +02:00
Marco Vinciguerra	f81442b817	removed unused if	2024-06-04 18:41:44 +02:00
Marco Vinciguerra	4c0d0e935b	Merge pull request #339 from seyf97/seyf97-link_extraction_patch ... Update cleanup_html.py	2024-06-04 14:28:03 +02:00
Marco Vinciguerra	7ed2fe8ef0	feat: add dynamic caching	2024-06-04 14:27:46 +02:00
seyf97	acece72c28	Update cleanup_html.py ... Remove redundant lines in Links extraction	2024-06-04 13:49:00 +03:00
Marco Vinciguerra	7a13a6819f	feat: refactoring of rag node	2024-06-04 12:01:21 +02:00
Marco Vinciguerra	244aada2de	feat: refactoring of an in if	2024-06-04 10:01:20 +02:00
Marco Vinciguerra	c8d556da4e	feat: fix an if	2024-06-03 21:49:34 +02:00
Marco Vinciguerra	8de720d379	feat: removed a bug	2024-06-03 21:45:37 +02:00
Marco Perini	79ace115c7	Merge pull request #323 from VinciGit00/refactoring-pdf_scraper ... Refactoring pdf scraper and json scrape	2024-06-03 13:26:11 +02:00
Marco Vinciguerra	b4086550cc	feat: add csv scraper and xml scraper multi	2024-06-02 22:57:33 +02:00
Marco Vinciguerra	5cfc10178a	feat: add forcing format as json	2024-06-02 12:24:54 +02:00
Marco Vinciguerra	fff1232b8a	add rag node	2024-06-01 13:06:20 +02:00
Marco Vinciguerra	5bda918a39	feat: add json multiscraper	2024-06-01 12:04:19 +02:00
Marco Vinciguerra	4d42d7bfc6	add example	2024-06-01 11:20:24 +02:00
Marco Vinciguerra	f5cbd80c97	feat: add pdf scraper multi graph	2024-06-01 09:52:21 +02:00
Marco Vinciguerra	25352a56f9	Merge branch 'pre/beta' into temp	2024-05-31 23:08:24 +02:00
Marco Perini	8be27bad80	fix(3.9): python 3.9 logging fix	2024-05-31 22:32:20 +02:00
Marco Vinciguerra	930f673747	feat: removed rag node	2024-05-31 21:03:48 +02:00
Marco Vinciguerra	c4ce36111f	fix: typo in generate_screper_node	2024-05-30 18:50:04 +02:00
Johan Karlsson	4639f0cac5	fix: typo in prompt	2024-05-30 13:47:59 +02:00
Marco Vinciguerra	4fcb9902fe	fix: oneapi model	2024-05-29 18:24:09 +02:00
Marco Vinciguerra	8f2c8d5d12	Fix: Update __init__.py	2024-05-26 16:24:32 +02:00
Marco Vinciguerra	3c7dedf9c8	Merge pull request #305 from VinciGit00/pdf_fix ... Pdf fix	2024-05-26 15:39:21 +02:00
Marco Vinciguerra	1d958be44d	Merge pull request #303 from VinciGit00/295-scrapegraph-ai接入oneapi模型qwen-turbo ... add OneAPI integration	2024-05-26 12:17:34 +02:00
Marco Perini	8d76c4b3cb	fix(schema): added schema	2024-05-26 10:51:48 +02:00
Marco Perini	5fb9115330	feat(version): python 3.12 is now supported 🚀	2024-05-26 08:49:27 +02:00
Marco Perini	f9f6b08753	Merge branch 'pre/beta' into burr_integration	2024-05-25 14:51:07 +02:00
Marco Perini	19b27bbe85	feat(burr): first burr integration and docs	2024-05-25 01:34:53 +02:00
Marco Perini	5684578fab	fix(kg): removed unused nodes and utils	2024-05-25 00:44:14 +02:00
Marco Perini	a4ee757507	Merge branch 'pre/beta' into pdf_scraper_refactoring	2024-05-25 00:16:38 +02:00
Marco Perini	8d5eb0bb0d	fix(local_file): fixed textual input pdf, csv, json and xml graph	2024-05-25 00:13:47 +02:00
Marco Vinciguerra	b6f1766bd1	add OneAPI integration ... Co-Authored-By: Federico Aguzzi <62149513+f-aguzzi@users.noreply.github.com> Co-Authored-By: wangdongpeng1 <74647183+wangdongpeng1@users.noreply.github.com>	2024-05-24 13:46:40 +02:00
Marco Vinciguerra	b913b51cca	Merge branch 'logger-integration' into pre/beta	2024-05-24 12:39:14 +02:00
Federico Minutoli	c251cc45d3	fix(node-logging): use centralized logger in each node for logging	2024-05-24 01:09:49 +02:00
Federico Minutoli	4348d4f4db	fix(logger): set up centralized root logger in base node	2024-05-24 01:09:26 +02:00
Federico Minutoli	c807695720	feat(verbose): centralized graph logging on debug or warning depending on verbose	2024-05-24 01:09:03 +02:00
Federico Minutoli	0790ecd208	fix(web-loader): use sublogger	2024-05-24 01:08:28 +02:00
Federico Minutoli	d1394809d7	fix(logging): source code citation	2024-05-24 01:08:08 +02:00
Marco Vinciguerra	5fd7633c63	Update pdf_scraper_graph.py	2024-05-23 21:09:09 +02:00
Marco Vinciguerra	d00cde6030	fix(pdf_scraper): fix the pdf scraper gaph	2024-05-23 20:03:16 +02:00
Marco Vinciguerra	c93dbe0a94	Update smart_scraper_graph.py	2024-05-23 18:44:52 +02:00
Marco Vinciguerra	6d33a8a25e	rollback	2024-05-23 18:44:04 +02:00
Marco Vinciguerra	909af8d912	refactor gen answ node	2024-05-23 13:45:23 +02:00
Marco Vinciguerra	b377467b29	add info	2024-05-23 12:51:08 +02:00
Marco Vinciguerra	1774b18059	refactor of embeddings	2024-05-23 12:32:49 +02:00
Marco Vinciguerra	b6f7b647f8	Merge pull request #290 from VinciGit00/pre/beta ... Pre/beta	2024-05-23 11:48:44 +02:00
Marco Vinciguerra	f00ed35f7b	Merge branch 'pre/beta' into patch-1	2024-05-23 11:47:51 +02:00
Marco Vinciguerra	0ba3a594fc	Update models_tokens.py	2024-05-23 11:43:54 +02:00
Joe Stone	ffd6015a50	Update abstract_graph.py ... Adapt to some ollama modles with "nam/modelname" like ( 893379029/piccolo-large-zh-v2)	2024-05-23 15:58:47 +08:00
Alok Saboo	58289eccc5	feat: Add new models and update existing ones	2024-05-22 15:07:27 -04:00
Marco Perini	ac10128ff3	feat(burr): added burr integration in graphs and optional burr installation	2024-05-22 18:30:49 +02:00
Marco Vinciguerra	f2dffe534f	fix: pdf scraper bug	2024-05-22 11:54:55 +02:00
PeriniM	654a042396	feat(burr-node): working burr bridge	2024-05-22 00:24:38 +02:00
elijahbenizzy	d96840fa65	Updates Burr bridge to use class-based API ... (also makes the App ID a unique one for debugging, it kept getting confused)	2024-05-21 12:57:47 -07:00
Marco Perini	6cbd84f254	feat(burr-bridge): BurrBridge class to integrate inside BaseGraph	2024-05-21 19:57:10 +02:00
Marco Vinciguerra	ca436abf3c	fix: error in jsons	2024-05-21 14:35:48 +02:00
Marco Vinciguerra	be4237a04d	Merge branch 'pre/beta' into multi_scraper_graph	2024-05-21 13:44:06 +02:00
Marco Perini	fc58e2d3a6	feat(smart-scraper-multi): add schema to graphs and created SmartScraperMultiGraph	2024-05-21 13:13:27 +02:00
Marco Vinciguerra	1ef63217dd	add deepseek embeddings	2024-05-21 10:41:01 +02:00
Marco Vinciguerra	659fad770a	fix: add deepseek embeddings	2024-05-21 10:36:40 +02:00
Marco Vinciguerra	c8c3201745	Merge pull request #273 from JGalego/bugfix/bedrock-runs ... bugfix: ScriptCreatorGraph verbose settings + RobotsNode handling model IDs	2024-05-20 19:40:23 +02:00
JGalego	3ffa896c39	Fixed model ID -> model name conversion	2024-05-20 17:35:55 +01:00
JGalego	d0a301d3c3	Moved common params up (verbose, headless and loader_kwargs)	2024-05-20 17:25:13 +01:00
JGalego	05ecc3ae20	Added missing logic to extract model_name from model_id	2024-05-20 17:17:11 +01:00
JGalego	9e92b03913	Added missing Titan text embedding models	2024-05-20 16:56:41 +01:00
Marco Perini	9096da7d47	Merge pull request #269 from QIN2DIM/feat-undetected-playwright ... feat(docloaders): undetected-playwright	2024-05-19 22:26:50 +02:00
Marco Vinciguerra	ec8cbcab23	Merge branch 'pre/beta' into try	2024-05-19 22:08:11 +02:00
QIN2DIM	7b3ee4e71e	feat(docloaders): undetected-playwright	2024-05-19 18:01:03 +08:00