528 Commits

Author	SHA1	Message	Date
Lorenzo Paleari	eb89549248	feat: updated pydantic to v2	2024-09-17 23:08:56 +02:00
Lorenzo Paleari	28b85a3b16	refactor: Output parser code	2024-09-17 23:07:16 +02:00
Marco Vinciguerra	3f45c17022	fix: fetch_node condition	2024-09-16 09:11:59 +02:00
Marco Vinciguerra	c717bb6d8e	Merge branch 'main' into pre/beta	2024-09-16 09:08:37 +02:00
Marco Vinciguerra	438b8127db	refctoring of the code	2024-09-15 11:20:08 +02:00
Marco Vinciguerra	2ae26e96a0	Merge pull request #664 from LorenzoPaleari/598-fix-pydantic-validation-error ... 598 - Fix pydantic validation error	2024-09-13 09:00:17 +02:00
Lorenzo Paleari	66ea166438	fix: Added support for nested structure	2024-09-13 04:18:53 +02:00
Lorenzo Paleari	039ba2e95a	fix: Fixed pydantic error on SearchGraphs ... Changed instatiation location of iterated graph classes	2024-09-13 01:56:58 +02:00
Lorenzo Paleari	c3d1b7c200	fix: OmniScraerGraph working. ... Added url scraping capability to ParseNode	2024-09-13 01:47:39 +02:00
Matteo Vedovati	2a760a1c6f	initial promptrefiner prompt	2024-09-12 21:17:46 +02:00
Matteo Vedovati	42318a1441	Creation of PromptRefiner	2024-09-12 18:35:19 +02:00
Matteo Vedovati	9862425fb1	JsonDescriptorNode created	2024-09-12 18:04:37 +02:00
Marco Vinciguerra	4a16f14b25	Merge pull request #660 from tm-robinson/651-add-tokenization-for-ollama-and-mistral ... 651 add tokenization for ollama and mistral	2024-09-12 10:22:20 +02:00
Marco Vinciguerra	c64ce88db8	refactoting of imports	2024-09-12 10:16:15 +02:00
Tom Robinson	da9726f738	updates to tokenization for #651 to implement for mistral and ollama	2024-09-12 08:28:30 +01:00
Marco Vinciguerra	fe3aa28fe7	refactoring of the code	2024-09-11 16:04:43 +02:00
Marco Vinciguerra	1a7f21fbf3	feat: removed semchunk and used tikton	2024-09-10 14:03:52 +02:00
Marco Vinciguerra	380174d490	add chunking functionn	2024-09-10 13:52:15 +02:00
Federico Aguzzi	4c14fd79b2	Merge pull request #650 from ScrapeGraphAI/637-it-can´t-scrape-urls-from-the-source ... fix: parse node	2024-09-10 11:19:24 +02:00
Marco Vinciguerra	947ebd2895	fix: parse node	2024-09-10 08:41:08 +02:00
Federico Aguzzi	32a102af3c	Merge pull request #648 from ScrapeGraphAI/637-it-can´t-scrape-urls-from-the-source ... 637 it can´t scrape urls from the source	2024-09-09 12:04:46 +02:00
Federico Aguzzi	8a0d46b714	Merge pull request #641 from ScrapeGraphAI/urls_search_graph ... feat: return urls in searchgraph	2024-09-09 12:03:36 +02:00
Marco Vinciguerra	f2bb22d8e9	fix: temporary fix for parse_node	2024-09-09 11:42:33 +02:00
Marco Vinciguerra	fc738cacac	Update parse_node.py	2024-09-08 11:54:11 +02:00
Federico Aguzzi	9016bb53fb	Merge pull request #639 from ScrapeGraphAI/scrape_do_integration ... Scrape do integration	2024-09-07 17:06:22 +02:00
Marco Vinciguerra	ef7a5891dc	fix: screenshot_scraper	2024-09-07 14:55:12 +02:00
Marco Vinciguerra	afb6eb7e47	feat: return urls in searchgraph	2024-09-07 11:43:10 +02:00
Marco Vinciguerra	167f97040f	feat: fetch_node improved	2024-09-07 09:33:57 +02:00
Marco Vinciguerra	f5e7a8bfaf	fix of the bug for fetching the code	2024-09-06 16:58:37 +02:00
Marco Vinciguerra	94e69a0515	feat: add scrape_do_integration	2024-09-06 16:51:48 +02:00
Marco Vinciguerra	8b02cb4460	Merge pull request #632 from ekinsenler/concat_node ... feat: Concat node implementation	2024-09-06 15:52:35 +02:00
ekinsenler	bd4b26d7d7	feat: ConcatNode.py added for heavy merge operations	2024-09-04 17:53:32 +03:00
Lorenzo Paleari	8e74ac55a1	fix: correctly parsing output when using structured_output	2024-09-02 17:03:14 +02:00
Lorenzo Paleari	66a3b6d6a3	fix: Parse Node scraping link and img urls allowing OmniScraper to work	2024-09-02 12:53:10 +02:00
Marco Vinciguerra	e741602996	Merge branch 'pre/beta' into 543-ScriptCreatorGraph-only-use-first-chunk	2024-09-02 11:35:53 +02:00
Marco Vinciguerra	1bcc0bff0a	Merge pull request #620 from goasleep/feature/export_search_engine ... feat:expose the search engine params to user	2024-09-02 11:27:59 +02:00
Tom Robinson	3d265a8631	change GenerateScraperNode to only use first chunk	2024-09-02 08:21:19 +01:00
Tom Robinson	a8b0e4a359	updated token calculation on parsenode	2024-09-02 08:01:21 +01:00
smith peng	8422463ca6	feat:expose the search engine params to user	2024-09-02 14:28:39 +08:00
Marco Vinciguerra	0e0b280bde	Merge branch 'pre/beta' into temp	2024-09-01 11:53:51 +02:00
Marco Vinciguerra	c348f674ad	fix: update generate answernode	2024-08-30 11:37:44 +02:00
elijahbenizzy	25d8fd2e72	Fixes node expression validator error message to make it easier to debug missing keys	2024-08-29 13:24:47 -07:00
Federico Aguzzi	83e71df2e2	fix: set up dynamic imports correctly	2024-08-27 16:54:57 +02:00
Marco Vinciguerra	26de5dd623	Merge branch 'pre/beta' into ligthweigthing_library	2024-08-23 12:22:31 +02:00
Marco Vinciguerra	f7ba1f30de	refactoring of the code	2024-08-23 11:33:22 +02:00
Marco Vinciguerra	0d2b7b37e0	update dependenceis	2024-08-21 20:48:47 +02:00
Federico Aguzzi	d2486469af	Merge branch 'pre/beta' into screenshot_scraper	2024-08-20 09:58:34 +02:00
Marco Vinciguerra	f774fe40e5	add try catch and robust integration	2024-08-19 14:24:27 +02:00
Federico Aguzzi	d29338b7c2	feat(GenerateAnswerNode): built-in structured output through LangChain ... Co-Authored-By: Marco Vinciguerra <88108002+VinciGit00@users.noreply.github.com>	2024-08-19 13:45:37 +02:00
Federico Aguzzi	b48ee825ee	Merge branch 'pre/beta' into support_structured_output_shema_openai	2024-08-19 13:31:01 +02:00
Federico Aguzzi	683bf57d89	fix(ParseNode): leave room for LLM reply in context window	2024-08-19 11:33:09 +02:00
Marco Vinciguerra	f60aa3acde	feat: add async call	2024-08-19 11:22:40 +02:00
ekinsenler	08e9d9d6a0	feat: Implemented a filter logic in search_link_node.py ... feat: Added dict entry for Llama3.1:8b	2024-08-19 11:05:31 +03:00
Marco Vinciguerra	0bf79b5926	Update generate_answer_from_image_node.py	2024-08-19 02:21:45 +02:00
Marco Vinciguerra	79fa3f6bd6	add if and cool stuff	2024-08-19 02:01:41 +02:00
Marco Vinciguerra	c72c077eb6	refactoring of the nodes	2024-08-19 01:15:45 +02:00
Marco Vinciguerra	8d6c0b7288	Update generate_answer_node.py	2024-08-18 22:01:39 +02:00
Marco Vinciguerra	103c21c86e	Update generate_answer_from_image_node.py	2024-08-18 20:54:35 +02:00
Marco Vinciguerra	5eb3cff64f	feat: refactoring of the code	2024-08-18 20:53:35 +02:00
Marco Vinciguerra	8e3d5deaaa	add screenshot scraper	2024-08-18 19:39:49 +02:00
Marco Vinciguerra	8b8d8f09b7	refactoring of the code according to pylint style	2024-08-18 11:51:34 +02:00
Marco Vinciguerra	ebdb74967d	add new model for gpt	2024-08-17 15:00:44 +02:00
Marco Vinciguerra	391d051a5e	updated	2024-08-17 14:39:19 +02:00
Marco Vinciguerra	c2c2e6582f	Update generate_answer_node.py	2024-08-17 13:30:44 +02:00
Marco Vinciguerra	f0587908e2	Merge branch 'pre/beta' of https://github.com/ScrapeGraphAI/Scrapegraph-ai into pre/beta	2024-08-17 12:21:03 +02:00
Marco Vinciguerra	1d7f30b65b	fix: browser-base integration	2024-08-17 12:21:00 +02:00
Federico Aguzzi	5eed0a7654	fix(FetchNode) add OpenAI optimization to GPT models on Azure ... closes #547	2024-08-17 10:33:54 +02:00
Marco Vinciguerra	faef3186f7	fix: model count	2024-08-16 17:38:55 +02:00
Marco Vinciguerra	de1ec250ef	refactoring pyproject.toml ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-08-11 18:04:31 +02:00
Marco Vinciguerra	cec5537f2a	add new tests ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-08-11 17:10:55 +02:00
Matteo Vedovati	734b740bf9	Template name refactoring	2024-08-11 11:46:01 +02:00
Matteo Vedovati	98779d193b	search_link_node prompts moved	2024-08-11 11:43:22 +02:00
Matteo Vedovati	ef966525c3	search_link_node prompt moved	2024-08-11 11:34:49 +02:00
Matteo Vedovati	86a4903e99	search_internet_node prompt moved	2024-08-11 11:31:51 +02:00
Matteo Vedovati	f455fcbc5f	robot_node prompt moved	2024-08-11 11:27:27 +02:00
Matteo Vedovati	9814b6dd85	fixed versioning errors	2024-08-11 10:04:15 +02:00
Matteo Vedovati	7c6210019e	Merge branch 'pre/beta' of https://github.com/vedovati-matteo/Scrapegraph-ai into pre/beta	2024-08-11 09:42:38 +02:00
Matteo Vedovati	f17cef94bb	fix: merge_anwser prompt import	2024-08-11 09:41:31 +02:00
Matteo Vedovati	9fa3df2390	Changed prompts import from helpers to prompts	2024-08-11 09:33:21 +02:00
Marco Vinciguerra	0cda020d48	Merge branch 'pre/beta' into temp	2024-08-10 17:45:52 +02:00
Marco Vinciguerra	8b2c266aff	refactoring of the code ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-08-10 17:44:35 +02:00
Marco Vinciguerra	778efd4c87	fix: conditional node	2024-08-10 12:21:09 +02:00
Marco Vinciguerra	f01b55e89b	fix: fetch node	2024-08-10 11:41:51 +02:00
Marco Vinciguerra	ce00345495	fix: conditional node	2024-08-09 23:38:26 +02:00
Marco Vinciguerra	1272273499	fix: broken node	2024-08-09 17:35:43 +02:00
Marco Vinciguerra	157d59d5c0	Merge branch 'pre/beta' into temp	2024-08-09 10:28:59 +02:00
Federico Aguzzi	50edbcc7f8	fix(FetchNode): missing bracket syntax error	2024-08-09 09:37:51 +02:00
Federico Aguzzi	5ec2de9e1a	fix(chunking): count tokens from words instead of characters ... closes #513	2024-08-08 10:50:52 +02:00
Marco Vinciguerra	7f03926432	Merge pull request #515 from vedovati-matteo/pre/beta ... fix: refactoring of merge_answer_node	2024-08-07 21:50:14 +02:00
Matteo Vedovati	898e5a7af5	fix: refactoring of merge_answer_node	2024-08-07 21:42:54 +02:00
Marco Vinciguerra	b6bff0be33	Merge branch 'pre/beta' into mistral-ai-integration	2024-08-07 21:24:18 +02:00
Marco Vinciguerra	a7db47ecad	refatoring	2024-08-07 21:23:42 +02:00
Federico Aguzzi	07720b6e0c	fix(FetchNode): handling of missing browser_base key	2024-08-07 15:15:13 +02:00
Marco Vinciguerra	b52e4a390b	fix: generate answer node omni	2024-08-07 14:09:02 +02:00
Marco Vinciguerra	625ca9f22a	fix: generate answer node pdf has a bug	2024-08-07 14:06:53 +02:00
Matteo Vedovati	bfc6852b77	fix: refactoring of fetch_node adding comment	2024-08-07 13:02:02 +02:00
Matteo Vedovati	1ea2ad8e79	fix: refactoring of fetch_node qixed error	2024-08-07 12:04:54 +02:00
Matteo Vedovati	29ad140fa3	fix: refactoring of fetch_node	2024-08-07 11:56:10 +02:00
Marco Vinciguerra	82e63213ae	Merge branch 'pre/beta' of https://github.com/ScrapeGraphAI/Scrapegraph-ai into pre/beta	2024-08-06 17:07:35 +02:00
Marco Vinciguerra	37dd6e9ba9	update reqs ... Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>	2024-08-06 17:07:33 +02:00
Marco Vinciguerra	886bab78e7	Merge branch 'main' into temp-1	2024-08-06 10:04:57 +02:00
Marco Vinciguerra	9355507a2d	feat: refactoring of the code	2024-08-02 12:00:00 +02:00
Marco Vinciguerra	fe099f9cbc	Merge branch 'pre/beta' into browserbase_integration	2024-08-01 15:14:49 +02:00
Marco Vinciguerra	be870a4316	Update fetch_node.py ... Co-Authored-By: Federico Minutoli <40361744+DiTo97@users.noreply.github.com>	2024-08-01 13:24:48 +02:00
Marco Vinciguerra	968c69e217	fix: fixed bug on fetch_node ... Co-Authored-By: Federico Minutoli <40361744+DiTo97@users.noreply.github.com>	2024-08-01 13:23:54 +02:00
Marco Vinciguerra	e21d461710	push	2024-08-01 13:16:49 +02:00
Marco Vinciguerra	5ecdbe715f	feat: add integration in the abstract grapgh	2024-08-01 12:51:18 +02:00
Federico Aguzzi	a94ebcde00	refactor: move embeddings code from AbstractGraph to RAGNode	2024-08-01 11:53:17 +02:00
Marco Vinciguerra	f6b7c6a430	refactoring ... Co-Authored-By: Federico Aguzzi <62149513+f-aguzzi@users.noreply.github.com>	2024-07-29 17:56:32 +02:00
Federico Aguzzi	25066b2bc5	refactor: remove LangChain wrappers for Ollama	2024-07-29 17:49:21 +02:00
Federico Aguzzi	2c5f934f10	refactor: remove LangChain wrappers	2024-07-29 17:46:17 +02:00
amosdinh	05cf9adaa7	Ollama: Use no json format when creating the search query	2024-07-26 12:10:14 +02:00
Marco Perini	5c08eea189	docs: prev version	2024-07-24 23:37:57 +02:00
Marco Vinciguerra	e08b30495d	boh	2024-07-24 14:33:55 +02:00
Federico Aguzzi	ddbfbe1205	Merge branch 'pre/beta' into generate_answer_parallel	2024-07-23 20:51:35 +02:00
Marco Perini	1d41f6eafe	fix: md conversion	2024-07-23 19:27:18 +02:00
Marco Perini	49b6468fb1	Merge branch 'main' into pre/beta	2024-07-23 16:14:15 +02:00
Marco Perini	0ebc387446	Merge pull request #487 from ScrapeGraphAI/update-generate-answer-node ... Update generate answer node	2024-07-23 15:45:20 +02:00
Marco Perini	12b5eada6e	fix(md_conversion): add absolute links md, added missing dependency	2024-07-23 15:34:12 +02:00
Marco Vinciguerra	df1ecc0019	Update generate_answer_node.py	2024-07-23 13:06:59 +02:00
Marco Vinciguerra	cf2734392c	removed unused function	2024-07-23 13:05:50 +02:00
Marco Vinciguerra	8dd941e39e	update generate answer nodes ... Co-Authored-By: Federico Aguzzi <62149513+f-aguzzi@users.noreply.github.com>	2024-07-23 12:40:11 +02:00
Marco Vinciguerra	70911f83be	Update generate_answer_pdf_node.py	2024-07-23 11:07:35 +02:00
Marco Vinciguerra	0c4b2908d9	feat: add generate_answer node paralellization ... Co-Authored-By: Federico Minutoli <40361744+DiTo97@users.noreply.github.com>	2024-07-22 19:58:33 +02:00
Marco Vinciguerra	2ae19aee56	update function	2024-07-22 16:50:58 +02:00
Marco Vinciguerra	09256f7b11	fix: parse node	2024-07-22 16:15:38 +02:00
Marco Vinciguerra	71f894eee3	fix: parse_html node have a bug	2024-07-22 16:01:27 +02:00
Marco Vinciguerra	7a3c21b58d	Update generate_answer_node.py	2024-07-20 20:19:16 +02:00
Marco Vinciguerra	6a9205920f	Merge branch 'pre/beta' into temp	2024-07-20 20:03:20 +02:00
Marco Perini	99dc8497d8	docs(gpt-4o-mini): added new gpt, fixed chromium lazy loading, ... add documentation and metrics	2024-07-20 20:02:26 +02:00
Marco Perini	61a51d4bce	Merge pull request #466 from ScrapeGraphAI/refactoring_convert_to_md ... refactoring of generate answer node	2024-07-20 19:40:57 +02:00
Marco Vinciguerra	19af78c7d8	Merge branch 'pre/beta' into main	2024-07-20 18:14:36 +02:00
DragonelRoland	e9ecf5f0dc	Update typo in conditional_node.py ... "thar" -> "that"	2024-07-20 15:06:52 +02:00
Marco Vinciguerra	032a491605	Update parse_node.py	2024-07-17 23:06:05 +02:00
Marco Vinciguerra	07f1e23d23	fix: parse_node	2024-07-17 22:58:21 +02:00
Marco Vinciguerra	68f58cc4dd	refactoring of generate answer node	2024-07-17 22:41:49 +02:00
Federico Aguzzi	dccb893fdf	Merge pull request #456 from ScrapeGraphAI/refactoring-of-search_link_node ... fix: search link node	2024-07-15 23:06:03 +02:00
Marco Vinciguerra	830daee1f3	Update search_link_node.py	2024-07-15 20:47:09 +02:00
Marco Vinciguerra	57fdaf9e3a	create search_link_graph	2024-07-15 12:36:11 +02:00
Marco Vinciguerra	cf3ab5564a	fix: search link node	2024-07-14 16:49:29 +02:00
Marco Vinciguerra	a7249685cb	removed rag node	2024-07-14 10:24:38 +02:00
Marco Vinciguerra	765b548c0e	Merge pull request #417 from ScrapeGraphAI/md_scraper_integration ... feat: add integrations for markdown files	2024-07-04 19:42:53 +02:00
Marco Vinciguerra	8a52914871	Merge pull request #429 from ScrapeGraphAI/421-default-prompt-template-customization ... 421 default prompt template customization	2024-07-04 19:42:35 +02:00
Federico Aguzzi	104d8692d6	Merge branch 'pre/beta' into support	2024-07-04 18:36:34 +02:00
Marco Vinciguerra	3ee1743687	update prompts	2024-07-04 14:22:45 +02:00
Marco Vinciguerra	ed2af51501	update the chunk size	2024-07-02 12:03:08 +02:00
Marco Vinciguerra	3bf5f570a8	feat: add integration for infos	2024-07-01 21:19:16 +02:00
AmosDinh	3f9bf5b4d4	fixed typo	2024-07-01 13:42:10 +00:00
Marco Vinciguerra	5fe694b6b4	feat: improve md prompt recognition	2024-06-30 18:10:00 +02:00
Marco Vinciguerra	30ca15ca28	Merge branch 'md_scraper_integration' into integration_markdown	2024-06-30 16:58:37 +02:00