164 Commits

Author	SHA1	Message	Date
PeriniM	5cdf0550fe	chore: made some libs optional	2025-01-06 03:42:45 +01:00
PeriniM	54c69a2b0b	chore: pandas package is now optional	2025-01-06 02:36:05 +01:00
PeriniM	f6009d1abf	fix: better playwright installation handling	2025-01-06 02:01:17 +01:00
Marco Vinciguerra	d4b26796d9	fix: search graph	2025-01-03 14:00:17 +01:00
Marco Vinciguerra	faf0c0123b	feat: revert search function	2024-11-24 10:02:10 +01:00
aziz-ullah-khan	c218546a3d	feat: Serper API integration for Google search	2024-11-02 15:02:08 +05:00
Umut CAN	827f7260ad	This commit focuses on optimizing the utility modules in the codebase for better performance and maintainability. Key improvements include: - More efficient HTML processing with combined regex operations and optimized tag handling - Enhanced deep copy functionality with better type handling and optimized recursion - Refactored web search with improved error handling and modular helper functions The changes maintain all existing functionality while improving code quality, performance, and maintainability. Documentation and type hints have been enhanced throughout. ... Optimize utils modules for better performance and maintainability - Improve HTML cleanup and minification: - Combine regex operations for better performance - Add better error handling for HTML processing - Optimize tag removal and attribute filtering - Enhance deep copy functionality: - Add special case handling for primitive types - Improve type checking and error handling - Optimize recursive copying for collections - Refactor web search functionality: - Add input validation and error handling - Split search logic into separate helper functions - Improve proxy handling and configuration - Add better timeout and error management - Optimize URL filtering and processing Technical improvements: - Better type hints and documentation - More efficient data structures - Improved error handling and validation - Reduced code duplication - Better separation of concerns No breaking changes - all existing functionality maintained	2024-10-28 22:40:32 +03:00
Marco Vinciguerra	6179ab99a4	Update data_export.py	2024-10-24 15:20:36 +02:00
Marco Vinciguerra	c8a000f1d9	fix: fix export function	2024-10-24 10:11:36 +02:00
Marco Vinciguerra	066e77dbe7	Merge branch 'main' into pre/beta	2024-10-24 08:54:17 +02:00
Marco Vinciguerra	0ea00c078f	feat: refactoring of export functions	2024-10-21 10:30:21 +02:00
Marco Vinciguerra	a18471688f	fix: removed tokenizer	2024-10-19 07:18:56 +02:00
yusefes	d291819be3	Fix tokenizer loading for GPT2 ... Fixes #752 Fix the issue with loading the tokenizer for 'gpt2'. * **scrapegraphai/utils/tokenizer.py** - Add a check for `GPT2TokenizerFast` in the `num_tokens_calculus` function. - Import `GPT2TokenizerFast` from `transformers`. * **scrapegraphai/utils/tokenizers/tokenizer_ollama.py** - Modify the `num_tokens_ollama` function to handle `GPT2TokenizerFast`. * **tests/graphs/smart_scraper_ollama_test.py** - Add a test case to verify the tokenizer loading for `GPT2TokenizerFast`. --- For more details, open the [Copilot Workspace session](https://copilot-workspace.githubnext.com/ScrapeGraphAI/Scrapegraph-ai/issues/752?shareId=XXXX-XXXX-XXXX-XXXX).	2024-10-17 16:34:13 +03:30
Marco Vinciguerra	c2179abc60	fix: async invocation	2024-10-13 11:30:39 +02:00
Marco Vinciguerra	e4421ffdcd	Merge branch '731-fetchnode-object-has-no-attribute-update_state' into pre/beta	2024-10-12 09:58:20 +02:00
Marco Vinciguerra	47645d92ce	removed unused files	2024-10-12 09:41:02 +02:00
Marco Vinciguerra	257f393761	feat: async invocation	2024-10-11 09:31:18 +02:00
aziz-ullah-khan	e828c7010a	perf: Proxy integration in googlesearch	2024-10-11 12:30:58 +05:00
aziz-ullah-khan	36875b67f1	Merge remote-tracking branch 'refs/remotes/origin/pre/beta' into pre/beta	2024-10-11 10:08:07 +05:00
aziz-ullah-khan	46195d65ca	Update: proxy integration in googlesearch	2024-10-11 10:04:28 +05:00
Marco Vinciguerra	a200f595de	refactoring of examples	2024-10-08 08:54:18 +02:00
Marco Vinciguerra	99aac5b055	Merge branch 'pre/beta' into tem	2024-10-05 09:35:31 +02:00
Marco Vinciguerra	6664da07f8	Merge pull request #720 from ScrapeGraphAI/deep_scraper_integration ... Deep scraper integration	2024-10-04 09:56:49 +02:00
Marco Vinciguerra	e017ee21f3	update readme and readibility of the code	2024-10-02 10:07:03 +02:00
Marco Vinciguerra	26f89d895d	feat: refactoring of research web	2024-10-01 10:25:09 +02:00
Matteo Vedovati	d80b792e15	fetching first level	2024-09-30 12:42:26 +02:00
Marco Vinciguerra	27ae896cb7	Update output_parser.py	2024-09-28 11:05:09 +02:00
Marco Vinciguerra	560f079d4c	refactoring of the code	2024-09-28 09:02:20 +02:00
Marco Vinciguerra	76ce257efb	fix: update to pydantic documentation	2024-09-25 15:12:29 +02:00
Marco Vinciguerra	d55f6bee47	fix: node refiner + examples	2024-09-25 14:53:51 +02:00
Matteo Vedovati	ce841e21fb	Code generation refactoring	2024-09-24 19:04:43 +02:00
Marco Vinciguerra	5a3ece9a1a	Merge branch 'pre/beta' into temp	2024-09-24 12:41:55 +02:00
Marco Vinciguerra	d0969feb7c	removed unused imports and comments + removed dead code	2024-09-23 09:25:13 +02:00
Marco Vinciguerra	7eda6bc06b	fix: issue about parser	2024-09-22 18:55:05 +02:00
Matteo Vedovati	5b579b323f	gode generator v0.1	2024-09-21 11:44:04 +02:00
Marco Vinciguerra	95a5ee2d35	Merge pull request #680 from LorenzoPaleari/exec-info-enhanced ... feat: added Bedrock and Mistral to exec info	2024-09-18 10:05:45 +02:00
Lorenzo Paleari	8a37c6b793	feat: added Bedrock and Mistral to exec info	2024-09-18 02:02:38 +02:00
Lorenzo Paleari	28b85a3b16	refactor: Output parser code	2024-09-17 23:07:16 +02:00
Marco Vinciguerra	c717bb6d8e	Merge branch 'main' into pre/beta	2024-09-16 09:08:37 +02:00
Marco Vinciguerra	438b8127db	refctoring of the code	2024-09-15 11:20:08 +02:00
Marco Vinciguerra	5d1fe68dea	Merge branch 'pre/beta' into temp	2024-09-14 10:53:43 +02:00
Marco Vinciguerra	dcef172e03	Update copy.py	2024-09-14 10:43:47 +02:00
smith peng	7e3f66e180	fix:fix boto3 client copy	2024-09-14 11:15:31 +08:00
Lorenzo Paleari	e657113ebc	fix: Refactor code to use CustomOpenAiCallbackManager for exclusive access to get_openai_callback	2024-09-14 02:06:52 +02:00
Lorenzo Paleari	66ea166438	fix: Added support for nested structure	2024-09-13 04:18:53 +02:00
Marco Vinciguerra	ec6b164653	feat: refactoring of the tokenization function	2024-09-12 20:21:00 +02:00
Tom Robinson	dc4a76b9c5	use semchunk by default as the other code is causing tokenizers to be called for every individual word which is very slow especially with the mistral tokenizer	2024-09-12 08:46:52 +01:00
Tom Robinson	da9726f738	updates to tokenization for #651 to implement for mistral and ollama	2024-09-12 08:28:30 +01:00
Marco Vinciguerra	fe3aa28fe7	refactoring of the code	2024-09-11 16:04:43 +02:00
Marco Vinciguerra	1a7f21fbf3	feat: removed semchunk and used tikton	2024-09-10 14:03:52 +02:00