aziz-ullah-khan
e8cabfd1ae
fix: resolved outparser issue
2024-11-03 20:23:29 +05:00
aziz-ullah-khan
c218546a3d
feat: Serper API integration for Google search
2024-11-02 15:02:08 +05:00
Federico Aguzzi
76e2820ff1
Merge pull request #778 from ScrapeGraphAI/refactoring-generate-answer
...
feat: update generate answer
2024-10-31 10:20:17 +01:00
Marco Vinciguerra
09c967822d
Merge branch 'pre/beta' into main
2024-10-30 09:00:20 +01:00
Marco Vinciguerra
7172b32a0f
feat: update generate answer
2024-10-30 08:59:13 +01:00
Umut CAN
827f7260ad
This commit focuses on optimizing the utility modules in the codebase for better performance and maintainability. Key improvements include: - More efficient HTML processing with combined regex operations and optimized tag handling - Enhanced deep copy functionality with better type handling and optimized recursion - Refactored web search with improved error handling and modular helper functions The changes maintain all existing functionality while improving code quality, performance, and maintainability. Documentation and type hints have been enhanced throughout.
...
Optimize utils modules for better performance and maintainability
- Improve HTML cleanup and minification:
- Combine regex operations for better performance
- Add better error handling for HTML processing
- Optimize tag removal and attribute filtering
- Enhance deep copy functionality:
- Add special case handling for primitive types
- Improve type checking and error handling
- Optimize recursive copying for collections
- Refactor web search functionality:
- Add input validation and error handling
- Split search logic into separate helper functions
- Improve proxy handling and configuration
- Add better timeout and error management
- Optimize URL filtering and processing
Technical improvements:
- Better type hints and documentation
- More efficient data structures
- Improved error handling and validation
- Reduced code duplication
- Better separation of concerns
No breaking changes - all existing functionality maintained
2024-10-28 22:40:32 +03:00
Federico Aguzzi
f79f399ee0
fix(AbstractGraph): manually select model tokens
...
closes #768
2024-10-28 14:51:39 +01:00
Marco Vinciguerra
12a6c18f6a
feat: refactoring of the base_graph
2024-10-28 09:58:03 +01:00
Marco Vinciguerra
6914170890
feat: add new mistral models
2024-10-26 16:07:43 +02:00
Marco Vinciguerra
b7d5a20ae0
Merge pull request #764 from ScrapeGraphAI/pre/beta
2024-10-26 10:05:15 +02:00
Marco Vinciguerra
f9c1432342
Merge pull request #767 from ScrapeGraphAI/fix-export-function
2024-10-25 08:43:40 +02:00
Marco Vinciguerra
4cd5ef296e
add docstring files
CodeQL / Analyze (python) (push) Has been cancelled
/ build (push) Has been cancelled
Release / Build (push) Has been cancelled
Release / Release (push) Has been cancelled
2024-10-24 15:28:27 +02:00
Marco Vinciguerra
6179ab99a4
Update data_export.py
2024-10-24 15:20:36 +02:00
Marco Vinciguerra
c8a000f1d9
fix: fix export function
2024-10-24 10:11:36 +02:00
Marco Vinciguerra
51c55eb3a2
feat: add model integration gpt4
2024-10-24 09:10:51 +02:00
Marco Vinciguerra
066e77dbe7
Merge branch 'main' into pre/beta
2024-10-24 08:54:17 +02:00
Marco Vinciguerra
f658092dff
feat: refactoring of get_probable_tags node
2024-10-23 12:15:16 +02:00
Marco Vinciguerra
ae275ec5e8
feat: add integration with scrape.do
2024-10-23 12:08:00 +02:00
Marco Vinciguerra
0ea00c078f
feat: refactoring of export functions
2024-10-21 10:30:21 +02:00
Marco Vinciguerra
52b6bf5fb8
feat: refactoring of ScrapeGraph to SmartScraperLiteGraph
2024-10-21 10:12:53 +02:00
Marco Vinciguerra
ffa1067f0d
Merge pull request #756 from shenghongtw/pre/beta
...
The smart_scraper_multi_graph method is too expensive
2024-10-20 10:13:47 +02:00
Marco Vinciguerra
a18471688f
fix: removed tokenizer
2024-10-19 07:18:56 +02:00
Federico Aguzzi
8cb9646a45
Merge branch 'main' into pre/beta
2024-10-18 22:16:39 +02:00
roryhaung
3e8f047ab6
Renamed smart_scraper_multi_abstract_graph back to smart_scraper_multi_graph.
2024-10-18 03:10:57 +08:00
roryhaung
974f88a77e
rename SmartScraperMultiGraph to SmartScraperMultiLiteGraph
2024-10-18 03:01:59 +08:00
roryhaung
6dbac93668
rename the SmartScraperMultiParseMergeFirstGraph to SmartScraperMultiGraph
2024-10-18 01:52:39 +08:00
roryhaung
78bd40c3b5
modify the graph name
2024-10-18 01:51:26 +08:00
roryhaung
dfc67c670d
rename the smart_scraper_multi_parse_merge_first_graph to smart_scraper_multi_graph,so delete this file
2024-10-18 01:49:54 +08:00
roryhaung
94d8042c2a
rename smart_scraper_multi_graph to smart_scraper_multi_abstract_graph
2024-10-18 01:39:42 +08:00
roryhaung
69ff649556
fix: fix the example variable name
2024-10-18 01:36:29 +08:00
yusefes
d291819be3
Fix tokenizer loading for GPT2
...
Fixes #752
Fix the issue with loading the tokenizer for 'gpt2'.
* **scrapegraphai/utils/tokenizer.py**
- Add a check for `GPT2TokenizerFast` in the `num_tokens_calculus` function.
- Import `GPT2TokenizerFast` from `transformers`.
* **scrapegraphai/utils/tokenizers/tokenizer_ollama.py**
- Modify the `num_tokens_ollama` function to handle `GPT2TokenizerFast`.
* **tests/graphs/smart_scraper_ollama_test.py**
- Add a test case to verify the tokenizer loading for `GPT2TokenizerFast`.
---
For more details, open the [Copilot Workspace session](https://copilot-workspace.githubnext.com/ScrapeGraphAI/Scrapegraph-ai/issues/752?shareId=XXXX-XXXX-XXXX-XXXX ).
2024-10-17 16:34:13 +03:30
roryhaung
9b78e2d755
Merge branch 'pre/beta' of https://github.com/shenghongtw/Scrapegraph-ai into pre/beta
2024-10-17 03:20:46 +08:00
Marco Vinciguerra
aaa011cc89
Merge pull request #754 from ekinsenler/cond_node_refactor
...
feat: add conditional node to the smart_scraper_graph
2024-10-16 17:53:10 +02:00
roryhaung
3e3e1b2f3a
feat: Implement SmartScraperMultiParseMergeFirstGraph class that scrapes a list of URLs and merge the content first and finally generates answers to a given prompt.
...
(Different from the SmartScraperMultiGraph is that in this case the content is merged before to be processed by the llm.)
2024-10-16 19:38:53 +08:00
roryhaung
612c644623
feat: implement ScrapeGraph class for only web scraping automation
2024-10-16 18:37:50 +08:00
ekinsenler
038d2ef916
refactor cond node structure to fit with the new implementation
2024-10-15 15:23:05 +03:00
ekinsenler
cacd9cde00
feat: add conditional node structure to the smart_scraper_graph and implemented a structured way to check condition
2024-10-15 14:23:02 +03:00
Marco Vinciguerra
db4b7cd3d1
refactoring of the code
2024-10-15 11:21:04 +02:00
Marco Vinciguerra
39c7352cea
Merge pull request #751 from shenghongtw/pre/beta
...
Since the variable "max_result" is not used in the code, it has been removed to avoid causing confusion for users. (The example code was tested, and the presence or absence of max_result does not affect the results.)
2024-10-14 14:13:28 +02:00
roryhaung
e76a68a782
fix: remove variable "max_result" not being used in the code
2024-10-13 23:04:27 +08:00
Marco Vinciguerra
c2179abc60
fix: async invocation
2024-10-13 11:30:39 +02:00
Marco Vinciguerra
b208ef792c
fix: csv_node
2024-10-13 10:31:12 +02:00
Marco Vinciguerra
431b2093be
fix: generate answer node
2024-10-13 10:25:36 +02:00
Marco Vinciguerra
e4421ffdcd
Merge branch '731-fetchnode-object-has-no-attribute-update_state' into pre/beta
2024-10-12 09:58:20 +02:00
Marco Vinciguerra
026a70bd3a
fix: bugs
2024-10-12 09:57:59 +02:00
Marco Vinciguerra
47645d92ce
removed unused files
2024-10-12 09:41:02 +02:00
aziz-ullah-khan
7f03ec15de
Fix: search_on_web paremter
2024-10-11 16:58:37 +05:00
Marco Vinciguerra
1fb618cba4
removed unused node
2024-10-11 12:46:39 +02:00
Marco Vinciguerra
0a275d5ac3
Merge pull request #743 from aziz-ullah-khan/pre/beta
...
Replacement of Google search to googlesearch-python and integration of Proxy.
2024-10-11 11:17:29 +02:00
Marco Vinciguerra
257f393761
feat: async invocation
2024-10-11 09:31:18 +02:00
