CodeCow/Scrapegraph-ai
1
0
Fork 0
mirror of https://github.com/VinciGit00/Scrapegraph-ai.git synced 2026-07-12 21:01:56 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,293 Commits 2 Branches 448 Tags 17 MiB
5faac9fb46
Commit Graph

1007 Commits

Author SHA1 Message Date
Mobin Chowdhury
7f30da9930
Update models_tokens.py 2024-05-18 23:32:27 +06:00
Marco Vinciguerra
5701afe927 add new import 2024-05-18 10:36:41 +02:00
Marco Perini
a338383399 feat(kg): removed import 2024-05-18 10:30:29 +02:00
Marco Perini
c75e6a06b1 feat(kg): working rag kg 2024-05-18 10:26:25 +02:00
Marco Vinciguerra
a0125b7e90 add orca mini integration 2024-05-18 09:31:40 +02:00
Marco Perini
58cc903d55 feat(multiple): quick fix working 2024-05-18 05:57:43 +02:00
Marco Vinciguerra
ff53771f87 add falcon model 2024-05-18 03:01:08 +02:00
Marco Vinciguerra
6f62b0560a Merge branch 'multi_scraper_graph' of https://github.com/VinciGit00/Scrapegraph-ai into multi_scraper_graph 2024-05-18 02:42:27 +02:00
Marco Vinciguerra
b82f33aee7 fix: template names 2024-05-18 02:42:25 +02:00
Marco Perini
bed3eed50c feat(multiple_search): working multiple example 2024-05-18 01:51:12 +02:00
Marco Vinciguerra
05e511e36f add new prompts 2024-05-18 00:22:52 +02:00
Marco Perini
0196423bde feat(knowledgegraph): add knowledge graph node 2024-05-17 23:41:44 +02:00
Marco Perini
73fa31db0f feat(base_graph): alligned with main 2024-05-17 18:54:27 +02:00
Marco Perini
8c33ea3fbc feat(node): knowledge graph node 2024-05-17 18:40:58 +02:00
Marco Vinciguerra
3453f72397 add graph 2024-05-17 18:23:50 +02:00
Marco Vinciguerra
02745a4f63 Merge branch 'main' into pre/beta 2024-05-17 10:40:19 +02:00
Marco Perini
d4d913c8a3 fix(deepcopy): switch whether we have obj in the config 2024-05-17 10:36:39 +02:00
VinciGit00
9e9f8f09d5 removed max depth 2024-05-16 12:39:08 +02:00
VinciGit00
9483afddd3 revert 2024-05-16 12:34:02 +02:00
Mayur Bhosale
1e0b2f7e70
Merge branch 'pre/beta' into nDeep 2024-05-15 23:36:23 +02:00
mayurdb
d60438cc89 Add a n-level deep search support 2024-05-15 23:32:15 +02:00
VinciGit00
ba8a4f7122 removed duplicates 2024-05-15 21:47:39 +02:00
VinciGit00
befa48c912 update lock 2024-05-15 21:28:56 +02:00
VinciGit00
4fe58d9ba4 fix logger 2024-05-15 16:12:13 +02:00
VinciGit00
a8fb851999 remove asdt 2024-05-15 16:11:08 +02:00
VinciGit00
40260d8d25 remove asdt 2024-05-15 16:00:47 +02:00
Marco Perini
999c930f42 fix(deepcopy): reaplced to shallow copy 2024-05-15 16:00:42 +02:00
VinciGit00
cc5adefd29 fix: come back to the old version 2024-05-15 15:54:00 +02:00
VinciGit00
5587a64d23 fix: removed unused 2024-05-15 15:51:27 +02:00
VinciGit00
7ced0d0564 remove asdt 2024-05-15 15:49:09 +02:00
VinciGit00
29d284e497 Merge branch 'main' into logger-integration 2024-05-15 15:28:20 +02:00
Marco Vinciguerra
008f8d9eae
Merge pull request #248 from VinciGit00/main 
reallignment
 2024-05-15 15:20:53 +02:00
Marco Vinciguerra
cffcf80a75
Merge branch '88-blockscraper-implementation' into main 2024-05-15 15:20:12 +02:00
Marco Vinciguerra
0b5cdd48fd
Merge pull request #246 from VinciGit00/main 
reallignment
 2024-05-15 15:19:05 +02:00
Marco Vinciguerra
22cd9e3605
Merge branch 'search_link_context' into main 2024-05-15 15:16:57 +02:00
Marco Perini
096b665c01 fix(searchgraph): used shallow copy to serialize obj 2024-05-15 15:05:47 +02:00
mayurdb
0b71b9a1e8 Add a new graph traversal that allows more than one edges out of a graph 2024-05-15 13:46:19 +02:00
VinciGit00
a4700bfc75 add robot node 2024-05-15 12:00:32 +02:00
VinciGit00
05890835f5 refactoring of loggers 2024-05-15 10:54:53 +02:00
Marco Vinciguerra
932df8d491
Merge pull request #238 from VinciGit00/gpt4-omni 2024-05-14 17:09:23 +02:00
mayurdb
a458ec4b9f Update the prompt for the search_link_node 2024-05-14 16:59:16 +02:00
Marco Perini
a6e1813ddd fix(fetch_node): bug in handling local files 2024-05-14 16:51:10 +02:00
Marco Perini
fcb3abb01d feat(omni-search): added omni search graph and updated docs 2024-05-14 16:26:43 +02:00
VinciGit00
e53766b16e feat: add logger integration 2024-05-14 15:20:39 +02:00
Marco Perini
a296927624 feat(omni-scraper): working OmniScraperGraph with images 2024-05-14 13:46:49 +02:00
Marco Perini
90955ca52f feat(gpt-4o): image to text single node test 2024-05-14 11:43:21 +02:00
Federico Aguzzi
52a4a3b22d feat: add gpt-4o 
closes #232
 2024-05-14 11:11:34 +02:00
VinciGit00
d6f5ca874f Merge branch 'main' into pre/beta 2024-05-14 10:24:29 +02:00
Marco Perini
367dea5cbd
Merge branch 'pre/beta' into feat/parallel-node-execution 2024-05-13 23:50:40 +02:00
Marco Perini
a8d5e7db05 feat(batchsize): tested different batch sizes and systems 2024-05-13 23:49:48 +02:00
Alok Saboo
d9752b1619 chore: update models_tokens.py with new model configurations 2024-05-13 17:22:32 -04:00
JGalego
859c5d5ba5 Refactored to include custom AWS client for bedrock; Added missing Anthropic class 2024-05-13 17:57:53 +01:00
Marco Perini
dedc733047 fix(asyncio): replaced deepcopy with copy due to serialization problems 2024-05-13 18:46:34 +02:00
Marco Vinciguerra
f3d44c09ae
Merge pull request #228 from JGalego/fix/bedrock-support 
fix: Amazon Bedrock Support
 2024-05-13 14:57:23 +02:00
VinciGit00
d0167dee71 fix: bug for claude 2024-05-13 13:43:42 +02:00
JGalego
39be38f292 Fixed anthropic/bedrock conflict; Removed duplicate class Claude; Updated BedrockEmbeddings class 2024-05-13 12:35:42 +01:00
Marco Perini
0c1594737f fix(fetch-node): removed isSoup from default 2024-05-13 12:09:55 +02:00
Marco Perini
7e8acd8e6a
Merge branch 'pre/beta' into fix/fetch-node-proxybroker 2024-05-13 11:17:37 +02:00
Marco Perini
5d6d996e8f fix(proxy-rotation): removed max_shape duplicate 2024-05-13 07:26:43 +02:00
Marco Perini
1e9a564616 fix(proxy-rotation): removed duplicated arg and passed the loader_kwarhs correctly to the node 2024-05-12 18:39:03 +02:00
VinciGit00
e2350eda62 feat: add new prompt info 2024-05-12 11:14:30 +02:00
Federico Aguzzi
156b67b91e feat: add support for deepseek-chat 
closes #222
 2024-05-12 01:00:11 +02:00
mayurdb
dd29c16cbe Merge branch 'deepScrape' of github.com:mayurdb/Scrapegraph-ai into deepScrape 2024-05-11 16:58:54 +05:30
mayurdb
9a67a26cd3 Update documentation 2024-05-11 16:57:22 +05:30
Mayur Bhosale
ea3b545573
Merge branch 'pre/beta' into deepScrape 2024-05-11 16:54:40 +05:30
mayurdb
8f1fbe7e10 minor changes 2024-05-11 16:52:36 +05:30
mayurdb
df271b6451 Add search link node that can find out relevant links in the webpage 2024-05-11 16:39:55 +05:30
Federico Minutoli
b54d984c13 fix(chromium-loader): ensure it subclasses langchain's base loader 2024-05-11 11:44:56 +02:00
VinciGit00
dc91719365 Update cleanup_html.py 2024-05-11 10:49:16 +02:00
Marco Vinciguerra
b752499fab
Merge pull request #217 from mayurdb/fetchLinkFix 
Fetch links in the page while parsing html
 2024-05-11 09:42:40 +02:00
Stefan Krawczyk
f2bb1cc4f8 Fixes LC document deserialization 
Depends on https://github.com/DAGWorks-Inc/burr/pull/175.
 2024-05-11 00:06:02 -07:00
mayurdb
300fd5d253 Fetch links in the page while parsing html 2024-05-11 09:46:51 +05:30
Eric Page
0683e78e78
Merge branch 'pre/beta' into fix-GenerateScraperGraph 2024-05-11 01:59:28 +02:00
Eric Page
24c3b0580c Removed nonfunctional RAG node from ScriptCreatorGraph 2024-05-11 01:38:40 +02:00
Eric Page
aac51ba290 Removed dead code, allows GenerateScraperNode to generate scraper with 
one chunk of context
 2024-05-11 01:34:51 +02:00
Eric Page
40884747c7 Added parse_html option in parse_node 2024-05-11 00:32:01 +02:00
Federico Minutoli
627cbeeb20 feat(parallel-exeuction): add asyncio event loop dispatcher with semaphore for parallel graph instances 
TODO: still untested
 2024-05-11 00:13:27 +02:00
Federico Minutoli
fc2aa3ac1c Merge branch 'pre/beta' of https://github.com/DiTo97/Scrapegraph-ai into fix/fetch-node-proxybroker 2024-05-10 21:20:40 +02:00
Federico Minutoli
768719cce8 feat(safe-web-driver): enchanced the original AsyncChromiumLoader web driver with proxy protection and flexible kwargs and backend 
the original class prevents passing kwargs down to the playwright backend, making some config unfeasible, including passing a proxy server to the web driver.

the new class has backward compatibility with the original, but 1) allows any kwarg to be passed down to the web driver, 2) allows specifying the web driver backend (only playwright is supported for now) in case more (e.g., selenium) will be supported in the future and 3) automatically fetches a suitable proxy if one is not passed already
 2024-05-10 21:13:38 +02:00
Federico Minutoli
217013181d feat(proxy-rotation): add parse (IP address) or search (from broker) functionality for proxy rotation 
the broker has been made fully configurable for anonymity level, admissible locations, scheme and max shape not to waste resources, unlike the original `free-proxy` package.

other options have been explored (e.g., `proxybroker`, `proxybroker2`) due to their built-in proxy server and rotation capabilities, but the former is no longer maintained, and the latter has issue with any python version outside of python 3.9
 2024-05-10 21:09:48 +02:00
Federico Minutoli
db2234bf5d feat(webdriver-backend): add dynamic import scripts from module and file 2024-05-10 21:06:05 +02:00
Stefan Krawczyk
82afa0ef3d Working smart scraper graph 
Bugs found in tracker persistence:
 - serializing inputs properly for tracker
 - deserializing state from a previous found
 2024-05-10 10:49:53 -07:00
Stefan Krawczyk
d94195f823 WIP 2024-05-10 09:48:46 -07:00
Stefan Krawczyk
be16fecb55 WIP 2024-05-10 09:48:44 -07:00
Marco Perini
864aa91326 feat: revert fetch_node 2024-05-10 15:11:54 +02:00
Marco Vinciguerra
99adc9799f
Merge branch 'pre/beta' into fetchNodeFix 2024-05-10 11:13:54 +02:00
mayurdb
f8ce3d5916 fix: Augment the information getting fetched from a webpage 2024-05-10 13:28:53 +05:30
VinciGit00
0ab31c3fdb fix: add json integration 2024-05-09 21:07:07 +02:00
VinciGit00
324e977b85 fix: fixed bugs for csv and xml 2024-05-09 20:46:46 +02:00
Shubham Kamboj
f10f3b1438 feat: Add support for passing pdf path as source 2024-05-09 21:55:05 +05:30
VinciGit00
94156755d1 Update abstract_graph.py 2024-05-09 12:29:39 +02:00
VinciGit00
403979337c Update abstract_graph.py 2024-05-09 10:46:31 +02:00
VinciGit00
0bb68d1500 Merge branch 'main' of https://github.com/VinciGit00/Scrapegraph-ai 2024-05-09 10:31:21 +02:00
VinciGit00
8272d736a6 add tokenizatio for mxbai-embed-large 2024-05-08 21:50:42 +02:00
VinciGit00
4ed0fb89c3 feat: update info 2024-05-08 21:25:03 +02:00
VinciGit00
e7d39a5daf fixed gemini embeddings 2024-05-08 21:24:17 +02:00
Marco Perini
6b71ec1d2b fix(examples): local, mixed models and fixed SearchGraph embeddings problem 2024-05-08 15:36:26 +02:00
Marco Perini
186c0d035d fix(examples): openai std examples 2024-05-08 14:56:44 +02:00
Marco Vinciguerra
b326886250
Merge branch '88-blockscraper-implementation' into asdt 2024-05-07 13:27:30 +02:00
VinciGit00
2258fe5ac0 add new search graph examples 2024-05-06 22:31:48 +02:00
VinciGit00
67d5fbf816 feat: new search_graph 2024-05-06 22:09:18 +02:00
VinciGit00
5a67bca0db Merge branch 'pre/beta' into pr/161 2024-05-06 14:50:04 +02:00
VinciGit00
ac6d2005bb Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta 2024-05-06 14:46:25 +02:00
VinciGit00
cbd77dfdb0 removed claude 2024-05-06 14:46:24 +02:00
Marco Vinciguerra
97c3fffaf3
Merge pull request #162 from f-aguzzi/patch-1 
feat: add new hugging_face models
 2024-05-06 14:43:29 +02:00
Marco Vinciguerra
f6442ccca3
Merge pull request #157 from VinciGit00/llava_integration 2024-05-06 14:34:32 +02:00
Federico Aguzzi
d5547a450c
feat: add new hugging_face models 2024-05-06 14:28:58 +02:00
Cem Uzunoglu
2ac9e16dd9 Fixed accidental reformatting. 2024-05-06 15:09:56 +03:00
Cem Uzunoglu
e264e92e72 Added support for Claude 3 models from Anthropic 2024-05-06 14:34:35 +03:00
VinciGit00
8c7c3e3ced Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta 2024-05-06 13:29:32 +02:00
VinciGit00
ac0a2e52fb Update models_tokens.py 2024-05-06 13:29:31 +02:00
Marco Perini
fd01b73b71 fix(llm): fixed gemini api_key 2024-05-06 13:17:47 +02:00
Shubham Kamboj
726de28898 feat: Fix bug for gemini case when embeddings config not passed 2024-05-06 15:40:03 +05:30
VinciGit00
89a1f99778 add lava integration for ollama 2024-05-06 12:03:39 +02:00
VinciGit00
51aa109e42 feat: add turboscraper (alfa) 2024-05-06 11:59:14 +02:00
VinciGit00
5e1d5db6da Update search_internet_node.py 2024-05-06 10:09:17 +02:00
VinciGit00
80053a2358 Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta 2024-05-06 10:07:56 +02:00
VinciGit00
389b52aabb removed examples 2024-05-06 10:07:31 +02:00
VinciGit00
88d999e15b add website content 2024-05-06 09:36:32 +02:00
Marco Perini
930adb38f2 feat(node): multiple url search in SearchGraph + fixes 
Implemented GraphIteratorNode and MergeAnswersNode to create multiple istances of a graph and merge the scraped content from multiple pages
 2024-05-06 00:30:09 +02:00
Marco Perini
dbb614a8dd feat: multiple graph instances 2024-05-05 23:51:04 +02:00
Marco Perini
1c4ba91620 exposed abstract_graph allowing the user to create new graphs 2024-05-05 22:05:11 +02:00
Marco Perini
84fcb44aaa feat: fixed custom_graphs example and robots_node 2024-05-05 22:02:24 +02:00
Marco Perini
36a1522bba
Merge pull request #153 from VinciGit00/google_embeddings 
feat: add gemini embeddings
 2024-05-05 17:54:07 +02:00
Lorenzo Padoan
e1b9d69360 dev basic class blockindentifier 2024-05-05 17:36:30 +02:00
Eric Page
cc27b21c3f
Merge branch 'pre/beta' into pass-common-params-graph 2024-05-05 16:18:32 +02:00
Eric Page
3ae2ea1dbd Miscellaneous "llm" -> "llm_model" refactors 2024-05-05 15:58:50 +02:00
Eric Page
a53e95cbf0 Corrected graphs to use common params 2024-05-05 14:36:16 +02:00
Eric Page
8d0e109a70 Added overwrite keyword to set_common_params` 2024-05-05 14:32:58 +02:00
VinciGit00
79daa4c112 feat: add gemini embeddings 2024-05-05 11:40:44 +02:00
Eric Page
4dc6049e76 Simplified create graph functions using common params 2024-05-05 10:40:04 +02:00
Eric Page
444a13a6a6 Created set_common_params function 2024-05-05 10:15:37 +02:00
Eric Page
2178485b3d Adjusted graphs to reflect node_config change 2024-05-05 09:47:13 +02:00
VinciGit00
cb1bd00a19 removed unused node 2024-05-05 09:40:35 +02:00
Eric Page
729d5d7597 Changed node_config["llm"] to node_config["llm_model"] 2024-05-05 09:36:11 +02:00
Eric Page
c2c61625b3 Corrected logic of update_config function in base_node.py 2024-05-05 06:37:31 +02:00
Eric Page
b7539d4725 Added update_config function to base_node.py 2024-05-05 06:25:43 +02:00
Marco Perini
0ab7272fd7
Merge branch 'pre/beta' into 133-support-claude3-haiku-and-others-using-litellm 2024-05-05 02:15:15 +02:00
Marco Perini
8c6dda6e27
Merge pull request #138 from VinciGit00/new-search-function 
feat: refactoring search function
 2024-05-05 02:02:41 +02:00
VinciGit00
2878695d5f fix: trailing whitespace 2024-05-04 21:26:19 +02:00
Marco Vinciguerra
2a8ff0f65d
Merge branch 'main' into pre/beta 2024-05-04 21:20:40 +02:00
Shubham Kamboj
fd59f282a8 Bug fixes 2024-05-04 22:18:54 +05:30
Shubham Kamboj
d830d1371b Resolve merge conflicts 2024-05-04 22:01:40 +05:30
Shubham Kamboj
7599234ab9 feat: Enable end users to pass model instances of HuggingFaceHub 2024-05-04 21:42:57 +05:30
Tamas Darvas
1630c66a58
Update models_tokens.py with ollama/nomic-embed-text 2024-05-04 09:36:19 +02:00
Marco Perini
df293fdfd2
Merge pull request #139 from VinciGit00/pdf_scraper_graph_introduction 
feat: add pdf scraper
 2024-05-03 21:45:09 +02:00
VinciGit00
10a94530e3 feat: add pdf scraper 2024-05-03 21:42:02 +02:00
Marco Perini
aa9271e7bc feat: Azure implementation + embeddings refactoring 2024-05-03 21:40:01 +02:00
VinciGit00
aeb1acbf05 feat: refactoring search function 2024-05-03 21:06:09 +02:00
VinciGit00
39f64e5d48 add claude model 2024-05-03 17:17:31 +02:00
Marco Vinciguerra
2abe05a0cf
Merge pull request #135 from S4mpl3r/feature 2024-05-03 16:49:07 +02:00
S4mpl3r
819cbcd3be refactor: Changed the way embedding model is created in AbstractGraph class and removed handling of embedding model creation from RAGNode. Now AbstractGraph will call a dedicated method for embedding models instead of _create_llm. This makes it easy to use any LLM with any supported embedding model. 2024-05-03 16:14:27 +03:30
Marco Perini
cb1cb616b7
feat: add base_node to __init__.py 
Useful for people to create custom nodes
 2024-05-03 14:18:12 +02:00
Marco Vinciguerra
96ebcfcc35
Merge branch 'pre/beta' into main 2024-05-02 21:05:28 +02:00
VinciGit00
2ea54eab1d fix: gemini errror 2024-05-02 18:56:37 +02:00
Marco Vinciguerra
579a27faf9
Merge pull request #126 from VinciGit00/bedrock_support 
feat: bedrock support
 2024-05-02 17:57:20 +02:00
Marco Vinciguerra
a2d5c75f10
Merge branch 'pre/beta' into main 2024-05-02 17:55:21 +02:00
Shubham Kamboj
b86aac2188 feat: Allow end users to pass model instances for llm and embedding model 2024-05-02 20:09:23 +05:30
VinciGit00
16de49fa08 add integration for bedrock 
Co-Authored-By: redrusty2 <15157208+redrusty2@users.noreply.github.com>
 2024-05-02 13:47:17 +02:00
VinciGit00
4a3bc37f2f fix: bug on script_creator_graph.py 2024-05-02 09:23:46 +02:00
VinciGit00
5cf4e4f92f fix: examples and graphs 2024-05-02 09:20:46 +02:00
Marco Perini
29427c2334 docs: fixed basegraph docstring 2024-05-02 01:05:20 +02:00
Marco Perini
c11fc28896 docs: refactor examples 2024-05-02 00:44:19 +02:00
Marco Perini
0631985e61 docs: graphs and helpers docstrings 2024-05-02 00:23:38 +02:00
Marco Perini
18c20eb03d docs: refactor models docstrings 2024-05-01 23:33:42 +02:00
Marco Perini
1409797475 docs: refactor nodes docstrings 2024-05-01 23:17:57 +02:00
Marco Perini
4d542a88f7 feat: added node and graph for CSV scraping 2024-05-01 21:34:40 +02:00
Marco Perini
02d1af006c feat: modified node name 2024-05-01 21:33:25 +02:00
Marco Perini
e9817963c8 docs: base and fetch node 2024-05-01 21:30:06 +02:00
VinciGit00
276ac8b40c changed name 2024-05-01 20:17:01 +02:00
VinciGit00
7c1b5796d7 add csv scraper 2024-05-01 17:20:04 +02:00
Perinim
cf038b33ea docs: update utils docstrings 2024-05-01 12:35:12 +02:00
EURAC\marperini
2dd7817cfb feat: added verbose flag to suppress print statements 2024-04-30 15:31:57 +02:00
Marco Perini
d592d27bb4
Merge pull request #115 from VinciGit00/101-scrape-json-files 
feat: add xml scraper and json scraper
 2024-04-30 14:29:57 +02:00
EURAC\marperini
8fba7e5490 feat(refactor): changed variable names 2024-04-30 14:28:30 +02:00
VinciGit00
e3d0194dc9 fix: script generator and add new benchmarks 2024-04-30 11:51:04 +02:00
VinciGit00
da2c82a2a2 add json and xml scraper 2024-04-30 10:52:22 +02:00
VinciGit00
e0ffc838b0 feat: add cluade integration 2024-04-30 10:08:54 +02:00
EURAC\marperini
42ab0aa1d2 feat(fetch): added playwright support 2024-04-30 04:02:58 +02:00
EURAC\marperini
dbbf10fc77 feat(llm): implemented groq model 2024-04-30 02:32:14 +02:00
Marco Vinciguerra
a449ed1199
Merge branch 'pre/beta' into 93-groq-model-implementation 2024-04-29 21:48:08 +02:00
Marco Vinciguerra
ecaef43bf3
Merge pull request #107 from VinciGit00/main 
merging for new version
 2024-04-29 21:46:04 +02:00
Lorenzo Padoan
7dd5b1a033 feat: base groq + requirements + toml update with groq 
Co-Authored-By: gioCarBo <109686531+giocarbo@users.noreply.github.com>
 2024-04-29 20:06:51 +02:00
VinciGit00
deb920a33e fixing json and example 2024-04-29 16:11:57 +02:00
VinciGit00
3eacc6fbc3 add paths 2024-04-29 16:07:11 +02:00
VinciGit00
674e64222e add first new graphs 2024-04-29 15:55:21 +02:00
VinciGit00
0b25d9a977 Update search_link_node.py 2024-04-29 15:49:46 +02:00
VinciGit00
4f10ac302e fixed gemma model 2024-04-29 11:06:30 +02:00
VinciGit00
ab72db3b50 if the conf is not present preset to 8192 the context window 2024-04-29 10:26:34 +02:00
EURAC\marperini
6e7283ed8f feat: add finalize_node() 2024-04-29 10:05:04 +02:00
PeriniM
7f1c3b7d83 fix: added missing dependecies 
include python v3.9.0 and change deprecated method langchain
 2024-04-28 20:29:23 +02:00
VinciGit00
a9b11e433a fix: bug for calculate costs 2024-04-28 14:36:49 +02:00
VinciGit00
9cd516507c fix: bug with fetch node 2024-04-27 21:23:35 +02:00
VinciGit00
adbc08f27b fix: robot node and proxyes 2024-04-27 19:07:37 +02:00
VinciGit00
b754dd909c fix: changed proxy function 2024-04-27 14:40:50 +02:00
VinciGit00
f6077d1f98 feat: add new proxy rotation function 2024-04-27 13:31:53 +02:00
Marco Vinciguerra
44bc9196ef
Merge pull request #84 from VinciGit00/79-add-getstate-function-to-abstract-graph 
complete get_state function
 2024-04-26 21:44:33 +02:00
Marco Vinciguerra
4f1d28d89a
Merge pull request #81 from VinciGit00/refactoring_engine 
Refactoring engine -> from set to list
 2024-04-26 21:37:57 +02:00
Marco Vinciguerra
922aa96b37
Merge branch 'pre/beta' into search_links_node 2024-04-26 19:23:42 +02:00
EURAC\marperini
e778d27169 added tree metadata 2024-04-26 15:17:44 +02:00
VinciGit00
cd8d3e7a4f refactoring of the graphs 2024-04-26 11:08:56 +02:00
VinciGit00
b2ebabd32f Update base_graph.py 2024-04-25 19:45:49 +02:00
VinciGit00
e714a59c2e refactoring of engine 2024-04-25 19:22:12 +02:00
VinciGit00
75255bb4af complete get_state function 2024-04-25 15:08:42 +02:00
EURAC\marperini
dd99ac595e structural and textual hashing 2024-04-25 11:56:41 +02:00
EURAC\marperini
c5f9fcaabe Merge branch 'asdt' of https://github.com/VinciGit00/Scrapegraph-ai into asdt 2024-04-25 11:06:54 +02:00
EURAC\marperini
c927f707e9 subtrees implementation 2024-04-25 11:06:05 +02:00
VinciGit00
f232717bf8 REFACTORING 2024-04-25 09:19:44 +02:00
VinciGit00
a73496df9b refactoring of nodes 2024-04-24 21:13:27 +02:00
EURAC\marperini
ae69e4b340 dom tree structure 2024-04-24 18:59:44 +02:00
VinciGit00
e68b707e0d changed color and return type 2024-04-24 17:29:49 +02:00
EURAC\marperini
ae49dee985 fixed robots_node and add test 2024-04-24 16:12:53 +02:00
VinciGit00
9b9a9f204c handling the agent model 2024-04-24 13:13:46 +02:00
VinciGit00
adf89e5e1b add new info for robot node 2024-04-24 10:14:26 +02:00
VinciGit00
8ab603220d changed node name 2024-04-24 10:02:09 +02:00
VinciGit00
726efb7f00 add new info 2024-04-23 20:51:39 +02:00
VinciGit00
8ef9dec2e2 add new node 2024-04-23 20:23:35 +02:00
Lorenzo Padoan
12dcc5065a ADD benchmarks m2 max llama3 2024-04-23 12:28:51 +02:00
VinciGit00
261ecbf3a5 fixed typos in the doc 2024-04-22 21:32:06 +02:00
VinciGit00
3960e56d29 fixed typos in the doc 2024-04-22 21:31:23 +02:00
VinciGit00
9e3dddcb44 add nodes that search all the links 2024-04-22 18:22:40 +02:00
VinciGit00
74d7ef614a refactor of conbert to json function 2024-04-21 20:30:56 +02:00
Alok Saboo
6cbf992870 chore: Remove .csv extension from filename in convert_to_csv.py 2024-04-21 14:13:34 -04:00
VinciGit00
8aa2cadb00 add integration for llama3 2024-04-21 13:11:10 +02:00
EURAC\marperini
ac099e2763 fixed ollama embedder params 2024-04-21 01:21:09 +02:00
VinciGit00
13669cc1ab revert to old naming 2024-04-19 13:43:51 +02:00
VinciGit00
a2349e56af recatoring of nodes excluive for openai 2024-04-19 13:21:48 +02:00
VinciGit00
3f95801737 Merge branch 'main' of https://github.com/VinciGit00/Scrapegraph-ai 2024-04-19 10:36:40 +02:00
EURAC\marperini
134c94e0ed added first asdt implementation 2024-04-17 17:12:59 +02:00
EURAC\marperini
c032131692 fixed chunk response 2024-04-17 15:20:45 +02:00
Marco Perini
b14527c08d
Merge branch 'main' into refactor_generate_answer_node 2024-04-17 12:08:43 +02:00
EURAC\marperini
c237e63631 removed unused variable 2024-04-17 12:05:51 +02:00
Marco Perini
edc439fd01
Merge pull request #64 from VinciGit00/generateScraperbranch 
add generateScraperBranch
 2024-04-17 11:51:06 +02:00
EURAC\marperini
b2d170cd86 refactored and fixed single chunk bug 2024-04-17 11:50:19 +02:00
EURAC\marperini
bdac2df4bb fixed single chunk bug 2024-04-17 11:34:10 +02:00
Andrea Rota
b0e446f014 feat: apply remove to the document before updating the state 2024-04-17 11:24:56 +02:00
EURAC\marperini
d5f1c82c91 Merge remote-tracking branch 'origin/refactor_genrerate_answer_node' 2024-04-17 10:55:28 +02:00
Marco Vinciguerra
4703a0b94c
Update remover.py 2024-04-16 12:27:17 +02:00
VinciGit00
4233430518 add integration on the fetch node 2024-04-16 12:19:23 +02:00
VinciGit00
9661c77ebe add minimizer function 2024-04-16 12:07:43 +02:00
Marco Vinciguerra
3fc18b2110
Update generate_answer_node.py 2024-04-16 12:00:33 +02:00
Matteo Vedovati
9c3b490b03
Update generate_scraper_node.py 2024-04-16 10:21:49 +02:00
VinciGit00
372981f153 Update generate_answer_node.py 2024-04-16 10:09:35 +02:00
VinciGit00
c56640034e add return statement and new answers 2024-04-15 10:49:02 +02:00
VinciGit00
8334737f10 Update generate_answer_node.py 2024-04-15 10:44:03 +02:00
VinciGit00
724aee6b23 add informations on prompt 2024-04-15 10:43:40 +02:00
VinciGit00
8fc41873a7 refactoring of generate answer node 2024-04-14 21:53:15 +02:00
VinciGit00
876223da56 add generateScraperBranch 
Co-Authored-By: Matteo Mei <52063123+FattiMei@users.noreply.github.com>
 2024-04-14 13:18:50 +02:00
Lorenzo Padoan
849f411844
Merge pull request #61 from VinciGit00/anti_malicious 
add anti not scraping task
 2024-04-12 13:47:10 +02:00
VinciGit00
e0978c53b3 Merge branch 'main' into ollama_integration 2024-04-12 13:00:35 +02:00
VinciGit00
3640434f5c add utils 2024-04-12 12:59:15 +02:00
Marco Perini
dde9eeec87
Merge pull request #60 from VinciGit00/huggingface_integration 
add hugginface integration (embeddings, models ...)
 2024-04-11 22:40:05 +02:00
VinciGit00
77581876b5 add anti not scrapong task 2024-04-11 19:31:54 +02:00
VinciGit00
dc149e6f63 fixing bug on rag node 2024-04-11 19:26:38 +02:00
EURAC\marperini
74ec928f96 added execution info of the graph, time, price etc 2024-04-11 14:29:32 +02:00
VinciGit00
53e317d46b add hugginface integration 2024-04-11 12:43:06 +02:00
EURAC\marperini
982f142f20 fixed embedder_model config for local models 2024-04-10 11:00:52 +02:00
EURAC\marperini
8b915e3ec6 fixed issue ollama embeddings 2024-04-10 09:29:03 +02:00
ftoppi
26616d25b1
abstract_graph.py: allow user to set Ollama model_tokens in config 2024-04-09 21:27:15 +02:00
VinciGit00
78a1e7aec4 new model 2024-04-09 13:54:10 +02:00
Marco Vinciguerra
b4e640879d
Merge branch 'main' into refactoring_examples_folder 2024-04-09 11:46:34 +02:00
VinciGit00
992e7f862f add models avaiables 2024-04-09 10:33:45 +02:00
VinciGit00
cec4c43c64 add new local models context window 2024-04-09 10:23:38 +02:00
VinciGit00
0ffefdd170 refactoring if examples ffolder 2024-04-08 22:28:55 +02:00
EURAC\marperini
a601695fd5 updated nodes metadata 2024-04-08 15:23:23 +02:00
EURAC\marperini
dee1a42629 fixed token models, added mistral support 2024-04-08 15:21:06 +02:00
EURAC\marperini
17add20c13 implemented node_config, add embedder model choice, add azure endpoint, refactor graphs and exmaples 2024-04-08 12:18:32 +02:00
EURAC\marperini
6a83c2306d docker compose working ollama implementation 2024-04-07 11:08:08 +02:00
VinciGit00
6dc4c6dec4 Merge branch 'main' into research_branch 2024-04-06 14:44:56 +02:00
VinciGit00
a5fc71f34e fixed internet research 2024-04-06 14:39:19 +02:00
EURAC\marperini
57a53bd6af refactor examples 2024-04-06 13:22:42 +02:00
EURAC\marperini
c2709aa50c working search graph 2024-04-06 13:01:04 +02:00
VinciGit00
3269b4aaa7 fixing infos for the pr 2024-04-03 13:33:17 +02:00
VinciGit00
c9951340b5 fixed image_to_tex_node and refactoring 2024-04-03 12:56:25 +02:00
VinciGit00
4c3ea8f4ac fixed image_to_tex_node and refactoring 2024-04-03 12:53:12 +02:00
VinciGit00
b53bfefeea removed junk if 2024-04-02 12:57:27 +02:00
VinciGit00
db6d988b2d refactoring for pylint 2024-04-01 13:41:50 +02:00
VinciGit00
471df48a22 add google search 2024-04-01 13:36:02 +02:00
VinciGit00
5e38c2e298 add superclasse for graphs 2024-03-28 20:46:17 +01:00
VinciGit00
7d0fd74bfc add search graph 2024-03-28 16:54:01 +01:00
VinciGit00
b8a3590235 add SearcInetnetNode 2024-03-28 13:14:39 +01:00
VinciGit00
fff1f9ee51 new function for searching 2024-03-28 12:41:45 +01:00
VinciGit00
a28ca993de refactoring of examples folder 2024-03-27 21:41:21 +01:00
VinciGit00
672e9a5191 refactoring of research function 2024-03-27 21:29:10 +01:00
VinciGit00
97d0a6887f add reserch web function 2024-03-27 21:22:04 +01:00
VinciGit00
587dfa5c68 refactoring of the saving functions 2024-03-26 14:15:36 +01:00
VinciGit00
be36ab8707 add new type of scraping through text 2024-03-24 21:22:26 +01:00
VinciGit00
f448b8dfb4 removed unused return 2024-03-18 22:55:24 +01:00
VinciGit00
e25f74ddc8 fixed bug for scraping from node + add example 2024-03-18 22:53:52 +01:00
VinciGit00
1afb950751 refactoring of code, add example on the readme 2024-03-18 14:35:47 +01:00
Perinim
cae38a0d67 fixed bugs after merge 2024-03-18 11:30:14 +01:00
Perinim
b5bdd4e729 Merge remote-tracking branch 'origin/main' into refactoring_nodes 2024-03-18 11:07:42 +01:00
Perinim
7d3190a636 fixed example graphs and utils files 2024-03-18 10:20:23 +01:00
Perinim
b547972968 fixed custom_graph, graph_builder and trulens_evaluator 2024-03-18 09:12:59 +01:00
Perinim
52934bf007 implemented graph_config, fixed smart_scraper and speech graph 2024-03-17 20:35:04 +01:00
Perinim
f27e0b4c5f removed fetch_html and fetch_text (merged) 2024-03-17 17:01:29 +01:00
Perinim
875a7cc4f0 refactored RagNode and GenerateAnswerNode 2024-03-17 16:48:58 +01:00
Perinim
3585cd81e5 refactor fetch node 2024-03-17 12:02:35 +01:00
Perinim
20d7b69008 refactored BaseNode and ParseNode 2024-03-17 11:40:35 +01:00
Lorenzo Padoan
d0f4b0159f DEV gemini support for simple custom graph 
-Not supported yet smartscrapergrapn with gemini
 2024-03-14 19:35:04 +01:00
Perinim
538133dfff added parse input string function 2024-03-13 19:34:51 +01:00
VinciGit00
a849315182 add new tests and refactoring of code 2024-03-12 21:40:09 +01:00
Perinim
308c2d28c8 fixed text input example 2024-03-12 12:22:47 +01:00
Perinim
aeff434878 fixed docstring 2024-03-12 11:53:03 +01:00
Perinim
4d2e872484 fixed speech graph 2024-03-12 11:14:14 +01:00
Perinim
9d31db2d35 working smart scraper example 2024-03-12 11:06:08 +01:00
Lorenzo Padoan
03e6881f00
Merge pull request #37 from VinciGit00/refactor-fetch-node 
Refactor fetch node, add integration for preprocessing
 2024-03-08 17:51:19 +01:00
VinciGit00
cffd954446 add integration in the fetch for preprocessing 2024-03-08 17:46:36 +01:00
VinciGit00
ce8bc08b3a refactoring of remove function and implement it in the fetch node 2024-03-08 09:43:09 +01:00
VinciGit00
cd176ff76c removed prints 2024-03-05 18:07:21 +01:00
VinciGit00
6663c9c5bc removed the duplicates 2024-03-05 14:59:48 +01:00
VinciGit00
ff7f8105bb add multiple merging generate answer node 2024-03-05 14:50:31 +01:00
VinciGit00
ab71a6bc5a fixed documentation and fetch_text_node 2024-03-03 18:50:03 +01:00
VinciGit00
e29454e5df add doc for the text node 2024-03-03 12:32:26 +01:00
VinciGit00
a0c77491bf add new progresbbar and finished the node 2024-03-02 20:51:29 +01:00
VinciGit00
1250aeeb8e add the node for reading the html as a txt 2024-03-02 13:32:34 +01:00
VinciGit00
79387d1617 refactoring for pylint score 2024-02-29 10:49:25 +01:00
VinciGit00
17d1f6a47b fix pylint score 2024-02-29 10:32:00 +01:00
VinciGit00
f4d5941831 refctoring of RAG node 2024-02-29 10:29:39 +01:00
Marco Perini
5735f3ec7f
Merge branch 'main' into refactoring-parse-html 2024-02-28 17:57:28 +01:00
VinciGit00
89d193392d refactor of parse_html_node 2024-02-27 17:02:06 +01:00
VinciGit00
cd405e4202 removed llm req 2024-02-27 12:28:46 +01:00
VinciGit00
a7c7526bf1 refactoring of parse_html node 2024-02-26 20:14:53 +01:00
VinciGit00
582c99d879 refactoring of the classes of constructors 2024-02-25 12:35:54 +01:00
VinciGit00
043ab48466 add integration for the RAG for ParseHTMLNode 2024-02-25 11:24:44 +01:00
VinciGit00
fc0f0957e1 add integration for the RAG for ParseHTMLNode 2024-02-25 11:12:44 +01:00
VinciGit00
9d6fcf7882 refactoring 2024-02-25 10:45:13 +01:00
PeriniM
ed64db24df working speech graph with rag 2024-02-23 14:17:46 +01:00
PeriniM
cb8ed4fc5b fixed eval output 2024-02-23 10:33:25 +01:00
PeriniM
284ba5b75d added trulens 2024-02-23 09:39:25 +01:00
VinciGit00
c02203068e add api_key as parameter instead of .env file 2024-02-22 21:13:01 +01:00
Lorenzo Padoan
665a91484b dev trulens_evaluetor 2024-02-22 19:19:10 +01:00
VinciGit00
ae929393b9 refactoring for pylint score 2024-02-22 10:19:10 +01:00
Marco Perini
ed48f273f8
Merge branch 'main' into multi-chunks-implementation 2024-02-22 00:00:01 +01:00
Perinim
62cc9170c8 updated SmartScraperGraph with rag 2024-02-21 23:52:45 +01:00
Perinim
b0c75dc7ed created rag example 2024-02-21 19:15:38 +01:00
Perinim
a626401f8a merged multiple chunks answer 2024-02-21 18:43:38 +01:00
Perinim
a17971b7ed implemented parallelrunnables 2024-02-21 18:17:34 +01:00
Perinim
15b468473b working test rag 2024-02-21 17:02:10 +01:00
VinciGit00
906fea92e9 add examples + module documentation 2024-02-20 16:50:32 +01:00
VinciGit00
7390dba0ab removed dict info and put in another file 2024-02-19 13:50:30 +01:00
VinciGit00
8a566ff588 bugfix of the action 2024-02-19 12:21:41 +01:00
VinciGit00
71bd53d254 refactoring of the code, add tests and examples 2024-02-19 11:54:36 +01:00
PeriniM
4e3005db0a implemented summary graph 2024-02-19 07:18:28 +01:00
PeriniM
da16360b1f working text2speech and image2text 2024-02-19 06:26:50 +01:00
PeriniM
3ed5fd5e48 added openai wrapper and examples 2024-02-19 04:45:43 +01:00
PeriniM
20db549d08 implemented graphbuilder 2024-02-19 04:11:35 +01:00
PeriniM
49c22ae155 implemented graphbuilder class 2024-02-19 03:39:50 +01:00
PeriniM
20398a3059 dev: refactor dict on graph state 2024-02-18 23:36:16 +01:00
VinciGit00
dc442b9ebd refactoring of fetch node 2024-02-18 21:41:42 +01:00
VinciGit00
f8bb32919b refactoring of the example and add 2 new nodes 2024-02-18 20:36:45 +01:00
VinciGit00
e57ec7002a add new node 2024-02-17 21:42:43 +01:00
VinciGit00
4e509e1f66 removed unused files 2024-02-16 19:43:48 +01:00
VinciGit00
63ecd3dd70 refactoring of the folder 2024-02-16 19:29:38 +01:00
VinciGit00
f225957ea9 fixed imports 2024-02-15 09:33:33 +01:00
VinciGit00
ce7949de1b refactoring of the folders 2024-02-15 09:27:39 +01:00
VinciGit00
2a234f2e4d refactoring of the code changing the name 2024-02-14 20:35:30 +01:00