CodeCow/Scrapegraph-ai
1
0
Fork 0
mirror of https://github.com/VinciGit00/Scrapegraph-ai.git synced 2026-07-12 21:01:56 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,293 Commits 2 Branches 448 Tags 17 MiB
5faac9fb46
Commit Graph

1007 Commits

Author SHA1 Message Date
Marco Vinciguerra
2abe05a0cf
Merge pull request #135 from S4mpl3r/feature 2024-05-03 16:49:07 +02:00
S4mpl3r
819cbcd3be refactor: Changed the way embedding model is created in AbstractGraph class and removed handling of embedding model creation from RAGNode. Now AbstractGraph will call a dedicated method for embedding models instead of _create_llm. This makes it easy to use any LLM with any supported embedding model. 2024-05-03 16:14:27 +03:30
Marco Perini
cb1cb616b7
feat: add base_node to __init__.py 
Useful for people to create custom nodes
 2024-05-03 14:18:12 +02:00
Marco Vinciguerra
96ebcfcc35
Merge branch 'pre/beta' into main 2024-05-02 21:05:28 +02:00
VinciGit00
2ea54eab1d fix: gemini errror 2024-05-02 18:56:37 +02:00
Marco Vinciguerra
579a27faf9
Merge pull request #126 from VinciGit00/bedrock_support 
feat: bedrock support
 2024-05-02 17:57:20 +02:00
Marco Vinciguerra
a2d5c75f10
Merge branch 'pre/beta' into main 2024-05-02 17:55:21 +02:00
Shubham Kamboj
b86aac2188 feat: Allow end users to pass model instances for llm and embedding model 2024-05-02 20:09:23 +05:30
VinciGit00
16de49fa08 add integration for bedrock 
Co-Authored-By: redrusty2 <15157208+redrusty2@users.noreply.github.com>
 2024-05-02 13:47:17 +02:00
VinciGit00
4a3bc37f2f fix: bug on script_creator_graph.py 2024-05-02 09:23:46 +02:00
VinciGit00
5cf4e4f92f fix: examples and graphs 2024-05-02 09:20:46 +02:00
Marco Perini
29427c2334 docs: fixed basegraph docstring 2024-05-02 01:05:20 +02:00
Marco Perini
c11fc28896 docs: refactor examples 2024-05-02 00:44:19 +02:00
Marco Perini
0631985e61 docs: graphs and helpers docstrings 2024-05-02 00:23:38 +02:00
Marco Perini
18c20eb03d docs: refactor models docstrings 2024-05-01 23:33:42 +02:00
Marco Perini
1409797475 docs: refactor nodes docstrings 2024-05-01 23:17:57 +02:00
Marco Perini
4d542a88f7 feat: added node and graph for CSV scraping 2024-05-01 21:34:40 +02:00
Marco Perini
02d1af006c feat: modified node name 2024-05-01 21:33:25 +02:00
Marco Perini
e9817963c8 docs: base and fetch node 2024-05-01 21:30:06 +02:00
VinciGit00
276ac8b40c changed name 2024-05-01 20:17:01 +02:00
VinciGit00
7c1b5796d7 add csv scraper 2024-05-01 17:20:04 +02:00
Perinim
cf038b33ea docs: update utils docstrings 2024-05-01 12:35:12 +02:00
EURAC\marperini
2dd7817cfb feat: added verbose flag to suppress print statements 2024-04-30 15:31:57 +02:00
Marco Perini
d592d27bb4
Merge pull request #115 from VinciGit00/101-scrape-json-files 
feat: add xml scraper and json scraper
 2024-04-30 14:29:57 +02:00
EURAC\marperini
8fba7e5490 feat(refactor): changed variable names 2024-04-30 14:28:30 +02:00
VinciGit00
e3d0194dc9 fix: script generator and add new benchmarks 2024-04-30 11:51:04 +02:00
VinciGit00
da2c82a2a2 add json and xml scraper 2024-04-30 10:52:22 +02:00
VinciGit00
e0ffc838b0 feat: add cluade integration 2024-04-30 10:08:54 +02:00
EURAC\marperini
42ab0aa1d2 feat(fetch): added playwright support 2024-04-30 04:02:58 +02:00
EURAC\marperini
dbbf10fc77 feat(llm): implemented groq model 2024-04-30 02:32:14 +02:00
Marco Vinciguerra
a449ed1199
Merge branch 'pre/beta' into 93-groq-model-implementation 2024-04-29 21:48:08 +02:00
Marco Vinciguerra
ecaef43bf3
Merge pull request #107 from VinciGit00/main 
merging for new version
 2024-04-29 21:46:04 +02:00
Lorenzo Padoan
7dd5b1a033 feat: base groq + requirements + toml update with groq 
Co-Authored-By: gioCarBo <109686531+giocarbo@users.noreply.github.com>
 2024-04-29 20:06:51 +02:00
VinciGit00
deb920a33e fixing json and example 2024-04-29 16:11:57 +02:00
VinciGit00
3eacc6fbc3 add paths 2024-04-29 16:07:11 +02:00
VinciGit00
674e64222e add first new graphs 2024-04-29 15:55:21 +02:00
VinciGit00
0b25d9a977 Update search_link_node.py 2024-04-29 15:49:46 +02:00
VinciGit00
4f10ac302e fixed gemma model 2024-04-29 11:06:30 +02:00
VinciGit00
ab72db3b50 if the conf is not present preset to 8192 the context window 2024-04-29 10:26:34 +02:00
EURAC\marperini
6e7283ed8f feat: add finalize_node() 2024-04-29 10:05:04 +02:00
PeriniM
7f1c3b7d83 fix: added missing dependecies 
include python v3.9.0 and change deprecated method langchain
 2024-04-28 20:29:23 +02:00
VinciGit00
a9b11e433a fix: bug for calculate costs 2024-04-28 14:36:49 +02:00
VinciGit00
9cd516507c fix: bug with fetch node 2024-04-27 21:23:35 +02:00
VinciGit00
adbc08f27b fix: robot node and proxyes 2024-04-27 19:07:37 +02:00
VinciGit00
b754dd909c fix: changed proxy function 2024-04-27 14:40:50 +02:00
VinciGit00
f6077d1f98 feat: add new proxy rotation function 2024-04-27 13:31:53 +02:00
Marco Vinciguerra
44bc9196ef
Merge pull request #84 from VinciGit00/79-add-getstate-function-to-abstract-graph 
complete get_state function
 2024-04-26 21:44:33 +02:00
Marco Vinciguerra
4f1d28d89a
Merge pull request #81 from VinciGit00/refactoring_engine 
Refactoring engine -> from set to list
 2024-04-26 21:37:57 +02:00
Marco Vinciguerra
922aa96b37
Merge branch 'pre/beta' into search_links_node 2024-04-26 19:23:42 +02:00
EURAC\marperini
e778d27169 added tree metadata 2024-04-26 15:17:44 +02:00
VinciGit00
cd8d3e7a4f refactoring of the graphs 2024-04-26 11:08:56 +02:00
VinciGit00
b2ebabd32f Update base_graph.py 2024-04-25 19:45:49 +02:00
VinciGit00
e714a59c2e refactoring of engine 2024-04-25 19:22:12 +02:00
VinciGit00
75255bb4af complete get_state function 2024-04-25 15:08:42 +02:00
EURAC\marperini
dd99ac595e structural and textual hashing 2024-04-25 11:56:41 +02:00
EURAC\marperini
c5f9fcaabe Merge branch 'asdt' of https://github.com/VinciGit00/Scrapegraph-ai into asdt 2024-04-25 11:06:54 +02:00
EURAC\marperini
c927f707e9 subtrees implementation 2024-04-25 11:06:05 +02:00
VinciGit00
f232717bf8 REFACTORING 2024-04-25 09:19:44 +02:00
VinciGit00
a73496df9b refactoring of nodes 2024-04-24 21:13:27 +02:00
EURAC\marperini
ae69e4b340 dom tree structure 2024-04-24 18:59:44 +02:00
VinciGit00
e68b707e0d changed color and return type 2024-04-24 17:29:49 +02:00
EURAC\marperini
ae49dee985 fixed robots_node and add test 2024-04-24 16:12:53 +02:00
VinciGit00
9b9a9f204c handling the agent model 2024-04-24 13:13:46 +02:00
VinciGit00
adf89e5e1b add new info for robot node 2024-04-24 10:14:26 +02:00
VinciGit00
8ab603220d changed node name 2024-04-24 10:02:09 +02:00
VinciGit00
726efb7f00 add new info 2024-04-23 20:51:39 +02:00
VinciGit00
8ef9dec2e2 add new node 2024-04-23 20:23:35 +02:00
Lorenzo Padoan
12dcc5065a ADD benchmarks m2 max llama3 2024-04-23 12:28:51 +02:00
VinciGit00
261ecbf3a5 fixed typos in the doc 2024-04-22 21:32:06 +02:00
VinciGit00
3960e56d29 fixed typos in the doc 2024-04-22 21:31:23 +02:00
VinciGit00
9e3dddcb44 add nodes that search all the links 2024-04-22 18:22:40 +02:00
VinciGit00
74d7ef614a refactor of conbert to json function 2024-04-21 20:30:56 +02:00
Alok Saboo
6cbf992870 chore: Remove .csv extension from filename in convert_to_csv.py 2024-04-21 14:13:34 -04:00
VinciGit00
8aa2cadb00 add integration for llama3 2024-04-21 13:11:10 +02:00
EURAC\marperini
ac099e2763 fixed ollama embedder params 2024-04-21 01:21:09 +02:00
VinciGit00
13669cc1ab revert to old naming 2024-04-19 13:43:51 +02:00
VinciGit00
a2349e56af recatoring of nodes excluive for openai 2024-04-19 13:21:48 +02:00
VinciGit00
3f95801737 Merge branch 'main' of https://github.com/VinciGit00/Scrapegraph-ai 2024-04-19 10:36:40 +02:00
EURAC\marperini
134c94e0ed added first asdt implementation 2024-04-17 17:12:59 +02:00
EURAC\marperini
c032131692 fixed chunk response 2024-04-17 15:20:45 +02:00
Marco Perini
b14527c08d
Merge branch 'main' into refactor_generate_answer_node 2024-04-17 12:08:43 +02:00
EURAC\marperini
c237e63631 removed unused variable 2024-04-17 12:05:51 +02:00
Marco Perini
edc439fd01
Merge pull request #64 from VinciGit00/generateScraperbranch 
add generateScraperBranch
 2024-04-17 11:51:06 +02:00
EURAC\marperini
b2d170cd86 refactored and fixed single chunk bug 2024-04-17 11:50:19 +02:00
EURAC\marperini
bdac2df4bb fixed single chunk bug 2024-04-17 11:34:10 +02:00
Andrea Rota
b0e446f014 feat: apply remove to the document before updating the state 2024-04-17 11:24:56 +02:00
EURAC\marperini
d5f1c82c91 Merge remote-tracking branch 'origin/refactor_genrerate_answer_node' 2024-04-17 10:55:28 +02:00
Marco Vinciguerra
4703a0b94c
Update remover.py 2024-04-16 12:27:17 +02:00
VinciGit00
4233430518 add integration on the fetch node 2024-04-16 12:19:23 +02:00
VinciGit00
9661c77ebe add minimizer function 2024-04-16 12:07:43 +02:00
Marco Vinciguerra
3fc18b2110
Update generate_answer_node.py 2024-04-16 12:00:33 +02:00
Matteo Vedovati
9c3b490b03
Update generate_scraper_node.py 2024-04-16 10:21:49 +02:00
VinciGit00
372981f153 Update generate_answer_node.py 2024-04-16 10:09:35 +02:00
VinciGit00
c56640034e add return statement and new answers 2024-04-15 10:49:02 +02:00
VinciGit00
8334737f10 Update generate_answer_node.py 2024-04-15 10:44:03 +02:00
VinciGit00
724aee6b23 add informations on prompt 2024-04-15 10:43:40 +02:00
VinciGit00
8fc41873a7 refactoring of generate answer node 2024-04-14 21:53:15 +02:00
VinciGit00
876223da56 add generateScraperBranch 
Co-Authored-By: Matteo Mei <52063123+FattiMei@users.noreply.github.com>
 2024-04-14 13:18:50 +02:00
Lorenzo Padoan
849f411844
Merge pull request #61 from VinciGit00/anti_malicious 
add anti not scraping task
 2024-04-12 13:47:10 +02:00
VinciGit00
e0978c53b3 Merge branch 'main' into ollama_integration 2024-04-12 13:00:35 +02:00
VinciGit00
3640434f5c add utils 2024-04-12 12:59:15 +02:00
Marco Perini
dde9eeec87
Merge pull request #60 from VinciGit00/huggingface_integration 
add hugginface integration (embeddings, models ...)
 2024-04-11 22:40:05 +02:00
VinciGit00
77581876b5 add anti not scrapong task 2024-04-11 19:31:54 +02:00
VinciGit00
dc149e6f63 fixing bug on rag node 2024-04-11 19:26:38 +02:00
EURAC\marperini
74ec928f96 added execution info of the graph, time, price etc 2024-04-11 14:29:32 +02:00
VinciGit00
53e317d46b add hugginface integration 2024-04-11 12:43:06 +02:00
EURAC\marperini
982f142f20 fixed embedder_model config for local models 2024-04-10 11:00:52 +02:00
EURAC\marperini
8b915e3ec6 fixed issue ollama embeddings 2024-04-10 09:29:03 +02:00
ftoppi
26616d25b1
abstract_graph.py: allow user to set Ollama model_tokens in config 2024-04-09 21:27:15 +02:00
VinciGit00
78a1e7aec4 new model 2024-04-09 13:54:10 +02:00
Marco Vinciguerra
b4e640879d
Merge branch 'main' into refactoring_examples_folder 2024-04-09 11:46:34 +02:00
VinciGit00
992e7f862f add models avaiables 2024-04-09 10:33:45 +02:00
VinciGit00
cec4c43c64 add new local models context window 2024-04-09 10:23:38 +02:00
VinciGit00
0ffefdd170 refactoring if examples ffolder 2024-04-08 22:28:55 +02:00
EURAC\marperini
a601695fd5 updated nodes metadata 2024-04-08 15:23:23 +02:00
EURAC\marperini
dee1a42629 fixed token models, added mistral support 2024-04-08 15:21:06 +02:00
EURAC\marperini
17add20c13 implemented node_config, add embedder model choice, add azure endpoint, refactor graphs and exmaples 2024-04-08 12:18:32 +02:00
EURAC\marperini
6a83c2306d docker compose working ollama implementation 2024-04-07 11:08:08 +02:00
VinciGit00
6dc4c6dec4 Merge branch 'main' into research_branch 2024-04-06 14:44:56 +02:00
VinciGit00
a5fc71f34e fixed internet research 2024-04-06 14:39:19 +02:00
EURAC\marperini
57a53bd6af refactor examples 2024-04-06 13:22:42 +02:00
EURAC\marperini
c2709aa50c working search graph 2024-04-06 13:01:04 +02:00
VinciGit00
3269b4aaa7 fixing infos for the pr 2024-04-03 13:33:17 +02:00
VinciGit00
c9951340b5 fixed image_to_tex_node and refactoring 2024-04-03 12:56:25 +02:00
VinciGit00
4c3ea8f4ac fixed image_to_tex_node and refactoring 2024-04-03 12:53:12 +02:00
VinciGit00
b53bfefeea removed junk if 2024-04-02 12:57:27 +02:00
VinciGit00
db6d988b2d refactoring for pylint 2024-04-01 13:41:50 +02:00
VinciGit00
471df48a22 add google search 2024-04-01 13:36:02 +02:00
VinciGit00
5e38c2e298 add superclasse for graphs 2024-03-28 20:46:17 +01:00
VinciGit00
7d0fd74bfc add search graph 2024-03-28 16:54:01 +01:00
VinciGit00
b8a3590235 add SearcInetnetNode 2024-03-28 13:14:39 +01:00
VinciGit00
fff1f9ee51 new function for searching 2024-03-28 12:41:45 +01:00
VinciGit00
a28ca993de refactoring of examples folder 2024-03-27 21:41:21 +01:00
VinciGit00
672e9a5191 refactoring of research function 2024-03-27 21:29:10 +01:00
VinciGit00
97d0a6887f add reserch web function 2024-03-27 21:22:04 +01:00
VinciGit00
587dfa5c68 refactoring of the saving functions 2024-03-26 14:15:36 +01:00
VinciGit00
be36ab8707 add new type of scraping through text 2024-03-24 21:22:26 +01:00
VinciGit00
f448b8dfb4 removed unused return 2024-03-18 22:55:24 +01:00
VinciGit00
e25f74ddc8 fixed bug for scraping from node + add example 2024-03-18 22:53:52 +01:00
VinciGit00
1afb950751 refactoring of code, add example on the readme 2024-03-18 14:35:47 +01:00
Perinim
cae38a0d67 fixed bugs after merge 2024-03-18 11:30:14 +01:00
Perinim
b5bdd4e729 Merge remote-tracking branch 'origin/main' into refactoring_nodes 2024-03-18 11:07:42 +01:00
Perinim
7d3190a636 fixed example graphs and utils files 2024-03-18 10:20:23 +01:00
Perinim
b547972968 fixed custom_graph, graph_builder and trulens_evaluator 2024-03-18 09:12:59 +01:00
Perinim
52934bf007 implemented graph_config, fixed smart_scraper and speech graph 2024-03-17 20:35:04 +01:00
Perinim
f27e0b4c5f removed fetch_html and fetch_text (merged) 2024-03-17 17:01:29 +01:00
Perinim
875a7cc4f0 refactored RagNode and GenerateAnswerNode 2024-03-17 16:48:58 +01:00
Perinim
3585cd81e5 refactor fetch node 2024-03-17 12:02:35 +01:00
Perinim
20d7b69008 refactored BaseNode and ParseNode 2024-03-17 11:40:35 +01:00
Lorenzo Padoan
d0f4b0159f DEV gemini support for simple custom graph 
-Not supported yet smartscrapergrapn with gemini
 2024-03-14 19:35:04 +01:00
Perinim
538133dfff added parse input string function 2024-03-13 19:34:51 +01:00
VinciGit00
a849315182 add new tests and refactoring of code 2024-03-12 21:40:09 +01:00
Perinim
308c2d28c8 fixed text input example 2024-03-12 12:22:47 +01:00
Perinim
aeff434878 fixed docstring 2024-03-12 11:53:03 +01:00
Perinim
4d2e872484 fixed speech graph 2024-03-12 11:14:14 +01:00
Perinim
9d31db2d35 working smart scraper example 2024-03-12 11:06:08 +01:00
Lorenzo Padoan
03e6881f00
Merge pull request #37 from VinciGit00/refactor-fetch-node 
Refactor fetch node, add integration for preprocessing
 2024-03-08 17:51:19 +01:00
VinciGit00
cffd954446 add integration in the fetch for preprocessing 2024-03-08 17:46:36 +01:00
VinciGit00
ce8bc08b3a refactoring of remove function and implement it in the fetch node 2024-03-08 09:43:09 +01:00
VinciGit00
cd176ff76c removed prints 2024-03-05 18:07:21 +01:00
VinciGit00
6663c9c5bc removed the duplicates 2024-03-05 14:59:48 +01:00
VinciGit00
ff7f8105bb add multiple merging generate answer node 2024-03-05 14:50:31 +01:00
VinciGit00
ab71a6bc5a fixed documentation and fetch_text_node 2024-03-03 18:50:03 +01:00
VinciGit00
e29454e5df add doc for the text node 2024-03-03 12:32:26 +01:00
VinciGit00
a0c77491bf add new progresbbar and finished the node 2024-03-02 20:51:29 +01:00
VinciGit00
1250aeeb8e add the node for reading the html as a txt 2024-03-02 13:32:34 +01:00
VinciGit00
79387d1617 refactoring for pylint score 2024-02-29 10:49:25 +01:00
VinciGit00
17d1f6a47b fix pylint score 2024-02-29 10:32:00 +01:00
VinciGit00
f4d5941831 refctoring of RAG node 2024-02-29 10:29:39 +01:00
Marco Perini
5735f3ec7f
Merge branch 'main' into refactoring-parse-html 2024-02-28 17:57:28 +01:00
VinciGit00
89d193392d refactor of parse_html_node 2024-02-27 17:02:06 +01:00
VinciGit00
cd405e4202 removed llm req 2024-02-27 12:28:46 +01:00
VinciGit00
a7c7526bf1 refactoring of parse_html node 2024-02-26 20:14:53 +01:00
VinciGit00
582c99d879 refactoring of the classes of constructors 2024-02-25 12:35:54 +01:00
VinciGit00
043ab48466 add integration for the RAG for ParseHTMLNode 2024-02-25 11:24:44 +01:00
VinciGit00
fc0f0957e1 add integration for the RAG for ParseHTMLNode 2024-02-25 11:12:44 +01:00
VinciGit00
9d6fcf7882 refactoring 2024-02-25 10:45:13 +01:00
PeriniM
ed64db24df working speech graph with rag 2024-02-23 14:17:46 +01:00
PeriniM
cb8ed4fc5b fixed eval output 2024-02-23 10:33:25 +01:00
PeriniM
284ba5b75d added trulens 2024-02-23 09:39:25 +01:00
VinciGit00
c02203068e add api_key as parameter instead of .env file 2024-02-22 21:13:01 +01:00
Lorenzo Padoan
665a91484b dev trulens_evaluetor 2024-02-22 19:19:10 +01:00
VinciGit00
ae929393b9 refactoring for pylint score 2024-02-22 10:19:10 +01:00
Marco Perini
ed48f273f8
Merge branch 'main' into multi-chunks-implementation 2024-02-22 00:00:01 +01:00
Perinim
62cc9170c8 updated SmartScraperGraph with rag 2024-02-21 23:52:45 +01:00
Perinim
b0c75dc7ed created rag example 2024-02-21 19:15:38 +01:00
Perinim
a626401f8a merged multiple chunks answer 2024-02-21 18:43:38 +01:00
Perinim
a17971b7ed implemented parallelrunnables 2024-02-21 18:17:34 +01:00
Perinim
15b468473b working test rag 2024-02-21 17:02:10 +01:00
VinciGit00
906fea92e9 add examples + module documentation 2024-02-20 16:50:32 +01:00
VinciGit00
7390dba0ab removed dict info and put in another file 2024-02-19 13:50:30 +01:00
VinciGit00
8a566ff588 bugfix of the action 2024-02-19 12:21:41 +01:00
VinciGit00
71bd53d254 refactoring of the code, add tests and examples 2024-02-19 11:54:36 +01:00
PeriniM
4e3005db0a implemented summary graph 2024-02-19 07:18:28 +01:00
PeriniM
da16360b1f working text2speech and image2text 2024-02-19 06:26:50 +01:00
PeriniM
3ed5fd5e48 added openai wrapper and examples 2024-02-19 04:45:43 +01:00
PeriniM
20db549d08 implemented graphbuilder 2024-02-19 04:11:35 +01:00
PeriniM
49c22ae155 implemented graphbuilder class 2024-02-19 03:39:50 +01:00
PeriniM
20398a3059 dev: refactor dict on graph state 2024-02-18 23:36:16 +01:00
VinciGit00
dc442b9ebd refactoring of fetch node 2024-02-18 21:41:42 +01:00
VinciGit00
f8bb32919b refactoring of the example and add 2 new nodes 2024-02-18 20:36:45 +01:00
VinciGit00
e57ec7002a add new node 2024-02-17 21:42:43 +01:00
VinciGit00
4e509e1f66 removed unused files 2024-02-16 19:43:48 +01:00
VinciGit00
63ecd3dd70 refactoring of the folder 2024-02-16 19:29:38 +01:00
VinciGit00
f225957ea9 fixed imports 2024-02-15 09:33:33 +01:00
VinciGit00
ce7949de1b refactoring of the folders 2024-02-15 09:27:39 +01:00
VinciGit00
2a234f2e4d refactoring of the code changing the name 2024-02-14 20:35:30 +01:00