CodeCow/Scrapegraph-ai
1
0
Fork 0
mirror of https://github.com/VinciGit00/Scrapegraph-ai.git synced 2026-07-12 21:01:56 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
2,293 Commits 2 Branches 448 Tags 17 MiB
5faac9fb46
Commit Graph

408 Commits

Author SHA1 Message Date
PeriniM
654a042396 feat(burr-node): working burr bridge 2024-05-22 00:24:38 +02:00
Marco Perini
6cbd84f254 feat(burr-bridge): BurrBridge class to integrate inside BaseGraph 2024-05-21 19:57:10 +02:00
Marco Vinciguerra
be4237a04d
Merge branch 'pre/beta' into multi_scraper_graph 2024-05-21 13:44:06 +02:00
Marco Perini
fc58e2d3a6 feat(smart-scraper-multi): add schema to graphs and created SmartScraperMultiGraph 2024-05-21 13:13:27 +02:00
Marco Vinciguerra
1ef63217dd add deepseek embeddings 2024-05-21 10:41:01 +02:00
Marco Vinciguerra
659fad770a fix: add deepseek embeddings 2024-05-21 10:36:40 +02:00
JGalego
d0a301d3c3 Moved common params up (verbose, headless and loader_kwargs) 2024-05-20 17:25:13 +01:00
Marco Perini
bed3eed50c feat(multiple_search): working multiple example 2024-05-18 01:51:12 +02:00
Marco Vinciguerra
05e511e36f add new prompts 2024-05-18 00:22:52 +02:00
Marco Perini
0196423bde feat(knowledgegraph): add knowledge graph node 2024-05-17 23:41:44 +02:00
Marco Perini
73fa31db0f feat(base_graph): alligned with main 2024-05-17 18:54:27 +02:00
Marco Vinciguerra
3453f72397 add graph 2024-05-17 18:23:50 +02:00
Marco Vinciguerra
02745a4f63 Merge branch 'main' into pre/beta 2024-05-17 10:40:19 +02:00
Marco Perini
d4d913c8a3 fix(deepcopy): switch whether we have obj in the config 2024-05-17 10:36:39 +02:00
VinciGit00
9e9f8f09d5 removed max depth 2024-05-16 12:39:08 +02:00
VinciGit00
9483afddd3 revert 2024-05-16 12:34:02 +02:00
Mayur Bhosale
1e0b2f7e70
Merge branch 'pre/beta' into nDeep 2024-05-15 23:36:23 +02:00
mayurdb
d60438cc89 Add a n-level deep search support 2024-05-15 23:32:15 +02:00
VinciGit00
ba8a4f7122 removed duplicates 2024-05-15 21:47:39 +02:00
VinciGit00
4fe58d9ba4 fix logger 2024-05-15 16:12:13 +02:00
VinciGit00
40260d8d25 remove asdt 2024-05-15 16:00:47 +02:00
Marco Perini
999c930f42 fix(deepcopy): reaplced to shallow copy 2024-05-15 16:00:42 +02:00
VinciGit00
cc5adefd29 fix: come back to the old version 2024-05-15 15:54:00 +02:00
VinciGit00
5587a64d23 fix: removed unused 2024-05-15 15:51:27 +02:00
VinciGit00
29d284e497 Merge branch 'main' into logger-integration 2024-05-15 15:28:20 +02:00
Marco Vinciguerra
008f8d9eae
Merge pull request #248 from VinciGit00/main 
reallignment
 2024-05-15 15:20:53 +02:00
Marco Vinciguerra
0b5cdd48fd
Merge pull request #246 from VinciGit00/main 
reallignment
 2024-05-15 15:19:05 +02:00
Marco Vinciguerra
22cd9e3605
Merge branch 'search_link_context' into main 2024-05-15 15:16:57 +02:00
Marco Perini
096b665c01 fix(searchgraph): used shallow copy to serialize obj 2024-05-15 15:05:47 +02:00
mayurdb
0b71b9a1e8 Add a new graph traversal that allows more than one edges out of a graph 2024-05-15 13:46:19 +02:00
VinciGit00
05890835f5 refactoring of loggers 2024-05-15 10:54:53 +02:00
Marco Perini
a6e1813ddd fix(fetch_node): bug in handling local files 2024-05-14 16:51:10 +02:00
Marco Perini
fcb3abb01d feat(omni-search): added omni search graph and updated docs 2024-05-14 16:26:43 +02:00
Marco Perini
a296927624 feat(omni-scraper): working OmniScraperGraph with images 2024-05-14 13:46:49 +02:00
VinciGit00
d6f5ca874f Merge branch 'main' into pre/beta 2024-05-14 10:24:29 +02:00
JGalego
859c5d5ba5 Refactored to include custom AWS client for bedrock; Added missing Anthropic class 2024-05-13 17:57:53 +01:00
Marco Vinciguerra
f3d44c09ae
Merge pull request #228 from JGalego/fix/bedrock-support 
fix: Amazon Bedrock Support
 2024-05-13 14:57:23 +02:00
VinciGit00
d0167dee71 fix: bug for claude 2024-05-13 13:43:42 +02:00
JGalego
39be38f292 Fixed anthropic/bedrock conflict; Removed duplicate class Claude; Updated BedrockEmbeddings class 2024-05-13 12:35:42 +01:00
Marco Perini
7e8acd8e6a
Merge branch 'pre/beta' into fix/fetch-node-proxybroker 2024-05-13 11:17:37 +02:00
Marco Perini
1e9a564616 fix(proxy-rotation): removed duplicated arg and passed the loader_kwarhs correctly to the node 2024-05-12 18:39:03 +02:00
Federico Aguzzi
156b67b91e feat: add support for deepseek-chat 
closes #222
 2024-05-12 01:00:11 +02:00
mayurdb
8f1fbe7e10 minor changes 2024-05-11 16:52:36 +05:30
mayurdb
df271b6451 Add search link node that can find out relevant links in the webpage 2024-05-11 16:39:55 +05:30
Stefan Krawczyk
f2bb1cc4f8 Fixes LC document deserialization 
Depends on https://github.com/DAGWorks-Inc/burr/pull/175.
 2024-05-11 00:06:02 -07:00
Eric Page
0683e78e78
Merge branch 'pre/beta' into fix-GenerateScraperGraph 2024-05-11 01:59:28 +02:00
Eric Page
24c3b0580c Removed nonfunctional RAG node from ScriptCreatorGraph 2024-05-11 01:38:40 +02:00
Stefan Krawczyk
82afa0ef3d Working smart scraper graph 
Bugs found in tracker persistence:
 - serializing inputs properly for tracker
 - deserializing state from a previous found
 2024-05-10 10:49:53 -07:00
Stefan Krawczyk
d94195f823 WIP 2024-05-10 09:48:46 -07:00
Stefan Krawczyk
be16fecb55 WIP 2024-05-10 09:48:44 -07:00
VinciGit00
0ab31c3fdb fix: add json integration 2024-05-09 21:07:07 +02:00
VinciGit00
324e977b85 fix: fixed bugs for csv and xml 2024-05-09 20:46:46 +02:00
Shubham Kamboj
f10f3b1438 feat: Add support for passing pdf path as source 2024-05-09 21:55:05 +05:30
VinciGit00
94156755d1 Update abstract_graph.py 2024-05-09 12:29:39 +02:00
VinciGit00
403979337c Update abstract_graph.py 2024-05-09 10:46:31 +02:00
VinciGit00
0bb68d1500 Merge branch 'main' of https://github.com/VinciGit00/Scrapegraph-ai 2024-05-09 10:31:21 +02:00
VinciGit00
8272d736a6 add tokenizatio for mxbai-embed-large 2024-05-08 21:50:42 +02:00
VinciGit00
4ed0fb89c3 feat: update info 2024-05-08 21:25:03 +02:00
VinciGit00
e7d39a5daf fixed gemini embeddings 2024-05-08 21:24:17 +02:00
Marco Perini
6b71ec1d2b fix(examples): local, mixed models and fixed SearchGraph embeddings problem 2024-05-08 15:36:26 +02:00
VinciGit00
2258fe5ac0 add new search graph examples 2024-05-06 22:31:48 +02:00
VinciGit00
67d5fbf816 feat: new search_graph 2024-05-06 22:09:18 +02:00
VinciGit00
5a67bca0db Merge branch 'pre/beta' into pr/161 2024-05-06 14:50:04 +02:00
Cem Uzunoglu
2ac9e16dd9 Fixed accidental reformatting. 2024-05-06 15:09:56 +03:00
Cem Uzunoglu
e264e92e72 Added support for Claude 3 models from Anthropic 2024-05-06 14:34:35 +03:00
Shubham Kamboj
726de28898 feat: Fix bug for gemini case when embeddings config not passed 2024-05-06 15:40:03 +05:30
VinciGit00
51aa109e42 feat: add turboscraper (alfa) 2024-05-06 11:59:14 +02:00
VinciGit00
80053a2358 Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta 2024-05-06 10:07:56 +02:00
VinciGit00
389b52aabb removed examples 2024-05-06 10:07:31 +02:00
Marco Perini
930adb38f2 feat(node): multiple url search in SearchGraph + fixes 
Implemented GraphIteratorNode and MergeAnswersNode to create multiple istances of a graph and merge the scraped content from multiple pages
 2024-05-06 00:30:09 +02:00
Marco Perini
1c4ba91620 exposed abstract_graph allowing the user to create new graphs 2024-05-05 22:05:11 +02:00
Marco Perini
84fcb44aaa feat: fixed custom_graphs example and robots_node 2024-05-05 22:02:24 +02:00
Marco Perini
36a1522bba
Merge pull request #153 from VinciGit00/google_embeddings 
feat: add gemini embeddings
 2024-05-05 17:54:07 +02:00
Eric Page
cc27b21c3f
Merge branch 'pre/beta' into pass-common-params-graph 2024-05-05 16:18:32 +02:00
Eric Page
3ae2ea1dbd Miscellaneous "llm" -> "llm_model" refactors 2024-05-05 15:58:50 +02:00
Eric Page
a53e95cbf0 Corrected graphs to use common params 2024-05-05 14:36:16 +02:00
Eric Page
8d0e109a70 Added overwrite keyword to set_common_params` 2024-05-05 14:32:58 +02:00
VinciGit00
79daa4c112 feat: add gemini embeddings 2024-05-05 11:40:44 +02:00
Eric Page
4dc6049e76 Simplified create graph functions using common params 2024-05-05 10:40:04 +02:00
Eric Page
444a13a6a6 Created set_common_params function 2024-05-05 10:15:37 +02:00
Eric Page
2178485b3d Adjusted graphs to reflect node_config change 2024-05-05 09:47:13 +02:00
Marco Perini
0ab7272fd7
Merge branch 'pre/beta' into 133-support-claude3-haiku-and-others-using-litellm 2024-05-05 02:15:15 +02:00
VinciGit00
2878695d5f fix: trailing whitespace 2024-05-04 21:26:19 +02:00
Shubham Kamboj
fd59f282a8 Bug fixes 2024-05-04 22:18:54 +05:30
Shubham Kamboj
d830d1371b Resolve merge conflicts 2024-05-04 22:01:40 +05:30
Shubham Kamboj
7599234ab9 feat: Enable end users to pass model instances of HuggingFaceHub 2024-05-04 21:42:57 +05:30
VinciGit00
10a94530e3 feat: add pdf scraper 2024-05-03 21:42:02 +02:00
VinciGit00
39f64e5d48 add claude model 2024-05-03 17:17:31 +02:00
S4mpl3r
819cbcd3be refactor: Changed the way embedding model is created in AbstractGraph class and removed handling of embedding model creation from RAGNode. Now AbstractGraph will call a dedicated method for embedding models instead of _create_llm. This makes it easy to use any LLM with any supported embedding model. 2024-05-03 16:14:27 +03:30
Marco Vinciguerra
579a27faf9
Merge pull request #126 from VinciGit00/bedrock_support 
feat: bedrock support
 2024-05-02 17:57:20 +02:00
Marco Vinciguerra
a2d5c75f10
Merge branch 'pre/beta' into main 2024-05-02 17:55:21 +02:00
Shubham Kamboj
b86aac2188 feat: Allow end users to pass model instances for llm and embedding model 2024-05-02 20:09:23 +05:30
VinciGit00
16de49fa08 add integration for bedrock 
Co-Authored-By: redrusty2 <15157208+redrusty2@users.noreply.github.com>
 2024-05-02 13:47:17 +02:00
VinciGit00
4a3bc37f2f fix: bug on script_creator_graph.py 2024-05-02 09:23:46 +02:00
VinciGit00
5cf4e4f92f fix: examples and graphs 2024-05-02 09:20:46 +02:00
Marco Perini
29427c2334 docs: fixed basegraph docstring 2024-05-02 01:05:20 +02:00
Marco Perini
c11fc28896 docs: refactor examples 2024-05-02 00:44:19 +02:00
Marco Perini
0631985e61 docs: graphs and helpers docstrings 2024-05-02 00:23:38 +02:00
Marco Perini
02d1af006c feat: modified node name 2024-05-01 21:33:25 +02:00
VinciGit00
276ac8b40c changed name 2024-05-01 20:17:01 +02:00
VinciGit00
7c1b5796d7 add csv scraper 2024-05-01 17:20:04 +02:00
EURAC\marperini
2dd7817cfb feat: added verbose flag to suppress print statements 2024-04-30 15:31:57 +02:00
Marco Perini
d592d27bb4
Merge pull request #115 from VinciGit00/101-scrape-json-files 
feat: add xml scraper and json scraper
 2024-04-30 14:29:57 +02:00
EURAC\marperini
8fba7e5490 feat(refactor): changed variable names 2024-04-30 14:28:30 +02:00
VinciGit00
e3d0194dc9 fix: script generator and add new benchmarks 2024-04-30 11:51:04 +02:00
VinciGit00
da2c82a2a2 add json and xml scraper 2024-04-30 10:52:22 +02:00
EURAC\marperini
42ab0aa1d2 feat(fetch): added playwright support 2024-04-30 04:02:58 +02:00
EURAC\marperini
dbbf10fc77 feat(llm): implemented groq model 2024-04-30 02:32:14 +02:00
VinciGit00
deb920a33e fixing json and example 2024-04-29 16:11:57 +02:00
VinciGit00
3eacc6fbc3 add paths 2024-04-29 16:07:11 +02:00
VinciGit00
674e64222e add first new graphs 2024-04-29 15:55:21 +02:00
VinciGit00
ab72db3b50 if the conf is not present preset to 8192 the context window 2024-04-29 10:26:34 +02:00
VinciGit00
a9b11e433a fix: bug for calculate costs 2024-04-28 14:36:49 +02:00
Marco Vinciguerra
44bc9196ef
Merge pull request #84 from VinciGit00/79-add-getstate-function-to-abstract-graph 
complete get_state function
 2024-04-26 21:44:33 +02:00
VinciGit00
cd8d3e7a4f refactoring of the graphs 2024-04-26 11:08:56 +02:00
VinciGit00
b2ebabd32f Update base_graph.py 2024-04-25 19:45:49 +02:00
VinciGit00
e714a59c2e refactoring of engine 2024-04-25 19:22:12 +02:00
VinciGit00
75255bb4af complete get_state function 2024-04-25 15:08:42 +02:00
VinciGit00
a73496df9b refactoring of nodes 2024-04-24 21:13:27 +02:00
VinciGit00
e68b707e0d changed color and return type 2024-04-24 17:29:49 +02:00
VinciGit00
3f95801737 Merge branch 'main' of https://github.com/VinciGit00/Scrapegraph-ai 2024-04-19 10:36:40 +02:00
EURAC\marperini
b2d170cd86 refactored and fixed single chunk bug 2024-04-17 11:50:19 +02:00
VinciGit00
876223da56 add generateScraperBranch 
Co-Authored-By: Matteo Mei <52063123+FattiMei@users.noreply.github.com>
 2024-04-14 13:18:50 +02:00
Marco Perini
dde9eeec87
Merge pull request #60 from VinciGit00/huggingface_integration 
add hugginface integration (embeddings, models ...)
 2024-04-11 22:40:05 +02:00
EURAC\marperini
74ec928f96 added execution info of the graph, time, price etc 2024-04-11 14:29:32 +02:00
VinciGit00
53e317d46b add hugginface integration 2024-04-11 12:43:06 +02:00
EURAC\marperini
982f142f20 fixed embedder_model config for local models 2024-04-10 11:00:52 +02:00
ftoppi
26616d25b1
abstract_graph.py: allow user to set Ollama model_tokens in config 2024-04-09 21:27:15 +02:00
Marco Vinciguerra
b4e640879d
Merge branch 'main' into refactoring_examples_folder 2024-04-09 11:46:34 +02:00
VinciGit00
992e7f862f add models avaiables 2024-04-09 10:33:45 +02:00
VinciGit00
cec4c43c64 add new local models context window 2024-04-09 10:23:38 +02:00
VinciGit00
0ffefdd170 refactoring if examples ffolder 2024-04-08 22:28:55 +02:00
EURAC\marperini
dee1a42629 fixed token models, added mistral support 2024-04-08 15:21:06 +02:00
EURAC\marperini
17add20c13 implemented node_config, add embedder model choice, add azure endpoint, refactor graphs and exmaples 2024-04-08 12:18:32 +02:00
EURAC\marperini
6a83c2306d docker compose working ollama implementation 2024-04-07 11:08:08 +02:00
VinciGit00
6dc4c6dec4 Merge branch 'main' into research_branch 2024-04-06 14:44:56 +02:00
EURAC\marperini
c2709aa50c working search graph 2024-04-06 13:01:04 +02:00
VinciGit00
c9951340b5 fixed image_to_tex_node and refactoring 2024-04-03 12:56:25 +02:00
VinciGit00
5e38c2e298 add superclasse for graphs 2024-03-28 20:46:17 +01:00
VinciGit00
7d0fd74bfc add search graph 2024-03-28 16:54:01 +01:00
VinciGit00
be36ab8707 add new type of scraping through text 2024-03-24 21:22:26 +01:00
VinciGit00
1afb950751 refactoring of code, add example on the readme 2024-03-18 14:35:47 +01:00
Perinim
cae38a0d67 fixed bugs after merge 2024-03-18 11:30:14 +01:00
Perinim
7d3190a636 fixed example graphs and utils files 2024-03-18 10:20:23 +01:00
Perinim
52934bf007 implemented graph_config, fixed smart_scraper and speech graph 2024-03-17 20:35:04 +01:00
Perinim
4d2e872484 fixed speech graph 2024-03-12 11:14:14 +01:00
Perinim
9d31db2d35 working smart scraper example 2024-03-12 11:06:08 +01:00
VinciGit00
17d1f6a47b fix pylint score 2024-02-29 10:32:00 +01:00
PeriniM
ed64db24df working speech graph with rag 2024-02-23 14:17:46 +01:00
Marco Perini
ed48f273f8
Merge branch 'main' into multi-chunks-implementation 2024-02-22 00:00:01 +01:00
Perinim
62cc9170c8 updated SmartScraperGraph with rag 2024-02-21 23:52:45 +01:00
VinciGit00
906fea92e9 add examples + module documentation 2024-02-20 16:50:32 +01:00
VinciGit00
71bd53d254 refactoring of the code, add tests and examples 2024-02-19 11:54:36 +01:00
PeriniM
4e3005db0a implemented summary graph 2024-02-19 07:18:28 +01:00
PeriniM
3ed5fd5e48 added openai wrapper and examples 2024-02-19 04:45:43 +01:00
PeriniM
49c22ae155 implemented graphbuilder class 2024-02-19 03:39:50 +01:00
PeriniM
20398a3059 dev: refactor dict on graph state 2024-02-18 23:36:16 +01:00
VinciGit00
2a234f2e4d refactoring of the code changing the name 2024-02-14 20:35:30 +01:00