Marco Vinciguerra
19af78c7d8
Merge branch 'pre/beta' into main
2024-07-20 18:14:36 +02:00
DragonelRoland
4b8360658e
Update models_tokens.py
...
added gpt4o mini to the models
2024-07-20 16:51:24 +02:00
DragonelRoland
e9ecf5f0dc
Update typo in conditional_node.py
...
"thar" -> "that"
2024-07-20 15:06:52 +02:00
Marco Vinciguerra
3cf7089e3f
Merge branch 'pre/beta' into temp
2024-07-20 13:27:37 +02:00
Marco Vinciguerra
77777c898d
fix: add gpt o mini for azure
2024-07-20 13:20:16 +02:00
Marco Vinciguerra
03f4a3aa29
fix: azure models
2024-07-20 13:19:21 +02:00
Marco Vinciguerra
3a9d556e1b
Merge branch 'pre/beta' into add-openai-supported-model-gpt-4o-mini
2024-07-19 10:08:41 +02:00
Kernick
4f079cfc74
Fixed a typo in the model name
2024-07-18 21:06:16 +03:00
Kernick
b91020ce56
Update models_tokens.py with gpt-4o-mini
...
OpenAI added a new available model - gpt-4o-mini. Adding this model to the supported models, so it would be possible to use it with the Scrapegraph-ai.
2024-07-18 21:03:55 +03:00
Federico Aguzzi
dfa759e834
style(models): fix module docstrings
2024-07-18 11:02:51 +02:00
Marco Vinciguerra
032a491605
Update parse_node.py
/ build (push) Has been cancelled
2024-07-17 23:06:05 +02:00
Marco Vinciguerra
07f1e23d23
fix: parse_node
2024-07-17 22:58:21 +02:00
Marco Vinciguerra
68f58cc4dd
refactoring of generate answer node
2024-07-17 22:41:49 +02:00
Marco Vinciguerra
602dd00209
feat: refactoring_to_md function
2024-07-16 12:39:48 +02:00
Federico Aguzzi
dccb893fdf
Merge pull request #456 from ScrapeGraphAI/refactoring-of-search_link_node
...
fix: search link node
2024-07-15 23:06:03 +02:00
Marco Vinciguerra
830daee1f3
Update search_link_node.py
/ build (push) Has been cancelled
2024-07-15 20:47:09 +02:00
Marco Vinciguerra
57fdaf9e3a
create search_link_graph
/ build (push) Waiting to run
2024-07-15 12:36:11 +02:00
Federico Aguzzi
789c16fb76
Merge pull request #458 from ScrapeGraphAI/searchngx_integration
...
merge: update research_web.py
2024-07-14 22:55:52 +02:00
Federico Aguzzi
7ba2f6ae0b
chore: correct search engine name
/ build (push) Has been cancelled
2024-07-14 22:39:34 +02:00
Federico Aguzzi
fd1b7cb24a
chore: remove unused import
/ build (push) Has been cancelled
2024-07-14 22:36:43 +02:00
Marco Vinciguerra
7e5789baa3
Update research_web.py
2024-07-14 19:31:22 +02:00
Marco Vinciguerra
cf3ab5564a
fix: search link node
2024-07-14 16:49:29 +02:00
Marco Vinciguerra
a7249685cb
removed rag node
2024-07-14 10:24:38 +02:00
salamanderxing
4b65392cca
Resolves #447
2024-07-12 10:56:25 +02:00
Marco Vinciguerra
765b548c0e
Merge pull request #417 from ScrapeGraphAI/md_scraper_integration
...
feat: add integrations for markdown files
2024-07-04 19:42:53 +02:00
Marco Vinciguerra
8a52914871
Merge pull request #429 from ScrapeGraphAI/421-default-prompt-template-customization
...
421 default prompt template customization
2024-07-04 19:42:35 +02:00
Federico Aguzzi
104d8692d6
Merge branch 'pre/beta' into support
/ build (push) Has been cancelled
2024-07-04 18:36:34 +02:00
Federico Aguzzi
1bbd25a63f
Merge pull request #407 from ScrapeGraphAI/404-split-unit-testing-from-src
...
Merge #404 (split unit testing from src)
2024-07-04 18:33:45 +02:00
Federico Aguzzi
d77a622b05
Merge branch '423-add-vertex-ai-integration' into support
2024-07-04 18:23:22 +02:00
Federico Aguzzi
720f18729b
Merge branch 'fireworks_integration' into support
2024-07-04 18:17:38 +02:00
Federico Aguzzi
2ab7ddc95a
Merge pull request #405 from ScrapeGraphAI/integration_markdown
...
Integration markdown
2024-07-04 18:13:12 +02:00
Marco Vinciguerra
3ee1743687
update prompts
/ build (3.10) (push) Has been cancelled
2024-07-04 14:22:45 +02:00
Marco Vinciguerra
ed2af51501
update the chunk size
/ build (3.10) (push) Waiting to run
2024-07-02 12:03:08 +02:00
Marco Vinciguerra
3bf5f570a8
feat: add integration for infos
2024-07-01 21:19:16 +02:00
AmosDinh
3f9bf5b4d4
fixed typo
2024-07-01 13:42:10 +00:00
Marco Vinciguerra
119514bdfc
feat: add vertexai integration
2024-07-01 12:21:47 +02:00
Marco Vinciguerra
5fe694b6b4
feat: improve md prompt recognition
/ build (3.10) (push) Waiting to run
2024-06-30 18:10:00 +02:00
Marco Vinciguerra
30ca15ca28
Merge branch 'md_scraper_integration' into integration_markdown
/ build (3.10) (push) Has been cancelled
2024-06-30 16:58:37 +02:00
Marco Vinciguerra
b6a78e926f
Merge branch 'main' into temp
2024-06-30 16:54:15 +02:00
Marco Vinciguerra
2804434a9e
feat: add integrations for markdown files
/ build (3.10) (push) Waiting to run
2024-06-29 13:35:39 +02:00
Marco Vinciguerra
9b45ebcdcf
modify fetch node with no cut mode
2024-06-28 14:38:36 +02:00
Marco Vinciguerra
f3cbbcee92
Merge pull request #412 from oviniciusfeitosa/patch-1
...
/ build (3.10) (push) Has been cancelled
Release / Build (push) Has been cancelled
Release / Release (push) Has been cancelled
🐛 Rename user_prompt parameter to prompt
2024-06-28 11:09:25 +02:00
Marco Vinciguerra
5d2e592a67
add new claude models
CodeQL / Analyze (python) (push) Has been cancelled
/ build (3.10) (push) Has been cancelled
Release / Build (push) Has been cancelled
Release / Release (push) Has been cancelled
2024-06-28 10:53:40 +02:00
Marco Vinciguerra
4d936410cc
fix: add new claude model
2024-06-28 10:50:01 +02:00
Vinícius Feitosa da Silva
a3c43c9a99
🐛 Rename user_prompt parameter to prompt
...
This adjustment makes the class consistent with the definition of the arguments.
2024-06-27 17:53:20 -03:00
Marco Vinciguerra
228a1de2be
add new force
2024-06-27 18:57:27 +02:00
Marco Vinciguerra
ff4ccb94a1
add claude 3.5 integration
CodeQL / Analyze (python) (push) Has been cancelled
/ build (3.10) (push) Has been cancelled
Release / Build (push) Has been cancelled
Release / Release (push) Has been cancelled
2024-06-25 11:21:06 +02:00
Marco Vinciguerra
0dfa5bf07e
Merge branch 'pre/beta' into main
2024-06-25 10:47:15 +02:00
Maorsg
f75e0835fc
added a function to the search_graph class to allow user to return URLs considered in the search
2024-06-24 21:31:28 -04:00
Marco Vinciguerra
df0e310829
feat: add fireworks integration
2024-06-24 23:11:28 +02:00
Marco Vinciguerra
3a537eec6f
fix: add test
2024-06-23 17:54:09 +02:00
Djamel Feddad
2b4f4f12be
doc: Fixed a small typo in the warning message
...
This PR fixes a small type in the warning message
2024-06-23 15:34:48 +08:00
Marco Vinciguerra
9917972c11
fixed request
/ build (3.10) (push) Waiting to run
2024-06-22 21:39:37 +02:00
Marco Vinciguerra
afd46ac77b
fixed generate_answer_node
/ build (3.10) (push) Waiting to run
2024-06-22 11:31:54 +02:00
Marco Vinciguerra
d1c3de777f
fixed a bug
2024-06-21 14:14:43 +02:00
Marco Vinciguerra
599512d2e5
fix: add new model for claude
2024-06-21 14:08:10 +02:00
Marco Vinciguerra
7af411aa99
add trigger
...
Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>
2024-06-21 13:36:27 +02:00
Marco Vinciguerra
5d6123847e
add new convert function
...
/ build (3.10) (push) Waiting to run
Co-Authored-By: Federico Minutoli <40361744+DiTo97@users.noreply.github.com>
2024-06-20 21:15:16 +02:00
Marco Vinciguerra
2f02830c81
refactoring of fetch node
/ build (3.10) (push) Waiting to run
2024-06-20 13:44:42 +02:00
Marco Vinciguerra
5664eb292b
Update generate_answer_node_prompts.py
2024-06-20 11:57:11 +02:00
Marco Vinciguerra
23bc6332d0
fixed a bug
2024-06-19 21:46:31 +02:00
Marco Vinciguerra
8bb560a489
add convert function
2024-06-19 20:17:45 +02:00
Marco Vinciguerra
88f9def69d
fix: reduced model tokens
2024-06-19 19:55:13 +02:00
Marco Vinciguerra
79a2f51c34
add new models tokens
2024-06-19 10:42:41 +02:00
Marco Vinciguerra
40b4b2d939
add new models tokens
2024-06-19 10:42:00 +02:00
Marco Perini
0cb46f5f30
Merge pull request #392 from VinciGit00/391-switch-between-search-engines
...
/ build (3.10) (push) Has been cancelled
Release / Build (push) Has been cancelled
Release / Release (push) Has been cancelled
feat: add new search engine avaiability and new tests
2024-06-18 23:46:28 +02:00
Marco Perini
c7870905e1
fix: total tokens and docs
2024-06-18 23:22:23 +02:00
Marco Vinciguerra
aa2160c108
feat: add research with bing + test function
/ build (3.10) (push) Has been cancelled
2024-06-18 21:28:29 +02:00
Jason Vertrees
aedda44868
fix: updated for schema changes
...
docs: updated for schema changes
2024-06-18 13:31:10 -05:00
Marco Vinciguerra
073d226723
feat: add new search engine avaiability and new tests
2024-06-18 14:35:13 +02:00
Marco Vinciguerra
1d0cbbc6d6
fix: add new embedding models
2024-06-18 09:13:15 +02:00
Marco Vinciguerra
48ff87f20a
Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta
2024-06-17 14:17:46 +02:00
Marco Vinciguerra
03ffebc52d
fix: add chinese embedding model
2024-06-17 14:17:44 +02:00
Marco Perini
080a318ff6
feat(telemetry): add telemetry module
2024-06-17 13:00:33 +02:00
Marco Vinciguerra
da93162b53
Merge branch 'pre/beta' into main
2024-06-17 11:02:56 +02:00
shubihu
d8d5cd267a
Update abstract_graph.py
...
fix: Incorrect API Key Error with OpenAI Proxy
2024-06-17 16:26:33 +08:00
Marco Perini
4c8becc721
overwrite common params to affect nodes config
/ build (3.10) (push) Has been cancelled
Release / Build (push) Has been cancelled
Release / Release (push) Has been cancelled
2024-06-16 15:19:40 +02:00
Marco Vinciguerra
2419003999
fix: fix robot node
/ build (3.10) (push) Waiting to run
Release / Build (push) Waiting to run
Release / Release (push) Blocked by required conditions
2024-06-16 14:04:36 +02:00
Marco Vinciguerra
3d34ea65df
Merge pull request #384 from liaoliaojun/fix/merge
...
fix: TypeError ‘Model provided by the configuration not supported’
2024-06-16 13:34:43 +02:00
Marco Vinciguerra
dcd216e345
fix: removed duplicate from ollama dictionary
...
Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>
2024-06-15 19:51:20 +02:00
liaoliaojun
62b372b675
fix: shallow copy config of create_embedder
2024-06-15 17:53:13 +08:00
Marco Perini
09cb6e964e
refactor: add missing schemas and renamed files
/ build (3.10) (push) Waiting to run
Release / Build (push) Waiting to run
Release / Release (push) Blocked by required conditions
2024-06-14 15:38:46 +02:00
Marco Perini
12f4386552
Merge branch 'pre/beta' into 349-problem-with-scrapegraphaigraphspdf_scraper_graphpy
2024-06-14 15:22:43 +02:00
Marco Perini
203de83405
fix(pdf): correctly read .pdf files
2024-06-14 15:20:30 +02:00
Marco Perini
91c5b5af43
fix(multi): updated multi pdf scraper with schema
2024-06-14 14:59:12 +02:00
Marco Vinciguerra
7a34562d50
refactoring of merging answers nodes
2024-06-13 21:41:54 +02:00
Marco Perini
283b61fafc
docs: better logging
2024-06-13 18:13:47 +02:00
Marco Vinciguerra
a6757aca57
Merge pull request #380 from Kshitij-Jande/main
...
Strip out the scheme from the proxy server address URI.
2024-06-13 13:23:26 +02:00
Kshitij Jande
2a9ab69f4b
Strip out the scheme from the server address URI
2024-06-13 12:18:41 +05:30
Marco Vinciguerra
1705046cc7
Update pdf_scraper_graph.py
2024-06-12 18:00:12 +02:00
Marco Vinciguerra
dc1340e302
Update generate_answer_pdf_node.py
2024-06-12 15:47:04 +02:00
Marco Vinciguerra
e45f159a31
enhanced performance and readibility
2024-06-12 14:59:10 +02:00
Marco Vinciguerra
58a257f05b
update model tokens
/ build (3.10) (push) Waiting to run
Release / Build (push) Waiting to run
Release / Release (push) Blocked by required conditions
2024-06-12 12:41:58 +02:00
Marco Vinciguerra
e6c7940a57
feat: add Parse_Node
2024-06-12 12:29:14 +02:00
Marco Vinciguerra
5065aa00e1
Merge branch 'pre/beta' into supercoder-327
2024-06-12 12:12:15 +02:00
supercoder-dev
d0e300af72
Update fetch_node.py
2024-06-12 14:32:01 +05:30
supercoder-dev
879c94a2b5
Update cleanup_html.py
2024-06-12 14:30:02 +05:30
supercoder-dev
828bdeedce
Update smart_scraper_graph.py
2024-06-12 14:28:33 +05:30
Marco Perini
6b4cdf92b8
fix: common params
2024-06-12 03:06:21 +02:00
Marco Perini
ab00f23d85
fix(node): fixed generate answer node pydantic schema
2024-06-12 01:40:49 +02:00
Marco Perini
a10b060409
Merge pull request #361 from VinciGit00/multi_scraper_implementation
...
Multi scraper implementation
2024-06-12 00:49:18 +02:00
Marco Perini
5d692bff9e
feat(schema): merge scripts to follow pydantic schema
2024-06-12 00:48:08 +02:00
Marco Perini
893aaddc5c
Merge pull request #359 from VinciGit00/semchunk_integration
...
feat: add new chunking function
2024-06-11 23:16:00 +02:00
Marco Perini
589da1d695
Merge pull request #351 from VinciGit00/faiss_integration
...
Faiss integration
2024-06-11 23:15:23 +02:00
Marco Perini
c881f64209
fix(cache): correctly pass the node arguments and logging
2024-06-11 22:56:09 +02:00
Marco Vinciguerra
fa951b4c8b
Merge pull request #368 from stevenmichaelthomas/wait-for-network-idle
...
Add the ability to specify load state
2024-06-11 19:59:23 +02:00
Steven Thomas
8f405ff87a
Add the ability to specify load state
2024-06-11 11:22:39 -04:00
Marco Vinciguerra
81f89d88be
Update search_link_node.py
2024-06-10 16:14:24 +02:00
Marco Vinciguerra
9326637cc3
Merge branch 'main' into pre/beta
2024-06-09 17:04:11 +02:00
Marco Vinciguerra
1981230e6f
add multi scraper integration
2024-06-08 12:13:18 +02:00
Marco Perini
8228225dff
Merge pull request #360 from VinciGit00/burr_integration
...
Burr integration
2024-06-08 12:10:34 +02:00
Marco Vinciguerra
e1f045b280
feat: add new chunking function
2024-06-08 11:44:09 +02:00
Marco Vinciguerra
543b48764a
add default folder for the cache
/ build (3.10) (push) Has been cancelled
2024-06-07 09:47:21 +02:00
Marco Vinciguerra
d79036149a
feat: add caching
/ build (3.10) (push) Waiting to run
2024-06-06 21:35:52 +02:00
Marco Vinciguerra
10672d6ebb
fix: update openai tts class
2024-06-06 10:00:58 +02:00
Marco Perini
5d1fbf806a
feat(indexify-node): add example
/ build (3.10) (push) Has been cancelled
2024-06-05 18:45:37 +02:00
Tin Do
d772453220
Refactor model_name attribute access in llm_model in robots_node.py
...
- Changed the access of model_name from dictionary-style to attribute-style in llm_model
to comply with langchain BaseChatModel.
- Updated the conditional and split operations accordingly.
2024-06-05 11:00:33 -04:00
Marco Perini
5c9843f141
fix(schema): fixed json output
2024-06-05 15:07:21 +02:00
duke147
1a404e358d
Merge remote-tracking branch 'upstream/main' into ernie
2024-06-05 19:14:44 +08:00
duke147
2b2b910a37
support ernie
2024-06-05 19:02:34 +08:00
Marco Vinciguerra
2ef6d67a9e
Merge pull request #346 from duke147/ernie
...
support ernie
2024-06-05 12:28:43 +02:00
Marco Vinciguerra
67d83cff46
fix: getter
2024-06-05 12:02:00 +02:00
duke147
4e16c9a81d
support ernie
2024-06-05 17:37:35 +08:00
Marco Vinciguerra
450fde601e
add get functions on the dictionary
2024-06-05 09:28:24 +02:00
Marco Vinciguerra
5d20186bf2
feat: add json as output
2024-06-05 09:20:20 +02:00
Marco Vinciguerra
1d38ed146a
fix: bug on generate_answer_node
2024-06-05 09:17:29 +02:00
Marco Vinciguerra
74fd530914
Merge branch 'pre/beta' into 332-pydantic-schema-validation
2024-06-05 09:05:19 +02:00
elijahbenizzy
e96b7018b6
Integrates with Burr's Forking/spawning ability
...
Requires this PR: https://github.com/DAGWorks-Inc/burr/pull/225
2024-06-04 21:17:52 -07:00
Marco Perini
f8b08e0b33
feat(append_node): append node to existing graph
/ build (3.10) (push) Has been cancelled
2024-06-04 23:34:43 +02:00
Marco Perini
376f758a76
feat(pydantic): added pydantic output schema
2024-06-04 23:07:49 +02:00
Marco Vinciguerra
fff89f431f
feat: refactoring of abstract graph
2024-06-04 19:41:11 +02:00
Marco Vinciguerra
58cd523bc5
Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta
2024-06-04 18:41:46 +02:00
Marco Vinciguerra
f81442b817
removed unused if
2024-06-04 18:41:44 +02:00
Marco Vinciguerra
4c0d0e935b
Merge pull request #339 from seyf97/seyf97-link_extraction_patch
...
/ build (3.10) (push) Has been cancelled
Release / Build (push) Has been cancelled
Release / Release (push) Has been cancelled
Update cleanup_html.py
2024-06-04 14:28:03 +02:00
Marco Vinciguerra
7ed2fe8ef0
feat: add dynamic caching
2024-06-04 14:27:46 +02:00
seyf97
acece72c28
Update cleanup_html.py
...
Remove redundant lines in Links extraction
2024-06-04 13:49:00 +03:00
Marco Vinciguerra
7a13a6819f
feat: refactoring of rag node
2024-06-04 12:01:21 +02:00
Marco Vinciguerra
244aada2de
feat: refactoring of an in if
2024-06-04 10:01:20 +02:00
Marco Vinciguerra
c8d556da4e
feat: fix an if
2024-06-03 21:49:34 +02:00
Marco Vinciguerra
8de720d379
feat: removed a bug
2024-06-03 21:45:37 +02:00
Marco Perini
79ace115c7
Merge pull request #323 from VinciGit00/refactoring-pdf_scraper
...
Refactoring pdf scraper and json scrape
2024-06-03 13:26:11 +02:00
Marco Vinciguerra
b4086550cc
feat: add csv scraper and xml scraper multi
2024-06-02 22:57:33 +02:00
Marco Vinciguerra
5cfc10178a
feat: add forcing format as json
2024-06-02 12:24:54 +02:00
Marco Vinciguerra
fff1232b8a
add rag node
2024-06-01 13:06:20 +02:00
Marco Vinciguerra
5bda918a39
feat: add json multiscraper
2024-06-01 12:04:19 +02:00
Marco Vinciguerra
4d42d7bfc6
add example
2024-06-01 11:20:24 +02:00
Marco Vinciguerra
f5cbd80c97
feat: add pdf scraper multi graph
2024-06-01 09:52:21 +02:00
Marco Vinciguerra
25352a56f9
Merge branch 'pre/beta' into temp
2024-05-31 23:08:24 +02:00
Marco Perini
8be27bad80
fix(3.9): python 3.9 logging fix
2024-05-31 22:32:20 +02:00
Marco Vinciguerra
930f673747
feat: removed rag node
2024-05-31 21:03:48 +02:00
Marco Vinciguerra
c4ce36111f
fix: typo in generate_screper_node
2024-05-30 18:50:04 +02:00
Johan Karlsson
4639f0cac5
fix: typo in prompt
2024-05-30 13:47:59 +02:00
Marco Vinciguerra
4fcb9902fe
fix: oneapi model
2024-05-29 18:24:09 +02:00
Marco Vinciguerra
8f2c8d5d12
Fix: Update __init__.py
2024-05-26 16:24:32 +02:00
Marco Vinciguerra
3c7dedf9c8
Merge pull request #305 from VinciGit00/pdf_fix
...
Pdf fix
2024-05-26 15:39:21 +02:00
Marco Vinciguerra
1d958be44d
Merge pull request #303 from VinciGit00/295-scrapegraph-ai接入oneapi模型qwen-turbo
...
add OneAPI integration
2024-05-26 12:17:34 +02:00
Marco Perini
8d76c4b3cb
fix(schema): added schema
2024-05-26 10:51:48 +02:00
Marco Perini
5fb9115330
feat(version): python 3.12 is now supported 🚀
2024-05-26 08:49:27 +02:00
Marco Perini
f9f6b08753
Merge branch 'pre/beta' into burr_integration
2024-05-25 14:51:07 +02:00
Marco Perini
19b27bbe85
feat(burr): first burr integration and docs
2024-05-25 01:34:53 +02:00
Marco Perini
5684578fab
fix(kg): removed unused nodes and utils
2024-05-25 00:44:14 +02:00
Marco Perini
a4ee757507
Merge branch 'pre/beta' into pdf_scraper_refactoring
2024-05-25 00:16:38 +02:00
Marco Perini
8d5eb0bb0d
fix(local_file): fixed textual input pdf, csv, json and xml graph
2024-05-25 00:13:47 +02:00
Marco Vinciguerra
b6f1766bd1
add OneAPI integration
...
Co-Authored-By: Federico Aguzzi <62149513+f-aguzzi@users.noreply.github.com>
Co-Authored-By: wangdongpeng1 <74647183+wangdongpeng1@users.noreply.github.com>
2024-05-24 13:46:40 +02:00
Marco Vinciguerra
b913b51cca
Merge branch 'logger-integration' into pre/beta
2024-05-24 12:39:14 +02:00
Federico Minutoli
c251cc45d3
fix(node-logging): use centralized logger in each node for logging
2024-05-24 01:09:49 +02:00
Federico Minutoli
4348d4f4db
fix(logger): set up centralized root logger in base node
2024-05-24 01:09:26 +02:00
Federico Minutoli
c807695720
feat(verbose): centralized graph logging on debug or warning depending on verbose
2024-05-24 01:09:03 +02:00
Federico Minutoli
0790ecd208
fix(web-loader): use sublogger
2024-05-24 01:08:28 +02:00
Federico Minutoli
d1394809d7
fix(logging): source code citation
2024-05-24 01:08:08 +02:00
Marco Vinciguerra
5fd7633c63
Update pdf_scraper_graph.py
2024-05-23 21:09:09 +02:00
Marco Vinciguerra
d00cde6030
fix(pdf_scraper): fix the pdf scraper gaph
2024-05-23 20:03:16 +02:00
Marco Vinciguerra
c93dbe0a94
Update smart_scraper_graph.py
2024-05-23 18:44:52 +02:00
Marco Vinciguerra
6d33a8a25e
rollback
2024-05-23 18:44:04 +02:00
Marco Vinciguerra
909af8d912
refactor gen answ node
2024-05-23 13:45:23 +02:00
Marco Vinciguerra
b377467b29
add info
2024-05-23 12:51:08 +02:00
Marco Vinciguerra
1774b18059
refactor of embeddings
2024-05-23 12:32:49 +02:00
Marco Vinciguerra
b6f7b647f8
Merge pull request #290 from VinciGit00/pre/beta
...
Pre/beta
2024-05-23 11:48:44 +02:00
Marco Vinciguerra
f00ed35f7b
Merge branch 'pre/beta' into patch-1
2024-05-23 11:47:51 +02:00
Marco Vinciguerra
0ba3a594fc
Update models_tokens.py
2024-05-23 11:43:54 +02:00
Joe Stone
ffd6015a50
Update abstract_graph.py
...
Adapt to some ollama modles with "nam/modelname" like ( 893379029/piccolo-large-zh-v2)
2024-05-23 15:58:47 +08:00
Alok Saboo
58289eccc5
feat: Add new models and update existing ones
2024-05-22 15:07:27 -04:00
Marco Perini
ac10128ff3
feat(burr): added burr integration in graphs and optional burr installation
2024-05-22 18:30:49 +02:00
Marco Vinciguerra
f2dffe534f
fix: pdf scraper bug
2024-05-22 11:54:55 +02:00
PeriniM
654a042396
feat(burr-node): working burr bridge
2024-05-22 00:24:38 +02:00
elijahbenizzy
d96840fa65
Updates Burr bridge to use class-based API
...
(also makes the App ID a unique one for debugging, it kept getting
confused)
2024-05-21 12:57:47 -07:00
Marco Perini
6cbd84f254
feat(burr-bridge): BurrBridge class to integrate inside BaseGraph
2024-05-21 19:57:10 +02:00
Marco Vinciguerra
ca436abf3c
fix: error in jsons
2024-05-21 14:35:48 +02:00
Marco Vinciguerra
be4237a04d
Merge branch 'pre/beta' into multi_scraper_graph
2024-05-21 13:44:06 +02:00
Marco Perini
fc58e2d3a6
feat(smart-scraper-multi): add schema to graphs and created SmartScraperMultiGraph
2024-05-21 13:13:27 +02:00
Marco Vinciguerra
1ef63217dd
add deepseek embeddings
2024-05-21 10:41:01 +02:00
Marco Vinciguerra
659fad770a
fix: add deepseek embeddings
2024-05-21 10:36:40 +02:00
Marco Vinciguerra
c8c3201745
Merge pull request #273 from JGalego/bugfix/bedrock-runs
...
bugfix: ScriptCreatorGraph verbose settings + RobotsNode handling model IDs
2024-05-20 19:40:23 +02:00
JGalego
3ffa896c39
Fixed model ID -> model name conversion
2024-05-20 17:35:55 +01:00
JGalego
d0a301d3c3
Moved common params up (verbose, headless and loader_kwargs)
2024-05-20 17:25:13 +01:00
JGalego
05ecc3ae20
Added missing logic to extract model_name from model_id
2024-05-20 17:17:11 +01:00
JGalego
9e92b03913
Added missing Titan text embedding models
2024-05-20 16:56:41 +01:00
Marco Perini
9096da7d47
Merge pull request #269 from QIN2DIM/feat-undetected-playwright
...
feat(docloaders): undetected-playwright
2024-05-19 22:26:50 +02:00
Marco Vinciguerra
ec8cbcab23
Merge branch 'pre/beta' into try
2024-05-19 22:08:11 +02:00
QIN2DIM
7b3ee4e71e
feat(docloaders): undetected-playwright
2024-05-19 18:01:03 +08:00
Mobin Chowdhury
7f30da9930
Update models_tokens.py
2024-05-18 23:32:27 +06:00
Marco Vinciguerra
5701afe927
add new import
2024-05-18 10:36:41 +02:00
Marco Perini
a338383399
feat(kg): removed import
2024-05-18 10:30:29 +02:00
Marco Perini
c75e6a06b1
feat(kg): working rag kg
2024-05-18 10:26:25 +02:00
Marco Vinciguerra
a0125b7e90
add orca mini integration
2024-05-18 09:31:40 +02:00
Marco Perini
58cc903d55
feat(multiple): quick fix working
2024-05-18 05:57:43 +02:00
Marco Vinciguerra
ff53771f87
add falcon model
2024-05-18 03:01:08 +02:00
Marco Vinciguerra
6f62b0560a
Merge branch 'multi_scraper_graph' of https://github.com/VinciGit00/Scrapegraph-ai into multi_scraper_graph
2024-05-18 02:42:27 +02:00
Marco Vinciguerra
b82f33aee7
fix: template names
2024-05-18 02:42:25 +02:00
Marco Perini
bed3eed50c
feat(multiple_search): working multiple example
2024-05-18 01:51:12 +02:00
Marco Vinciguerra
05e511e36f
add new prompts
2024-05-18 00:22:52 +02:00
Marco Perini
0196423bde
feat(knowledgegraph): add knowledge graph node
2024-05-17 23:41:44 +02:00
Marco Perini
73fa31db0f
feat(base_graph): alligned with main
2024-05-17 18:54:27 +02:00
Marco Perini
8c33ea3fbc
feat(node): knowledge graph node
2024-05-17 18:40:58 +02:00
Marco Vinciguerra
3453f72397
add graph
2024-05-17 18:23:50 +02:00
Marco Vinciguerra
02745a4f63
Merge branch 'main' into pre/beta
2024-05-17 10:40:19 +02:00
Marco Perini
d4d913c8a3
fix(deepcopy): switch whether we have obj in the config
2024-05-17 10:36:39 +02:00
VinciGit00
9e9f8f09d5
removed max depth
2024-05-16 12:39:08 +02:00
VinciGit00
9483afddd3
revert
2024-05-16 12:34:02 +02:00
Mayur Bhosale
1e0b2f7e70
Merge branch 'pre/beta' into nDeep
2024-05-15 23:36:23 +02:00
mayurdb
d60438cc89
Add a n-level deep search support
2024-05-15 23:32:15 +02:00
VinciGit00
ba8a4f7122
removed duplicates
2024-05-15 21:47:39 +02:00
VinciGit00
befa48c912
update lock
2024-05-15 21:28:56 +02:00
VinciGit00
4fe58d9ba4
fix logger
2024-05-15 16:12:13 +02:00
VinciGit00
a8fb851999
remove asdt
2024-05-15 16:11:08 +02:00
VinciGit00
40260d8d25
remove asdt
2024-05-15 16:00:47 +02:00
Marco Perini
999c930f42
fix(deepcopy): reaplced to shallow copy
2024-05-15 16:00:42 +02:00
VinciGit00
cc5adefd29
fix: come back to the old version
2024-05-15 15:54:00 +02:00
VinciGit00
5587a64d23
fix: removed unused
2024-05-15 15:51:27 +02:00
VinciGit00
7ced0d0564
remove asdt
2024-05-15 15:49:09 +02:00
VinciGit00
29d284e497
Merge branch 'main' into logger-integration
2024-05-15 15:28:20 +02:00
Marco Vinciguerra
008f8d9eae
Merge pull request #248 from VinciGit00/main
...
reallignment
2024-05-15 15:20:53 +02:00
Marco Vinciguerra
cffcf80a75
Merge branch '88-blockscraper-implementation' into main
2024-05-15 15:20:12 +02:00
Marco Vinciguerra
0b5cdd48fd
Merge pull request #246 from VinciGit00/main
...
reallignment
2024-05-15 15:19:05 +02:00
Marco Vinciguerra
22cd9e3605
Merge branch 'search_link_context' into main
2024-05-15 15:16:57 +02:00
Marco Perini
096b665c01
fix(searchgraph): used shallow copy to serialize obj
2024-05-15 15:05:47 +02:00
mayurdb
0b71b9a1e8
Add a new graph traversal that allows more than one edges out of a graph
2024-05-15 13:46:19 +02:00
VinciGit00
a4700bfc75
add robot node
2024-05-15 12:00:32 +02:00
VinciGit00
05890835f5
refactoring of loggers
2024-05-15 10:54:53 +02:00
Marco Vinciguerra
932df8d491
Merge pull request #238 from VinciGit00/gpt4-omni
2024-05-14 17:09:23 +02:00
mayurdb
a458ec4b9f
Update the prompt for the search_link_node
2024-05-14 16:59:16 +02:00
Marco Perini
a6e1813ddd
fix(fetch_node): bug in handling local files
2024-05-14 16:51:10 +02:00
Marco Perini
fcb3abb01d
feat(omni-search): added omni search graph and updated docs
2024-05-14 16:26:43 +02:00
VinciGit00
e53766b16e
feat: add logger integration
2024-05-14 15:20:39 +02:00
Marco Perini
a296927624
feat(omni-scraper): working OmniScraperGraph with images
2024-05-14 13:46:49 +02:00
Marco Perini
90955ca52f
feat(gpt-4o): image to text single node test
2024-05-14 11:43:21 +02:00
Federico Aguzzi
52a4a3b22d
feat: add gpt-4o
...
closes #232
2024-05-14 11:11:34 +02:00
VinciGit00
d6f5ca874f
Merge branch 'main' into pre/beta
2024-05-14 10:24:29 +02:00
Marco Perini
367dea5cbd
Merge branch 'pre/beta' into feat/parallel-node-execution
2024-05-13 23:50:40 +02:00
Marco Perini
a8d5e7db05
feat(batchsize): tested different batch sizes and systems
2024-05-13 23:49:48 +02:00
Alok Saboo
d9752b1619
chore: update models_tokens.py with new model configurations
2024-05-13 17:22:32 -04:00
JGalego
859c5d5ba5
Refactored to include custom AWS client for bedrock; Added missing Anthropic class
2024-05-13 17:57:53 +01:00
Marco Perini
dedc733047
fix(asyncio): replaced deepcopy with copy due to serialization problems
2024-05-13 18:46:34 +02:00
Marco Vinciguerra
f3d44c09ae
Merge pull request #228 from JGalego/fix/bedrock-support
...
fix: Amazon Bedrock Support
2024-05-13 14:57:23 +02:00
VinciGit00
d0167dee71
fix: bug for claude
2024-05-13 13:43:42 +02:00
JGalego
39be38f292
Fixed anthropic/bedrock conflict; Removed duplicate class Claude; Updated BedrockEmbeddings class
2024-05-13 12:35:42 +01:00
Marco Perini
0c1594737f
fix(fetch-node): removed isSoup from default
2024-05-13 12:09:55 +02:00
Marco Perini
7e8acd8e6a
Merge branch 'pre/beta' into fix/fetch-node-proxybroker
2024-05-13 11:17:37 +02:00
Marco Perini
5d6d996e8f
fix(proxy-rotation): removed max_shape duplicate
2024-05-13 07:26:43 +02:00
Marco Perini
1e9a564616
fix(proxy-rotation): removed duplicated arg and passed the loader_kwarhs correctly to the node
2024-05-12 18:39:03 +02:00
VinciGit00
e2350eda62
feat: add new prompt info
2024-05-12 11:14:30 +02:00
Federico Aguzzi
156b67b91e
feat: add support for deepseek-chat
...
closes #222
2024-05-12 01:00:11 +02:00
mayurdb
dd29c16cbe
Merge branch 'deepScrape' of github.com:mayurdb/Scrapegraph-ai into deepScrape
2024-05-11 16:58:54 +05:30
mayurdb
9a67a26cd3
Update documentation
2024-05-11 16:57:22 +05:30
Mayur Bhosale
ea3b545573
Merge branch 'pre/beta' into deepScrape
2024-05-11 16:54:40 +05:30
mayurdb
8f1fbe7e10
minor changes
2024-05-11 16:52:36 +05:30
mayurdb
df271b6451
Add search link node that can find out relevant links in the webpage
2024-05-11 16:39:55 +05:30
Federico Minutoli
b54d984c13
fix(chromium-loader): ensure it subclasses langchain's base loader
2024-05-11 11:44:56 +02:00
VinciGit00
dc91719365
Update cleanup_html.py
2024-05-11 10:49:16 +02:00
Marco Vinciguerra
b752499fab
Merge pull request #217 from mayurdb/fetchLinkFix
...
Fetch links in the page while parsing html
2024-05-11 09:42:40 +02:00
Stefan Krawczyk
f2bb1cc4f8
Fixes LC document deserialization
...
Depends on https://github.com/DAGWorks-Inc/burr/pull/175 .
2024-05-11 00:06:02 -07:00
mayurdb
300fd5d253
Fetch links in the page while parsing html
2024-05-11 09:46:51 +05:30
Eric Page
0683e78e78
Merge branch 'pre/beta' into fix-GenerateScraperGraph
2024-05-11 01:59:28 +02:00
Eric Page
24c3b0580c
Removed nonfunctional RAG node from ScriptCreatorGraph
2024-05-11 01:38:40 +02:00
Eric Page
aac51ba290
Removed dead code, allows GenerateScraperNode to generate scraper with
...
one chunk of context
2024-05-11 01:34:51 +02:00
Eric Page
40884747c7
Added parse_html option in parse_node
2024-05-11 00:32:01 +02:00
Federico Minutoli
627cbeeb20
feat(parallel-exeuction): add asyncio event loop dispatcher with semaphore for parallel graph instances
...
TODO: still untested
2024-05-11 00:13:27 +02:00
Federico Minutoli
fc2aa3ac1c
Merge branch 'pre/beta' of https://github.com/DiTo97/Scrapegraph-ai into fix/fetch-node-proxybroker
2024-05-10 21:20:40 +02:00
Federico Minutoli
768719cce8
feat(safe-web-driver): enchanced the original AsyncChromiumLoader web driver with proxy protection and flexible kwargs and backend
...
the original class prevents passing kwargs down to the playwright backend, making some config unfeasible, including passing a proxy server to the web driver.
the new class has backward compatibility with the original, but 1) allows any kwarg to be passed down to the web driver, 2) allows specifying the web driver backend (only playwright is supported for now) in case more (e.g., selenium) will be supported in the future and 3) automatically fetches a suitable proxy if one is not passed already
2024-05-10 21:13:38 +02:00
Federico Minutoli
217013181d
feat(proxy-rotation): add parse (IP address) or search (from broker) functionality for proxy rotation
...
the broker has been made fully configurable for anonymity level, admissible locations, scheme and max shape not to waste resources, unlike the original `free-proxy` package.
other options have been explored (e.g., `proxybroker`, `proxybroker2`) due to their built-in proxy server and rotation capabilities, but the former is no longer maintained, and the latter has issue with any python version outside of python 3.9
2024-05-10 21:09:48 +02:00
Federico Minutoli
db2234bf5d
feat(webdriver-backend): add dynamic import scripts from module and file
2024-05-10 21:06:05 +02:00
Stefan Krawczyk
82afa0ef3d
Working smart scraper graph
...
Bugs found in tracker persistence:
- serializing inputs properly for tracker
- deserializing state from a previous found
2024-05-10 10:49:53 -07:00
Stefan Krawczyk
d94195f823
WIP
2024-05-10 09:48:46 -07:00
Stefan Krawczyk
be16fecb55
WIP
2024-05-10 09:48:44 -07:00
Marco Perini
864aa91326
feat: revert fetch_node
2024-05-10 15:11:54 +02:00
Marco Vinciguerra
99adc9799f
Merge branch 'pre/beta' into fetchNodeFix
2024-05-10 11:13:54 +02:00
mayurdb
f8ce3d5916
fix: Augment the information getting fetched from a webpage
2024-05-10 13:28:53 +05:30
VinciGit00
0ab31c3fdb
fix: add json integration
2024-05-09 21:07:07 +02:00
VinciGit00
324e977b85
fix: fixed bugs for csv and xml
2024-05-09 20:46:46 +02:00
Shubham Kamboj
f10f3b1438
feat: Add support for passing pdf path as source
2024-05-09 21:55:05 +05:30
VinciGit00
94156755d1
Update abstract_graph.py
2024-05-09 12:29:39 +02:00
VinciGit00
403979337c
Update abstract_graph.py
2024-05-09 10:46:31 +02:00
VinciGit00
0bb68d1500
Merge branch 'main' of https://github.com/VinciGit00/Scrapegraph-ai
2024-05-09 10:31:21 +02:00
VinciGit00
8272d736a6
add tokenizatio for mxbai-embed-large
2024-05-08 21:50:42 +02:00
VinciGit00
4ed0fb89c3
feat: update info
2024-05-08 21:25:03 +02:00
VinciGit00
e7d39a5daf
fixed gemini embeddings
2024-05-08 21:24:17 +02:00
Marco Perini
6b71ec1d2b
fix(examples): local, mixed models and fixed SearchGraph embeddings problem
2024-05-08 15:36:26 +02:00
Marco Perini
186c0d035d
fix(examples): openai std examples
2024-05-08 14:56:44 +02:00
Marco Vinciguerra
b326886250
Merge branch '88-blockscraper-implementation' into asdt
2024-05-07 13:27:30 +02:00
VinciGit00
2258fe5ac0
add new search graph examples
2024-05-06 22:31:48 +02:00
VinciGit00
67d5fbf816
feat: new search_graph
2024-05-06 22:09:18 +02:00
VinciGit00
5a67bca0db
Merge branch 'pre/beta' into pr/161
2024-05-06 14:50:04 +02:00
VinciGit00
ac6d2005bb
Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta
2024-05-06 14:46:25 +02:00
VinciGit00
cbd77dfdb0
removed claude
2024-05-06 14:46:24 +02:00
Marco Vinciguerra
97c3fffaf3
Merge pull request #162 from f-aguzzi/patch-1
...
feat: add new hugging_face models
2024-05-06 14:43:29 +02:00
Marco Vinciguerra
f6442ccca3
Merge pull request #157 from VinciGit00/llava_integration
2024-05-06 14:34:32 +02:00
Federico Aguzzi
d5547a450c
feat: add new hugging_face models
2024-05-06 14:28:58 +02:00
Cem Uzunoglu
2ac9e16dd9
Fixed accidental reformatting.
2024-05-06 15:09:56 +03:00
Cem Uzunoglu
e264e92e72
Added support for Claude 3 models from Anthropic
2024-05-06 14:34:35 +03:00
VinciGit00
8c7c3e3ced
Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta
2024-05-06 13:29:32 +02:00
VinciGit00
ac0a2e52fb
Update models_tokens.py
2024-05-06 13:29:31 +02:00
Marco Perini
fd01b73b71
fix(llm): fixed gemini api_key
2024-05-06 13:17:47 +02:00
Shubham Kamboj
726de28898
feat: Fix bug for gemini case when embeddings config not passed
2024-05-06 15:40:03 +05:30
VinciGit00
89a1f99778
add lava integration for ollama
2024-05-06 12:03:39 +02:00
VinciGit00
51aa109e42
feat: add turboscraper (alfa)
2024-05-06 11:59:14 +02:00
VinciGit00
5e1d5db6da
Update search_internet_node.py
2024-05-06 10:09:17 +02:00
VinciGit00
80053a2358
Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta
2024-05-06 10:07:56 +02:00
VinciGit00
389b52aabb
removed examples
2024-05-06 10:07:31 +02:00
VinciGit00
88d999e15b
add website content
2024-05-06 09:36:32 +02:00
Marco Perini
930adb38f2
feat(node): multiple url search in SearchGraph + fixes
...
Implemented GraphIteratorNode and MergeAnswersNode to create multiple istances of a graph and merge the scraped content from multiple pages
2024-05-06 00:30:09 +02:00
Marco Perini
dbb614a8dd
feat: multiple graph instances
2024-05-05 23:51:04 +02:00
Marco Perini
1c4ba91620
exposed abstract_graph allowing the user to create new graphs
2024-05-05 22:05:11 +02:00
Marco Perini
84fcb44aaa
feat: fixed custom_graphs example and robots_node
2024-05-05 22:02:24 +02:00
Marco Perini
36a1522bba
Merge pull request #153 from VinciGit00/google_embeddings
...
feat: add gemini embeddings
2024-05-05 17:54:07 +02:00
Lorenzo Padoan
e1b9d69360
dev basic class blockindentifier
2024-05-05 17:36:30 +02:00
Eric Page
cc27b21c3f
Merge branch 'pre/beta' into pass-common-params-graph
2024-05-05 16:18:32 +02:00
Eric Page
3ae2ea1dbd
Miscellaneous "llm" -> "llm_model" refactors
2024-05-05 15:58:50 +02:00
Eric Page
a53e95cbf0
Corrected graphs to use common params
2024-05-05 14:36:16 +02:00
Eric Page
8d0e109a70
Added overwrite keyword to set_common_params`
2024-05-05 14:32:58 +02:00
VinciGit00
79daa4c112
feat: add gemini embeddings
2024-05-05 11:40:44 +02:00
Eric Page
4dc6049e76
Simplified create graph functions using common params
2024-05-05 10:40:04 +02:00
Eric Page
444a13a6a6
Created set_common_params function
2024-05-05 10:15:37 +02:00
Eric Page
2178485b3d
Adjusted graphs to reflect node_config change
2024-05-05 09:47:13 +02:00
VinciGit00
cb1bd00a19
removed unused node
2024-05-05 09:40:35 +02:00
Eric Page
729d5d7597
Changed node_config["llm"] to node_config["llm_model"]
2024-05-05 09:36:11 +02:00
Eric Page
c2c61625b3
Corrected logic of update_config function in base_node.py
2024-05-05 06:37:31 +02:00
Eric Page
b7539d4725
Added update_config function to base_node.py
2024-05-05 06:25:43 +02:00
Marco Perini
0ab7272fd7
Merge branch 'pre/beta' into 133-support-claude3-haiku-and-others-using-litellm
2024-05-05 02:15:15 +02:00
Marco Perini
8c6dda6e27
Merge pull request #138 from VinciGit00/new-search-function
...
feat: refactoring search function
2024-05-05 02:02:41 +02:00
VinciGit00
2878695d5f
fix: trailing whitespace
2024-05-04 21:26:19 +02:00
Marco Vinciguerra
2a8ff0f65d
Merge branch 'main' into pre/beta
2024-05-04 21:20:40 +02:00
Shubham Kamboj
fd59f282a8
Bug fixes
2024-05-04 22:18:54 +05:30
Shubham Kamboj
d830d1371b
Resolve merge conflicts
2024-05-04 22:01:40 +05:30
Shubham Kamboj
7599234ab9
feat: Enable end users to pass model instances of HuggingFaceHub
2024-05-04 21:42:57 +05:30
Tamas Darvas
1630c66a58
Update models_tokens.py with ollama/nomic-embed-text
2024-05-04 09:36:19 +02:00
Marco Perini
df293fdfd2
Merge pull request #139 from VinciGit00/pdf_scraper_graph_introduction
...
feat: add pdf scraper
2024-05-03 21:45:09 +02:00
VinciGit00
10a94530e3
feat: add pdf scraper
2024-05-03 21:42:02 +02:00
Marco Perini
aa9271e7bc
feat: Azure implementation + embeddings refactoring
2024-05-03 21:40:01 +02:00
VinciGit00
aeb1acbf05
feat: refactoring search function
2024-05-03 21:06:09 +02:00
VinciGit00
39f64e5d48
add claude model
2024-05-03 17:17:31 +02:00
Marco Vinciguerra
2abe05a0cf
Merge pull request #135 from S4mpl3r/feature
2024-05-03 16:49:07 +02:00
S4mpl3r
819cbcd3be
refactor: Changed the way embedding model is created in AbstractGraph class and removed handling of embedding model creation from RAGNode. Now AbstractGraph will call a dedicated method for embedding models instead of _create_llm. This makes it easy to use any LLM with any supported embedding model.
2024-05-03 16:14:27 +03:30
Marco Perini
cb1cb616b7
feat: add base_node to __init__.py
...
Useful for people to create custom nodes
2024-05-03 14:18:12 +02:00
Marco Vinciguerra
96ebcfcc35
Merge branch 'pre/beta' into main
2024-05-02 21:05:28 +02:00
VinciGit00
2ea54eab1d
fix: gemini errror
2024-05-02 18:56:37 +02:00
Marco Vinciguerra
579a27faf9
Merge pull request #126 from VinciGit00/bedrock_support
...
feat: bedrock support
2024-05-02 17:57:20 +02:00
Marco Vinciguerra
a2d5c75f10
Merge branch 'pre/beta' into main
2024-05-02 17:55:21 +02:00
Shubham Kamboj
b86aac2188
feat: Allow end users to pass model instances for llm and embedding model
2024-05-02 20:09:23 +05:30
VinciGit00
16de49fa08
add integration for bedrock
...
Co-Authored-By: redrusty2 <15157208+redrusty2@users.noreply.github.com>
2024-05-02 13:47:17 +02:00
VinciGit00
4a3bc37f2f
fix: bug on script_creator_graph.py
2024-05-02 09:23:46 +02:00
VinciGit00
5cf4e4f92f
fix: examples and graphs
2024-05-02 09:20:46 +02:00
Marco Perini
29427c2334
docs: fixed basegraph docstring
2024-05-02 01:05:20 +02:00
Marco Perini
c11fc28896
docs: refactor examples
2024-05-02 00:44:19 +02:00
Marco Perini
0631985e61
docs: graphs and helpers docstrings
2024-05-02 00:23:38 +02:00
Marco Perini
18c20eb03d
docs: refactor models docstrings
2024-05-01 23:33:42 +02:00
Marco Perini
1409797475
docs: refactor nodes docstrings
2024-05-01 23:17:57 +02:00
Marco Perini
4d542a88f7
feat: added node and graph for CSV scraping
2024-05-01 21:34:40 +02:00
Marco Perini
02d1af006c
feat: modified node name
2024-05-01 21:33:25 +02:00
Marco Perini
e9817963c8
docs: base and fetch node
2024-05-01 21:30:06 +02:00
VinciGit00
276ac8b40c
changed name
2024-05-01 20:17:01 +02:00
VinciGit00
7c1b5796d7
add csv scraper
2024-05-01 17:20:04 +02:00
Perinim
cf038b33ea
docs: update utils docstrings
2024-05-01 12:35:12 +02:00
EURAC\marperini
2dd7817cfb
feat: added verbose flag to suppress print statements
2024-04-30 15:31:57 +02:00
Marco Perini
d592d27bb4
Merge pull request #115 from VinciGit00/101-scrape-json-files
...
feat: add xml scraper and json scraper
2024-04-30 14:29:57 +02:00
EURAC\marperini
8fba7e5490
feat(refactor): changed variable names
2024-04-30 14:28:30 +02:00
VinciGit00
e3d0194dc9
fix: script generator and add new benchmarks
2024-04-30 11:51:04 +02:00
VinciGit00
da2c82a2a2
add json and xml scraper
2024-04-30 10:52:22 +02:00
VinciGit00
e0ffc838b0
feat: add cluade integration
2024-04-30 10:08:54 +02:00
EURAC\marperini
42ab0aa1d2
feat(fetch): added playwright support
2024-04-30 04:02:58 +02:00
EURAC\marperini
dbbf10fc77
feat(llm): implemented groq model
2024-04-30 02:32:14 +02:00
Marco Vinciguerra
a449ed1199
Merge branch 'pre/beta' into 93-groq-model-implementation
2024-04-29 21:48:08 +02:00
Marco Vinciguerra
ecaef43bf3
Merge pull request #107 from VinciGit00/main
...
merging for new version
2024-04-29 21:46:04 +02:00
Lorenzo Padoan
7dd5b1a033
feat: base groq + requirements + toml update with groq
...
Co-Authored-By: gioCarBo <109686531+giocarbo@users.noreply.github.com>
2024-04-29 20:06:51 +02:00
VinciGit00
deb920a33e
fixing json and example
2024-04-29 16:11:57 +02:00
VinciGit00
3eacc6fbc3
add paths
2024-04-29 16:07:11 +02:00
VinciGit00
674e64222e
add first new graphs
2024-04-29 15:55:21 +02:00
VinciGit00
0b25d9a977
Update search_link_node.py
2024-04-29 15:49:46 +02:00
VinciGit00
4f10ac302e
fixed gemma model
2024-04-29 11:06:30 +02:00
VinciGit00
ab72db3b50
if the conf is not present preset to 8192 the context window
2024-04-29 10:26:34 +02:00
EURAC\marperini
6e7283ed8f
feat: add finalize_node()
2024-04-29 10:05:04 +02:00
PeriniM
7f1c3b7d83
fix: added missing dependecies
...
include python v3.9.0 and change deprecated method langchain
2024-04-28 20:29:23 +02:00
VinciGit00
a9b11e433a
fix: bug for calculate costs
2024-04-28 14:36:49 +02:00
VinciGit00
9cd516507c
fix: bug with fetch node
2024-04-27 21:23:35 +02:00
VinciGit00
adbc08f27b
fix: robot node and proxyes
2024-04-27 19:07:37 +02:00
VinciGit00
b754dd909c
fix: changed proxy function
2024-04-27 14:40:50 +02:00
VinciGit00
f6077d1f98
feat: add new proxy rotation function
2024-04-27 13:31:53 +02:00
Marco Vinciguerra
44bc9196ef
Merge pull request #84 from VinciGit00/79-add-getstate-function-to-abstract-graph
...
complete get_state function
2024-04-26 21:44:33 +02:00
Marco Vinciguerra
4f1d28d89a
Merge pull request #81 from VinciGit00/refactoring_engine
...
Refactoring engine -> from set to list
2024-04-26 21:37:57 +02:00
Marco Vinciguerra
922aa96b37
Merge branch 'pre/beta' into search_links_node
2024-04-26 19:23:42 +02:00
EURAC\marperini
e778d27169
added tree metadata
2024-04-26 15:17:44 +02:00
VinciGit00
cd8d3e7a4f
refactoring of the graphs
2024-04-26 11:08:56 +02:00
VinciGit00
b2ebabd32f
Update base_graph.py
2024-04-25 19:45:49 +02:00
VinciGit00
e714a59c2e
refactoring of engine
2024-04-25 19:22:12 +02:00
VinciGit00
75255bb4af
complete get_state function
2024-04-25 15:08:42 +02:00
EURAC\marperini
dd99ac595e
structural and textual hashing
2024-04-25 11:56:41 +02:00
EURAC\marperini
c5f9fcaabe
Merge branch 'asdt' of https://github.com/VinciGit00/Scrapegraph-ai into asdt
2024-04-25 11:06:54 +02:00
EURAC\marperini
c927f707e9
subtrees implementation
2024-04-25 11:06:05 +02:00
VinciGit00
f232717bf8
REFACTORING
2024-04-25 09:19:44 +02:00
VinciGit00
a73496df9b
refactoring of nodes
2024-04-24 21:13:27 +02:00
EURAC\marperini
ae69e4b340
dom tree structure
2024-04-24 18:59:44 +02:00
VinciGit00
e68b707e0d
changed color and return type
2024-04-24 17:29:49 +02:00
EURAC\marperini
ae49dee985
fixed robots_node and add test
2024-04-24 16:12:53 +02:00
VinciGit00
9b9a9f204c
handling the agent model
2024-04-24 13:13:46 +02:00
VinciGit00
adf89e5e1b
add new info for robot node
2024-04-24 10:14:26 +02:00
VinciGit00
8ab603220d
changed node name
2024-04-24 10:02:09 +02:00
VinciGit00
726efb7f00
add new info
2024-04-23 20:51:39 +02:00
VinciGit00
8ef9dec2e2
add new node
2024-04-23 20:23:35 +02:00
Lorenzo Padoan
12dcc5065a
ADD benchmarks m2 max llama3
2024-04-23 12:28:51 +02:00
VinciGit00
261ecbf3a5
fixed typos in the doc
2024-04-22 21:32:06 +02:00
VinciGit00
3960e56d29
fixed typos in the doc
2024-04-22 21:31:23 +02:00
VinciGit00
9e3dddcb44
add nodes that search all the links
2024-04-22 18:22:40 +02:00
VinciGit00
74d7ef614a
refactor of conbert to json function
2024-04-21 20:30:56 +02:00
Alok Saboo
6cbf992870
chore: Remove .csv extension from filename in convert_to_csv.py
2024-04-21 14:13:34 -04:00
VinciGit00
8aa2cadb00
add integration for llama3
2024-04-21 13:11:10 +02:00
EURAC\marperini
ac099e2763
fixed ollama embedder params
2024-04-21 01:21:09 +02:00
VinciGit00
13669cc1ab
revert to old naming
2024-04-19 13:43:51 +02:00
VinciGit00
a2349e56af
recatoring of nodes excluive for openai
2024-04-19 13:21:48 +02:00
VinciGit00
3f95801737
Merge branch 'main' of https://github.com/VinciGit00/Scrapegraph-ai
2024-04-19 10:36:40 +02:00
EURAC\marperini
134c94e0ed
added first asdt implementation
2024-04-17 17:12:59 +02:00
EURAC\marperini
c032131692
fixed chunk response
2024-04-17 15:20:45 +02:00
Marco Perini
b14527c08d
Merge branch 'main' into refactor_generate_answer_node
2024-04-17 12:08:43 +02:00
EURAC\marperini
c237e63631
removed unused variable
2024-04-17 12:05:51 +02:00
Marco Perini
edc439fd01
Merge pull request #64 from VinciGit00/generateScraperbranch
...
add generateScraperBranch
2024-04-17 11:51:06 +02:00
EURAC\marperini
b2d170cd86
refactored and fixed single chunk bug
2024-04-17 11:50:19 +02:00
EURAC\marperini
bdac2df4bb
fixed single chunk bug
2024-04-17 11:34:10 +02:00
Andrea Rota
b0e446f014
feat: apply remove to the document before updating the state
2024-04-17 11:24:56 +02:00
EURAC\marperini
d5f1c82c91
Merge remote-tracking branch 'origin/refactor_genrerate_answer_node'
2024-04-17 10:55:28 +02:00
Marco Vinciguerra
4703a0b94c
Update remover.py
2024-04-16 12:27:17 +02:00
VinciGit00
4233430518
add integration on the fetch node
2024-04-16 12:19:23 +02:00
VinciGit00
9661c77ebe
add minimizer function
2024-04-16 12:07:43 +02:00
Marco Vinciguerra
3fc18b2110
Update generate_answer_node.py
2024-04-16 12:00:33 +02:00
Matteo Vedovati
9c3b490b03
Update generate_scraper_node.py
2024-04-16 10:21:49 +02:00
VinciGit00
372981f153
Update generate_answer_node.py
2024-04-16 10:09:35 +02:00
VinciGit00
c56640034e
add return statement and new answers
2024-04-15 10:49:02 +02:00
VinciGit00
8334737f10
Update generate_answer_node.py
2024-04-15 10:44:03 +02:00
VinciGit00
724aee6b23
add informations on prompt
2024-04-15 10:43:40 +02:00
VinciGit00
8fc41873a7
refactoring of generate answer node
2024-04-14 21:53:15 +02:00
VinciGit00
876223da56
add generateScraperBranch
...
Co-Authored-By: Matteo Mei <52063123+FattiMei@users.noreply.github.com>
2024-04-14 13:18:50 +02:00
Lorenzo Padoan
849f411844
Merge pull request #61 from VinciGit00/anti_malicious
...
add anti not scraping task
2024-04-12 13:47:10 +02:00
VinciGit00
e0978c53b3
Merge branch 'main' into ollama_integration
2024-04-12 13:00:35 +02:00
VinciGit00
3640434f5c
add utils
2024-04-12 12:59:15 +02:00
Marco Perini
dde9eeec87
Merge pull request #60 from VinciGit00/huggingface_integration
...
add hugginface integration (embeddings, models ...)
2024-04-11 22:40:05 +02:00
VinciGit00
77581876b5
add anti not scrapong task
2024-04-11 19:31:54 +02:00
VinciGit00
dc149e6f63
fixing bug on rag node
2024-04-11 19:26:38 +02:00
EURAC\marperini
74ec928f96
added execution info of the graph, time, price etc
2024-04-11 14:29:32 +02:00
VinciGit00
53e317d46b
add hugginface integration
2024-04-11 12:43:06 +02:00
EURAC\marperini
982f142f20
fixed embedder_model config for local models
2024-04-10 11:00:52 +02:00
EURAC\marperini
8b915e3ec6
fixed issue ollama embeddings
2024-04-10 09:29:03 +02:00
ftoppi
26616d25b1
abstract_graph.py: allow user to set Ollama model_tokens in config
2024-04-09 21:27:15 +02:00
VinciGit00
78a1e7aec4
new model
2024-04-09 13:54:10 +02:00
Marco Vinciguerra
b4e640879d
Merge branch 'main' into refactoring_examples_folder
2024-04-09 11:46:34 +02:00
VinciGit00
992e7f862f
add models avaiables
2024-04-09 10:33:45 +02:00
VinciGit00
cec4c43c64
add new local models context window
2024-04-09 10:23:38 +02:00
VinciGit00
0ffefdd170
refactoring if examples ffolder
2024-04-08 22:28:55 +02:00
EURAC\marperini
a601695fd5
updated nodes metadata
2024-04-08 15:23:23 +02:00
EURAC\marperini
dee1a42629
fixed token models, added mistral support
2024-04-08 15:21:06 +02:00
EURAC\marperini
17add20c13
implemented node_config, add embedder model choice, add azure endpoint, refactor graphs and exmaples
2024-04-08 12:18:32 +02:00
EURAC\marperini
6a83c2306d
docker compose working ollama implementation
2024-04-07 11:08:08 +02:00
VinciGit00
6dc4c6dec4
Merge branch 'main' into research_branch
2024-04-06 14:44:56 +02:00
VinciGit00
a5fc71f34e
fixed internet research
2024-04-06 14:39:19 +02:00
EURAC\marperini
57a53bd6af
refactor examples
2024-04-06 13:22:42 +02:00
EURAC\marperini
c2709aa50c
working search graph
2024-04-06 13:01:04 +02:00
VinciGit00
3269b4aaa7
fixing infos for the pr
2024-04-03 13:33:17 +02:00
VinciGit00
c9951340b5
fixed image_to_tex_node and refactoring
2024-04-03 12:56:25 +02:00
VinciGit00
4c3ea8f4ac
fixed image_to_tex_node and refactoring
2024-04-03 12:53:12 +02:00
VinciGit00
b53bfefeea
removed junk if
2024-04-02 12:57:27 +02:00
VinciGit00
db6d988b2d
refactoring for pylint
2024-04-01 13:41:50 +02:00
VinciGit00
471df48a22
add google search
2024-04-01 13:36:02 +02:00
VinciGit00
5e38c2e298
add superclasse for graphs
2024-03-28 20:46:17 +01:00
VinciGit00
7d0fd74bfc
add search graph
2024-03-28 16:54:01 +01:00
VinciGit00
b8a3590235
add SearcInetnetNode
2024-03-28 13:14:39 +01:00
VinciGit00
fff1f9ee51
new function for searching
2024-03-28 12:41:45 +01:00
VinciGit00
a28ca993de
refactoring of examples folder
2024-03-27 21:41:21 +01:00
VinciGit00
672e9a5191
refactoring of research function
2024-03-27 21:29:10 +01:00
VinciGit00
97d0a6887f
add reserch web function
2024-03-27 21:22:04 +01:00
VinciGit00
587dfa5c68
refactoring of the saving functions
2024-03-26 14:15:36 +01:00
VinciGit00
be36ab8707
add new type of scraping through text
2024-03-24 21:22:26 +01:00
VinciGit00
f448b8dfb4
removed unused return
2024-03-18 22:55:24 +01:00
VinciGit00
e25f74ddc8
fixed bug for scraping from node + add example
2024-03-18 22:53:52 +01:00
VinciGit00
1afb950751
refactoring of code, add example on the readme
2024-03-18 14:35:47 +01:00
Perinim
cae38a0d67
fixed bugs after merge
2024-03-18 11:30:14 +01:00
Perinim
b5bdd4e729
Merge remote-tracking branch 'origin/main' into refactoring_nodes
2024-03-18 11:07:42 +01:00
Perinim
7d3190a636
fixed example graphs and utils files
2024-03-18 10:20:23 +01:00
Perinim
b547972968
fixed custom_graph, graph_builder and trulens_evaluator
2024-03-18 09:12:59 +01:00
Perinim
52934bf007
implemented graph_config, fixed smart_scraper and speech graph
2024-03-17 20:35:04 +01:00
Perinim
f27e0b4c5f
removed fetch_html and fetch_text (merged)
2024-03-17 17:01:29 +01:00
Perinim
875a7cc4f0
refactored RagNode and GenerateAnswerNode
2024-03-17 16:48:58 +01:00
Perinim
3585cd81e5
refactor fetch node
2024-03-17 12:02:35 +01:00
Perinim
20d7b69008
refactored BaseNode and ParseNode
2024-03-17 11:40:35 +01:00
Lorenzo Padoan
d0f4b0159f
DEV gemini support for simple custom graph
...
-Not supported yet smartscrapergrapn with gemini
2024-03-14 19:35:04 +01:00
