Marco Vinciguerra
886bab78e7
Merge branch 'main' into temp-1
2024-08-06 10:04:57 +02:00
Marco Vinciguerra
9355507a2d
feat: refactoring of the code
2024-08-02 12:00:00 +02:00
Marco Vinciguerra
fe099f9cbc
Merge branch 'pre/beta' into browserbase_integration
/ build (push) Has been cancelled
2024-08-01 15:14:49 +02:00
Marco Vinciguerra
be870a4316
Update fetch_node.py
...
Co-Authored-By: Federico Minutoli <40361744+DiTo97@users.noreply.github.com>
2024-08-01 13:24:48 +02:00
Marco Vinciguerra
968c69e217
fix: fixed bug on fetch_node
...
Co-Authored-By: Federico Minutoli <40361744+DiTo97@users.noreply.github.com>
2024-08-01 13:23:54 +02:00
Marco Vinciguerra
e21d461710
push
2024-08-01 13:16:49 +02:00
Marco Vinciguerra
5ecdbe715f
feat: add integration in the abstract grapgh
2024-08-01 12:51:18 +02:00
Federico Aguzzi
a94ebcde00
refactor: move embeddings code from AbstractGraph to RAGNode
/ build (push) Waiting to run
2024-08-01 11:53:17 +02:00
Marco Vinciguerra
f6b7c6a430
refactoring
...
Co-Authored-By: Federico Aguzzi <62149513+f-aguzzi@users.noreply.github.com>
2024-07-29 17:56:32 +02:00
Federico Aguzzi
25066b2bc5
refactor: remove LangChain wrappers for Ollama
2024-07-29 17:49:21 +02:00
Federico Aguzzi
2c5f934f10
refactor: remove LangChain wrappers
2024-07-29 17:46:17 +02:00
amosdinh
05cf9adaa7
Ollama: Use no json format when creating the search query
2024-07-26 12:10:14 +02:00
Marco Perini
5c08eea189
docs: prev version
CodeQL / Analyze (python) (push) Has been cancelled
/ build (push) Has been cancelled
Release / Build (push) Has been cancelled
Release / Release (push) Has been cancelled
2024-07-24 23:37:57 +02:00
Marco Vinciguerra
e08b30495d
boh
CodeQL / Analyze (python) (push) Waiting to run
/ build (push) Waiting to run
Release / Build (push) Waiting to run
Release / Release (push) Blocked by required conditions
2024-07-24 14:33:55 +02:00
Federico Aguzzi
ddbfbe1205
Merge branch 'pre/beta' into generate_answer_parallel
2024-07-23 20:51:35 +02:00
Marco Perini
1d41f6eafe
fix: md conversion
2024-07-23 19:27:18 +02:00
Marco Perini
49b6468fb1
Merge branch 'main' into pre/beta
2024-07-23 16:14:15 +02:00
Marco Perini
0ebc387446
Merge pull request #487 from ScrapeGraphAI/update-generate-answer-node
...
Update generate answer node
2024-07-23 15:45:20 +02:00
Marco Perini
12b5eada6e
fix(md_conversion): add absolute links md, added missing dependency
2024-07-23 15:34:12 +02:00
Marco Vinciguerra
df1ecc0019
Update generate_answer_node.py
/ build (push) Waiting to run
2024-07-23 13:06:59 +02:00
Marco Vinciguerra
cf2734392c
removed unused function
2024-07-23 13:05:50 +02:00
Marco Vinciguerra
8dd941e39e
update generate answer nodes
...
Co-Authored-By: Federico Aguzzi <62149513+f-aguzzi@users.noreply.github.com>
2024-07-23 12:40:11 +02:00
Marco Vinciguerra
70911f83be
Update generate_answer_pdf_node.py
2024-07-23 11:07:35 +02:00
Marco Vinciguerra
0c4b2908d9
feat: add generate_answer node paralellization
...
/ build (push) Waiting to run
Co-Authored-By: Federico Minutoli <40361744+DiTo97@users.noreply.github.com>
2024-07-22 19:58:33 +02:00
Marco Vinciguerra
2ae19aee56
update function
2024-07-22 16:50:58 +02:00
Marco Vinciguerra
09256f7b11
fix: parse node
2024-07-22 16:15:38 +02:00
Marco Vinciguerra
71f894eee3
fix: parse_html node have a bug
2024-07-22 16:01:27 +02:00
Marco Vinciguerra
7a3c21b58d
Update generate_answer_node.py
2024-07-20 20:19:16 +02:00
Marco Vinciguerra
6a9205920f
Merge branch 'pre/beta' into temp
2024-07-20 20:03:20 +02:00
Marco Perini
99dc8497d8
docs(gpt-4o-mini): added new gpt, fixed chromium lazy loading,
...
add documentation and metrics
2024-07-20 20:02:26 +02:00
Marco Perini
61a51d4bce
Merge pull request #466 from ScrapeGraphAI/refactoring_convert_to_md
...
refactoring of generate answer node
2024-07-20 19:40:57 +02:00
Marco Vinciguerra
19af78c7d8
Merge branch 'pre/beta' into main
2024-07-20 18:14:36 +02:00
DragonelRoland
e9ecf5f0dc
Update typo in conditional_node.py
...
"thar" -> "that"
2024-07-20 15:06:52 +02:00
Marco Vinciguerra
032a491605
Update parse_node.py
/ build (push) Has been cancelled
2024-07-17 23:06:05 +02:00
Marco Vinciguerra
07f1e23d23
fix: parse_node
2024-07-17 22:58:21 +02:00
Marco Vinciguerra
68f58cc4dd
refactoring of generate answer node
2024-07-17 22:41:49 +02:00
Federico Aguzzi
dccb893fdf
Merge pull request #456 from ScrapeGraphAI/refactoring-of-search_link_node
...
fix: search link node
2024-07-15 23:06:03 +02:00
Marco Vinciguerra
830daee1f3
Update search_link_node.py
/ build (push) Has been cancelled
2024-07-15 20:47:09 +02:00
Marco Vinciguerra
57fdaf9e3a
create search_link_graph
/ build (push) Waiting to run
2024-07-15 12:36:11 +02:00
Marco Vinciguerra
cf3ab5564a
fix: search link node
2024-07-14 16:49:29 +02:00
Marco Vinciguerra
a7249685cb
removed rag node
2024-07-14 10:24:38 +02:00
Marco Vinciguerra
765b548c0e
Merge pull request #417 from ScrapeGraphAI/md_scraper_integration
...
feat: add integrations for markdown files
2024-07-04 19:42:53 +02:00
Marco Vinciguerra
8a52914871
Merge pull request #429 from ScrapeGraphAI/421-default-prompt-template-customization
...
421 default prompt template customization
2024-07-04 19:42:35 +02:00
Federico Aguzzi
104d8692d6
Merge branch 'pre/beta' into support
/ build (push) Has been cancelled
2024-07-04 18:36:34 +02:00
Marco Vinciguerra
3ee1743687
update prompts
/ build (3.10) (push) Has been cancelled
2024-07-04 14:22:45 +02:00
Marco Vinciguerra
ed2af51501
update the chunk size
/ build (3.10) (push) Waiting to run
2024-07-02 12:03:08 +02:00
Marco Vinciguerra
3bf5f570a8
feat: add integration for infos
2024-07-01 21:19:16 +02:00
AmosDinh
3f9bf5b4d4
fixed typo
2024-07-01 13:42:10 +00:00
Marco Vinciguerra
5fe694b6b4
feat: improve md prompt recognition
/ build (3.10) (push) Waiting to run
2024-06-30 18:10:00 +02:00
Marco Vinciguerra
30ca15ca28
Merge branch 'md_scraper_integration' into integration_markdown
/ build (3.10) (push) Has been cancelled
2024-06-30 16:58:37 +02:00
Marco Vinciguerra
2804434a9e
feat: add integrations for markdown files
/ build (3.10) (push) Waiting to run
2024-06-29 13:35:39 +02:00
Marco Vinciguerra
9b45ebcdcf
modify fetch node with no cut mode
2024-06-28 14:38:36 +02:00
Marco Vinciguerra
228a1de2be
add new force
2024-06-27 18:57:27 +02:00
Marco Vinciguerra
9917972c11
fixed request
/ build (3.10) (push) Waiting to run
2024-06-22 21:39:37 +02:00
Marco Vinciguerra
afd46ac77b
fixed generate_answer_node
/ build (3.10) (push) Waiting to run
2024-06-22 11:31:54 +02:00
Marco Vinciguerra
d1c3de777f
fixed a bug
2024-06-21 14:14:43 +02:00
Marco Vinciguerra
7af411aa99
add trigger
...
Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>
2024-06-21 13:36:27 +02:00
Marco Vinciguerra
2f02830c81
refactoring of fetch node
/ build (3.10) (push) Waiting to run
2024-06-20 13:44:42 +02:00
Marco Vinciguerra
23bc6332d0
fixed a bug
2024-06-19 21:46:31 +02:00
Marco Vinciguerra
8bb560a489
add convert function
2024-06-19 20:17:45 +02:00
Marco Vinciguerra
073d226723
feat: add new search engine avaiability and new tests
2024-06-18 14:35:13 +02:00
Marco Perini
4c8becc721
overwrite common params to affect nodes config
/ build (3.10) (push) Has been cancelled
Release / Build (push) Has been cancelled
Release / Release (push) Has been cancelled
2024-06-16 15:19:40 +02:00
Marco Vinciguerra
2419003999
fix: fix robot node
/ build (3.10) (push) Waiting to run
Release / Build (push) Waiting to run
Release / Release (push) Blocked by required conditions
2024-06-16 14:04:36 +02:00
Marco Perini
12f4386552
Merge branch 'pre/beta' into 349-problem-with-scrapegraphaigraphspdf_scraper_graphpy
2024-06-14 15:22:43 +02:00
Marco Perini
203de83405
fix(pdf): correctly read .pdf files
2024-06-14 15:20:30 +02:00
Marco Perini
91c5b5af43
fix(multi): updated multi pdf scraper with schema
2024-06-14 14:59:12 +02:00
Marco Vinciguerra
7a34562d50
refactoring of merging answers nodes
2024-06-13 21:41:54 +02:00
Marco Perini
283b61fafc
docs: better logging
2024-06-13 18:13:47 +02:00
Marco Vinciguerra
dc1340e302
Update generate_answer_pdf_node.py
2024-06-12 15:47:04 +02:00
Marco Vinciguerra
e45f159a31
enhanced performance and readibility
2024-06-12 14:59:10 +02:00
Marco Vinciguerra
e6c7940a57
feat: add Parse_Node
2024-06-12 12:29:14 +02:00
Marco Vinciguerra
5065aa00e1
Merge branch 'pre/beta' into supercoder-327
2024-06-12 12:12:15 +02:00
supercoder-dev
d0e300af72
Update fetch_node.py
2024-06-12 14:32:01 +05:30
Marco Perini
6b4cdf92b8
fix: common params
2024-06-12 03:06:21 +02:00
Marco Perini
ab00f23d85
fix(node): fixed generate answer node pydantic schema
2024-06-12 01:40:49 +02:00
Marco Perini
a10b060409
Merge pull request #361 from VinciGit00/multi_scraper_implementation
...
Multi scraper implementation
2024-06-12 00:49:18 +02:00
Marco Perini
5d692bff9e
feat(schema): merge scripts to follow pydantic schema
2024-06-12 00:48:08 +02:00
Marco Perini
893aaddc5c
Merge pull request #359 from VinciGit00/semchunk_integration
...
feat: add new chunking function
2024-06-11 23:16:00 +02:00
Marco Perini
589da1d695
Merge pull request #351 from VinciGit00/faiss_integration
...
Faiss integration
2024-06-11 23:15:23 +02:00
Marco Perini
c881f64209
fix(cache): correctly pass the node arguments and logging
2024-06-11 22:56:09 +02:00
Marco Vinciguerra
fa951b4c8b
Merge pull request #368 from stevenmichaelthomas/wait-for-network-idle
...
Add the ability to specify load state
2024-06-11 19:59:23 +02:00
Marco Vinciguerra
81f89d88be
Update search_link_node.py
2024-06-10 16:14:24 +02:00
Marco Vinciguerra
9326637cc3
Merge branch 'main' into pre/beta
2024-06-09 17:04:11 +02:00
Marco Vinciguerra
1981230e6f
add multi scraper integration
2024-06-08 12:13:18 +02:00
Marco Vinciguerra
e1f045b280
feat: add new chunking function
2024-06-08 11:44:09 +02:00
Marco Vinciguerra
543b48764a
add default folder for the cache
/ build (3.10) (push) Has been cancelled
2024-06-07 09:47:21 +02:00
Marco Vinciguerra
d79036149a
feat: add caching
/ build (3.10) (push) Waiting to run
2024-06-06 21:35:52 +02:00
Tin Do
d772453220
Refactor model_name attribute access in llm_model in robots_node.py
...
- Changed the access of model_name from dictionary-style to attribute-style in llm_model
to comply with langchain BaseChatModel.
- Updated the conditional and split operations accordingly.
2024-06-05 11:00:33 -04:00
Marco Perini
5c9843f141
fix(schema): fixed json output
2024-06-05 15:07:21 +02:00
Marco Vinciguerra
67d83cff46
fix: getter
2024-06-05 12:02:00 +02:00
Marco Vinciguerra
450fde601e
add get functions on the dictionary
2024-06-05 09:28:24 +02:00
Marco Vinciguerra
5d20186bf2
feat: add json as output
2024-06-05 09:20:20 +02:00
Marco Vinciguerra
1d38ed146a
fix: bug on generate_answer_node
2024-06-05 09:17:29 +02:00
Marco Vinciguerra
74fd530914
Merge branch 'pre/beta' into 332-pydantic-schema-validation
2024-06-05 09:05:19 +02:00
Marco Perini
376f758a76
feat(pydantic): added pydantic output schema
2024-06-04 23:07:49 +02:00
Marco Vinciguerra
7ed2fe8ef0
feat: add dynamic caching
2024-06-04 14:27:46 +02:00
Marco Vinciguerra
7a13a6819f
feat: refactoring of rag node
2024-06-04 12:01:21 +02:00
Marco Vinciguerra
244aada2de
feat: refactoring of an in if
2024-06-04 10:01:20 +02:00
Marco Vinciguerra
c8d556da4e
feat: fix an if
2024-06-03 21:49:34 +02:00
Marco Vinciguerra
8de720d379
feat: removed a bug
2024-06-03 21:45:37 +02:00
Marco Perini
79ace115c7
Merge pull request #323 from VinciGit00/refactoring-pdf_scraper
...
Refactoring pdf scraper and json scrape
2024-06-03 13:26:11 +02:00
Marco Vinciguerra
5cfc10178a
feat: add forcing format as json
2024-06-02 12:24:54 +02:00
Marco Vinciguerra
5bda918a39
feat: add json multiscraper
2024-06-01 12:04:19 +02:00
Marco Vinciguerra
f5cbd80c97
feat: add pdf scraper multi graph
2024-06-01 09:52:21 +02:00
Marco Vinciguerra
25352a56f9
Merge branch 'pre/beta' into temp
2024-05-31 23:08:24 +02:00
Marco Vinciguerra
930f673747
feat: removed rag node
2024-05-31 21:03:48 +02:00
Marco Vinciguerra
c4ce36111f
fix: typo in generate_screper_node
2024-05-30 18:50:04 +02:00
Johan Karlsson
4639f0cac5
fix: typo in prompt
2024-05-30 13:47:59 +02:00
Marco Perini
8d76c4b3cb
fix(schema): added schema
2024-05-26 10:51:48 +02:00
Marco Perini
5684578fab
fix(kg): removed unused nodes and utils
2024-05-25 00:44:14 +02:00
Marco Perini
a4ee757507
Merge branch 'pre/beta' into pdf_scraper_refactoring
2024-05-25 00:16:38 +02:00
Marco Perini
8d5eb0bb0d
fix(local_file): fixed textual input pdf, csv, json and xml graph
2024-05-25 00:13:47 +02:00
Marco Vinciguerra
b913b51cca
Merge branch 'logger-integration' into pre/beta
2024-05-24 12:39:14 +02:00
Federico Minutoli
c251cc45d3
fix(node-logging): use centralized logger in each node for logging
2024-05-24 01:09:49 +02:00
Federico Minutoli
4348d4f4db
fix(logger): set up centralized root logger in base node
2024-05-24 01:09:26 +02:00
Marco Vinciguerra
909af8d912
refactor gen answ node
2024-05-23 13:45:23 +02:00
Marco Vinciguerra
b377467b29
add info
2024-05-23 12:51:08 +02:00
Marco Vinciguerra
f00ed35f7b
Merge branch 'pre/beta' into patch-1
2024-05-23 11:47:51 +02:00
Marco Vinciguerra
f2dffe534f
fix: pdf scraper bug
2024-05-22 11:54:55 +02:00
Marco Vinciguerra
be4237a04d
Merge branch 'pre/beta' into multi_scraper_graph
2024-05-21 13:44:06 +02:00
Marco Perini
fc58e2d3a6
feat(smart-scraper-multi): add schema to graphs and created SmartScraperMultiGraph
2024-05-21 13:13:27 +02:00
JGalego
3ffa896c39
Fixed model ID -> model name conversion
2024-05-20 17:35:55 +01:00
JGalego
05ecc3ae20
Added missing logic to extract model_name from model_id
2024-05-20 17:17:11 +01:00
Marco Perini
a338383399
feat(kg): removed import
2024-05-18 10:30:29 +02:00
Marco Perini
58cc903d55
feat(multiple): quick fix working
2024-05-18 05:57:43 +02:00
Marco Vinciguerra
6f62b0560a
Merge branch 'multi_scraper_graph' of https://github.com/VinciGit00/Scrapegraph-ai into multi_scraper_graph
2024-05-18 02:42:27 +02:00
Marco Vinciguerra
b82f33aee7
fix: template names
2024-05-18 02:42:25 +02:00
Marco Perini
bed3eed50c
feat(multiple_search): working multiple example
2024-05-18 01:51:12 +02:00
Marco Vinciguerra
05e511e36f
add new prompts
2024-05-18 00:22:52 +02:00
Marco Perini
0196423bde
feat(knowledgegraph): add knowledge graph node
2024-05-17 23:41:44 +02:00
Marco Perini
8c33ea3fbc
feat(node): knowledge graph node
2024-05-17 18:40:58 +02:00
Marco Vinciguerra
3453f72397
add graph
2024-05-17 18:23:50 +02:00
Marco Vinciguerra
02745a4f63
Merge branch 'main' into pre/beta
2024-05-17 10:40:19 +02:00
VinciGit00
9e9f8f09d5
removed max depth
2024-05-16 12:39:08 +02:00
VinciGit00
9483afddd3
revert
2024-05-16 12:34:02 +02:00
Mayur Bhosale
1e0b2f7e70
Merge branch 'pre/beta' into nDeep
2024-05-15 23:36:23 +02:00
mayurdb
d60438cc89
Add a n-level deep search support
2024-05-15 23:32:15 +02:00
VinciGit00
ba8a4f7122
removed duplicates
2024-05-15 21:47:39 +02:00
VinciGit00
4fe58d9ba4
fix logger
2024-05-15 16:12:13 +02:00
VinciGit00
cc5adefd29
fix: come back to the old version
2024-05-15 15:54:00 +02:00
VinciGit00
29d284e497
Merge branch 'main' into logger-integration
2024-05-15 15:28:20 +02:00
Marco Vinciguerra
008f8d9eae
Merge pull request #248 from VinciGit00/main
...
reallignment
2024-05-15 15:20:53 +02:00
Marco Vinciguerra
cffcf80a75
Merge branch '88-blockscraper-implementation' into main
2024-05-15 15:20:12 +02:00
Marco Vinciguerra
22cd9e3605
Merge branch 'search_link_context' into main
2024-05-15 15:16:57 +02:00
mayurdb
0b71b9a1e8
Add a new graph traversal that allows more than one edges out of a graph
2024-05-15 13:46:19 +02:00
VinciGit00
a4700bfc75
add robot node
2024-05-15 12:00:32 +02:00
VinciGit00
05890835f5
refactoring of loggers
2024-05-15 10:54:53 +02:00
Marco Vinciguerra
932df8d491
Merge pull request #238 from VinciGit00/gpt4-omni
2024-05-14 17:09:23 +02:00
mayurdb
a458ec4b9f
Update the prompt for the search_link_node
2024-05-14 16:59:16 +02:00
Marco Perini
a6e1813ddd
fix(fetch_node): bug in handling local files
2024-05-14 16:51:10 +02:00
VinciGit00
e53766b16e
feat: add logger integration
2024-05-14 15:20:39 +02:00
Marco Perini
a296927624
feat(omni-scraper): working OmniScraperGraph with images
2024-05-14 13:46:49 +02:00
Marco Perini
90955ca52f
feat(gpt-4o): image to text single node test
2024-05-14 11:43:21 +02:00
Marco Perini
367dea5cbd
Merge branch 'pre/beta' into feat/parallel-node-execution
2024-05-13 23:50:40 +02:00
Marco Perini
a8d5e7db05
feat(batchsize): tested different batch sizes and systems
2024-05-13 23:49:48 +02:00
Marco Perini
dedc733047
fix(asyncio): replaced deepcopy with copy due to serialization problems
2024-05-13 18:46:34 +02:00
Marco Perini
0c1594737f
fix(fetch-node): removed isSoup from default
2024-05-13 12:09:55 +02:00
Marco Perini
7e8acd8e6a
Merge branch 'pre/beta' into fix/fetch-node-proxybroker
2024-05-13 11:17:37 +02:00
Marco Perini
1e9a564616
fix(proxy-rotation): removed duplicated arg and passed the loader_kwarhs correctly to the node
2024-05-12 18:39:03 +02:00
VinciGit00
e2350eda62
feat: add new prompt info
2024-05-12 11:14:30 +02:00
mayurdb
9a67a26cd3
Update documentation
2024-05-11 16:57:22 +05:30
mayurdb
df271b6451
Add search link node that can find out relevant links in the webpage
2024-05-11 16:39:55 +05:30
Marco Vinciguerra
b752499fab
Merge pull request #217 from mayurdb/fetchLinkFix
...
Fetch links in the page while parsing html
2024-05-11 09:42:40 +02:00
mayurdb
300fd5d253
Fetch links in the page while parsing html
2024-05-11 09:46:51 +05:30
Eric Page
0683e78e78
Merge branch 'pre/beta' into fix-GenerateScraperGraph
2024-05-11 01:59:28 +02:00
Eric Page
aac51ba290
Removed dead code, allows GenerateScraperNode to generate scraper with
...
one chunk of context
2024-05-11 01:34:51 +02:00
Eric Page
40884747c7
Added parse_html option in parse_node
2024-05-11 00:32:01 +02:00
Federico Minutoli
627cbeeb20
feat(parallel-exeuction): add asyncio event loop dispatcher with semaphore for parallel graph instances
...
TODO: still untested
2024-05-11 00:13:27 +02:00
Federico Minutoli
fc2aa3ac1c
Merge branch 'pre/beta' of https://github.com/DiTo97/Scrapegraph-ai into fix/fetch-node-proxybroker
2024-05-10 21:20:40 +02:00
Federico Minutoli
768719cce8
feat(safe-web-driver): enchanced the original AsyncChromiumLoader web driver with proxy protection and flexible kwargs and backend
...
the original class prevents passing kwargs down to the playwright backend, making some config unfeasible, including passing a proxy server to the web driver.
the new class has backward compatibility with the original, but 1) allows any kwarg to be passed down to the web driver, 2) allows specifying the web driver backend (only playwright is supported for now) in case more (e.g., selenium) will be supported in the future and 3) automatically fetches a suitable proxy if one is not passed already
2024-05-10 21:13:38 +02:00
Marco Perini
864aa91326
feat: revert fetch_node
2024-05-10 15:11:54 +02:00
Marco Vinciguerra
99adc9799f
Merge branch 'pre/beta' into fetchNodeFix
2024-05-10 11:13:54 +02:00
mayurdb
f8ce3d5916
fix: Augment the information getting fetched from a webpage
2024-05-10 13:28:53 +05:30
VinciGit00
0ab31c3fdb
fix: add json integration
2024-05-09 21:07:07 +02:00
VinciGit00
324e977b85
fix: fixed bugs for csv and xml
2024-05-09 20:46:46 +02:00
Shubham Kamboj
f10f3b1438
feat: Add support for passing pdf path as source
2024-05-09 21:55:05 +05:30
Marco Perini
6b71ec1d2b
fix(examples): local, mixed models and fixed SearchGraph embeddings problem
2024-05-08 15:36:26 +02:00
Marco Perini
186c0d035d
fix(examples): openai std examples
2024-05-08 14:56:44 +02:00
Marco Vinciguerra
b326886250
Merge branch '88-blockscraper-implementation' into asdt
2024-05-07 13:27:30 +02:00
VinciGit00
67d5fbf816
feat: new search_graph
2024-05-06 22:09:18 +02:00
VinciGit00
5a67bca0db
Merge branch 'pre/beta' into pr/161
2024-05-06 14:50:04 +02:00
VinciGit00
51aa109e42
feat: add turboscraper (alfa)
2024-05-06 11:59:14 +02:00
VinciGit00
5e1d5db6da
Update search_internet_node.py
2024-05-06 10:09:17 +02:00
VinciGit00
80053a2358
Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta
2024-05-06 10:07:56 +02:00
VinciGit00
389b52aabb
removed examples
2024-05-06 10:07:31 +02:00
VinciGit00
88d999e15b
add website content
2024-05-06 09:36:32 +02:00
Marco Perini
930adb38f2
feat(node): multiple url search in SearchGraph + fixes
...
Implemented GraphIteratorNode and MergeAnswersNode to create multiple istances of a graph and merge the scraped content from multiple pages
2024-05-06 00:30:09 +02:00
Marco Perini
dbb614a8dd
feat: multiple graph instances
2024-05-05 23:51:04 +02:00
Marco Perini
84fcb44aaa
feat: fixed custom_graphs example and robots_node
2024-05-05 22:02:24 +02:00
Lorenzo Padoan
e1b9d69360
dev basic class blockindentifier
2024-05-05 17:36:30 +02:00
Eric Page
cc27b21c3f
Merge branch 'pre/beta' into pass-common-params-graph
2024-05-05 16:18:32 +02:00
Eric Page
3ae2ea1dbd
Miscellaneous "llm" -> "llm_model" refactors
2024-05-05 15:58:50 +02:00
VinciGit00
cb1bd00a19
removed unused node
2024-05-05 09:40:35 +02:00
Eric Page
729d5d7597
Changed node_config["llm"] to node_config["llm_model"]
2024-05-05 09:36:11 +02:00
Eric Page
c2c61625b3
Corrected logic of update_config function in base_node.py
2024-05-05 06:37:31 +02:00
Eric Page
b7539d4725
Added update_config function to base_node.py
2024-05-05 06:25:43 +02:00
Shubham Kamboj
fd59f282a8
Bug fixes
2024-05-04 22:18:54 +05:30
Shubham Kamboj
d830d1371b
Resolve merge conflicts
2024-05-04 22:01:40 +05:30
Shubham Kamboj
7599234ab9
feat: Enable end users to pass model instances of HuggingFaceHub
2024-05-04 21:42:57 +05:30
Marco Perini
df293fdfd2
Merge pull request #139 from VinciGit00/pdf_scraper_graph_introduction
...
feat: add pdf scraper
2024-05-03 21:45:09 +02:00
VinciGit00
10a94530e3
feat: add pdf scraper
2024-05-03 21:42:02 +02:00
Marco Perini
aa9271e7bc
feat: Azure implementation + embeddings refactoring
2024-05-03 21:40:01 +02:00
Marco Vinciguerra
2abe05a0cf
Merge pull request #135 from S4mpl3r/feature
2024-05-03 16:49:07 +02:00
S4mpl3r
819cbcd3be
refactor: Changed the way embedding model is created in AbstractGraph class and removed handling of embedding model creation from RAGNode. Now AbstractGraph will call a dedicated method for embedding models instead of _create_llm. This makes it easy to use any LLM with any supported embedding model.
2024-05-03 16:14:27 +03:30
Marco Perini
cb1cb616b7
feat: add base_node to __init__.py
...
Useful for people to create custom nodes
2024-05-03 14:18:12 +02:00
Marco Vinciguerra
579a27faf9
Merge pull request #126 from VinciGit00/bedrock_support
...
feat: bedrock support
2024-05-02 17:57:20 +02:00
Marco Vinciguerra
a2d5c75f10
Merge branch 'pre/beta' into main
2024-05-02 17:55:21 +02:00
Shubham Kamboj
b86aac2188
feat: Allow end users to pass model instances for llm and embedding model
2024-05-02 20:09:23 +05:30
VinciGit00
16de49fa08
add integration for bedrock
...
Co-Authored-By: redrusty2 <15157208+redrusty2@users.noreply.github.com>
2024-05-02 13:47:17 +02:00
Marco Perini
0631985e61
docs: graphs and helpers docstrings
2024-05-02 00:23:38 +02:00
Marco Perini
1409797475
docs: refactor nodes docstrings
2024-05-01 23:17:57 +02:00
Marco Perini
4d542a88f7
feat: added node and graph for CSV scraping
2024-05-01 21:34:40 +02:00
Marco Perini
02d1af006c
feat: modified node name
2024-05-01 21:33:25 +02:00
Marco Perini
e9817963c8
docs: base and fetch node
2024-05-01 21:30:06 +02:00
VinciGit00
7c1b5796d7
add csv scraper
2024-05-01 17:20:04 +02:00
EURAC\marperini
2dd7817cfb
feat: added verbose flag to suppress print statements
2024-04-30 15:31:57 +02:00
Marco Perini
d592d27bb4
Merge pull request #115 from VinciGit00/101-scrape-json-files
...
feat: add xml scraper and json scraper
2024-04-30 14:29:57 +02:00
EURAC\marperini
42ab0aa1d2
feat(fetch): added playwright support
2024-04-30 04:02:58 +02:00
VinciGit00
deb920a33e
fixing json and example
2024-04-29 16:11:57 +02:00
VinciGit00
0b25d9a977
Update search_link_node.py
2024-04-29 15:49:46 +02:00
EURAC\marperini
6e7283ed8f
feat: add finalize_node()
2024-04-29 10:05:04 +02:00
PeriniM
7f1c3b7d83
fix: added missing dependecies
...
include python v3.9.0 and change deprecated method langchain
2024-04-28 20:29:23 +02:00
VinciGit00
9cd516507c
fix: bug with fetch node
2024-04-27 21:23:35 +02:00
VinciGit00
adbc08f27b
fix: robot node and proxyes
2024-04-27 19:07:37 +02:00
VinciGit00
b754dd909c
fix: changed proxy function
2024-04-27 14:40:50 +02:00
VinciGit00
f6077d1f98
feat: add new proxy rotation function
2024-04-27 13:31:53 +02:00
Marco Vinciguerra
922aa96b37
Merge branch 'pre/beta' into search_links_node
2024-04-26 19:23:42 +02:00
VinciGit00
a73496df9b
refactoring of nodes
2024-04-24 21:13:27 +02:00
VinciGit00
e68b707e0d
changed color and return type
2024-04-24 17:29:49 +02:00
EURAC\marperini
ae49dee985
fixed robots_node and add test
2024-04-24 16:12:53 +02:00
VinciGit00
9b9a9f204c
handling the agent model
2024-04-24 13:13:46 +02:00
VinciGit00
adf89e5e1b
add new info for robot node
2024-04-24 10:14:26 +02:00
VinciGit00
8ab603220d
changed node name
2024-04-24 10:02:09 +02:00
VinciGit00
726efb7f00
add new info
2024-04-23 20:51:39 +02:00
VinciGit00
8ef9dec2e2
add new node
2024-04-23 20:23:35 +02:00
VinciGit00
261ecbf3a5
fixed typos in the doc
2024-04-22 21:32:06 +02:00
VinciGit00
3960e56d29
fixed typos in the doc
2024-04-22 21:31:23 +02:00
VinciGit00
9e3dddcb44
add nodes that search all the links
2024-04-22 18:22:40 +02:00
EURAC\marperini
ac099e2763
fixed ollama embedder params
2024-04-21 01:21:09 +02:00
VinciGit00
13669cc1ab
revert to old naming
2024-04-19 13:43:51 +02:00
VinciGit00
a2349e56af
recatoring of nodes excluive for openai
2024-04-19 13:21:48 +02:00
VinciGit00
3f95801737
Merge branch 'main' of https://github.com/VinciGit00/Scrapegraph-ai
2024-04-19 10:36:40 +02:00
EURAC\marperini
c032131692
fixed chunk response
2024-04-17 15:20:45 +02:00
Marco Perini
b14527c08d
Merge branch 'main' into refactor_generate_answer_node
2024-04-17 12:08:43 +02:00
EURAC\marperini
c237e63631
removed unused variable
2024-04-17 12:05:51 +02:00
Marco Perini
edc439fd01
Merge pull request #64 from VinciGit00/generateScraperbranch
...
add generateScraperBranch
2024-04-17 11:51:06 +02:00
EURAC\marperini
b2d170cd86
refactored and fixed single chunk bug
2024-04-17 11:50:19 +02:00
EURAC\marperini
bdac2df4bb
fixed single chunk bug
2024-04-17 11:34:10 +02:00
Andrea Rota
b0e446f014
feat: apply remove to the document before updating the state
2024-04-17 11:24:56 +02:00
EURAC\marperini
d5f1c82c91
Merge remote-tracking branch 'origin/refactor_genrerate_answer_node'
2024-04-17 10:55:28 +02:00
