mirror of
https://github.com/VinciGit00/Scrapegraph-ai.git
synced 2026-07-12 21:01:56 +08:00
docs(scriptcreator): enhance documentation
This commit is contained in:
Marco Perini
parent
a10b060409
commit
650c3aaa60
BIN
docs/assets/scriptcreatorgraph.png
Normal file
BIN
docs/assets/scriptcreatorgraph.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
@ -6,11 +6,15 @@ Graphs are scraping pipelines aimed at solving specific tasks. They are composed
|
||||
There are several types of graphs available in the library, each with its own purpose and functionality. The most common ones are:
|
||||
|
||||
- **SmartScraperGraph**: one-page scraper that requires a user-defined prompt and a URL (or local file) to extract information using LLM.
|
||||
- **SmartScraperMultiGraph**: multi-page scraper that requires a user-defined prompt and a list of URLs (or local files) to extract information using LLM. It is built on top of SmartScraperGraph.
|
||||
- **SearchGraph**: multi-page scraper that only requires a user-defined prompt to extract information from a search engine using LLM. It is built on top of SmartScraperGraph.
|
||||
- **SpeechGraph**: text-to-speech pipeline that generates an answer as well as a requested audio file. It is built on top of SmartScraperGraph and requires a user-defined prompt and a URL (or local file).
|
||||
- **ScriptCreatorGraph**: script generator that creates a Python script to scrape a website using the specified library (e.g. BeautifulSoup). It requires a user-defined prompt and a URL (or local file).
|
||||
|
||||
There are also two additional graphs that can handle multiple sources:
|
||||
|
||||
- **SmartScraperMultiGraph**: similar to `SmartScraperGraph`, but with the ability to handle multiple sources.
|
||||
- **ScriptCreatorMultiGraph**: similar to `ScriptCreatorGraph`, but with the ability to handle multiple sources.
|
||||
|
||||
With the introduction of `GPT-4o`, two new powerful graphs have been created:
|
||||
|
||||
- **OmniScraperGraph**: similar to `SmartScraperGraph`, but with the ability to scrape images and describe them.
|
||||
@ -186,4 +190,37 @@ It will fetch the data from the source, extract the information based on the pro
|
||||
)
|
||||
|
||||
result = speech_graph.run()
|
||||
print(result)
|
||||
print(result)
|
||||
|
||||
|
||||
ScriptCreatorGraph & ScriptCreatorMultiGraph
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
.. image:: ../../assets/scriptcreatorgraph.png
|
||||
:align: center
|
||||
:width: 90%
|
||||
:alt: ScriptCreatorGraph
|
||||
|
||||
First we define the graph configuration, which includes the LLM model and other parameters.
|
||||
Then we create an instance of the ScriptCreatorGraph class, passing the prompt, source, and configuration as arguments. Finally, we run the graph and print the result.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from scrapegraphai.graphs import ScriptCreatorGraph
|
||||
|
||||
graph_config = {
|
||||
"llm": {...},
|
||||
"library": "beautifulsoup4"
|
||||
}
|
||||
|
||||
script_creator_graph = ScriptCreatorGraph(
|
||||
prompt="Create a Python script to scrape the projects.",
|
||||
source="https://perinim.github.io/projects/",
|
||||
config=graph_config,
|
||||
schema=schema
|
||||
)
|
||||
|
||||
result = script_creator_graph.run()
|
||||
print(result)
|
||||
|
||||
**ScriptCreatorMultiGraph** is similar to ScriptCreatorGraph, but it can handle multiple sources. We define the graph configuration, create an instance of the ScriptCreatorMultiGraph class, and run the graph.
|
||||
|
||||
@ -30,6 +30,9 @@ anyio==4.3.0
|
||||
# via openai
|
||||
# via starlette
|
||||
# via watchfiles
|
||||
async-timeout==4.0.3
|
||||
# via aiohttp
|
||||
# via langchain
|
||||
attrs==23.2.0
|
||||
# via aiohttp
|
||||
# via jsonschema
|
||||
@ -47,7 +50,8 @@ boto3==1.34.113
|
||||
botocore==1.34.113
|
||||
# via boto3
|
||||
# via s3transfer
|
||||
burr==0.19.1
|
||||
burr==0.22.1
|
||||
# via burr
|
||||
# via scrapegraphai
|
||||
cachetools==5.3.3
|
||||
# via google-auth
|
||||
@ -63,6 +67,13 @@ click==8.1.7
|
||||
# via streamlit
|
||||
# via typer
|
||||
# via uvicorn
|
||||
colorama==0.4.6
|
||||
# via click
|
||||
# via loguru
|
||||
# via pytest
|
||||
# via sphinx
|
||||
# via tqdm
|
||||
# via uvicorn
|
||||
contourpy==1.2.1
|
||||
# via matplotlib
|
||||
cycler==0.12.1
|
||||
@ -82,6 +93,9 @@ docutils==0.19
|
||||
# via sphinx
|
||||
email-validator==2.1.1
|
||||
# via fastapi
|
||||
exceptiongroup==1.2.1
|
||||
# via anyio
|
||||
# via pytest
|
||||
faiss-cpu==1.8.0
|
||||
# via scrapegraphai
|
||||
fastapi==0.111.0
|
||||
@ -136,6 +150,7 @@ graphviz==0.20.3
|
||||
# via scrapegraphai
|
||||
greenlet==3.0.3
|
||||
# via playwright
|
||||
# via sqlalchemy
|
||||
groq==0.8.0
|
||||
# via langchain-groq
|
||||
grpcio==1.64.0
|
||||
@ -170,6 +185,10 @@ idna==3.7
|
||||
# via yarl
|
||||
imagesize==1.4.1
|
||||
# via sphinx
|
||||
importlib-metadata==7.1.0
|
||||
# via sphinx
|
||||
importlib-resources==6.4.0
|
||||
# via matplotlib
|
||||
iniconfig==2.0.0
|
||||
# via pytest
|
||||
jinja2==3.1.4
|
||||
@ -430,6 +449,8 @@ tokenizers==0.19.1
|
||||
# via anthropic
|
||||
toml==0.10.2
|
||||
# via streamlit
|
||||
tomli==2.0.1
|
||||
# via pytest
|
||||
toolz==0.12.1
|
||||
# via altair
|
||||
tornado==6.4
|
||||
@ -443,7 +464,9 @@ tqdm==4.66.4
|
||||
typer==0.12.3
|
||||
# via fastapi-cli
|
||||
typing-extensions==4.12.0
|
||||
# via altair
|
||||
# via anthropic
|
||||
# via anyio
|
||||
# via fastapi
|
||||
# via fastapi-pagination
|
||||
# via google-generativeai
|
||||
@ -455,9 +478,11 @@ typing-extensions==4.12.0
|
||||
# via pyee
|
||||
# via sf-hamilton
|
||||
# via sqlalchemy
|
||||
# via starlette
|
||||
# via streamlit
|
||||
# via typer
|
||||
# via typing-inspect
|
||||
# via uvicorn
|
||||
typing-inspect==0.9.0
|
||||
# via dataclasses-json
|
||||
# via sf-hamilton
|
||||
@ -475,11 +500,16 @@ urllib3==1.26.18
|
||||
uvicorn==0.29.0
|
||||
# via burr
|
||||
# via fastapi
|
||||
uvloop==0.19.0
|
||||
# via uvicorn
|
||||
watchdog==4.0.1
|
||||
# via streamlit
|
||||
watchfiles==0.21.0
|
||||
# via uvicorn
|
||||
websockets==12.0
|
||||
# via uvicorn
|
||||
win32-setctime==1.1.0
|
||||
# via loguru
|
||||
yarl==1.9.4
|
||||
# via aiohttp
|
||||
zipp==3.19.2
|
||||
# via importlib-metadata
|
||||
# via importlib-resources
|
||||
|
||||
@ -22,6 +22,9 @@ anyio==4.3.0
|
||||
# via groq
|
||||
# via httpx
|
||||
# via openai
|
||||
async-timeout==4.0.3
|
||||
# via aiohttp
|
||||
# via langchain
|
||||
attrs==23.2.0
|
||||
# via aiohttp
|
||||
beautifulsoup4==4.12.3
|
||||
@ -40,6 +43,8 @@ certifi==2024.2.2
|
||||
# via requests
|
||||
charset-normalizer==3.3.2
|
||||
# via requests
|
||||
colorama==0.4.6
|
||||
# via tqdm
|
||||
dataclasses-json==0.6.6
|
||||
# via langchain
|
||||
# via langchain-community
|
||||
@ -49,6 +54,8 @@ distro==1.9.0
|
||||
# via anthropic
|
||||
# via groq
|
||||
# via openai
|
||||
exceptiongroup==1.2.1
|
||||
# via anyio
|
||||
faiss-cpu==1.8.0
|
||||
# via scrapegraphai
|
||||
filelock==3.14.0
|
||||
@ -87,6 +94,7 @@ graphviz==0.20.3
|
||||
# via scrapegraphai
|
||||
greenlet==3.0.3
|
||||
# via playwright
|
||||
# via sqlalchemy
|
||||
groq==0.8.0
|
||||
# via langchain-groq
|
||||
grpcio==1.64.0
|
||||
@ -270,6 +278,7 @@ tqdm==4.66.4
|
||||
# via semchunk
|
||||
typing-extensions==4.12.0
|
||||
# via anthropic
|
||||
# via anyio
|
||||
# via google-generativeai
|
||||
# via groq
|
||||
# via huggingface-hub
|
||||
|
||||
@ -16,6 +16,5 @@ free-proxy==1.1.1
|
||||
langchain-groq==0.1.3
|
||||
playwright==1.43.0
|
||||
langchain-aws==0.1.2
|
||||
yahoo-search-py==0.3
|
||||
undetected-playwright==0.3.0
|
||||
semchunk==1.0.1
|
||||
Loading…
Reference in New Issue
Block a user