docs(scriptcreator): enhance documentation

This commit is contained in:
Marco Perini 2024-06-12 01:16:50 +02:00
parent a10b060409
commit 650c3aaa60
5 changed files with 81 additions and 6 deletions

Binary file not shown.

After

Width:  |  Height:  |  Size: 54 KiB

View File

@ -6,11 +6,15 @@ Graphs are scraping pipelines aimed at solving specific tasks. They are composed
There are several types of graphs available in the library, each with its own purpose and functionality. The most common ones are: There are several types of graphs available in the library, each with its own purpose and functionality. The most common ones are:
- **SmartScraperGraph**: one-page scraper that requires a user-defined prompt and a URL (or local file) to extract information using LLM. - **SmartScraperGraph**: one-page scraper that requires a user-defined prompt and a URL (or local file) to extract information using LLM.
- **SmartScraperMultiGraph**: multi-page scraper that requires a user-defined prompt and a list of URLs (or local files) to extract information using LLM. It is built on top of SmartScraperGraph.
- **SearchGraph**: multi-page scraper that only requires a user-defined prompt to extract information from a search engine using LLM. It is built on top of SmartScraperGraph. - **SearchGraph**: multi-page scraper that only requires a user-defined prompt to extract information from a search engine using LLM. It is built on top of SmartScraperGraph.
- **SpeechGraph**: text-to-speech pipeline that generates an answer as well as a requested audio file. It is built on top of SmartScraperGraph and requires a user-defined prompt and a URL (or local file). - **SpeechGraph**: text-to-speech pipeline that generates an answer as well as a requested audio file. It is built on top of SmartScraperGraph and requires a user-defined prompt and a URL (or local file).
- **ScriptCreatorGraph**: script generator that creates a Python script to scrape a website using the specified library (e.g. BeautifulSoup). It requires a user-defined prompt and a URL (or local file). - **ScriptCreatorGraph**: script generator that creates a Python script to scrape a website using the specified library (e.g. BeautifulSoup). It requires a user-defined prompt and a URL (or local file).
There are also two additional graphs that can handle multiple sources:
- **SmartScraperMultiGraph**: similar to `SmartScraperGraph`, but with the ability to handle multiple sources.
- **ScriptCreatorMultiGraph**: similar to `ScriptCreatorGraph`, but with the ability to handle multiple sources.
With the introduction of `GPT-4o`, two new powerful graphs have been created: With the introduction of `GPT-4o`, two new powerful graphs have been created:
- **OmniScraperGraph**: similar to `SmartScraperGraph`, but with the ability to scrape images and describe them. - **OmniScraperGraph**: similar to `SmartScraperGraph`, but with the ability to scrape images and describe them.
@ -186,4 +190,37 @@ It will fetch the data from the source, extract the information based on the pro
) )
result = speech_graph.run() result = speech_graph.run()
print(result) print(result)
ScriptCreatorGraph & ScriptCreatorMultiGraph
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. image:: ../../assets/scriptcreatorgraph.png
:align: center
:width: 90%
:alt: ScriptCreatorGraph
First we define the graph configuration, which includes the LLM model and other parameters.
Then we create an instance of the ScriptCreatorGraph class, passing the prompt, source, and configuration as arguments. Finally, we run the graph and print the result.
.. code-block:: python
from scrapegraphai.graphs import ScriptCreatorGraph
graph_config = {
"llm": {...},
"library": "beautifulsoup4"
}
script_creator_graph = ScriptCreatorGraph(
prompt="Create a Python script to scrape the projects.",
source="https://perinim.github.io/projects/",
config=graph_config,
schema=schema
)
result = script_creator_graph.run()
print(result)
**ScriptCreatorMultiGraph** is similar to ScriptCreatorGraph, but it can handle multiple sources. We define the graph configuration, create an instance of the ScriptCreatorMultiGraph class, and run the graph.

View File

@ -30,6 +30,9 @@ anyio==4.3.0
# via openai # via openai
# via starlette # via starlette
# via watchfiles # via watchfiles
async-timeout==4.0.3
# via aiohttp
# via langchain
attrs==23.2.0 attrs==23.2.0
# via aiohttp # via aiohttp
# via jsonschema # via jsonschema
@ -47,7 +50,8 @@ boto3==1.34.113
botocore==1.34.113 botocore==1.34.113
# via boto3 # via boto3
# via s3transfer # via s3transfer
burr==0.19.1 burr==0.22.1
# via burr
# via scrapegraphai # via scrapegraphai
cachetools==5.3.3 cachetools==5.3.3
# via google-auth # via google-auth
@ -63,6 +67,13 @@ click==8.1.7
# via streamlit # via streamlit
# via typer # via typer
# via uvicorn # via uvicorn
colorama==0.4.6
# via click
# via loguru
# via pytest
# via sphinx
# via tqdm
# via uvicorn
contourpy==1.2.1 contourpy==1.2.1
# via matplotlib # via matplotlib
cycler==0.12.1 cycler==0.12.1
@ -82,6 +93,9 @@ docutils==0.19
# via sphinx # via sphinx
email-validator==2.1.1 email-validator==2.1.1
# via fastapi # via fastapi
exceptiongroup==1.2.1
# via anyio
# via pytest
faiss-cpu==1.8.0 faiss-cpu==1.8.0
# via scrapegraphai # via scrapegraphai
fastapi==0.111.0 fastapi==0.111.0
@ -136,6 +150,7 @@ graphviz==0.20.3
# via scrapegraphai # via scrapegraphai
greenlet==3.0.3 greenlet==3.0.3
# via playwright # via playwright
# via sqlalchemy
groq==0.8.0 groq==0.8.0
# via langchain-groq # via langchain-groq
grpcio==1.64.0 grpcio==1.64.0
@ -170,6 +185,10 @@ idna==3.7
# via yarl # via yarl
imagesize==1.4.1 imagesize==1.4.1
# via sphinx # via sphinx
importlib-metadata==7.1.0
# via sphinx
importlib-resources==6.4.0
# via matplotlib
iniconfig==2.0.0 iniconfig==2.0.0
# via pytest # via pytest
jinja2==3.1.4 jinja2==3.1.4
@ -430,6 +449,8 @@ tokenizers==0.19.1
# via anthropic # via anthropic
toml==0.10.2 toml==0.10.2
# via streamlit # via streamlit
tomli==2.0.1
# via pytest
toolz==0.12.1 toolz==0.12.1
# via altair # via altair
tornado==6.4 tornado==6.4
@ -443,7 +464,9 @@ tqdm==4.66.4
typer==0.12.3 typer==0.12.3
# via fastapi-cli # via fastapi-cli
typing-extensions==4.12.0 typing-extensions==4.12.0
# via altair
# via anthropic # via anthropic
# via anyio
# via fastapi # via fastapi
# via fastapi-pagination # via fastapi-pagination
# via google-generativeai # via google-generativeai
@ -455,9 +478,11 @@ typing-extensions==4.12.0
# via pyee # via pyee
# via sf-hamilton # via sf-hamilton
# via sqlalchemy # via sqlalchemy
# via starlette
# via streamlit # via streamlit
# via typer # via typer
# via typing-inspect # via typing-inspect
# via uvicorn
typing-inspect==0.9.0 typing-inspect==0.9.0
# via dataclasses-json # via dataclasses-json
# via sf-hamilton # via sf-hamilton
@ -475,11 +500,16 @@ urllib3==1.26.18
uvicorn==0.29.0 uvicorn==0.29.0
# via burr # via burr
# via fastapi # via fastapi
uvloop==0.19.0 watchdog==4.0.1
# via uvicorn # via streamlit
watchfiles==0.21.0 watchfiles==0.21.0
# via uvicorn # via uvicorn
websockets==12.0 websockets==12.0
# via uvicorn # via uvicorn
win32-setctime==1.1.0
# via loguru
yarl==1.9.4 yarl==1.9.4
# via aiohttp # via aiohttp
zipp==3.19.2
# via importlib-metadata
# via importlib-resources

View File

@ -22,6 +22,9 @@ anyio==4.3.0
# via groq # via groq
# via httpx # via httpx
# via openai # via openai
async-timeout==4.0.3
# via aiohttp
# via langchain
attrs==23.2.0 attrs==23.2.0
# via aiohttp # via aiohttp
beautifulsoup4==4.12.3 beautifulsoup4==4.12.3
@ -40,6 +43,8 @@ certifi==2024.2.2
# via requests # via requests
charset-normalizer==3.3.2 charset-normalizer==3.3.2
# via requests # via requests
colorama==0.4.6
# via tqdm
dataclasses-json==0.6.6 dataclasses-json==0.6.6
# via langchain # via langchain
# via langchain-community # via langchain-community
@ -49,6 +54,8 @@ distro==1.9.0
# via anthropic # via anthropic
# via groq # via groq
# via openai # via openai
exceptiongroup==1.2.1
# via anyio
faiss-cpu==1.8.0 faiss-cpu==1.8.0
# via scrapegraphai # via scrapegraphai
filelock==3.14.0 filelock==3.14.0
@ -87,6 +94,7 @@ graphviz==0.20.3
# via scrapegraphai # via scrapegraphai
greenlet==3.0.3 greenlet==3.0.3
# via playwright # via playwright
# via sqlalchemy
groq==0.8.0 groq==0.8.0
# via langchain-groq # via langchain-groq
grpcio==1.64.0 grpcio==1.64.0
@ -270,6 +278,7 @@ tqdm==4.66.4
# via semchunk # via semchunk
typing-extensions==4.12.0 typing-extensions==4.12.0
# via anthropic # via anthropic
# via anyio
# via google-generativeai # via google-generativeai
# via groq # via groq
# via huggingface-hub # via huggingface-hub

View File

@ -16,6 +16,5 @@ free-proxy==1.1.1
langchain-groq==0.1.3 langchain-groq==0.1.3
playwright==1.43.0 playwright==1.43.0
langchain-aws==0.1.2 langchain-aws==0.1.2
yahoo-search-py==0.3
undetected-playwright==0.3.0 undetected-playwright==0.3.0
semchunk==1.0.1 semchunk==1.0.1