mirror of
https://github.com/VinciGit00/Scrapegraph-ai.git
synced 2026-07-09 21:19:20 +08:00
Marco Perini
GitHub
commit
220156e996
26
.github/update-requirements.yml
vendored
26
.github/update-requirements.yml
vendored
@ -1,26 +0,0 @@
|
||||
name: Update requirements
|
||||
on:
|
||||
push:
|
||||
paths:
|
||||
- 'pyproject.toml'
|
||||
- '.github/workflows/update-requirements.yml'
|
||||
|
||||
jobs:
|
||||
update:
|
||||
name: Update requirements
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Install the latest version of rye
|
||||
uses: eifinger/setup-rye@v3
|
||||
- name: Build app
|
||||
run: rye run update-requirements
|
||||
commit:
|
||||
name: Commit changes
|
||||
run: |
|
||||
git config --global user.name 'github-actions'
|
||||
git config --global user.email 'github-actions[bot]@users.noreply.github.com'
|
||||
git add .
|
||||
git commit -m "ci: update requirements.txt [skip ci]"
|
||||
git push
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
32
.github/workflows/python-publish.yml
vendored
32
.github/workflows/python-publish.yml
vendored
@ -1,32 +0,0 @@
|
||||
# This workflow will upload a Python Package using Twine when a release is created
|
||||
# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
|
||||
|
||||
name: Upload Python Package
|
||||
|
||||
on:
|
||||
release:
|
||||
types: [published]
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.x'
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install setuptools wheel twine
|
||||
- name: Build and publish
|
||||
env:
|
||||
TWINE_USERNAME: mvincig11
|
||||
TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
|
||||
run: |
|
||||
git fetch --all --tags
|
||||
python setup.py sdist bdist_wheel
|
||||
twine upload dist/*
|
||||
19
CHANGELOG.md
19
CHANGELOG.md
@ -1,8 +1,24 @@
|
||||
## [1.34.0-beta.16](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.34.0-beta.15...v1.34.0-beta.16) (2025-01-06)
|
||||
## [1.34.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.34.0...v1.34.1) (2025-01-04)
|
||||
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* add back poethepoet for pylint ([a82af04](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/a82af04afed2e4ba309b5e98b5df351d9b79ca2e))
|
||||
* better playwright installation handling ([f6009d1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f6009d1abf9e2c83999de0c9b03a41aa1bf8f2a4))
|
||||
* disallow mailto: ([#861](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/861)) ([8d9c909](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8d9c909923dff1c247c85099db20e2a6dabb93f5))
|
||||
* removed requirements files ([25861b0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/25861b04be8a6fc60c900a46033aed91d1fef1f9))
|
||||
* selenium import in ChromiumLoader ([e374e05](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e374e055d64b7fa4c5a4c7694384dd15e6361bbd))
|
||||
|
||||
|
||||
### chore
|
||||
|
||||
* chromium browser asnc handling ([5be7c49](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5be7c497cd44fbd0c026bf3d833f572b34661b08))
|
||||
* made some libs optional ([5cdf055](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5cdf0550fe9dcd519d274bb343cf65c845e8a608))
|
||||
* pandas package is now optional ([54c69a2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/54c69a2b0b1677286b840be95ce482bcee881413))
|
||||
|
||||
## [1.34.0-beta.15](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.34.0-beta.14...v1.34.0-beta.15) (2025-01-03)
|
||||
* add new models ([72684a9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/72684a9476e255d5e20550f82daf3e7462fb8f5a))
|
||||
|
||||
## [1.34.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.33.11...v1.34.0) (2025-01-03)
|
||||
@ -14,8 +30,11 @@
|
||||
* added scrolling method to chromium docloader ([1c8b910](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1c8b910562112947a357277bca9dc81619b72e61))
|
||||
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
|
||||
* search graph ([d4b2679](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d4b26796d94d314af135d2d1bbd538e1d4be7593))
|
||||
* added license-files = [ ([9150e4c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/9150e4c95fa468afe9ddda3f1278b5037a2d0f38))
|
||||
* added twine ([df07da9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/df07da9bcc59cbccf1c45d69e3a3e904eaed565b))
|
||||
* build config ([b186a4f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b186a4f1c73fe29fa706158cc3c61812d6b16343))
|
||||
|
||||
113
README.md
113
README.md
@ -24,21 +24,6 @@ Just say which information you want to extract and the library will do it for yo
|
||||
<img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/sgai-hero.png" alt="ScrapeGraphAI Hero" style="width: 100%;">
|
||||
</p>
|
||||
|
||||
## 🔗 ScrapeGraph API & SDKs
|
||||
If you are looking for a quick solution to integrate ScrapeGraph in your system, check out our powerful API [here!](https://dashboard.scrapegraphai.com/login)
|
||||
|
||||
<p align="center">
|
||||
<img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/api-banner.png" alt="ScrapeGraph API Banner" style="width: 100%;">
|
||||
</p>
|
||||
|
||||
We offer SDKs in both Python and Node.js, making it easy to integrate into your projects. Check them out below:
|
||||
|
||||
| SDK | Language | GitHub Link |
|
||||
|-----------|----------|-----------------------------------------------------------------------------|
|
||||
| Python SDK | Python | [scrapegraph-py](https://github.com/ScrapeGraphAI/scrapegraph-sdk/tree/main/scrapegraph-py) |
|
||||
| Node.js SDK | Node.js | [scrapegraph-js](https://github.com/ScrapeGraphAI/scrapegraph-sdk/tree/main/scrapegraph-js) |
|
||||
|
||||
The Official API Documentation can be found [here](https://docs.scrapegraphai.com/).
|
||||
|
||||
## 🚀 Quick install
|
||||
|
||||
@ -47,35 +32,12 @@ The reference page for Scrapegraph-ai is available on the official page of PyPI:
|
||||
```bash
|
||||
pip install scrapegraphai
|
||||
|
||||
# IMPORTANT (to fetch websites content)
|
||||
playwright install
|
||||
```
|
||||
|
||||
**Note**: it is recommended to install the library in a virtual environment to avoid conflicts with other libraries 🐱
|
||||
|
||||
<details>
|
||||
<summary><b>Optional Dependencies</b></summary>
|
||||
Additional dependecies can be added while installing the library:
|
||||
|
||||
- <b>More Language Models</b>: additional language models are installed, such as Fireworks, Groq, Anthropic, Hugging Face, and Nvidia AI Endpoints.
|
||||
|
||||
This group allows you to use additional language models like Fireworks, Groq, Anthropic, Together AI, Hugging Face, and Nvidia AI Endpoints.
|
||||
```bash
|
||||
pip install scrapegraphai[other-language-models]
|
||||
```
|
||||
- <b>Semantic Options</b>: this group includes tools for advanced semantic processing, such as Graphviz.
|
||||
|
||||
```bash
|
||||
pip install scrapegraphai[more-semantic-options]
|
||||
```
|
||||
|
||||
- <b>Browsers Options</b>: this group includes additional browser management tools/services, such as Browserbase.
|
||||
|
||||
```bash
|
||||
pip install scrapegraphai[more-browser-options]
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
## 💻 Usage
|
||||
There are multiple standard scraping pipelines that can be used to extract information from a website (or local file).
|
||||
@ -84,13 +46,12 @@ The most common one is the `SmartScraperGraph`, which extracts information from
|
||||
|
||||
|
||||
```python
|
||||
import json
|
||||
from scrapegraphai.graphs import SmartScraperGraph
|
||||
|
||||
# Define the configuration for the scraping pipeline
|
||||
graph_config = {
|
||||
"llm": {
|
||||
"api_key": "YOUR_OPENAI_APIKEY",
|
||||
"api_key": "YOUR_OPENAI_API_KEY",
|
||||
"model": "openai/gpt-4o-mini",
|
||||
},
|
||||
"verbose": True,
|
||||
@ -99,33 +60,45 @@ graph_config = {
|
||||
|
||||
# Create the SmartScraperGraph instance
|
||||
smart_scraper_graph = SmartScraperGraph(
|
||||
prompt="Extract me all the news from the website",
|
||||
source="https://www.wired.com",
|
||||
prompt="Extract useful information from the webpage, including a description of what the company does, founders and social media links",
|
||||
source="https://scrapegraphai.com/",
|
||||
config=graph_config
|
||||
)
|
||||
|
||||
# Run the pipeline
|
||||
result = smart_scraper_graph.run()
|
||||
|
||||
import json
|
||||
print(json.dumps(result, indent=4))
|
||||
```
|
||||
|
||||
The output will be a dictionary like the following:
|
||||
|
||||
```python
|
||||
"result": {
|
||||
"news": [
|
||||
{
|
||||
"title": "The New Jersey Drone Mystery May Not Actually Be That Mysterious",
|
||||
"link": "https://www.wired.com/story/new-jersey-drone-mystery-maybe-not-drones/",
|
||||
"author": "Lily Hay Newman"
|
||||
},
|
||||
{
|
||||
"title": "Former ByteDance Intern Accused of Sabotage Among Winners of Prestigious AI Award",
|
||||
"link": "https://www.wired.com/story/bytedance-intern-best-paper-neurips/",
|
||||
"author": "Louise Matsakis"
|
||||
},
|
||||
...
|
||||
]
|
||||
{
|
||||
"description": "ScrapeGraphAI transforms websites into clean, organized data for AI agents and data analytics. It offers an AI-powered API for effortless and cost-effective data extraction.",
|
||||
"founders": [
|
||||
{
|
||||
"name": "Marco Perini",
|
||||
"role": "Founder & Technical Lead",
|
||||
"linkedin": "https://www.linkedin.com/in/perinim/"
|
||||
},
|
||||
{
|
||||
"name": "Marco Vinciguerra",
|
||||
"role": "Founder & Software Engineer",
|
||||
"linkedin": "https://www.linkedin.com/in/marco-vinciguerra-7ba365242/"
|
||||
},
|
||||
{
|
||||
"name": "Lorenzo Padoan",
|
||||
"role": "Founder & Product Engineer",
|
||||
"linkedin": "https://www.linkedin.com/in/lorenzo-padoan-4521a2154/"
|
||||
}
|
||||
],
|
||||
"social_media_links": {
|
||||
"linkedin": "https://www.linkedin.com/company/101881123",
|
||||
"twitter": "https://x.com/scrapegraphai",
|
||||
"github": "https://github.com/ScrapeGraphAI/Scrapegraph-ai"
|
||||
}
|
||||
}
|
||||
```
|
||||
There are other pipelines that can be used to extract information from multiple pages, generate Python scripts, or even generate audio files.
|
||||
@ -145,20 +118,30 @@ It is possible to use different LLM through APIs, such as **OpenAI**, **Groq**,
|
||||
|
||||
Remember to have [Ollama](https://ollama.com/) installed and download the models using the **ollama pull** command, if you want to use local models.
|
||||
|
||||
## 🔍 Demo
|
||||
Official streamlit demo:
|
||||
|
||||
[](https://scrapegraph-demo-demo.streamlit.app)
|
||||
|
||||
Try it directly on the web using Google Colab:
|
||||
|
||||
[](https://colab.research.google.com/drive/1sEZBonBMGP44CtO6GQTwAlL0BGJXjtfd?usp=sharing)
|
||||
|
||||
## 📖 Documentation
|
||||
|
||||
[](https://colab.research.google.com/drive/1sEZBonBMGP44CtO6GQTwAlL0BGJXjtfd?usp=sharing)
|
||||
|
||||
The documentation for ScrapeGraphAI can be found [here](https://scrapegraph-ai.readthedocs.io/en/latest/).
|
||||
Check out also the Docusaurus [here](https://docs-oss.scrapegraphai.com/).
|
||||
|
||||
## 🔗 ScrapeGraph API & SDKs
|
||||
If you are looking for a quick solution to integrate ScrapeGraph in your system, check out our powerful API [here!](https://dashboard.scrapegraphai.com/login)
|
||||
|
||||
<p align="center">
|
||||
<img src="https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/api-banner.png" alt="ScrapeGraph API Banner" style="width: 100%;">
|
||||
</p>
|
||||
|
||||
We offer SDKs in both Python and Node.js, making it easy to integrate into your projects. Check them out below:
|
||||
|
||||
| SDK | Language | GitHub Link |
|
||||
|-----------|----------|-----------------------------------------------------------------------------|
|
||||
| Python SDK | Python | [scrapegraph-py](https://github.com/ScrapeGraphAI/scrapegraph-sdk/tree/main/scrapegraph-py) |
|
||||
| Node.js SDK | Node.js | [scrapegraph-js](https://github.com/ScrapeGraphAI/scrapegraph-sdk/tree/main/scrapegraph-js) |
|
||||
|
||||
The Official API Documentation can be found [here](https://docs.scrapegraphai.com/).
|
||||
|
||||
## 🏆 Sponsors
|
||||
<div style="text-align: center;">
|
||||
<a href="https://2ly.link/1zaXG">
|
||||
|
||||
@ -31,31 +31,6 @@ playwright install
|
||||
|
||||
**Not**: Diğer kütüphanelerle çakışmaları önlemek için kütüphaneyi sanal bir ortamda kurmanız önerilir 🐱
|
||||
|
||||
<details>
|
||||
<summary><b>Opsiyonel Bağımlılıklar</b></summary>
|
||||
Kütüphaneyi kurarken ek bağımlılıklar ekleyebilirsiniz:
|
||||
|
||||
- **Daha Fazla Dil Modeli**: Fireworks, Groq, Anthropic, Hugging Face ve Nvidia AI Endpoints gibi ek dil modelleri kurulur.
|
||||
|
||||
Bu grup, Fireworks, Groq, Anthropic, Together AI, Hugging Face ve Nvidia AI Endpoints gibi ek dil modellerini kullanmanızı sağlar.
|
||||
|
||||
```bash
|
||||
pip install scrapegraphai[other-language-models]
|
||||
```
|
||||
|
||||
- **Semantik Seçenekler**: Graphviz gibi gelişmiş semantik işleme araçlarını içerir.
|
||||
|
||||
```bash
|
||||
pip install scrapegraphai[more-semantic-options]
|
||||
```
|
||||
|
||||
- **Tarayıcı Seçenekleri**: Browserbase gibi ek tarayıcı yönetim araçları/hizmetlerini içerir.
|
||||
|
||||
```bash
|
||||
pip install scrapegraphai[more-browser-options]
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## 💻 Kullanım
|
||||
|
||||
|
||||
@ -3,9 +3,8 @@ Basic example of scraping pipeline using CSVScraperGraph from CSV documents
|
||||
"""
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
import pandas as pd
|
||||
from scrapegraphai.graphs import CSVScraperGraph
|
||||
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
|
||||
from scrapegraphai.utils import prettify_exec_info
|
||||
|
||||
load_dotenv()
|
||||
|
||||
@ -17,7 +16,8 @@ FILE_NAME = "inputs/username.csv"
|
||||
curr_dir = os.path.dirname(os.path.realpath(__file__))
|
||||
file_path = os.path.join(curr_dir, FILE_NAME)
|
||||
|
||||
text = pd.read_csv(file_path)
|
||||
with open(file_path, 'r') as file:
|
||||
text = file.read()
|
||||
|
||||
# ************************************************
|
||||
# Define the configuration for the graph
|
||||
@ -41,7 +41,7 @@ graph_config = {
|
||||
|
||||
csv_scraper_graph = CSVScraperGraph(
|
||||
prompt="List me all the last names",
|
||||
source=str(text), # Pass the content of the file, not the file object
|
||||
source=text, # Pass the content of the file
|
||||
config=graph_config
|
||||
)
|
||||
|
||||
@ -53,8 +53,4 @@ print(result)
|
||||
# ************************************************
|
||||
|
||||
graph_exec_info = csv_scraper_graph.get_execution_info()
|
||||
print(prettify_exec_info(graph_exec_info))
|
||||
|
||||
# Save to json or csv
|
||||
convert_to_csv(result, "result")
|
||||
convert_to_json(result, "result")
|
||||
print(prettify_exec_info(graph_exec_info))
|
||||
@ -3,9 +3,8 @@ Basic example of scraping pipeline using CSVScraperMultiGraph from CSV documents
|
||||
"""
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
import pandas as pd
|
||||
from scrapegraphai.graphs import CSVScraperMultiGraph
|
||||
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
|
||||
from scrapegraphai.utils import prettify_exec_info
|
||||
|
||||
load_dotenv()
|
||||
# ************************************************
|
||||
@ -16,7 +15,8 @@ FILE_NAME = "inputs/username.csv"
|
||||
curr_dir = os.path.dirname(os.path.realpath(__file__))
|
||||
file_path = os.path.join(curr_dir, FILE_NAME)
|
||||
|
||||
text = pd.read_csv(file_path)
|
||||
with open(file_path, 'r') as file:
|
||||
text = file.read()
|
||||
|
||||
# ************************************************
|
||||
# Define the configuration for the graph
|
||||
@ -48,7 +48,3 @@ print(result)
|
||||
|
||||
graph_exec_info = csv_scraper_graph.get_execution_info()
|
||||
print(prettify_exec_info(graph_exec_info))
|
||||
|
||||
# Save to json or csv
|
||||
convert_to_csv(result, "result")
|
||||
convert_to_json(result, "result")
|
||||
|
||||
@ -7,7 +7,7 @@ from scrapegraphai.graphs import DepthSearchGraph
|
||||
|
||||
load_dotenv()
|
||||
|
||||
openai_key = os.getenv("OPENAI_APIKEY")
|
||||
openai_key = os.getenv("OPENAI_API_KEY")
|
||||
|
||||
graph_config = {
|
||||
"llm": {
|
||||
|
||||
@ -11,7 +11,7 @@ load_dotenv()
|
||||
# Define the configuration for the graph
|
||||
# ************************************************
|
||||
|
||||
openai_key = os.getenv("OPENAI_APIKEY")
|
||||
openai_key = os.getenv("OPENAI_API_KEY")
|
||||
|
||||
graph_config = {
|
||||
"llm": {
|
||||
|
||||
@ -28,7 +28,7 @@ graph_config = {
|
||||
# ************************************************
|
||||
|
||||
smart_scraper_graph = SmartScraperGraph(
|
||||
prompt="Extract me all the articles",
|
||||
prompt="Extract me the first article",
|
||||
source="https://www.wired.com",
|
||||
config=graph_config
|
||||
)
|
||||
|
||||
@ -20,7 +20,7 @@ output_path = os.path.join(curr_dir, FILE_NAME)
|
||||
# Define the configuration for the graph
|
||||
# ************************************************
|
||||
|
||||
openai_key = os.getenv("OPENAI_APIKEY")
|
||||
openai_key = os.getenv("OPENAI_API_KEY")
|
||||
|
||||
graph_config = {
|
||||
"llm": {
|
||||
|
||||
83
funding.json
83
funding.json
@ -1,83 +0,0 @@
|
||||
{
|
||||
"id": 0,
|
||||
"guid": "",
|
||||
"version": "v1.0.0",
|
||||
"url": "",
|
||||
"meta": {},
|
||||
"status": "",
|
||||
"status_message": null,
|
||||
"crawl_errors": 0,
|
||||
"crawl_message": null,
|
||||
"created_at": "2024-10-31T10:00:00Z",
|
||||
"updated_at": "2024-10-31T10:00:00Z",
|
||||
"entity": {
|
||||
"type": "organisation",
|
||||
"role": "owner",
|
||||
"name": "ScrapeGraphAI, Inc.",
|
||||
"email": "contact@scrapegraphai.com",
|
||||
"phone": "",
|
||||
"description": "An AI-powered web scraping framework that intelligently extracts structured data from websites with automatic pattern recognition, adaptive scraping strategies, and built-in rate limiting. Recognized as a top 200 open-source AI project globally.",
|
||||
"webpageUrl": {
|
||||
"url": "https://github.com/ScrapeGraphAI/Scrapegraph-ai/blob/main/funding.json"
|
||||
}
|
||||
},
|
||||
"projects": [
|
||||
{
|
||||
"guid": "scrapegraph-core",
|
||||
"name": "ScrapeGraphAI Core",
|
||||
"description": "An AI-powered web scraping framework that intelligently extracts structured data from websites with automatic pattern recognition, adaptive scraping strategies, and built-in rate limiting. Recognized as a top 200 open-source AI project globally.",
|
||||
"webpageUrl": {
|
||||
"url": "https://github.com/ScrapeGraphAI/Scrapegraph-ai/blob/main/funding.json"
|
||||
},
|
||||
"repositoryUrl": {
|
||||
"url": "https://github.com/ScrapeGraphAI/Scrapegraph-ai"
|
||||
},
|
||||
"licenses": [
|
||||
"spdx:MIT"
|
||||
],
|
||||
"tags": [
|
||||
"web-scraping",
|
||||
"ai",
|
||||
"data-extraction",
|
||||
"python",
|
||||
"machine-learning",
|
||||
"open-source",
|
||||
"llm"
|
||||
]
|
||||
}
|
||||
],
|
||||
"funding": {
|
||||
"channels": [
|
||||
{
|
||||
"guid": "stripe",
|
||||
"type": "bank",
|
||||
"address": "https://buy.stripe.com/5kAaGW2E5gHH4vK3ce",
|
||||
"description": "Will accept direct bank transfers via Stripe in the address link, for more info contact us via email contact@scrapegraphai.com"
|
||||
}
|
||||
],
|
||||
"plans": [
|
||||
{
|
||||
"guid": "developer-compensation",
|
||||
"status": "active",
|
||||
"name": "Developer Compensation",
|
||||
"description": "Provides financial support for developers working on maintenance, updates, and feature additions for the projects.",
|
||||
"amount": 3000,
|
||||
"currency": "USD",
|
||||
"frequency": "monthly",
|
||||
"channels": [
|
||||
"stripe"
|
||||
]
|
||||
}
|
||||
],
|
||||
"history": [
|
||||
{
|
||||
"year": 2024,
|
||||
"income": 15000,
|
||||
"expenses": 15000,
|
||||
"taxes": 0,
|
||||
"currency": "USD",
|
||||
"description": "From some companies that sponsor us in our Github repo page"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "scrapegraphai"
|
||||
version = "1.34.1"
|
||||
version = "1.35.0b1"
|
||||
|
||||
description = "A web scraping library based on LangChain which uses LLM and direct graph logic to create scraping pipelines."
|
||||
authors = [
|
||||
@ -11,15 +11,13 @@ authors = [
|
||||
|
||||
dependencies = [
|
||||
"langchain>=0.3.0",
|
||||
"langchain-google-genai>=1.0.7",
|
||||
"langchain-openai>=0.1.22",
|
||||
"langchain-mistralai>=0.1.12",
|
||||
"langchain_community>=0.2.9",
|
||||
"langchain-aws>=0.1.3",
|
||||
"mistral-common>=1.4.0",
|
||||
"langchain-ollama>=0.1.3",
|
||||
"html2text>=2024.2.26",
|
||||
"beautifulsoup4>=4.12.3",
|
||||
"pandas>=2.2.2",
|
||||
"python-dotenv>=1.0.1",
|
||||
"tiktoken>=0.7",
|
||||
"tqdm>=4.66.4",
|
||||
@ -27,19 +25,11 @@ dependencies = [
|
||||
"free-proxy>=1.1.1",
|
||||
"playwright>=1.43.0",
|
||||
"undetected-playwright>=0.3.0",
|
||||
"langchain-ollama>=0.1.3",
|
||||
"qdrant-client>=1.11.3",
|
||||
"fastembed>=0.3.6",
|
||||
"semchunk>=2.2.0",
|
||||
"transformers>=4.44.2",
|
||||
"transformers>=4.44.2",
|
||||
"googlesearch-python>=1.2.5",
|
||||
"async-timeout>=4.0.3",
|
||||
"transformers>=4.44.2",
|
||||
"googlesearch-python>=1.2.5",
|
||||
"simpleeval>=1.0.0",
|
||||
"async_timeout>=4.0.3",
|
||||
"scrapegraph-py>=1.7.0"
|
||||
"jsonschema>=4.23.0",
|
||||
]
|
||||
|
||||
readme = "README.md"
|
||||
@ -77,30 +67,7 @@ requires-python = ">=3.10,<4.0"
|
||||
[project.optional-dependencies]
|
||||
burr = ["burr[start]==0.22.1"]
|
||||
docs = ["sphinx==6.0", "furo==2024.5.6"]
|
||||
|
||||
# Group 1: Other Language Models
|
||||
other-language-models = [
|
||||
"langchain-google-vertexai>=1.0.7",
|
||||
"langchain-fireworks>=0.1.3",
|
||||
"langchain-groq>=0.1.3",
|
||||
"langchain-anthropic>=0.1.11",
|
||||
"langchain-huggingface>=0.0.3",
|
||||
"langchain-nvidia-ai-endpoints>=0.1.6",
|
||||
"langchain_together>=0.2.0"
|
||||
]
|
||||
|
||||
# Group 2: More Semantic Options
|
||||
more-semantic-options = [
|
||||
"graphviz>=0.20.3",
|
||||
]
|
||||
|
||||
# Group 3: More Browser Options
|
||||
more-browser-options = [
|
||||
"browserbase>=0.3.0",
|
||||
]
|
||||
|
||||
# Group 4: Surya Library
|
||||
screenshot_scraper = [
|
||||
ocr = [
|
||||
"surya-ocr>=0.5.0",
|
||||
"matplotlib>=3.7.2",
|
||||
"ipywidgets>=8.1.0",
|
||||
@ -109,22 +76,15 @@ screenshot_scraper = [
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling==1.26.3"]
|
||||
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
"burr[start]==0.22.1",
|
||||
"sphinx==6.0",
|
||||
"furo==2024.5.6",
|
||||
]
|
||||
|
||||
[tool.uv]
|
||||
dev-dependencies = [
|
||||
"poethepoet>=0.31.1",
|
||||
"pytest==8.0.0",
|
||||
"pytest-mock==3.14.0",
|
||||
"pytest>=8.0.0",
|
||||
"pytest-mock>=3.14.0",
|
||||
"pytest-asyncio>=0.25.0",
|
||||
"pylint>=3.2.5",
|
||||
"poethepoet>=0.32.0"
|
||||
]
|
||||
|
||||
[tool.poe.tasks]
|
||||
|
||||
@ -1,565 +0,0 @@
|
||||
# generated by rye
|
||||
# use `rye lock` or `rye sync` to update this lockfile
|
||||
#
|
||||
# last locked with the following flags:
|
||||
# pre: false
|
||||
# features: []
|
||||
# all-features: false
|
||||
# with-sources: false
|
||||
|
||||
-e file:.
|
||||
aiofiles==24.1.0
|
||||
# via burr
|
||||
aiohappyeyeballs==2.3.5
|
||||
# via aiohttp
|
||||
aiohttp==3.10.3
|
||||
# via langchain
|
||||
# via langchain-community
|
||||
aiosignal==1.3.1
|
||||
# via aiohttp
|
||||
alabaster==0.7.16
|
||||
# via sphinx
|
||||
altair==5.4.0
|
||||
# via streamlit
|
||||
annotated-types==0.7.0
|
||||
# via pydantic
|
||||
anyio==4.4.0
|
||||
# via httpx
|
||||
# via openai
|
||||
# via starlette
|
||||
astroid==3.2.4
|
||||
# via pylint
|
||||
async-timeout==4.0.3
|
||||
# via aiohttp
|
||||
# via langchain
|
||||
# via scrapegraphai
|
||||
attrs==24.2.0
|
||||
# via aiohttp
|
||||
# via jsonschema
|
||||
# via referencing
|
||||
babel==2.16.0
|
||||
# via sphinx
|
||||
beautifulsoup4==4.12.3
|
||||
# via furo
|
||||
# via googlesearch-python
|
||||
# via scrapegraphai
|
||||
blinker==1.8.2
|
||||
# via streamlit
|
||||
boto3==1.34.158
|
||||
# via langchain-aws
|
||||
botocore==1.34.158
|
||||
# via boto3
|
||||
# via s3transfer
|
||||
burr==0.22.1
|
||||
# via scrapegraphai
|
||||
cachetools==5.4.0
|
||||
# via google-auth
|
||||
# via streamlit
|
||||
certifi==2024.7.4
|
||||
# via httpcore
|
||||
# via httpx
|
||||
# via requests
|
||||
charset-normalizer==3.3.2
|
||||
# via requests
|
||||
click==8.1.7
|
||||
# via burr
|
||||
# via streamlit
|
||||
# via uvicorn
|
||||
coloredlogs==15.0.1
|
||||
# via onnxruntime
|
||||
contourpy==1.2.1
|
||||
# via matplotlib
|
||||
cycler==0.12.1
|
||||
# via matplotlib
|
||||
dataclasses-json==0.6.7
|
||||
# via langchain-community
|
||||
dill==0.3.8
|
||||
# via multiprocess
|
||||
# via pylint
|
||||
distro==1.9.0
|
||||
# via openai
|
||||
docutils==0.19
|
||||
# via sphinx
|
||||
exceptiongroup==1.2.2
|
||||
# via anyio
|
||||
# via pytest
|
||||
fastapi==0.112.0
|
||||
# via burr
|
||||
fastapi-pagination==0.12.26
|
||||
# via burr
|
||||
fastembed==0.3.6
|
||||
# via scrapegraphai
|
||||
filelock==3.15.4
|
||||
# via huggingface-hub
|
||||
# via transformers
|
||||
flatbuffers==24.3.25
|
||||
# via onnxruntime
|
||||
fonttools==4.53.1
|
||||
# via matplotlib
|
||||
free-proxy==1.1.1
|
||||
# via scrapegraphai
|
||||
frozenlist==1.4.1
|
||||
# via aiohttp
|
||||
# via aiosignal
|
||||
fsspec==2024.6.1
|
||||
# via huggingface-hub
|
||||
furo==2024.5.6
|
||||
# via scrapegraphai
|
||||
gitdb==4.0.11
|
||||
# via gitpython
|
||||
gitpython==3.1.43
|
||||
# via streamlit
|
||||
google-ai-generativelanguage==0.6.6
|
||||
# via google-generativeai
|
||||
google-api-core==2.19.1
|
||||
# via google-ai-generativelanguage
|
||||
# via google-api-python-client
|
||||
# via google-generativeai
|
||||
google-api-python-client==2.140.0
|
||||
# via google-generativeai
|
||||
google-auth==2.33.0
|
||||
# via google-ai-generativelanguage
|
||||
# via google-api-core
|
||||
# via google-api-python-client
|
||||
# via google-auth-httplib2
|
||||
# via google-generativeai
|
||||
google-auth-httplib2==0.2.0
|
||||
# via google-api-python-client
|
||||
google-generativeai==0.7.2
|
||||
# via langchain-google-genai
|
||||
googleapis-common-protos==1.63.2
|
||||
# via google-api-core
|
||||
# via grpcio-status
|
||||
googlesearch-python==1.2.5
|
||||
# via scrapegraphai
|
||||
graphviz==0.20.3
|
||||
# via burr
|
||||
greenlet==3.0.3
|
||||
# via playwright
|
||||
grpcio==1.65.4
|
||||
# via google-api-core
|
||||
# via grpcio-status
|
||||
# via grpcio-tools
|
||||
# via qdrant-client
|
||||
grpcio-status==1.62.3
|
||||
# via google-api-core
|
||||
grpcio-tools==1.62.3
|
||||
# via qdrant-client
|
||||
h11==0.14.0
|
||||
# via httpcore
|
||||
# via uvicorn
|
||||
h2==4.1.0
|
||||
# via httpx
|
||||
hpack==4.0.0
|
||||
# via h2
|
||||
html2text==2024.2.26
|
||||
# via scrapegraphai
|
||||
httpcore==1.0.5
|
||||
# via httpx
|
||||
httplib2==0.22.0
|
||||
# via google-api-python-client
|
||||
# via google-auth-httplib2
|
||||
httpx==0.27.0
|
||||
# via langchain-mistralai
|
||||
# via langsmith
|
||||
# via ollama
|
||||
# via openai
|
||||
# via qdrant-client
|
||||
httpx-sse==0.4.0
|
||||
# via langchain-mistralai
|
||||
huggingface-hub==0.24.5
|
||||
# via fastembed
|
||||
# via tokenizers
|
||||
# via transformers
|
||||
humanfriendly==10.0
|
||||
# via coloredlogs
|
||||
hyperframe==6.0.1
|
||||
# via h2
|
||||
idna==3.7
|
||||
# via anyio
|
||||
# via httpx
|
||||
# via requests
|
||||
# via yarl
|
||||
imagesize==1.4.1
|
||||
# via sphinx
|
||||
iniconfig==2.0.0
|
||||
# via pytest
|
||||
isort==5.13.2
|
||||
# via pylint
|
||||
jinja2==3.1.4
|
||||
# via altair
|
||||
# via burr
|
||||
# via pydeck
|
||||
# via sphinx
|
||||
jiter==0.5.0
|
||||
# via openai
|
||||
jmespath==1.0.1
|
||||
# via boto3
|
||||
# via botocore
|
||||
jsonpatch==1.33
|
||||
# via langchain-core
|
||||
jsonpointer==3.0.0
|
||||
# via jsonpatch
|
||||
jsonschema==4.23.0
|
||||
# via altair
|
||||
# via mistral-common
|
||||
jsonschema-specifications==2023.12.1
|
||||
# via jsonschema
|
||||
kiwisolver==1.4.5
|
||||
# via matplotlib
|
||||
langchain==0.3.0
|
||||
# via langchain-community
|
||||
# via scrapegraphai
|
||||
langchain-aws==0.2.0
|
||||
# via scrapegraphai
|
||||
langchain-community==0.3.0
|
||||
# via scrapegraphai
|
||||
langchain-core==0.3.1
|
||||
# via langchain
|
||||
# via langchain-aws
|
||||
# via langchain-community
|
||||
# via langchain-google-genai
|
||||
# via langchain-mistralai
|
||||
# via langchain-ollama
|
||||
# via langchain-openai
|
||||
# via langchain-text-splitters
|
||||
langchain-google-genai==2.0.0
|
||||
# via scrapegraphai
|
||||
langchain-mistralai==0.2.0
|
||||
# via scrapegraphai
|
||||
langchain-ollama==0.2.0
|
||||
# via scrapegraphai
|
||||
langchain-openai==0.2.0
|
||||
# via scrapegraphai
|
||||
langchain-text-splitters==0.3.0
|
||||
# via langchain
|
||||
langsmith==0.1.121
|
||||
# via langchain
|
||||
# via langchain-community
|
||||
# via langchain-core
|
||||
loguru==0.7.2
|
||||
# via burr
|
||||
# via fastembed
|
||||
lxml==5.3.0
|
||||
# via free-proxy
|
||||
markdown-it-py==3.0.0
|
||||
# via rich
|
||||
markupsafe==2.1.5
|
||||
# via jinja2
|
||||
marshmallow==3.21.3
|
||||
# via dataclasses-json
|
||||
matplotlib==3.9.1.post1
|
||||
# via burr
|
||||
mccabe==0.7.0
|
||||
# via pylint
|
||||
mdurl==0.1.2
|
||||
# via markdown-it-py
|
||||
minify-html==0.15.0
|
||||
# via scrapegraphai
|
||||
mistral-common==1.4.1
|
||||
# via scrapegraphai
|
||||
mmh3==4.1.0
|
||||
# via fastembed
|
||||
mpire==2.10.2
|
||||
# via semchunk
|
||||
mpmath==1.3.0
|
||||
# via sympy
|
||||
multidict==6.0.5
|
||||
# via aiohttp
|
||||
# via yarl
|
||||
multiprocess==0.70.16
|
||||
# via mpire
|
||||
mypy-extensions==1.0.0
|
||||
# via typing-inspect
|
||||
narwhals==1.3.0
|
||||
# via altair
|
||||
numpy==1.26.4
|
||||
# via contourpy
|
||||
# via fastembed
|
||||
# via langchain
|
||||
# via langchain-aws
|
||||
# via langchain-community
|
||||
# via matplotlib
|
||||
# via onnx
|
||||
# via onnxruntime
|
||||
# via opencv-python-headless
|
||||
# via pandas
|
||||
# via pyarrow
|
||||
# via pydeck
|
||||
# via qdrant-client
|
||||
# via sf-hamilton
|
||||
# via streamlit
|
||||
# via transformers
|
||||
ollama==0.3.2
|
||||
# via langchain-ollama
|
||||
onnx==1.17.0
|
||||
# via fastembed
|
||||
onnxruntime==1.19.2
|
||||
# via fastembed
|
||||
openai==1.40.3
|
||||
# via burr
|
||||
# via langchain-openai
|
||||
opencv-python-headless==4.10.0.84
|
||||
# via mistral-common
|
||||
orjson==3.10.7
|
||||
# via langsmith
|
||||
packaging==24.1
|
||||
# via altair
|
||||
# via huggingface-hub
|
||||
# via langchain-core
|
||||
# via marshmallow
|
||||
# via matplotlib
|
||||
# via onnxruntime
|
||||
# via pytest
|
||||
# via sphinx
|
||||
# via streamlit
|
||||
# via transformers
|
||||
pandas==2.2.2
|
||||
# via scrapegraphai
|
||||
# via sf-hamilton
|
||||
# via streamlit
|
||||
pillow==10.4.0
|
||||
# via fastembed
|
||||
# via matplotlib
|
||||
# via mistral-common
|
||||
# via streamlit
|
||||
platformdirs==4.2.2
|
||||
# via pylint
|
||||
playwright==1.45.1
|
||||
# via scrapegraphai
|
||||
# via undetected-playwright
|
||||
pluggy==1.5.0
|
||||
# via pytest
|
||||
portalocker==2.10.1
|
||||
# via qdrant-client
|
||||
proto-plus==1.24.0
|
||||
# via google-ai-generativelanguage
|
||||
# via google-api-core
|
||||
protobuf==4.25.4
|
||||
# via google-ai-generativelanguage
|
||||
# via google-api-core
|
||||
# via google-generativeai
|
||||
# via googleapis-common-protos
|
||||
# via grpcio-status
|
||||
# via grpcio-tools
|
||||
# via onnx
|
||||
# via onnxruntime
|
||||
# via proto-plus
|
||||
# via streamlit
|
||||
pyarrow==17.0.0
|
||||
# via streamlit
|
||||
pyasn1==0.6.0
|
||||
# via pyasn1-modules
|
||||
# via rsa
|
||||
pyasn1-modules==0.4.0
|
||||
# via google-auth
|
||||
pydantic==2.10.1
|
||||
# via burr
|
||||
# via fastapi
|
||||
# via fastapi-pagination
|
||||
# via google-generativeai
|
||||
# via langchain
|
||||
# via langchain-aws
|
||||
# via langchain-core
|
||||
# via langchain-google-genai
|
||||
# via langchain-mistralai
|
||||
# via langsmith
|
||||
# via mistral-common
|
||||
# via openai
|
||||
# via pydantic-settings
|
||||
# via qdrant-client
|
||||
# via scrapegraph-py
|
||||
pydantic-core==2.27.1
|
||||
# via pydantic
|
||||
pydantic-settings==2.5.2
|
||||
# via langchain-community
|
||||
pydeck==0.9.1
|
||||
# via streamlit
|
||||
pyee==11.1.0
|
||||
# via playwright
|
||||
pygments==2.18.0
|
||||
# via furo
|
||||
# via mpire
|
||||
# via rich
|
||||
# via sphinx
|
||||
pylint==3.2.6
|
||||
pyparsing==3.1.2
|
||||
# via httplib2
|
||||
# via matplotlib
|
||||
pystemmer==2.2.0.1
|
||||
# via fastembed
|
||||
pytest==8.0.0
|
||||
# via pytest-mock
|
||||
pytest-mock==3.14.0
|
||||
python-dateutil==2.9.0.post0
|
||||
# via botocore
|
||||
# via matplotlib
|
||||
# via pandas
|
||||
python-dotenv==1.0.1
|
||||
# via pydantic-settings
|
||||
# via scrapegraph-py
|
||||
# via scrapegraphai
|
||||
pytz==2024.1
|
||||
# via pandas
|
||||
pyyaml==6.0.2
|
||||
# via huggingface-hub
|
||||
# via langchain
|
||||
# via langchain-community
|
||||
# via langchain-core
|
||||
# via transformers
|
||||
qdrant-client==1.11.3
|
||||
# via scrapegraphai
|
||||
referencing==0.35.1
|
||||
# via jsonschema
|
||||
# via jsonschema-specifications
|
||||
regex==2024.7.24
|
||||
# via tiktoken
|
||||
# via transformers
|
||||
requests==2.32.3
|
||||
# via burr
|
||||
# via fastembed
|
||||
# via free-proxy
|
||||
# via google-api-core
|
||||
# via googlesearch-python
|
||||
# via huggingface-hub
|
||||
# via langchain
|
||||
# via langchain-community
|
||||
# via langsmith
|
||||
# via mistral-common
|
||||
# via scrapegraph-py
|
||||
# via sphinx
|
||||
# via streamlit
|
||||
# via tiktoken
|
||||
# via transformers
|
||||
rich==13.7.1
|
||||
# via streamlit
|
||||
rpds-py==0.20.0
|
||||
# via jsonschema
|
||||
# via referencing
|
||||
rsa==4.9
|
||||
# via google-auth
|
||||
s3transfer==0.10.2
|
||||
# via boto3
|
||||
safetensors==0.4.5
|
||||
# via transformers
|
||||
scrapegraph-py==0.0.3
|
||||
# via scrapegraphai
|
||||
semchunk==2.2.0
|
||||
# via scrapegraphai
|
||||
sentencepiece==0.2.0
|
||||
# via mistral-common
|
||||
setuptools==75.1.0
|
||||
# via grpcio-tools
|
||||
sf-hamilton==1.73.1
|
||||
# via burr
|
||||
simpleeval==1.0.0
|
||||
# via scrapegraphai
|
||||
six==1.16.0
|
||||
# via python-dateutil
|
||||
smmap==5.0.1
|
||||
# via gitdb
|
||||
sniffio==1.3.1
|
||||
# via anyio
|
||||
# via httpx
|
||||
# via openai
|
||||
snowballstemmer==2.2.0
|
||||
# via fastembed
|
||||
# via sphinx
|
||||
soupsieve==2.5
|
||||
# via beautifulsoup4
|
||||
sphinx==6.0.0
|
||||
# via furo
|
||||
# via scrapegraphai
|
||||
# via sphinx-basic-ng
|
||||
sphinx-basic-ng==1.0.0b2
|
||||
# via furo
|
||||
sphinxcontrib-applehelp==2.0.0
|
||||
# via sphinx
|
||||
sphinxcontrib-devhelp==2.0.0
|
||||
# via sphinx
|
||||
sphinxcontrib-htmlhelp==2.1.0
|
||||
# via sphinx
|
||||
sphinxcontrib-jsmath==1.0.1
|
||||
# via sphinx
|
||||
sphinxcontrib-qthelp==2.0.0
|
||||
# via sphinx
|
||||
sphinxcontrib-serializinghtml==2.0.0
|
||||
# via sphinx
|
||||
sqlalchemy==2.0.32
|
||||
# via langchain
|
||||
# via langchain-community
|
||||
starlette==0.37.2
|
||||
# via fastapi
|
||||
streamlit==1.37.1
|
||||
# via burr
|
||||
sympy==1.13.3
|
||||
# via onnxruntime
|
||||
tenacity==8.5.0
|
||||
# via langchain
|
||||
# via langchain-community
|
||||
# via langchain-core
|
||||
# via streamlit
|
||||
tiktoken==0.7.0
|
||||
# via langchain-openai
|
||||
# via mistral-common
|
||||
# via scrapegraphai
|
||||
tokenizers==0.19.1
|
||||
# via fastembed
|
||||
# via langchain-mistralai
|
||||
# via transformers
|
||||
toml==0.10.2
|
||||
# via streamlit
|
||||
tomli==2.1.0
|
||||
# via pylint
|
||||
# via pytest
|
||||
tomlkit==0.13.0
|
||||
# via pylint
|
||||
tornado==6.4.1
|
||||
# via streamlit
|
||||
tqdm==4.66.5
|
||||
# via fastembed
|
||||
# via google-generativeai
|
||||
# via huggingface-hub
|
||||
# via mpire
|
||||
# via openai
|
||||
# via scrapegraphai
|
||||
# via semchunk
|
||||
# via transformers
|
||||
transformers==4.44.2
|
||||
# via scrapegraphai
|
||||
typing-extensions==4.12.2
|
||||
# via altair
|
||||
# via anyio
|
||||
# via astroid
|
||||
# via fastapi
|
||||
# via fastapi-pagination
|
||||
# via google-generativeai
|
||||
# via huggingface-hub
|
||||
# via langchain-core
|
||||
# via mistral-common
|
||||
# via openai
|
||||
# via pydantic
|
||||
# via pydantic-core
|
||||
# via pyee
|
||||
# via sf-hamilton
|
||||
# via sqlalchemy
|
||||
# via streamlit
|
||||
# via typing-inspect
|
||||
# via uvicorn
|
||||
typing-inspect==0.9.0
|
||||
# via dataclasses-json
|
||||
# via sf-hamilton
|
||||
tzdata==2024.1
|
||||
# via pandas
|
||||
undetected-playwright==0.3.0
|
||||
# via scrapegraphai
|
||||
uritemplate==4.1.1
|
||||
# via google-api-python-client
|
||||
urllib3==1.26.19
|
||||
# via botocore
|
||||
# via qdrant-client
|
||||
# via requests
|
||||
uvicorn==0.30.5
|
||||
# via burr
|
||||
yarl==1.9.4
|
||||
# via aiohttp
|
||||
@ -1,7 +0,0 @@
|
||||
pytest==8.0.0
|
||||
pytest-asyncio==0.25.0
|
||||
pytest-mock==3.14.0
|
||||
burr[start]==0.22.1
|
||||
sphinx==6.0
|
||||
furo==2024.5.6
|
||||
pylint>=3.2.5
|
||||
@ -1,403 +0,0 @@
|
||||
# generated by rye
|
||||
# use `rye lock` or `rye sync` to update this lockfile
|
||||
#
|
||||
# last locked with the following flags:
|
||||
# pre: false
|
||||
# features: []
|
||||
# all-features: false
|
||||
# with-sources: false
|
||||
|
||||
-e file:.
|
||||
aiohttp==3.9.5
|
||||
# via langchain
|
||||
# via langchain-community
|
||||
aiosignal==1.3.1
|
||||
# via aiohttp
|
||||
annotated-types==0.7.0
|
||||
# via pydantic
|
||||
anyio==4.4.0
|
||||
# via httpx
|
||||
# via openai
|
||||
async-timeout==4.0.3
|
||||
# via aiohttp
|
||||
# via langchain
|
||||
# via scrapegraphai
|
||||
attrs==23.2.0
|
||||
# via aiohttp
|
||||
# via jsonschema
|
||||
# via referencing
|
||||
beautifulsoup4==4.12.3
|
||||
# via googlesearch-python
|
||||
# via scrapegraphai
|
||||
boto3==1.34.146
|
||||
# via langchain-aws
|
||||
botocore==1.34.146
|
||||
# via boto3
|
||||
# via s3transfer
|
||||
cachetools==5.4.0
|
||||
# via google-auth
|
||||
certifi==2024.7.4
|
||||
# via httpcore
|
||||
# via httpx
|
||||
# via requests
|
||||
charset-normalizer==3.3.2
|
||||
# via requests
|
||||
coloredlogs==15.0.1
|
||||
# via onnxruntime
|
||||
dataclasses-json==0.6.7
|
||||
# via langchain-community
|
||||
dill==0.3.8
|
||||
# via multiprocess
|
||||
distro==1.9.0
|
||||
# via openai
|
||||
exceptiongroup==1.2.2
|
||||
# via anyio
|
||||
fastembed==0.3.6
|
||||
# via scrapegraphai
|
||||
filelock==3.15.4
|
||||
# via huggingface-hub
|
||||
# via transformers
|
||||
flatbuffers==24.3.25
|
||||
# via onnxruntime
|
||||
free-proxy==1.1.1
|
||||
# via scrapegraphai
|
||||
frozenlist==1.4.1
|
||||
# via aiohttp
|
||||
# via aiosignal
|
||||
fsspec==2024.6.1
|
||||
# via huggingface-hub
|
||||
google-ai-generativelanguage==0.6.6
|
||||
# via google-generativeai
|
||||
google-api-core==2.19.1
|
||||
# via google-ai-generativelanguage
|
||||
# via google-api-python-client
|
||||
# via google-generativeai
|
||||
google-api-python-client==2.137.0
|
||||
# via google-generativeai
|
||||
google-auth==2.32.0
|
||||
# via google-ai-generativelanguage
|
||||
# via google-api-core
|
||||
# via google-api-python-client
|
||||
# via google-auth-httplib2
|
||||
# via google-generativeai
|
||||
google-auth-httplib2==0.2.0
|
||||
# via google-api-python-client
|
||||
google-generativeai==0.7.2
|
||||
# via langchain-google-genai
|
||||
googleapis-common-protos==1.63.2
|
||||
# via google-api-core
|
||||
# via grpcio-status
|
||||
googlesearch-python==1.2.5
|
||||
# via scrapegraphai
|
||||
greenlet==3.0.3
|
||||
# via playwright
|
||||
grpcio==1.65.1
|
||||
# via google-api-core
|
||||
# via grpcio-status
|
||||
# via grpcio-tools
|
||||
# via qdrant-client
|
||||
grpcio-status==1.62.2
|
||||
# via google-api-core
|
||||
grpcio-tools==1.62.3
|
||||
# via qdrant-client
|
||||
h11==0.14.0
|
||||
# via httpcore
|
||||
h2==4.1.0
|
||||
# via httpx
|
||||
hpack==4.0.0
|
||||
# via h2
|
||||
html2text==2024.2.26
|
||||
# via scrapegraphai
|
||||
httpcore==1.0.5
|
||||
# via httpx
|
||||
httplib2==0.22.0
|
||||
# via google-api-python-client
|
||||
# via google-auth-httplib2
|
||||
httpx==0.27.0
|
||||
# via langchain-mistralai
|
||||
# via langsmith
|
||||
# via ollama
|
||||
# via openai
|
||||
# via qdrant-client
|
||||
httpx-sse==0.4.0
|
||||
# via langchain-mistralai
|
||||
huggingface-hub==0.24.1
|
||||
# via fastembed
|
||||
# via tokenizers
|
||||
# via transformers
|
||||
humanfriendly==10.0
|
||||
# via coloredlogs
|
||||
hyperframe==6.0.1
|
||||
# via h2
|
||||
idna==3.7
|
||||
# via anyio
|
||||
# via httpx
|
||||
# via requests
|
||||
# via yarl
|
||||
jiter==0.5.0
|
||||
# via openai
|
||||
jmespath==1.0.1
|
||||
# via boto3
|
||||
# via botocore
|
||||
jsonpatch==1.33
|
||||
# via langchain-core
|
||||
jsonpointer==3.0.0
|
||||
# via jsonpatch
|
||||
jsonschema==4.23.0
|
||||
# via mistral-common
|
||||
jsonschema-specifications==2023.12.1
|
||||
# via jsonschema
|
||||
langchain==0.3.0
|
||||
# via langchain-community
|
||||
# via scrapegraphai
|
||||
langchain-aws==0.2.0
|
||||
# via scrapegraphai
|
||||
langchain-community==0.3.0
|
||||
# via scrapegraphai
|
||||
langchain-core==0.3.1
|
||||
# via langchain
|
||||
# via langchain-aws
|
||||
# via langchain-community
|
||||
# via langchain-google-genai
|
||||
# via langchain-mistralai
|
||||
# via langchain-ollama
|
||||
# via langchain-openai
|
||||
# via langchain-text-splitters
|
||||
langchain-google-genai==2.0.0
|
||||
# via scrapegraphai
|
||||
langchain-mistralai==0.2.0
|
||||
# via scrapegraphai
|
||||
langchain-ollama==0.2.0
|
||||
# via scrapegraphai
|
||||
langchain-openai==0.2.0
|
||||
# via scrapegraphai
|
||||
langchain-text-splitters==0.3.0
|
||||
# via langchain
|
||||
langsmith==0.1.121
|
||||
# via langchain
|
||||
# via langchain-community
|
||||
# via langchain-core
|
||||
loguru==0.7.2
|
||||
# via fastembed
|
||||
lxml==5.2.2
|
||||
# via free-proxy
|
||||
marshmallow==3.21.3
|
||||
# via dataclasses-json
|
||||
minify-html==0.15.0
|
||||
# via scrapegraphai
|
||||
mistral-common==1.4.1
|
||||
# via scrapegraphai
|
||||
mmh3==4.1.0
|
||||
# via fastembed
|
||||
mpire==2.10.2
|
||||
# via semchunk
|
||||
mpmath==1.3.0
|
||||
# via sympy
|
||||
multidict==6.0.5
|
||||
# via aiohttp
|
||||
# via yarl
|
||||
multiprocess==0.70.16
|
||||
# via mpire
|
||||
mypy-extensions==1.0.0
|
||||
# via typing-inspect
|
||||
numpy==1.26.4
|
||||
# via fastembed
|
||||
# via langchain
|
||||
# via langchain-aws
|
||||
# via langchain-community
|
||||
# via onnx
|
||||
# via onnxruntime
|
||||
# via opencv-python-headless
|
||||
# via pandas
|
||||
# via qdrant-client
|
||||
# via transformers
|
||||
ollama==0.3.2
|
||||
# via langchain-ollama
|
||||
onnx==1.17.0
|
||||
# via fastembed
|
||||
onnxruntime==1.19.2
|
||||
# via fastembed
|
||||
openai==1.41.0
|
||||
# via langchain-openai
|
||||
opencv-python-headless==4.10.0.84
|
||||
# via mistral-common
|
||||
orjson==3.10.6
|
||||
# via langsmith
|
||||
packaging==24.1
|
||||
# via huggingface-hub
|
||||
# via langchain-core
|
||||
# via marshmallow
|
||||
# via onnxruntime
|
||||
# via transformers
|
||||
pandas==2.2.2
|
||||
# via scrapegraphai
|
||||
pillow==10.4.0
|
||||
# via fastembed
|
||||
# via mistral-common
|
||||
playwright==1.45.1
|
||||
# via scrapegraphai
|
||||
# via undetected-playwright
|
||||
portalocker==2.10.1
|
||||
# via qdrant-client
|
||||
proto-plus==1.24.0
|
||||
# via google-ai-generativelanguage
|
||||
# via google-api-core
|
||||
protobuf==4.25.3
|
||||
# via google-ai-generativelanguage
|
||||
# via google-api-core
|
||||
# via google-generativeai
|
||||
# via googleapis-common-protos
|
||||
# via grpcio-status
|
||||
# via grpcio-tools
|
||||
# via onnx
|
||||
# via onnxruntime
|
||||
# via proto-plus
|
||||
pyasn1==0.6.0
|
||||
# via pyasn1-modules
|
||||
# via rsa
|
||||
pyasn1-modules==0.4.0
|
||||
# via google-auth
|
||||
pydantic==2.10.1
|
||||
# via google-generativeai
|
||||
# via langchain
|
||||
# via langchain-aws
|
||||
# via langchain-core
|
||||
# via langchain-google-genai
|
||||
# via langchain-mistralai
|
||||
# via langsmith
|
||||
# via mistral-common
|
||||
# via openai
|
||||
# via pydantic-settings
|
||||
# via qdrant-client
|
||||
# via scrapegraph-py
|
||||
pydantic-core==2.27.1
|
||||
# via pydantic
|
||||
pydantic-settings==2.5.2
|
||||
# via langchain-community
|
||||
pyee==11.1.0
|
||||
# via playwright
|
||||
pygments==2.18.0
|
||||
# via mpire
|
||||
pyparsing==3.1.2
|
||||
# via httplib2
|
||||
pystemmer==2.2.0.1
|
||||
# via fastembed
|
||||
python-dateutil==2.9.0.post0
|
||||
# via botocore
|
||||
# via pandas
|
||||
python-dotenv==1.0.1
|
||||
# via pydantic-settings
|
||||
# via scrapegraph-py
|
||||
# via scrapegraphai
|
||||
pytz==2024.1
|
||||
# via pandas
|
||||
pyyaml==6.0.1
|
||||
# via huggingface-hub
|
||||
# via langchain
|
||||
# via langchain-community
|
||||
# via langchain-core
|
||||
# via transformers
|
||||
qdrant-client==1.11.3
|
||||
# via scrapegraphai
|
||||
referencing==0.35.1
|
||||
# via jsonschema
|
||||
# via jsonschema-specifications
|
||||
regex==2024.5.15
|
||||
# via tiktoken
|
||||
# via transformers
|
||||
requests==2.32.3
|
||||
# via fastembed
|
||||
# via free-proxy
|
||||
# via google-api-core
|
||||
# via googlesearch-python
|
||||
# via huggingface-hub
|
||||
# via langchain
|
||||
# via langchain-community
|
||||
# via langsmith
|
||||
# via mistral-common
|
||||
# via scrapegraph-py
|
||||
# via tiktoken
|
||||
# via transformers
|
||||
rpds-py==0.20.0
|
||||
# via jsonschema
|
||||
# via referencing
|
||||
rsa==4.9
|
||||
# via google-auth
|
||||
s3transfer==0.10.2
|
||||
# via boto3
|
||||
safetensors==0.4.5
|
||||
# via transformers
|
||||
scrapegraph-py==0.0.3
|
||||
# via scrapegraphai
|
||||
semchunk==2.2.0
|
||||
# via scrapegraphai
|
||||
sentencepiece==0.2.0
|
||||
# via mistral-common
|
||||
setuptools==75.1.0
|
||||
# via grpcio-tools
|
||||
simpleeval==1.0.0
|
||||
# via scrapegraphai
|
||||
six==1.16.0
|
||||
# via python-dateutil
|
||||
sniffio==1.3.1
|
||||
# via anyio
|
||||
# via httpx
|
||||
# via openai
|
||||
snowballstemmer==2.2.0
|
||||
# via fastembed
|
||||
soupsieve==2.5
|
||||
# via beautifulsoup4
|
||||
sqlalchemy==2.0.31
|
||||
# via langchain
|
||||
# via langchain-community
|
||||
sympy==1.13.3
|
||||
# via onnxruntime
|
||||
tenacity==8.5.0
|
||||
# via langchain
|
||||
# via langchain-community
|
||||
# via langchain-core
|
||||
tiktoken==0.7.0
|
||||
# via langchain-openai
|
||||
# via mistral-common
|
||||
# via scrapegraphai
|
||||
tokenizers==0.19.1
|
||||
# via fastembed
|
||||
# via langchain-mistralai
|
||||
# via transformers
|
||||
tqdm==4.66.4
|
||||
# via fastembed
|
||||
# via google-generativeai
|
||||
# via huggingface-hub
|
||||
# via mpire
|
||||
# via openai
|
||||
# via scrapegraphai
|
||||
# via semchunk
|
||||
# via transformers
|
||||
transformers==4.44.2
|
||||
# via scrapegraphai
|
||||
typing-extensions==4.12.2
|
||||
# via anyio
|
||||
# via google-generativeai
|
||||
# via huggingface-hub
|
||||
# via langchain-core
|
||||
# via mistral-common
|
||||
# via openai
|
||||
# via pydantic
|
||||
# via pydantic-core
|
||||
# via pyee
|
||||
# via sqlalchemy
|
||||
# via typing-inspect
|
||||
typing-inspect==0.9.0
|
||||
# via dataclasses-json
|
||||
tzdata==2024.1
|
||||
# via pandas
|
||||
undetected-playwright==0.3.0
|
||||
# via scrapegraphai
|
||||
uritemplate==4.1.1
|
||||
# via google-api-python-client
|
||||
urllib3==1.26.19
|
||||
# via botocore
|
||||
# via qdrant-client
|
||||
# via requests
|
||||
yarl==1.9.4
|
||||
# via aiohttp
|
||||
@ -1,22 +0,0 @@
|
||||
langchain>=0.2.14
|
||||
langchain-google-genai>=1.0.7
|
||||
langchain-openai>=0.1.22
|
||||
langchain-mistralai>=0.1.12
|
||||
langchain_community>=0.2.9
|
||||
langchain-aws>=0.1.3
|
||||
html2text>=2024.2.26
|
||||
faiss-cpu>=1.8.0
|
||||
beautifulsoup4>=4.12.3
|
||||
pandas>=2.2.2
|
||||
python-dotenv>=1.0.1
|
||||
tiktoken>=0.7
|
||||
tqdm>=4.66.4
|
||||
minify-html>=0.15.0
|
||||
free-proxy>=1.1.1
|
||||
playwright>=1.43.0
|
||||
undetected-playwright>=0.3.0
|
||||
semchunk>=1.0.1
|
||||
langchain-ollama>=0.1.3
|
||||
simpleeval>=0.9.13
|
||||
googlesearch-python>=1.2.5
|
||||
async_timeout>=4.0.3
|
||||
@ -4,7 +4,6 @@ GraphBuilder Module
|
||||
from langchain_core.prompts import ChatPromptTemplate
|
||||
from langchain.chains import create_extraction_chain
|
||||
from langchain_community.chat_models import ErnieBotChat
|
||||
from langchain_google_genai import ChatGoogleGenerativeAI
|
||||
from langchain_openai import ChatOpenAI
|
||||
from ..helpers import nodes_metadata, graph_schema
|
||||
|
||||
@ -70,6 +69,10 @@ class GraphBuilder:
|
||||
if "gpt-" in llm_params["model"]:
|
||||
return ChatOpenAI(llm_params)
|
||||
elif "gemini" in llm_params["model"]:
|
||||
try:
|
||||
from langchain_google_genai import ChatGoogleGenerativeAI
|
||||
except ImportError:
|
||||
raise ImportError("langchain_google_genai is not installed. Please install it using 'pip install langchain-google-genai'.")
|
||||
return ChatGoogleGenerativeAI(llm_params)
|
||||
elif "ernie" in llm_params["model"]:
|
||||
return ErnieBotChat(llm_params)
|
||||
|
||||
@ -4,8 +4,6 @@ from langchain_community.document_loaders.base import BaseLoader
|
||||
from langchain_core.documents import Document
|
||||
import aiohttp
|
||||
import async_timeout
|
||||
from selenium import webdriver
|
||||
from selenium.webdriver.chrome.options import Options as ChromeOptions
|
||||
from typing import Union
|
||||
from ..utils import Proxy, dynamic_import, get_logger, parse_or_search_proxy
|
||||
|
||||
@ -25,9 +23,6 @@ class ChromiumLoader(BaseLoader):
|
||||
requires_js_support: Flag to determine if JS rendering is required.
|
||||
"""
|
||||
|
||||
RETRY_LIMIT = 3
|
||||
TIMEOUT = 10
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
urls: List[str],
|
||||
@ -39,6 +34,8 @@ class ChromiumLoader(BaseLoader):
|
||||
requires_js_support: bool = False,
|
||||
storage_state: Optional[str] = None,
|
||||
browser_name: str = "chromium", #default chromium
|
||||
retry_limit: int = 1,
|
||||
timeout: int = 10,
|
||||
**kwargs: Any,
|
||||
):
|
||||
"""Initialize the loader with a list of URL paths.
|
||||
@ -49,6 +46,8 @@ class ChromiumLoader(BaseLoader):
|
||||
proxy: A dictionary containing proxy information; None disables protection.
|
||||
urls: A list of URLs to scrape content from.
|
||||
requires_js_support: Whether to use JS rendering for scraping.
|
||||
retry_limit: Maximum number of retry attempts for scraping. Defaults to 3.
|
||||
timeout: Maximum time in seconds to wait for scraping. Defaults to 10.
|
||||
kwargs: A dictionary containing additional browser kwargs.
|
||||
|
||||
Raises:
|
||||
@ -70,12 +69,17 @@ class ChromiumLoader(BaseLoader):
|
||||
self.requires_js_support = requires_js_support
|
||||
self.storage_state = storage_state
|
||||
self.browser_name = browser_name
|
||||
self.retry_limit = retry_limit
|
||||
self.timeout = timeout
|
||||
|
||||
async def scrape(self, url:str) -> str:
|
||||
if self.backend == "playwright":
|
||||
return await self.ascrape_playwright(url)
|
||||
elif self.backend == "selenium":
|
||||
return await self.ascrape_undetected_chromedriver(url)
|
||||
try:
|
||||
return await self.ascrape_undetected_chromedriver(url)
|
||||
except Exception as e:
|
||||
raise ValueError(f"Failed to scrape with undetected chromedriver: {e}")
|
||||
else:
|
||||
raise ValueError(f"Unsupported backend: {self.backend}")
|
||||
|
||||
@ -90,18 +94,22 @@ class ChromiumLoader(BaseLoader):
|
||||
Returns:
|
||||
str: The scraped HTML content or an error message if an exception occurs.
|
||||
"""
|
||||
import undetected_chromedriver as uc
|
||||
try:
|
||||
import undetected_chromedriver as uc
|
||||
except ImportError:
|
||||
raise ImportError("undetected_chromedriver is required for ChromiumLoader. Please install it with `pip install undetected-chromedriver`.")
|
||||
|
||||
logger.info(f"Starting scraping with {self.backend}...")
|
||||
results = ""
|
||||
attempt = 0
|
||||
|
||||
while attempt < self.RETRY_LIMIT:
|
||||
while attempt < self.retry_limit:
|
||||
try:
|
||||
async with async_timeout.timeout(self.TIMEOUT):
|
||||
async with async_timeout.timeout(self.timeout):
|
||||
# Handling browser selection
|
||||
if self.backend == "selenium":
|
||||
if self.browser_name == "chromium":
|
||||
from selenium.webdriver.chrome.options import Options as ChromeOptions
|
||||
options = ChromeOptions()
|
||||
options.headless = self.headless
|
||||
# Initialize undetected chromedriver for Selenium
|
||||
@ -112,6 +120,7 @@ class ChromiumLoader(BaseLoader):
|
||||
break
|
||||
elif self.browser_name == "firefox":
|
||||
from selenium.webdriver.firefox.options import Options as FirefoxOptions
|
||||
from selenium import webdriver
|
||||
options = FirefoxOptions()
|
||||
options.headless = self.headless
|
||||
# Initialize undetected Firefox driver (if required)
|
||||
@ -131,9 +140,9 @@ class ChromiumLoader(BaseLoader):
|
||||
except (aiohttp.ClientError, asyncio.TimeoutError) as e:
|
||||
attempt += 1
|
||||
logger.error(f"Attempt {attempt} failed: {e}")
|
||||
if attempt == self.RETRY_LIMIT:
|
||||
if attempt == self.retry_limit:
|
||||
results = (
|
||||
f"Error: Network error after {self.RETRY_LIMIT} attempts - {e}"
|
||||
f"Error: Network error after {self.retry_limit} attempts - {e}"
|
||||
)
|
||||
finally:
|
||||
driver.quit()
|
||||
@ -201,7 +210,7 @@ class ChromiumLoader(BaseLoader):
|
||||
results = ""
|
||||
attempt = 0
|
||||
|
||||
while attempt < self.RETRY_LIMIT:
|
||||
while attempt < self.retry_limit:
|
||||
try:
|
||||
async with async_playwright() as p:
|
||||
browser = None
|
||||
@ -265,8 +274,8 @@ class ChromiumLoader(BaseLoader):
|
||||
except (aiohttp.ClientError, asyncio.TimeoutError, Exception) as e:
|
||||
attempt += 1
|
||||
logger.error(f"Attempt {attempt} failed: {e}")
|
||||
if attempt == self.RETRY_LIMIT:
|
||||
results = f"Error: Network error after {self.RETRY_LIMIT} attempts - {e}"
|
||||
if attempt == self.retry_limit:
|
||||
results = f"Error: Network error after {self.retry_limit} attempts - {e}"
|
||||
finally:
|
||||
await browser.close()
|
||||
|
||||
@ -280,7 +289,11 @@ class ChromiumLoader(BaseLoader):
|
||||
url (str): The URL to scrape.
|
||||
|
||||
Returns:
|
||||
str: The scraped HTML content or an error message if an exception occurs.
|
||||
str: The scraped HTML content
|
||||
|
||||
Raises:
|
||||
RuntimeError: When retry limit is reached without successful scraping
|
||||
ValueError: When an invalid browser name is provided
|
||||
"""
|
||||
from playwright.async_api import async_playwright
|
||||
from undetected_playwright import Malenia
|
||||
@ -289,9 +302,9 @@ class ChromiumLoader(BaseLoader):
|
||||
results = ""
|
||||
attempt = 0
|
||||
|
||||
while attempt < self.RETRY_LIMIT:
|
||||
while attempt < self.retry_limit:
|
||||
try:
|
||||
async with async_playwright() as p, async_timeout.timeout(self.TIMEOUT):
|
||||
async with async_playwright() as p, async_timeout.timeout(self.timeout):
|
||||
browser = None
|
||||
if browser_name == "chromium":
|
||||
browser = await p.chromium.launch(
|
||||
@ -312,22 +325,15 @@ class ChromiumLoader(BaseLoader):
|
||||
await page.wait_for_load_state(self.load_state)
|
||||
results = await page.content()
|
||||
logger.info("Content scraped")
|
||||
break
|
||||
await browser.close()
|
||||
return results
|
||||
except (aiohttp.ClientError, asyncio.TimeoutError, Exception) as e:
|
||||
attempt += 1
|
||||
logger.error(f"Attempt {attempt} failed: {e}")
|
||||
if attempt == self.RETRY_LIMIT:
|
||||
results = f"Error: Network error after {self.RETRY_LIMIT} attempts - {e}"
|
||||
finally:
|
||||
if "browser" in locals():
|
||||
await browser.close()
|
||||
if attempt == self.retry_limit:
|
||||
raise RuntimeError(f"Failed to scrape after {self.retry_limit} attempts: {str(e)}")
|
||||
|
||||
|
||||
return results
|
||||
|
||||
|
||||
|
||||
async def ascrape_with_js_support(self, url: str , browser_name:str = "chromium") -> str:
|
||||
async def ascrape_with_js_support(self, url: str, browser_name: str = "chromium") -> str:
|
||||
"""
|
||||
Asynchronously scrape the content of a given URL by rendering JavaScript using Playwright.
|
||||
|
||||
@ -335,18 +341,20 @@ class ChromiumLoader(BaseLoader):
|
||||
url (str): The URL to scrape.
|
||||
|
||||
Returns:
|
||||
str: The fully rendered HTML content after JavaScript execution,
|
||||
or an error message if an exception occurs.
|
||||
str: The fully rendered HTML content after JavaScript execution
|
||||
|
||||
Raises:
|
||||
RuntimeError: When retry limit is reached without successful scraping
|
||||
ValueError: When an invalid browser name is provided
|
||||
"""
|
||||
from playwright.async_api import async_playwright
|
||||
|
||||
logger.info(f"Starting scraping with JavaScript support for {url}...")
|
||||
results = ""
|
||||
attempt = 0
|
||||
|
||||
while attempt < self.RETRY_LIMIT:
|
||||
while attempt < self.retry_limit:
|
||||
try:
|
||||
async with async_playwright() as p, async_timeout.timeout(self.TIMEOUT):
|
||||
async with async_playwright() as p, async_timeout.timeout(self.timeout):
|
||||
browser = None
|
||||
if browser_name == "chromium":
|
||||
browser = await p.chromium.launch(
|
||||
@ -365,19 +373,15 @@ class ChromiumLoader(BaseLoader):
|
||||
await page.goto(url, wait_until="networkidle")
|
||||
results = await page.content()
|
||||
logger.info("Content scraped after JavaScript rendering")
|
||||
break
|
||||
return results
|
||||
except (aiohttp.ClientError, asyncio.TimeoutError, Exception) as e:
|
||||
attempt += 1
|
||||
logger.error(f"Attempt {attempt} failed: {e}")
|
||||
if attempt == self.RETRY_LIMIT:
|
||||
results = (
|
||||
f"Error: Network error after {self.RETRY_LIMIT} attempts - {e}"
|
||||
)
|
||||
if attempt == self.retry_limit:
|
||||
raise RuntimeError(f"Failed to scrape after {self.retry_limit} attempts: {str(e)}")
|
||||
finally:
|
||||
await browser.close()
|
||||
|
||||
return results
|
||||
|
||||
def lazy_load(self) -> Iterator[Document]:
|
||||
"""
|
||||
Lazily load text content from the provided URLs.
|
||||
|
||||
@ -234,7 +234,7 @@ class AbstractGraph(ABC):
|
||||
from langchain_together import ChatTogether
|
||||
except ImportError:
|
||||
raise ImportError("""The langchain_together module is not installed.
|
||||
Please install it using `pip install scrapegraphai[other-language-models]`.""")
|
||||
Please install it using `pip install langchain-together`.""")
|
||||
return ChatTogether(**llm_params)
|
||||
|
||||
elif model_provider == "nvidia":
|
||||
@ -242,7 +242,7 @@ class AbstractGraph(ABC):
|
||||
from langchain_nvidia_ai_endpoints import ChatNVIDIA
|
||||
except ImportError:
|
||||
raise ImportError("""The langchain_nvidia_ai_endpoints module is not installed.
|
||||
Please install it using `pip install scrapegraphai[other-language-models]`.""")
|
||||
Please install it using `pip install langchain-nvidia-ai-endpoints`.""")
|
||||
return ChatNVIDIA(**llm_params)
|
||||
|
||||
except Exception as e:
|
||||
|
||||
@ -3,8 +3,6 @@ SmartScraperGraph Module
|
||||
"""
|
||||
from typing import Optional
|
||||
from pydantic import BaseModel
|
||||
from scrapegraph_py import Client
|
||||
from scrapegraph_py.logger import sgai_logger
|
||||
from .base_graph import BaseGraph
|
||||
from .abstract_graph import AbstractGraph
|
||||
from ..nodes import (
|
||||
@ -55,6 +53,9 @@ class SmartScraperGraph(AbstractGraph):
|
||||
super().__init__(prompt, config, source, schema)
|
||||
|
||||
self.input_key = "url" if source.startswith("http") else "local_dir"
|
||||
|
||||
# for detailed logging of the SmartScraper API set it to True
|
||||
self.verbose = config.get("verbose", False)
|
||||
|
||||
def _create_graph(self) -> BaseGraph:
|
||||
"""
|
||||
@ -64,7 +65,12 @@ class SmartScraperGraph(AbstractGraph):
|
||||
BaseGraph: A graph instance representing the web scraping workflow.
|
||||
"""
|
||||
if self.llm_model == "scrapegraphai/smart-scraper":
|
||||
|
||||
try:
|
||||
from scrapegraph_py import Client
|
||||
from scrapegraph_py.logger import sgai_logger
|
||||
except ImportError:
|
||||
raise ImportError("scrapegraph_py is not installed. Please install it using 'pip install scrapegraph-py'.")
|
||||
|
||||
sgai_logger.set_logging(level="INFO")
|
||||
|
||||
# Initialize the client with explicit API key
|
||||
|
||||
@ -4,14 +4,12 @@ FetchNode Module
|
||||
import json
|
||||
from typing import List, Optional
|
||||
from langchain_openai import ChatOpenAI, AzureChatOpenAI
|
||||
import pandas as pd
|
||||
import requests
|
||||
from langchain_community.document_loaders import PyPDFLoader
|
||||
from langchain_core.documents import Document
|
||||
from ..utils.cleanup_html import cleanup_html
|
||||
from ..docloaders import ChromiumLoader
|
||||
from ..utils.convert_to_md import convert_to_md
|
||||
from ..utils.logging import get_logger
|
||||
from .base_node import BaseNode
|
||||
|
||||
class FetchNode(BaseNode):
|
||||
@ -80,24 +78,6 @@ class FetchNode(BaseNode):
|
||||
None if node_config is None else node_config.get("storage_state", None)
|
||||
)
|
||||
|
||||
def is_valid_url(self, source: str) -> bool:
|
||||
"""
|
||||
Validates if the source string is a valid URL using regex.
|
||||
|
||||
Parameters:
|
||||
source (str): The URL string to validate
|
||||
|
||||
Raises:
|
||||
ValueError: If the URL is invalid
|
||||
"""
|
||||
import re
|
||||
|
||||
url_pattern = r"^https?://[^\s/$.?#].[^\s]*$"
|
||||
if not bool(re.match(url_pattern, source)):
|
||||
raise ValueError(
|
||||
f"Invalid URL format: {source}. URL must start with http(s):// and contain a valid domain."
|
||||
)
|
||||
return True
|
||||
|
||||
def execute(self, state):
|
||||
"""
|
||||
@ -130,12 +110,9 @@ class FetchNode(BaseNode):
|
||||
elif self.input == "pdf_dir":
|
||||
return state
|
||||
|
||||
# For web sources, validate URL before proceeding
|
||||
try:
|
||||
if self.is_valid_url(source):
|
||||
return self.handle_web_source(state, source)
|
||||
return self.handle_web_source(state, source)
|
||||
except ValueError as e:
|
||||
# Re-raise the exception from is_valid_url
|
||||
raise
|
||||
|
||||
return self.handle_local_source(state, source)
|
||||
@ -199,6 +176,10 @@ class FetchNode(BaseNode):
|
||||
loader = PyPDFLoader(source)
|
||||
return loader.load()
|
||||
elif input_type == "csv":
|
||||
try:
|
||||
import pandas as pd
|
||||
except ImportError:
|
||||
raise ImportError("pandas is not installed. Please install it using `pip install pandas`.")
|
||||
return [
|
||||
Document(
|
||||
page_content=str(pd.read_csv(source)), metadata={"source": "csv"}
|
||||
|
||||
@ -160,20 +160,42 @@ class FetchNodeLevelK(BaseNode):
|
||||
def get_full_links(self, base_url: str, links: list) -> list:
|
||||
"""
|
||||
Converts relative URLs to full URLs based on the base URL.
|
||||
Filters out non-web links (mailto:, tel:, javascript:, etc.).
|
||||
|
||||
Args:
|
||||
base_url (str): The base URL for resolving relative links.
|
||||
links (list): A list of links to convert.
|
||||
|
||||
Returns:
|
||||
list: A list of full URLs.
|
||||
list: A list of valid full URLs.
|
||||
"""
|
||||
# List of invalid URL schemes to filter out
|
||||
invalid_schemes = {
|
||||
'mailto:', 'tel:', 'fax:', 'sms:', 'callto:', 'wtai:', 'javascript:',
|
||||
'data:', 'file:', 'ftp:', 'irc:', 'news:', 'nntp:', 'feed:', 'webcal:',
|
||||
'skype:', 'im:', 'mtps:', 'spotify:', 'steam:', 'teamspeak:', 'udp:',
|
||||
'unreal:', 'ut2004:', 'ventrilo:', 'view-source:', 'ws:', 'wss:'
|
||||
}
|
||||
|
||||
full_links = []
|
||||
for link in links:
|
||||
if self.only_inside_links and link.startswith("http"):
|
||||
# Skip if link starts with any invalid scheme
|
||||
if any(link.lower().startswith(scheme) for scheme in invalid_schemes):
|
||||
continue
|
||||
full_link = link if link.startswith("http") else urljoin(base_url, link)
|
||||
full_links.append(full_link)
|
||||
|
||||
# Skip if it's an external link and only_inside_links is True
|
||||
if self.only_inside_links and link.startswith(('http://', 'https://')):
|
||||
continue
|
||||
|
||||
# Convert relative URLs to absolute URLs
|
||||
try:
|
||||
full_link = link if link.startswith(('http://', 'https://')) else urljoin(base_url, link)
|
||||
# Ensure the final URL starts with http:// or https://
|
||||
if full_link.startswith(('http://', 'https://')):
|
||||
full_links.append(full_link)
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Failed to process link {link}: {str(e)}")
|
||||
|
||||
return full_links
|
||||
|
||||
def obtain_content(self, documents: List, loader_kwargs) -> List:
|
||||
@ -191,7 +213,11 @@ class FetchNodeLevelK(BaseNode):
|
||||
for doc in documents:
|
||||
source = doc["source"]
|
||||
if "document" not in doc:
|
||||
document = self.fetch_content(source, loader_kwargs)
|
||||
try:
|
||||
document = self.fetch_content(source, loader_kwargs)
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Failed to fetch content for {source}: {str(e)}")
|
||||
continue
|
||||
|
||||
if not document or not document[0].page_content.strip():
|
||||
self.logger.warning(f"Failed to fetch content for {source}")
|
||||
|
||||
@ -12,10 +12,8 @@ from langchain_aws import ChatBedrock
|
||||
from langchain_community.chat_models import ChatOllama
|
||||
from tqdm import tqdm
|
||||
from .base_node import BaseNode
|
||||
from ..utils.output_parser import get_structured_output_parser, get_pydantic_output_parser
|
||||
from ..utils.output_parser import get_pydantic_output_parser
|
||||
from requests.exceptions import Timeout
|
||||
from langchain.callbacks.manager import CallbackManager
|
||||
from langchain.callbacks import get_openai_callback
|
||||
from ..prompts import (
|
||||
TEMPLATE_CHUNKS, TEMPLATE_NO_CHUNKS, TEMPLATE_MERGE,
|
||||
TEMPLATE_CHUNKS_MD, TEMPLATE_NO_CHUNKS_MD, TEMPLATE_MERGE_MD
|
||||
|
||||
@ -96,7 +96,17 @@ class MergeAnswersNode(BaseNode):
|
||||
|
||||
merge_chain = prompt_template | self.llm_model | output_parser
|
||||
answer = merge_chain.invoke({"user_prompt": user_prompt})
|
||||
answer["sources"] = state.get("urls", [])
|
||||
|
||||
# Get the URLs from the state, ensuring we get the actual URLs used for scraping
|
||||
urls = []
|
||||
if "urls" in state:
|
||||
urls = state["urls"]
|
||||
elif "considered_urls" in state:
|
||||
urls = state["considered_urls"]
|
||||
|
||||
# Only add sources if we actually have URLs
|
||||
if urls:
|
||||
answer["sources"] = urls
|
||||
|
||||
state.update({self.output[0]: answer})
|
||||
return state
|
||||
|
||||
@ -3,8 +3,6 @@ RAGNode Module
|
||||
"""
|
||||
from typing import List, Optional
|
||||
from .base_node import BaseNode
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.models import PointStruct, VectorParams, Distance
|
||||
|
||||
class RAGNode(BaseNode):
|
||||
"""
|
||||
@ -42,6 +40,14 @@ class RAGNode(BaseNode):
|
||||
def execute(self, state: dict) -> dict:
|
||||
self.logger.info(f"--- Executing {self.node_name} Node ---")
|
||||
|
||||
try:
|
||||
import qdrant_client
|
||||
except ImportError:
|
||||
raise ImportError("qdrant_client is not installed. Please install it using 'pip install qdrant-client'.")
|
||||
|
||||
from qdrant_client import QdrantClient
|
||||
from qdrant_client.models import PointStruct, VectorParams, Distance
|
||||
|
||||
if self.node_config.get("client_type") in ["memory", None]:
|
||||
client = QdrantClient(":memory:")
|
||||
elif self.node_config.get("client_type") == "local_db":
|
||||
|
||||
@ -99,5 +99,8 @@ class SearchInternetNode(BaseNode):
|
||||
if len(answer) == 0:
|
||||
raise ValueError("Zero results found for the search query.")
|
||||
|
||||
# Store both the URLs and considered_urls in the state
|
||||
state.update({self.output[0]: answer})
|
||||
state["considered_urls"] = answer # Add this as a backup
|
||||
|
||||
return state
|
||||
@ -7,8 +7,7 @@ Classes:
|
||||
|
||||
import threading
|
||||
from contextlib import contextmanager
|
||||
from langchain_community.callbacks import get_openai_callback
|
||||
from langchain_community.callbacks.manager import get_bedrock_anthropic_callback
|
||||
from langchain_community.callbacks.manager import get_openai_callback, get_bedrock_anthropic_callback
|
||||
from langchain_openai import ChatOpenAI, AzureChatOpenAI
|
||||
from langchain_aws import ChatBedrock
|
||||
from .custom_callback import get_custom_callback
|
||||
|
||||
@ -1,25 +1,45 @@
|
||||
"""
|
||||
Prettify the execution information of the graph.
|
||||
"""
|
||||
import pandas as pd
|
||||
from typing import Union
|
||||
|
||||
def prettify_exec_info(complete_result: list[dict]) -> pd.DataFrame:
|
||||
def prettify_exec_info(complete_result: list[dict], as_string: bool = True) -> Union[str, list[dict]]:
|
||||
"""
|
||||
Transforms the execution information of a graph into a DataFrame for enhanced visualization.
|
||||
Formats the execution information of a graph showing node statistics.
|
||||
|
||||
Args:
|
||||
complete_result (list[dict]): The complete execution information of the graph.
|
||||
complete_result (list[dict]): The execution information containing node statistics.
|
||||
as_string (bool, optional): If True, returns a formatted string table.
|
||||
If False, returns the original list. Defaults to True.
|
||||
|
||||
Returns:
|
||||
pd.DataFrame: A DataFrame that organizes the execution information
|
||||
for better readability and analysis.
|
||||
|
||||
Example:
|
||||
>>> prettify_exec_info([{'node': 'A', 'status': 'success'},
|
||||
{'node': 'B', 'status': 'failure'}])
|
||||
DataFrame with columns 'node' and 'status' showing execution results for each node.
|
||||
Union[str, list[dict]]: A formatted string table if as_string=True,
|
||||
otherwise the original list of dictionaries.
|
||||
"""
|
||||
if not as_string:
|
||||
return complete_result
|
||||
|
||||
df_nodes = pd.DataFrame(complete_result)
|
||||
if not complete_result:
|
||||
return "Empty result"
|
||||
|
||||
return df_nodes
|
||||
# Format the table
|
||||
lines = []
|
||||
lines.append("Node Statistics:")
|
||||
lines.append("-" * 100)
|
||||
lines.append(f"{'Node':<20} {'Tokens':<10} {'Prompt':<10} {'Compl.':<10} {'Requests':<10} {'Cost ($)':<10} {'Time (s)':<10}")
|
||||
lines.append("-" * 100)
|
||||
|
||||
for item in complete_result:
|
||||
node = item['node_name']
|
||||
tokens = item['total_tokens']
|
||||
prompt = item['prompt_tokens']
|
||||
completion = item['completion_tokens']
|
||||
requests = item['successful_requests']
|
||||
cost = f"{item['total_cost_USD']:.4f}"
|
||||
time = f"{item['exec_time']:.2f}"
|
||||
|
||||
lines.append(
|
||||
f"{node:<20} {tokens:<10} {prompt:<10} {completion:<10} {requests:<10} {cost:<10} {time:<10}"
|
||||
)
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
@ -41,7 +41,7 @@ def search_on_web(query: str, search_engine: str = "Google",
|
||||
research = DuckDuckGoSearchResults(max_results=max_results)
|
||||
res = research.run(query)
|
||||
links = re.findall(r'https?://[^\s,\]]+', res)
|
||||
return links
|
||||
return links[:max_results]
|
||||
|
||||
elif search_engine.lower() == "bing":
|
||||
headers = {
|
||||
@ -66,7 +66,7 @@ def search_on_web(query: str, search_engine: str = "Google",
|
||||
response = requests.get(url, params=params)
|
||||
|
||||
data = response.json()
|
||||
limited_results = data["results"][:max_results]
|
||||
limited_results = [result['url'] for result in data["results"][:max_results]]
|
||||
return limited_results
|
||||
|
||||
else:
|
||||
|
||||
@ -22,7 +22,7 @@ def detect_text(image, languages: list = ["en"]):
|
||||
from surya.model.recognition.model import load_model as load_rec_model
|
||||
from surya.model.recognition.processor import load_processor as load_rec_processor
|
||||
except:
|
||||
raise ImportError("The dependencies for screenshot scraping are not installed. Please install them using `pip install scrapegraphai[screenshot_scraper]`.")
|
||||
raise ImportError("The dependencies for OCR are not installed. Please install them using `pip install scrapegraphai[ocr]`.")
|
||||
|
||||
|
||||
langs = languages
|
||||
|
||||
@ -1,10 +1,6 @@
|
||||
"""
|
||||
Tokenization utilities for Mistral models
|
||||
"""
|
||||
from mistral_common.protocol.instruct.messages import UserMessage
|
||||
from mistral_common.protocol.instruct.request import ChatCompletionRequest
|
||||
from mistral_common.protocol.instruct.tool_calls import Function, Tool
|
||||
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
|
||||
from langchain_core.language_models.chat_models import BaseChatModel
|
||||
from ..logging import get_logger
|
||||
|
||||
@ -31,6 +27,13 @@ def num_tokens_mistral(text: str, llm_model:BaseChatModel) -> int:
|
||||
raise NotImplementedError(f"The model provider you are using ('{llm_model}') "
|
||||
"does not give us a model name so we cannot identify which encoding to use")
|
||||
|
||||
try:
|
||||
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
|
||||
from mistral_common.protocol.instruct.messages import UserMessage
|
||||
from mistral_common.protocol.instruct.request import ChatCompletionRequest
|
||||
except ImportError:
|
||||
raise ImportError("mistral_common is not installed. Please install it using 'pip install mistral-common'.")
|
||||
|
||||
tokenizer = MistralTokenizer.from_model(model)
|
||||
|
||||
tokenized = tokenizer.encode_chat_completion(
|
||||
|
||||
@ -11,7 +11,6 @@ from scrapegraphai.nodes import (
|
||||
from scrapegraphai.models import OneApi, DeepSeek
|
||||
from langchain_openai import ChatOpenAI, AzureChatOpenAI
|
||||
from langchain_ollama import ChatOllama
|
||||
from langchain_google_genai import ChatGoogleGenerativeAI
|
||||
from langchain_aws import ChatBedrock
|
||||
|
||||
|
||||
@ -68,7 +67,6 @@ class TestAbstractGraph:
|
||||
"api_version": "no version",
|
||||
"azure_endpoint": "https://www.example.com/"},
|
||||
AzureChatOpenAI),
|
||||
({"model": "google_genai/gemini-pro", "google_api_key": "google-key-test"}, ChatGoogleGenerativeAI),
|
||||
({"model": "ollama/llama2"}, ChatOllama),
|
||||
({"model": "oneapi/qwen-turbo", "api_key": "oneapi-api-key"}, OneApi),
|
||||
({"model": "deepseek/deepseek-coder", "api_key": "deepseek-api-key"}, DeepSeek),
|
||||
@ -86,7 +84,6 @@ class TestAbstractGraph:
|
||||
@pytest.mark.parametrize("llm_config, expected_model", [
|
||||
({"model": "openai/gpt-3.5-turbo", "openai_api_key": "sk-randomtest001", "rate_limit": {"requests_per_second": 1}}, ChatOpenAI),
|
||||
({"model": "azure_openai/gpt-3.5-turbo", "api_key": "random-api-key", "api_version": "no version", "azure_endpoint": "https://www.example.com/", "rate_limit": {"requests_per_second": 1}}, AzureChatOpenAI),
|
||||
({"model": "google_genai/gemini-pro", "google_api_key": "google-key-test", "rate_limit": {"requests_per_second": 1}}, ChatGoogleGenerativeAI),
|
||||
({"model": "ollama/llama2", "rate_limit": {"requests_per_second": 1}}, ChatOllama),
|
||||
({"model": "oneapi/qwen-turbo", "api_key": "oneapi-api-key", "rate_limit": {"requests_per_second": 1}}, OneApi),
|
||||
({"model": "deepseek/deepseek-coder", "api_key": "deepseek-api-key", "rate_limit": {"requests_per_second": 1}}, DeepSeek),
|
||||
|
||||
@ -4,10 +4,8 @@ Module for testing the scrape graph class
|
||||
|
||||
import os
|
||||
import pytest
|
||||
import pandas as pd
|
||||
from dotenv import load_dotenv
|
||||
from scrapegraphai.graphs import ScrapeGraph
|
||||
from scrapegraphai.utils import prettify_exec_info
|
||||
|
||||
load_dotenv()
|
||||
|
||||
|
||||
@ -4,10 +4,8 @@ Module for testing the smart scraper class
|
||||
|
||||
import os
|
||||
import pytest
|
||||
import pandas as pd
|
||||
from dotenv import load_dotenv
|
||||
from scrapegraphai.graphs import SmartScraperGraph
|
||||
from scrapegraphai.utils import prettify_exec_info
|
||||
|
||||
load_dotenv()
|
||||
|
||||
|
||||
@ -4,10 +4,8 @@ Module for testing the smart scraper class
|
||||
|
||||
import os
|
||||
import pytest
|
||||
import pandas as pd
|
||||
from dotenv import load_dotenv
|
||||
from scrapegraphai.graphs import SmartScraperMultiLiteGraph
|
||||
from scrapegraphai.utils import prettify_exec_info
|
||||
|
||||
load_dotenv()
|
||||
|
||||
|
||||
@ -3,7 +3,6 @@ Module for testing th smart scraper class
|
||||
"""
|
||||
import pytest
|
||||
from scrapegraphai.graphs import SmartScraperGraph
|
||||
from transformers import GPT2TokenizerFast
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@ -52,10 +51,3 @@ def test_get_execution_info(graph_config: dict):
|
||||
|
||||
assert graph_exec_info is not None
|
||||
|
||||
|
||||
def test_gpt2_tokenizer_loading():
|
||||
"""
|
||||
Test loading of GPT2TokenizerFast
|
||||
"""
|
||||
tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
|
||||
assert tokenizer is not None
|
||||
|
||||
@ -4,10 +4,8 @@ Module for testing the smart scraper class
|
||||
|
||||
import os
|
||||
import pytest
|
||||
import pandas as pd
|
||||
from dotenv import load_dotenv
|
||||
from scrapegraphai.graphs import SmartScraperGraph
|
||||
from scrapegraphai.utils import prettify_exec_info
|
||||
|
||||
load_dotenv()
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user