CodeCow/Scrapegraph-ai
1
0
Fork 0
mirror of https://github.com/VinciGit00/Scrapegraph-ai.git synced 2026-07-09 21:19:20 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #747 from ScrapeGraphAI/pre/beta

Browse Source
This commit is contained in:
Marco Vinciguerra 2024-10-13 09:33:28 +02:00 committed by GitHub
commit 5799124883
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
335 changed files with 1490 additions and 3083 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

157
CHANGELOG.md
View File

@ -1,3 +1,123 @@
## [1.26.0-beta.17](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.16...v1.26.0-beta.17) (2024-10-12)
### Features
* async invocation ([257f393](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/257f393761e8ff823e37c72659c8b55925c4aecb))
* refactoring of mdscraper ([3b7b701](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3b7b701a89aad503dea771db3f043167f7203d46))
### Bug Fixes
* bugs ([026a70b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/026a70bd3a01b0ebab4d175ae4005e7f3ba3a833))
* search_on_web paremter ([7f03ec1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7f03ec15de20fc2d6c2aad2655cc5348cced1951))
## [1.26.0-beta.16](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.15...v1.26.0-beta.16) (2024-10-11)
### Features
* add google proxy support ([a986523](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/a9865238847e2edccde579ace7ba226f7012e95d))
### Bug Fixes
* typo ([e285127](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e28512720c3d47917814cf388912aef0e2230188))
### Perf
* Proxy integration in googlesearch ([e828c70](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e828c7010acb1bd04498e027da69f35d53a37890))
## [1.26.0-beta.15](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.14...v1.26.0-beta.15) (2024-10-11)
### Features
* prompt refactoring ([5a2f6d9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5a2f6d9a77a814d5c3756e85cabde8af978f4c06))
## [1.26.0-beta.14](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.13...v1.26.0-beta.14) (2024-10-10)
### Features
* refactoring fetch_node ([39a029e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/39a029ed9a8cd7c2277ba1386b976738e99d231b))
## [1.26.0-beta.13](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.12...v1.26.0-beta.13) (2024-10-10)
### Features
* update chromium loader ([4f816f3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4f816f3b04974e90ca4208158f05724cfe68ffb8))
## [1.26.0-beta.12](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.11...v1.26.0-beta.12) (2024-10-09)
### Bug Fixes
* nodes prompt ([8753537](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8753537ecd2a0ba480cda482b6dc50c090b418d6))
## [1.26.0-beta.11](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.10...v1.26.0-beta.11) (2024-10-09)
### Bug Fixes
* refactoring prompts ([c655642](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c65564257798a5ccdc2bdf92487cd9b069e6d951))
## [1.26.0-beta.10](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.9...v1.26.0-beta.10) (2024-10-09)
### Bug Fixes
* removed pdf_scraper graph and created document scraper ([a57da96](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/a57da96175a09a16d990eeee679988d10832ce13))
## [1.26.0-beta.9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.8...v1.26.0-beta.9) (2024-10-08)
### Bug Fixes
* pyproject.toml ([3b27c5e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3b27c5e88c0b0744438e8b604f40929e22d722bc))
## [1.26.0-beta.8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.7...v1.26.0-beta.8) (2024-10-08)
### Features
* undected_chromedriver support ([80ece21](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/80ece2179ac47a7ea42fbae4b61504a49ca18daa))
## [1.26.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.6...v1.26.0-beta.7) (2024-10-07)
### Bug Fixes
* import error ([37b6ba0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/37b6ba08ae9972240fc00a15efe43233fd093f3b))
## [1.26.0-beta.6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.5...v1.26.0-beta.6) (2024-10-07)
### Features
* refactoring of the conditional node ([420c71b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/420c71ba2ca0fc77465dd533a807b887c6a87f52))
## [1.26.0-beta.5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.4...v1.26.0-beta.5) (2024-10-05)
### Features
* conditional_node ([f837dc1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f837dc16ce6db0f38fd181822748ca413b7ab4b0))
## [1.26.0-beta.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.3...v1.26.0-beta.4) (2024-10-05)
### Bug Fixes
* update dependencies ([7579d0e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7579d0e2599d63c0003b1b7a0918132511a9c8f1))
### CI
* **release:** 1.25.2 [skip ci] ([5db4c51](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5db4c518056e9946c00f2fdab612786e0db9ce95))
## [1.25.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.25.1...v1.25.2) (2024-10-03)
@ -6,12 +126,49 @@
* update dependencies ([7579d0e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7579d0e2599d63c0003b1b7a0918132511a9c8f1))
## [1.25.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.25.0...v1.25.1) (2024-09-29)
## [1.26.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.2...v1.26.0-beta.3) (2024-10-04)
### Features
* add deep scraper implementation ([4b371f4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4b371f4d94dae47986aad751508813d89ce87b93))
* finished basic version of deep scraper ([85cb957](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/85cb9572971719f9f7c66171f5e2246376b6aed2))
## [1.26.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.26.0-beta.1...v1.26.0-beta.2) (2024-10-01)
### Features
* refactoring of research web ([26f89d8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/26f89d895d547ef2463492f82da7ac21b57b9d1b))
### CI
* **release:** 1.25.1 [skip ci] ([a98328c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/a98328c7f2f39bdd609615247cb71ecf912a3bd8))
## [1.26.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.25.0...v1.26.0-beta.1) (2024-09-29)
* add html_mode to smart_scraper ([bdcffd6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/bdcffd6360237b27797546a198ceece55ce4bc81))
* add reasoning integration ([b2822f6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b2822f620a610e61d295cbf4b670aa08fde9de24))
### Bug Fixes
* removed deep scraper ([9aa8c88](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/9aa8c889fb32f2eb2005a2fb04f05dc188092279))
* integration with html_mode ([f87ffa1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f87ffa1d8db32b38c47d9f5aa2ae88f1d7978a04))
* removed deep scraper ([9aa8c88](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/9aa8c889fb32f2eb2005a2fb04f05dc188092279))
### CI
* **release:** 1.22.0-beta.4 [skip ci] ([4330179](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4330179cb65674d65423c1763f90182e85c15a74))
* **release:** 1.22.0-beta.5 [skip ci] ([6d8f543](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/6d8f5435d1ecd2d90b06aade50abc064f75c9d78))
* **release:** 1.22.0-beta.6 [skip ci] ([39f7815](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/39f78154a6f1123fa8aca5e169c803111c175473))
## [1.25.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.24.1...v1.25.0) (2024-09-27)

1
examples/anthropic/code_generator_graph_anthropic.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using Code Generator with schema
"""
import os, json
from typing import List
from dotenv import load_dotenv

1
examples/anthropic/csv_scraper_anthropic.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using CSVScraperGraph from CSV documents
"""
import os
from dotenv import load_dotenv
import pandas as pd

1
examples/anthropic/csv_scraper_graph_multi_anthropic.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using CSVScraperMultiGraph from CSV documents
"""
import os
from dotenv import load_dotenv
import pandas as pd

2
examples/anthropic/custom_graph_anthropic.py
View File

@ -1,10 +1,8 @@
"""
Example of custom graph using existing nodes
"""
import os
from dotenv import load_dotenv
from langchain_anthropic import ChatAnthropic
from scrapegraphai.graphs import BaseGraph
from scrapegraphai.nodes import FetchNode, ParseNode, GenerateAnswerNode, RobotsNode

2
examples/anthropic/json_scraper_anthropic.py
View File

@ -1,11 +1,11 @@
"""
Basic example of scraping pipeline using JSONScraperGraph from JSON documents
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import JSONScraperGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
load_dotenv()
# ************************************************

39
examples/anthropic/pdf_scraper_graph_anthropic.py
View File

@ -1,39 +0,0 @@
"""
Module for showing how PDFScraper multi works
"""
import os, json
from dotenv import load_dotenv
from scrapegraphai.graphs import PDFScraperGraph
load_dotenv()
# ************************************************
# Define the configuration for the graph
# ************************************************
graph_config = {
"llm": {
"api_key": os.getenv("ANTHROPIC_API_KEY"),
"model": "anthropic/claude-3-haiku-20240307",
},
}
source = """
The Divine Comedy, Italian La Divina Commedia, original name La commedia, long narrative poem written in Italian
circa 1308/21 by Dante. It is usually held to be one of the world s great works of literature.
Divided into three major sections—Inferno, Purgatorio, and Paradiso—the narrative traces the journey of Dante
from darkness and error to the revelation of the divine light, culminating in the Beatific Vision of God.
Dante is guided by the Roman poet Virgil, who represents the epitome of human knowledge, from the dark wood
through the descending circles of the pit of Hell (Inferno). He then climbs the mountain of Purgatory, guided
by the Roman poet Statius, who represents the fulfilment of human knowledge, and is finally led by his lifelong love,
the Beatrice of his earlier poetry, through the celestial spheres of Paradise.
"""
pdf_scraper_graph = PDFScraperGraph(
prompt="Summarize the text and find the main topics",
source=source,
config=graph_config,
)
result = pdf_scraper_graph.run()
print(json.dumps(result, indent=4))

71
examples/anthropic/pdf_scraper_multi_anthropic.py
View File

@ -1,71 +0,0 @@
"""
Module for showing how PDFScraper multi works
"""
import os
import json
from dotenv import load_dotenv
from scrapegraphai.graphs import PdfScraperMultiGraph
load_dotenv()
graph_config = {
"llm": {
"api_key": os.getenv("ANTHROPIC_API_KEY"),
"model": "anthropic/claude-3-haiku-20240307",
},
}
# ***************
# Covert to list
# ***************
sources = [
"This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather—the interaction between call center architecture and outdoor weather conditions—in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity – largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.",
"This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather—the interaction between call center architecture and outdoor weather conditions—in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity – largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.",
"This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather—the interaction between call center architecture and outdoor weather conditions—in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity – largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.",
"This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather—the interaction between call center architecture and outdoor weather conditions—in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity – largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.",
]
prompt = """
You are an expert in reviewing academic manuscripts. Please analyze the abstracts provided from an academic journal article to extract and clearly identify the following elements:
Independent Variable (IV): The variable that is manipulated or considered as the primary cause affecting other variables.
Dependent Variable (DV): The variable that is measured or observed, which is expected to change as a result of variations in the Independent Variable.
Exogenous Shock: Identify any external or unexpected events used in the study that serve as a natural experiment or provide a unique setting for observing the effects on the IV and DV.
Response Format: For each abstract, present your response in the following structured format:
Independent Variable (IV):
Dependent Variable (DV):
Exogenous Shock:
Example Queries and Responses:
Query: This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather the interaction between call center architecture and outdoor weather conditions in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.
Response:
Independent Variable (IV): Employee happiness.
Dependent Variable (DV): Overall firm productivity.
Exogenous Shock: Sudden company-wide increase in bonus payments.
Query: The diffusion of social media coincided with a worsening of mental health conditions among adolescents and young adults in the United States, giving rise to speculation that social media might be detrimental to mental health. In this paper, we provide quasi-experimental estimates of the impact of social media on mental health by leveraging a unique natural experiment: the staggered introduction of Facebook across U.S. colleges. Our analysis couples data on student mental health around the years of Facebook's expansion with a generalized difference-in-differences empirical strategy. We find that the roll-out of Facebook at a college increased symptoms of poor mental health, especially depression. We also find that, among students predicted to be most susceptible to mental illness, the introduction of Facebook led to increased utilization of mental healthcare services. Lastly, we find that, after the introduction of Facebook, students were more likely to report experiencing impairments to academic performance resulting from poor mental health. Additional evidence on mechanisms suggests that the results are due to Facebook fostering unfavorable social comparisons.
Response:
Independent Variable (IV): Exposure to social media.
Dependent Variable (DV): Mental health outcomes.
Exogenous Shock: staggered introduction of Facebook across U.S. colleges.
"""
# *******************************************************
# Create the SmartScraperMultiGraph instance and run it
# *******************************************************
multiple_search_graph = PdfScraperMultiGraph(
prompt=prompt,
source= sources,
schema=None,
config=graph_config
)
result = multiple_search_graph.run()
print(json.dumps(result, indent=4))

1
examples/anthropic/rate_limit_anthropic.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper while setting an API rate limit.
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph

1
examples/anthropic/scrape_plain_text_anthropic.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper from text
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph

1
examples/anthropic/script_generator_anthropic.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import ScriptCreatorGraph

1
examples/anthropic/script_multi_generator_anthropic.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import ScriptCreatorMultiGraph

2
examples/anthropic/search_graph_anthropic.py
View File

@ -1,11 +1,11 @@
"""
Example of Search Graph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SearchGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
load_dotenv()
# ************************************************

1
examples/anthropic/search_graph_schema_anthropic.py
View File

@ -1,7 +1,6 @@
"""
Example of Search Graph
"""
import os
from typing import List
from dotenv import load_dotenv

4
examples/anthropic/smart_scraper_anthropic.py
View File

@ -1,15 +1,11 @@
"""
Basic example of scraping pipeline using SmartScraper using Azure OpenAI Key
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info
# required environment variables in .env
# ANTHROPIC_API_KEY
load_dotenv()
# ************************************************

4
examples/anthropic/smart_scraper_multi_anthropic.py
View File

@ -1,8 +1,8 @@
"""
Basic example of scraping pipeline using SmartScraper
"""
import os, json
import os
import json
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperMultiGraph

1
examples/anthropic/smart_scraper_multi_concat_anthropic.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper
"""
import os
import json
from dotenv import load_dotenv

5
examples/anthropic/smart_scraper_schema_anthropic.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper using Azure OpenAI Key
"""
import os
from typing import List
from pydantic import BaseModel, Field
@ -9,10 +8,6 @@ from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info
# required environment variables in .env
# HUGGINGFACEHUB_API_TOKEN
# ANTHROPIC_API_KEY
load_dotenv()
# ************************************************

2
examples/anthropic/xml_scraper_anthropic.py
View File

@ -1,11 +1,11 @@
"""
Basic example of scraping pipeline using XMLScraperGraph from XML documents
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import XMLScraperGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
load_dotenv()
# ************************************************

2
examples/anthropic/xml_scraper_graph_multi_anthropic.py
View File

@ -1,11 +1,11 @@
"""
Basic example of scraping pipeline using XMLScraperMultiGraph from XML documents
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import XMLScraperMultiGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
load_dotenv()
# ************************************************

5
examples/azure/code_generator_graph_azure.py
View File

@ -1,8 +1,7 @@
"""
Basic example of scraping pipeline using Code Generator with schema
"""
import os, json
import os
from typing import List
from dotenv import load_dotenv
from pydantic import BaseModel, Field
@ -55,4 +54,4 @@ code_generator_graph = CodeGeneratorGraph(
)
result = code_generator_graph.run()
print(result)
print(result)

2
examples/azure/csv_scraper_azure.py
View File

@ -1,12 +1,12 @@
"""
Basic example of scraping pipeline using CSVScraperGraph from CSV documents
"""
import os
from dotenv import load_dotenv
import pandas as pd
from scrapegraphai.graphs import CSVScraperGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
load_dotenv()
# ************************************************

1
examples/azure/csv_scraper_graph_multi_azure.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using CSVScraperMultiGraph from CSV documents
"""
import os
from dotenv import load_dotenv
import pandas as pd

1
examples/azure/json_scraper_azure.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper using Azure OpenAI Key
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import JSONScraperGraph

2
examples/azure/json_scraper_multi_azure.py
View File

@ -2,8 +2,8 @@
Module for showing how JSONScraperMultiGraph multi works
"""
import os
from dotenv import load_dotenv
import json
from dotenv import load_dotenv
from scrapegraphai.graphs import JSONScraperMultiGraph
load_dotenv()

37
examples/azure/pdf_scraper_azure.py
View File

@ -1,37 +0,0 @@
import os, json
from dotenv import load_dotenv
from scrapegraphai.graphs import PDFScraperGraph
load_dotenv()
# ************************************************
# Define the configuration for the graph
# ************************************************
graph_config = {
"llm": {
"api_key": os.environ["AZURE_OPENAI_KEY"],
"model": "azure_openai/gpt-4o"
},
"verbose": True,
"headless": False
}
source = """
The Divine Comedy, Italian La Divina Commedia, original name La commedia, long narrative poem written in Italian
circa 1308/21 by Dante. It is usually held to be one of the world s great works of literature.
Divided into three major sections—Inferno, Purgatorio, and Paradiso—the narrative traces the journey of Dante
from darkness and error to the revelation of the divine light, culminating in the Beatific Vision of God.
Dante is guided by the Roman poet Virgil, who represents the epitome of human knowledge, from the dark wood
through the descending circles of the pit of Hell (Inferno). He then climbs the mountain of Purgatory, guided
by the Roman poet Statius, who represents the fulfilment of human knowledge, and is finally led by his lifelong love,
the Beatrice of his earlier poetry, through the celestial spheres of Paradise.
"""
pdf_scraper_graph = PDFScraperGraph(
prompt="Summarize the text and find the main topics",
source=source,
config=graph_config,
)
result = pdf_scraper_graph.run()
print(json.dumps(result, indent=4))

1
examples/azure/rate_limit_azure.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper with a custom rate limit
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph

1
examples/azure/scrape_plain_text_azure.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper from text
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph

1
examples/azure/script_generator_azure.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import ScriptCreatorGraph

1
examples/azure/script_multi_generator_azure.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import ScriptCreatorMultiGraph

1
examples/azure/search_graph_azure.py
View File

@ -1,7 +1,6 @@
"""
Example of Search Graph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SearchGraph

7
examples/azure/search_graph_schema_azure.py
View File

@ -1,16 +1,15 @@
"""
Example of Search Graph
"""
import os
from typing import List
from dotenv import load_dotenv
load_dotenv()
from scrapegraphai.graphs import SearchGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
from pydantic import BaseModel, Field
from typing import List
load_dotenv()
# ************************************************
# Define the output schema for the graph

14
examples/azure/smart_scraper_azure.py
View File

@ -1,24 +1,13 @@
"""
Basic example of scraping pipeline using SmartScraper using Azure OpenAI Key
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info
# required environment variable in .env
# AZURE_OPENAI_ENDPOINT
# AZURE_OPENAI_CHAT_DEPLOYMENT_NAME
# MODEL_NAME
# AZURE_OPENAI_API_KEY
# OPENAI_API_TYPE
# AZURE_OPENAI_API_VERSION
# AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME
load_dotenv()
# ************************************************
# Initialize the model instances
# ************************************************
@ -33,7 +22,8 @@ graph_config = {
}
smart_scraper_graph = SmartScraperGraph(
prompt="""List me all the events, with the following fields: company_name, event_name, event_start_date, event_start_time,
prompt="""List me all the events, with the following fields:
company_name, event_name, event_start_date, event_start_time,
event_end_date, event_end_time, location, event_mode, event_category,
third_party_redirect, no_of_days,
time_in_hours, hosted_or_attending, refreshments_type,

1
examples/azure/smart_scraper_multi_concat_azure.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper
"""
import os
import json
from dotenv import load_dotenv

1
examples/azure/smart_scraper_schema_azure.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper with schema
"""
import os
import json
from typing import List

1
examples/azure/xml_scraper_azure.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper using Azure OpenAI Key
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import XMLScraperGraph

2
examples/azure/xml_scraper_graph_multi_azure.py
View File

@ -1,11 +1,11 @@
"""
Basic example of scraping pipeline using XMLScraperMultiGraph from XML documents
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import XMLScraperMultiGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
load_dotenv()
# ************************************************

6
examples/bedrock/depth_search_graph_bedrock.py
View File

@ -1,14 +1,8 @@
"""
depth_search_graph_opeani example
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import DepthSearchGraph
load_dotenv()
openai_key = os.getenv("OPENAI_APIKEY")
graph_config = {
"llm": {
"client": "client_name",

4
examples/bedrock/json_scraper_bedrock.py
View File

@ -1,12 +1,9 @@
"""
Basic example of scraping pipeline using JSONScraperGraph from JSON documents
"""
import os
import json
from dotenv import load_dotenv
from scrapegraphai.graphs import JSONScraperGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
@ -58,4 +55,3 @@ print(prettify_exec_info(graph_exec_info))
# Save to json or csv
convert_to_csv(result, "result")
convert_to_json(result, "result")

42
examples/bedrock/pdf_scraper_graph_bedrock.py
View File

@ -1,42 +0,0 @@
"""
Basic example of scraping pipeline using SmartScraper
"""
import os, json
from dotenv import load_dotenv
from scrapegraphai.utils import prettify_exec_info
from scrapegraphai.graphs import PDFScraperGraph
load_dotenv()
# ************************************************
# Define the configuration for the graph
# ************************************************
graph_config = {
"llm": {
"client": "client_name",
"model": "bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
"temperature": 0.0
}
}
source = """
The Divine Comedy, Italian La Divina Commedia, original name La commedia, long narrative poem written in Italian
circa 1308/21 by Dante. It is usually held to be one of the world s great works of literature.
Divided into three major sections—Inferno, Purgatorio, and Paradiso—the narrative traces the journey of Dante
from darkness and error to the revelation of the divine light, culminating in the Beatific Vision of God.
Dante is guided by the Roman poet Virgil, who represents the epitome of human knowledge, from the dark wood
through the descending circles of the pit of Hell (Inferno). He then climbs the mountain of Purgatory, guided
by the Roman poet Statius, who represents the fulfilment of human knowledge, and is finally led by his lifelong love,
the Beatrice of his earlier poetry, through the celestial spheres of Paradise.
"""
pdf_scraper_graph = PDFScraperGraph(
prompt="Summarize the text and find the main topics",
source=source,
config=graph_config,
)
result = pdf_scraper_graph.run()
print(json.dumps(result, indent=4))

69
examples/bedrock/pdf_scraper_graph_multi_bedrock.py
View File

@ -1,69 +0,0 @@
"""
Module for showing how PDFScraper multi works
"""
import os
import json
from dotenv import load_dotenv
from scrapegraphai.graphs import PdfScraperMultiGraph
graph_config = {
"llm": {
"client": "client_name",
"model": "bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
"temperature": 0.0
}
}
# ***************
# Covert to list
# ***************
sources = [
"This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather—the interaction between call center architecture and outdoor weather conditions—in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity – largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.",
"This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather—the interaction between call center architecture and outdoor weather conditions—in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity – largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.",
"This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather—the interaction between call center architecture and outdoor weather conditions—in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity – largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.",
"This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather—the interaction between call center architecture and outdoor weather conditions—in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity – largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.",
]
prompt = """
You are an expert in reviewing academic manuscripts. Please analyze the abstracts provided from an academic journal article to extract and clearly identify the following elements:
Independent Variable (IV): The variable that is manipulated or considered as the primary cause affecting other variables.
Dependent Variable (DV): The variable that is measured or observed, which is expected to change as a result of variations in the Independent Variable.
Exogenous Shock: Identify any external or unexpected events used in the study that serve as a natural experiment or provide a unique setting for observing the effects on the IV and DV.
Response Format: For each abstract, present your response in the following structured format:
Independent Variable (IV):
Dependent Variable (DV):
Exogenous Shock:
Example Queries and Responses:
Query: This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather the interaction between call center architecture and outdoor weather conditions in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.
Response:
Independent Variable (IV): Employee happiness.
Dependent Variable (DV): Overall firm productivity.
Exogenous Shock: Sudden company-wide increase in bonus payments.
Query: The diffusion of social media coincided with a worsening of mental health conditions among adolescents and young adults in the United States, giving rise to speculation that social media might be detrimental to mental health. In this paper, we provide quasi-experimental estimates of the impact of social media on mental health by leveraging a unique natural experiment: the staggered introduction of Facebook across U.S. colleges. Our analysis couples data on student mental health around the years of Facebook's expansion with a generalized difference-in-differences empirical strategy. We find that the roll-out of Facebook at a college increased symptoms of poor mental health, especially depression. We also find that, among students predicted to be most susceptible to mental illness, the introduction of Facebook led to increased utilization of mental healthcare services. Lastly, we find that, after the introduction of Facebook, students were more likely to report experiencing impairments to academic performance resulting from poor mental health. Additional evidence on mechanisms suggests that the results are due to Facebook fostering unfavorable social comparisons.
Response:
Independent Variable (IV): Exposure to social media.
Dependent Variable (DV): Mental health outcomes.
Exogenous Shock: staggered introduction of Facebook across U.S. colleges.
"""
# *******************************************************
# Create the SmartScraperMultiGraph instance and run it
# *******************************************************
multiple_search_graph = PdfScraperMultiGraph(
prompt=prompt,
source= sources,
schema=None,
config=graph_config
)
result = multiple_search_graph.run()
print(json.dumps(result, indent=4))

3
examples/bedrock/rate_limit_bedrock.py
View File

@ -1,15 +1,12 @@
"""
Basic example of scraping pipeline using SmartScraper with a custom rate limit
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info
load_dotenv()
# ************************************************
# Define the configuration for the graph
# ************************************************

3
examples/bedrock/scrape_plain_text_bedrock.py
View File

@ -1,12 +1,9 @@
"""
Basic example of scraping pipeline using SmartScraper from text
"""
import os
import json
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info

2
examples/bedrock/script_generator_bedrock.py
View File

@ -1,9 +1,7 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""
from dotenv import load_dotenv
from scrapegraphai.graphs import ScriptCreatorGraph
from scrapegraphai.utils import prettify_exec_info

1
examples/bedrock/script_multi_generator_bedrock.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""
from scrapegraphai.graphs import ScriptCreatorMultiGraph
from scrapegraphai.utils import prettify_exec_info

4
examples/bedrock/search_graph_bedrock.py
View File

@ -1,12 +1,8 @@
"""
Example of Search Graph
"""
from dotenv import load_dotenv
from scrapegraphai.graphs import SearchGraph
load_dotenv()
# ************************************************
# Define the configuration for the graph
# ************************************************

5
examples/bedrock/search_graph_schema_bedrock.py
View File

@ -1,12 +1,11 @@
"""
Example of Search Graph
"""
from typing import List
from pydantic import BaseModel, Field
from scrapegraphai.graphs import SearchGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
from pydantic import BaseModel, Field
from typing import List
# ************************************************
# Define the output schema for the graph
# ************************************************

2
examples/bedrock/search_link_graph_bedrock.py
View File

@ -1,8 +1,6 @@
"""
Example of Search Graph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SearchGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info

2
examples/bedrock/smart_scraper_bedrock.py
View File

@ -1,8 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info

1
examples/bedrock/smart_scraper_multi_bedrock.py
View File

@ -4,7 +4,6 @@ Basic example of scraping pipeline using SmartScraper
import json
from scrapegraphai.graphs import SmartScraperMultiGraph
# ************************************************
# Define the configuration for the graph
# ************************************************

1
examples/bedrock/xml_scraper_bedrock.py
View File

@ -4,7 +4,6 @@ Basic example of scraping pipeline using XMLScraperGraph from XML documents
import os
import json
from dotenv import load_dotenv
from scrapegraphai.graphs import XMLScraperGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info

2
examples/bedrock/xml_scraper_graph_multi_bedrock.py
View File

@ -1,11 +1,11 @@
"""
Basic example of scraping pipeline using XMLScraperMultiGraph from XML documents
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import XMLScraperMultiGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
load_dotenv()
# ************************************************

5
examples/deepseek/code_generator_graph_deepseek.py
View File

@ -1,8 +1,7 @@
"""
Basic example of scraping pipeline using Code Generator with schema
"""
import os, json
import os
from typing import List
from dotenv import load_dotenv
from pydantic import BaseModel, Field
@ -57,4 +56,4 @@ code_generator_graph = CodeGeneratorGraph(
)
result = code_generator_graph.run()
print(result)
print(result)

1
examples/deepseek/csv_scraper_deepseek.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using CSVScraperGraph from CSV documents
"""
import os
from dotenv import load_dotenv
import pandas as pd

1
examples/deepseek/csv_scraper_graph_multi_deepseek.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using CSVScraperMultiGraph from CSV documents
"""
import os
from dotenv import load_dotenv
import pandas as pd

2
examples/deepseek/json_scraper_deepseek.py
View File

@ -1,11 +1,11 @@
"""
Basic example of scraping pipeline using JSONScraperGraph from JSON documents
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import JSONScraperGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
load_dotenv()
# ************************************************

44
examples/deepseek/pdf_scraper_graph_deepseek.py
View File

@ -1,44 +0,0 @@
"""
Basic example of scraping pipeline using SmartScraper
"""
import os, json
from dotenv import load_dotenv
from scrapegraphai.utils import prettify_exec_info
from scrapegraphai.graphs import PDFScraperGraph
load_dotenv()
# ************************************************
# Define the configuration for the graph
# ************************************************
deepseek_key = os.getenv("DEEPSEEK_APIKEY")
graph_config = {
"llm": {
"model": "deepseek/deepseek-chat",
"api_key": deepseek_key,
},
"verbose": True,
}
source = """
The Divine Comedy, Italian La Divina Commedia, original name La commedia, long narrative poem written in Italian
circa 1308/21 by Dante. It is usually held to be one of the world s great works of literature.
Divided into three major sections—Inferno, Purgatorio, and Paradiso—the narrative traces the journey of Dante
from darkness and error to the revelation of the divine light, culminating in the Beatific Vision of God.
Dante is guided by the Roman poet Virgil, who represents the epitome of human knowledge, from the dark wood
through the descending circles of the pit of Hell (Inferno). He then climbs the mountain of Purgatory, guided
by the Roman poet Statius, who represents the fulfilment of human knowledge, and is finally led by his lifelong love,
the Beatrice of his earlier poetry, through the celestial spheres of Paradise.
"""
pdf_scraper_graph = PDFScraperGraph(
prompt="Summarize the text and find the main topics",
source=source,
config=graph_config,
)
result = pdf_scraper_graph.run()
print(json.dumps(result, indent=4))

74
examples/deepseek/pdf_scraper_multi_deepseek.py
View File

@ -1,74 +0,0 @@
"""
Module for showing how PDFScraper multi works
"""
import os
import json
from dotenv import load_dotenv
from scrapegraphai.graphs import PdfScraperMultiGraph
load_dotenv()
deepseek_key = os.getenv("DEEPSEEK_APIKEY")
graph_config = {
"llm": {
"model": "deepseek/deepseek-chat",
"api_key": deepseek_key,
},
"verbose": True,
}
# ***************
# Covert to list
# ***************
sources = [
"This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather—the interaction between call center architecture and outdoor weather conditions—in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity – largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.",
"This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather—the interaction between call center architecture and outdoor weather conditions—in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity – largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.",
"This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather—the interaction between call center architecture and outdoor weather conditions—in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity – largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.",
"This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather—the interaction between call center architecture and outdoor weather conditions—in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity – largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.",
]
prompt = """
You are an expert in reviewing academic manuscripts. Please analyze the abstracts provided from an academic journal article to extract and clearly identify the following elements:
Independent Variable (IV): The variable that is manipulated or considered as the primary cause affecting other variables.
Dependent Variable (DV): The variable that is measured or observed, which is expected to change as a result of variations in the Independent Variable.
Exogenous Shock: Identify any external or unexpected events used in the study that serve as a natural experiment or provide a unique setting for observing the effects on the IV and DV.
Response Format: For each abstract, present your response in the following structured format:
Independent Variable (IV):
Dependent Variable (DV):
Exogenous Shock:
Example Queries and Responses:
Query: This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather the interaction between call center architecture and outdoor weather conditions in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.
Response:
Independent Variable (IV): Employee happiness.
Dependent Variable (DV): Overall firm productivity.
Exogenous Shock: Sudden company-wide increase in bonus payments.
Query: The diffusion of social media coincided with a worsening of mental health conditions among adolescents and young adults in the United States, giving rise to speculation that social media might be detrimental to mental health. In this paper, we provide quasi-experimental estimates of the impact of social media on mental health by leveraging a unique natural experiment: the staggered introduction of Facebook across U.S. colleges. Our analysis couples data on student mental health around the years of Facebook's expansion with a generalized difference-in-differences empirical strategy. We find that the roll-out of Facebook at a college increased symptoms of poor mental health, especially depression. We also find that, among students predicted to be most susceptible to mental illness, the introduction of Facebook led to increased utilization of mental healthcare services. Lastly, we find that, after the introduction of Facebook, students were more likely to report experiencing impairments to academic performance resulting from poor mental health. Additional evidence on mechanisms suggests that the results are due to Facebook fostering unfavorable social comparisons.
Response:
Independent Variable (IV): Exposure to social media.
Dependent Variable (DV): Mental health outcomes.
Exogenous Shock: staggered introduction of Facebook across U.S. colleges.
"""
# *******************************************************
# Create the SmartScraperMultiGraph instance and run it
# *******************************************************
multiple_search_graph = PdfScraperMultiGraph(
prompt=prompt,
source= sources,
schema=None,
config=graph_config
)
result = multiple_search_graph.run()
print(json.dumps(result, indent=4))

2
examples/deepseek/rate_limit_deepseek.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper with a custom rate limit
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
@ -9,7 +8,6 @@ from scrapegraphai.utils import prettify_exec_info
load_dotenv()
# ************************************************
# Define the configuration for the graph
# ************************************************

2
examples/deepseek/scrape_plain_text_deepseek.py
View File

@ -1,11 +1,11 @@
"""
Basic example of scraping pipeline using SmartScraper from text
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info
load_dotenv()
# ************************************************

1
examples/deepseek/script_generator_deepseek.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import ScriptCreatorGraph

1
examples/deepseek/script_multi_generator_deepseek.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import ScriptCreatorMultiGraph

2
examples/deepseek/search_graph_deepseek.py
View File

@ -1,10 +1,10 @@
"""
Example of Search Graph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SearchGraph
load_dotenv()
# ************************************************

8
examples/deepseek/search_graph_schema_deepseek.py
View File

@ -1,16 +1,14 @@
"""
Example of Search Graph
"""
import os
from typing import List
from dotenv import load_dotenv
load_dotenv()
from pydantic import BaseModel, Field
from scrapegraphai.graphs import SearchGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
from pydantic import BaseModel, Field
from typing import List
load_dotenv()
# ************************************************
# Define the output schema for the graph

1
examples/deepseek/smart_scraper_deepseek.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph

1
examples/deepseek/smart_scraper_multi_concat_deepseek.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper
"""
import os
import json
from dotenv import load_dotenv

4
examples/deepseek/smart_scraper_multi_deepseek.py
View File

@ -1,8 +1,8 @@
"""
Basic example of scraping pipeline using SmartScraper
"""
import os, json
import os
import json
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperMultiGraph

1
examples/deepseek/smart_scraper_schema_deepseek.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper
"""
import os
from typing import List
from pydantic import BaseModel, Field

3
examples/deepseek/xml_scraper_deepseek.py
View File

@ -1,11 +1,11 @@
"""
Basic example of scraping pipeline using XMLScraperGraph from XML documents
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import XMLScraperGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
load_dotenv()
# ************************************************
@ -34,7 +34,6 @@ graph_config = {
"verbose": True,
}
# ************************************************
# Create the XMLScraperGraph instance and run it
# ************************************************

2
examples/deepseek/xml_scraper_graph_multi_deepseek.py
View File

@ -1,11 +1,11 @@
"""
Basic example of scraping pipeline using XMLScraperMultiGraph from XML documents
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import XMLScraperMultiGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
load_dotenv()
# ************************************************

3
examples/ernie/code_generator_graph_ernie.py
View File

@ -1,8 +1,7 @@
"""
Basic example of scraping pipeline using Code Generator with schema
"""
import os, json
import os
from typing import List
from dotenv import load_dotenv
from pydantic import BaseModel, Field

4
examples/ernie/csv_scraper_ernie.py
View File

@ -1,12 +1,12 @@
"""
Basic example of scraping pipeline using CSVScraperGraph from CSV documents
"""
import os
from dotenv import load_dotenv
import pandas as pd
from scrapegraphai.graphs import CSVScraperGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
load_dotenv()
# ************************************************
@ -23,7 +23,7 @@ text = pd.read_csv(file_path)
# Define the configuration for the graph
# ************************************************
graph_config = {
graph_config = {
"llm": {
"model": "ernie/ernie-bot-turbo",
"ernie_client_id": "<ernie_client_id>",

4
examples/ernie/custom_graph_ernie.py
View File

@ -1,10 +1,6 @@
"""
Example of custom graph using existing nodes
"""
import os
from dotenv import load_dotenv
from langchain_openai import OpenAIEmbeddings
from langchain_openai import ChatOpenAI
from scrapegraphai.graphs import BaseGraph

4
examples/ernie/json_scraper_ernie.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using JSONScraperGraph from JSON documents
"""
import os
from scrapegraphai.graphs import JSONScraperGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
@ -21,7 +20,7 @@ with open(file_path, 'r', encoding="utf-8") as file:
# Define the configuration for the graph
# ************************************************
graph_config = {
graph_config = {
"llm": {
"model": "ernie/ernie-bot-turbo",
"ernie_client_id": "<ernie_client_id>",
@ -53,4 +52,3 @@ print(prettify_exec_info(graph_exec_info))
# Save to json or csv
convert_to_csv(result, "result")
convert_to_json(result, "result")

35
examples/ernie/pdf_scraper_graph_ernie.py
View File

@ -1,35 +0,0 @@
import os, json
from scrapegraphai.graphs import PDFScraperGraph
# ************************************************
# Define the configuration for the graph
# ************************************************
graph_config = {
"llm": {
"model": "ernie/ernie-bot-turbo",
"ernie_client_id": "<ernie_client_id>",
"ernie_client_secret": "<ernie_client_secret>",
"temperature": 0.1
}
}
source = """
The Divine Comedy, Italian La Divina Commedia, original name La commedia, long narrative poem written in Italian
circa 1308/21 by Dante. It is usually held to be one of the world s great works of literature.
Divided into three major sections—Inferno, Purgatorio, and Paradiso—the narrative traces the journey of Dante
from darkness and error to the revelation of the divine light, culminating in the Beatific Vision of God.
Dante is guided by the Roman poet Virgil, who represents the epitome of human knowledge, from the dark wood
through the descending circles of the pit of Hell (Inferno). He then climbs the mountain of Purgatory, guided
by the Roman poet Statius, who represents the fulfilment of human knowledge, and is finally led by his lifelong love,
the Beatrice of his earlier poetry, through the celestial spheres of Paradise.
"""
pdf_scraper_graph = PDFScraperGraph(
prompt="Summarize the text and find the main topics",
source=source,
config=graph_config
)
result = pdf_scraper_graph.run()
print(json.dumps(result, indent=4))

4
examples/ernie/rate_limit_ernie.py
View File

@ -1,8 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper with a custom rate limit
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info
@ -14,7 +12,7 @@ load_dotenv()
# Define the configuration for the graph
# ************************************************
graph_config = {
graph_config = {
"llm": {
"model": "ernie/ernie-bot-turbo",
"ernie_client_id": "<ernie_client_id>",

1
examples/ernie/scrape_plain_text_ernie.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper from text
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph

3
examples/ernie/script_generator_ernie.py
View File

@ -1,8 +1,6 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import ScriptCreatorGraph
from scrapegraphai.utils import prettify_exec_info
@ -43,4 +41,3 @@ print(result)
graph_exec_info = script_creator_graph.get_execution_info()
print(prettify_exec_info(graph_exec_info))

1
examples/ernie/script_multi_generator_ernie.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""
from scrapegraphai.graphs import ScriptCreatorMultiGraph
from scrapegraphai.utils import prettify_exec_info

4
examples/ernie/search_graph_ernie.py
View File

@ -1,8 +1,6 @@
"""
Example of Search Graph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SearchGraph
@ -12,7 +10,7 @@ load_dotenv()
# Define the configuration for the graph
# ************************************************
graph_config = {
graph_config = {
"llm": {
"model": "ernie/ernie-bot-turbo",
"ernie_client_id": "<ernie_client_id>",

2
examples/ernie/search_link_graph_ernie.py
View File

@ -8,7 +8,7 @@ from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_i
# Define the configuration for the graph
# ************************************************
graph_config = {
graph_config = {
"llm": {
"model": "ernie/ernie-bot-turbo",
"ernie_client_id": "<ernie_client_id>",

1
examples/ernie/smart_scraper_ernie.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper
"""
from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info

4
examples/ernie/smart_scraper_multi_ernie.py
View File

@ -1,8 +1,8 @@
"""
Basic example of scraping pipeline using SmartScraper
"""
import os, json
import os
import json
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperMultiGraph

6
examples/ernie/smart_scraper_schema_ernie.py
View File

@ -1,24 +1,18 @@
"""
Basic example of scraping pipeline using SmartScraper with schema
"""
import json
import os
from typing import Dict
from dotenv import load_dotenv
from pydantic import BaseModel
from scrapegraphai.graphs import SmartScraperGraph
load_dotenv()
# ************************************************
# Define the output schema for the graph
# ************************************************
class Project(BaseModel):
title: str
description: str

2
examples/ernie/speech_graph_ernie.py
View File

@ -1,11 +1,11 @@
"""
Basic example of scraping pipeline using SpeechSummaryGraph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SpeechGraph
from scrapegraphai.utils import prettify_exec_info
load_dotenv()
# ************************************************

2
examples/ernie/xml_scraper_ernie.py
View File

@ -1,11 +1,11 @@
"""
Basic example of scraping pipeline using XMLScraperGraph from XML documents
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import XMLScraperGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
load_dotenv()
# ************************************************

41
examples/extras/conditional_usage.py Normal file
View File

@ -0,0 +1,41 @@
"""
Basic example of scraping pipeline using SmartScraperMultiConcatGraph with Groq
"""
import os
import json
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperMultiGraph
load_dotenv()
# ************************************************
# Define the configuration for the graph
# ************************************************
graph_config = {
"llm": {
"api_key": os.getenv("OPENAI_API_KEY"),
"model": "openai/gpt-4o",
},
"verbose": True,
"headless": False,
}
# *******************************************************
# Create the SmartScraperMultiCondGraph instance and run it
# *******************************************************
multiple_search_graph = SmartScraperMultiGraph(
prompt="Who is Marco Perini?",
source=[
"https://perinim.github.io/",
"https://perinim.github.io/cv/"
],
schema=None,
config=graph_config
)
result = multiple_search_graph.run()
print(json.dumps(result, indent=4))

47
examples/extras/undected_playwrigth.py Normal file
View File

@ -0,0 +1,47 @@
"""
Basic example of scraping pipeline using SmartScraper
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info
load_dotenv()
# ************************************************
# Define the configuration for the graph
# ************************************************
groq_key = os.getenv("GROQ_APIKEY")
graph_config = {
"llm": {
"model": "groq/gemma-7b-it",
"api_key": groq_key,
"temperature": 0
},
"headless": False,
"backend": "undetected_chromedriver"
}
# ************************************************
# Create the SmartScraperGraph instance and run it
# ************************************************
smart_scraper_graph = SmartScraperGraph(
prompt="List me all the projects with their description.",
# also accepts a string with the already downloaded HTML code
source="https://perinim.github.io/projects/",
config=graph_config
)
result = smart_scraper_graph.run()
print(result)
# ************************************************
# Get graph execution info
# ************************************************
graph_exec_info = smart_scraper_graph.get_execution_info()
print(prettify_exec_info(graph_exec_info))

4
examples/fireworks/code_generator_graph_fireworks.py
View File

@ -1,8 +1,8 @@
"""
Basic example of scraping pipeline using Code Generator with schema
"""
import os, json
import os
import json
from typing import List
from dotenv import load_dotenv
from pydantic import BaseModel, Field

2
examples/fireworks/csv_scraper_fireworks.py
View File

@ -1,12 +1,12 @@
"""
Basic example of scraping pipeline using CSVScraperGraph from CSV documents
"""
import os
from dotenv import load_dotenv
import pandas as pd
from scrapegraphai.graphs import CSVScraperGraph
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
load_dotenv()
# ************************************************

1
examples/fireworks/csv_scraper_graph_multi_fireworks.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using CSVScraperMultiGraph from CSV documents
"""
import os
from dotenv import load_dotenv
import pandas as pd

3
examples/fireworks/custom_graph_fireworks.py
View File

@ -1,12 +1,11 @@
"""
Example of custom graph using existing nodes
"""
import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from scrapegraphai.graphs import BaseGraph
from scrapegraphai.nodes import FetchNode, ParseNode, RAGNode, GenerateAnswerNode, RobotsNode
from scrapegraphai.nodes import FetchNode, ParseNode, GenerateAnswerNode, RobotsNode
load_dotenv()
# ************************************************

1
examples/fireworks/json_scraper_fireworkspy.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using JSONScraperGraph from JSON documents
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import JSONScraperGraph

40
examples/fireworks/pdf_scraper_fireworks.py
View File

@ -1,40 +0,0 @@
import os, json
from dotenv import load_dotenv
from scrapegraphai.graphs import PDFScraperGraph
load_dotenv()
# ************************************************
# Define the configuration for the graph
# ************************************************
fireworks_api_key = os.getenv("FIREWORKS_APIKEY")
graph_config = {
"llm": {
"api_key": fireworks_api_key,
"model": "fireworks/accounts/fireworks/models/mixtral-8x7b-instruct"
},
"verbose": True,
}
source = """
The Divine Comedy, Italian La Divina Commedia, original name La commedia, long narrative poem written in Italian
circa 1308/21 by Dante. It is usually held to be one of the world s great works of literature.
Divided into three major sections—Inferno, Purgatorio, and Paradiso—the narrative traces the journey of Dante
from darkness and error to the revelation of the divine light, culminating in the Beatific Vision of God.
Dante is guided by the Roman poet Virgil, who represents the epitome of human knowledge, from the dark wood
through the descending circles of the pit of Hell (Inferno). He then climbs the mountain of Purgatory, guided
by the Roman poet Statius, who represents the fulfilment of human knowledge, and is finally led by his lifelong love,
the Beatrice of his earlier poetry, through the celestial spheres of Paradise.
"""
pdf_scraper_graph = PDFScraperGraph(
prompt="Summarize the text and find the main topics",
source=source,
config=graph_config,
)
result = pdf_scraper_graph.run()
print(json.dumps(result, indent=4))

64
examples/fireworks/pdf_scraper_multi_fireworks.py
View File

@ -1,64 +0,0 @@
"""
Module for showing how PDFScraper multi works
"""
import os
import json
from typing import List
from dotenv import load_dotenv
from pydantic import BaseModel, Field
from scrapegraphai.graphs import PdfScraperMultiGraph
load_dotenv()
# ************************************************
# Define the configuration for the graph
# ************************************************
fireworks_api_key = os.getenv("FIREWORKS_APIKEY")
graph_config = {
"llm": {
"api_key": fireworks_api_key,
"model": "fireworks/accounts/fireworks/models/mixtral-8x7b-instruct"
},
"verbose": True,
}
# ************************************************
# Define the output schema for the graph
# ************************************************
class Article(BaseModel):
independent_variable: str = Field(description="(IV): The variable that is manipulated or considered as the primary cause affecting other variables.")
dependent_variable: str = Field(description="(DV) The variable that is measured or observed, which is expected to change as a result of variations in the Independent Variable.")
exogenous_shock: str = Field(description="Identify any external or unexpected events used in the study that serve as a natural experiment or provide a unique setting for observing the effects on the IV and DV.")
class Articles(BaseModel):
articles: List[Article]
# ************************************************
# Define the sources for the graph
# ************************************************
sources = [
"This paper provides evidence from a natural experiment on the relationship between positive affect and productivity. We link highly detailed administrative data on the behaviors and performance of all telesales workers at a large telecommunications company with survey reports of employee happiness that we collected on a weekly basis. We use variation in worker mood arising from visual exposure to weather the interaction between call center architecture and outdoor weather conditions in order to provide a quasi-experimental test of the effect of happiness on productivity. We find evidence of a positive impact on sales performance, which is driven by changes in labor productivity largely through workers converting more calls into sales, and to a lesser extent by making more calls per hour and adhering more closely to their schedule. We find no evidence in our setting of effects on measures of high-frequency labor supply such as attendance and break-taking.",
"The diffusion of social media coincided with a worsening of mental health conditions among adolescents and young adults in the United States, giving rise to speculation that social media might be detrimental to mental health. Our analysis couples data on student mental health around the years of Facebook's expansion with a generalized difference-in-differences empirical strategy. We find that the roll-out of Facebook at a college increased symptoms of poor mental health, especially depression. We also find that, among students predicted to be most susceptible to mental illness, the introduction of Facebook led to increased utilization of mental healthcare services. Lastly, we find that, after the introduction of Facebook, students were more likely to report experiencing impairments to academic performance resulting from poor mental health. Additional evidence on mechanisms suggests that the results are due to Facebook fostering unfavorable social comparisons."
]
prompt = """
Analyze the abstracts provided from an academic journal article to extract and clearly identify the Independent Variable (IV), Dependent Variable (DV), and Exogenous Shock.
"""
# *******************************************************
# Create the SmartScraperMultiGraph instance and run it
# *******************************************************
multiple_search_graph = PdfScraperMultiGraph(
prompt=prompt,
source= sources,
schema=Articles,
config=graph_config
)
result = multiple_search_graph.run()
print(json.dumps(result, indent=4))

5
examples/fireworks/rate_limit_fireworks.py
View File

@ -1,15 +1,14 @@
"""
Basic example of scraping pipeline using SmartScraper with a custom rate limit
"""
import os, json
import os
import json
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info
load_dotenv()
# ************************************************
# Define the configuration for the graph
# ************************************************

3
examples/fireworks/scrape_plain_text_fireworks.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using SmartScraper from text
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
@ -34,8 +33,6 @@ graph_config = {
},
}
# ************************************************
# Create the SmartScraperGraph instance and run it
# ************************************************

2
examples/fireworks/script_generator_fireworks.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import ScriptCreatorGraph
@ -46,4 +45,3 @@ print(result)
graph_exec_info = script_creator_graph.get_execution_info()
print(prettify_exec_info(graph_exec_info))

1
examples/fireworks/script_generator_schema_fireworks.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""
import os
from typing import List
from dotenv import load_dotenv

1
examples/fireworks/script_multi_generator_fireworks.py
View File

@ -1,7 +1,6 @@
"""
Basic example of scraping pipeline using ScriptCreatorGraph
"""
import os
from dotenv import load_dotenv
from scrapegraphai.graphs import ScriptCreatorMultiGraph

Some files were not shown because too many files have changed in this diff Show More