From 8392518293afc9e3d0a5116cedc38ec2dd454593 Mon Sep 17 00:00:00 2001 From: Jafar Saad Date: Tue, 9 Jul 2024 21:19:05 +0700 Subject: [PATCH 01/10] Create python-app.yml --- .github/workflows/python-app.yml | 39 ++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 .github/workflows/python-app.yml diff --git a/.github/workflows/python-app.yml b/.github/workflows/python-app.yml new file mode 100644 index 00000000..1168bd9a --- /dev/null +++ b/.github/workflows/python-app.yml @@ -0,0 +1,39 @@ +# This workflow will install Python dependencies, run tests and lint with a single version of Python +# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python + +name: Python application + +on: + push: + branches: [ "main" ] + pull_request: + branches: [ "main" ] + +permissions: + contents: read + +jobs: + build: + + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + - name: Set up Python 3.10 + uses: actions/setup-python@v3 + with: + python-version: "3.10" + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install flake8 pytest + if [ -f requirements.txt ]; then pip install -r requirements.txt; fi + - name: Lint with flake8 + run: | + # stop the build if there are Python syntax errors or undefined names + flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics + # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide + flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics + - name: Test with pytest + run: | + pytest From 40bb0246d3078f52727450935f6c174b4a7add0c Mon Sep 17 00:00:00 2001 From: semantic-release-bot Date: Tue, 9 Jul 2024 16:52:52 +0000 Subject: [PATCH 02/10] ci(release): 1.9.0 [skip ci] ## [1.9.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.8.0...v1.9.0) (2024-07-09) ### Features * add fireworks integration ([df0e310](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/df0e3108299071b849d7e055bd11d72764d24f08)) * add integration for infos ([3bf5f57](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3bf5f570a8f8e1b037a7ad3c9f583261a1536421)) * add integrations for markdown files ([2804434](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2804434a9ee12c52ae8956a88b1778a4dd3ec32f)) * add vertexai integration ([119514b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/119514bdfc2a16dfb8918b0c34ae7cc43a01384c)) * improve md prompt recognition ([5fe694b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5fe694b6b4545a5091d16110318b992acfca4f58)) ### Bug Fixes * add test ([3a537ee](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3a537eec6fef1743924a9aa5cef0ba2f8d44bf11)) * fix pyproject.toml ([7570bf8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7570bf8294e49bc54ec9e296aaadb763873390ca)) ### chore * **Docker:** fix port number ([afeb81f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/afeb81f77a884799192d79dcac85666190fb1c9d)) * **CI:** fix pylint workflow ([583c321](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/583c32106e827f50235d8fc69511652fd4b07a35)) * **rye:** rebuild lockfiles ([27c2dd2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/27c2dd23517a7e4b14fafd00320a8b81f73145dc)) ### Docs * **roadmap:** fix urls ([14faba4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/14faba4f00dd9f947f8dc5e0b51be49ea684179f)) * **roadmap:** next steps ([3e644f4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3e644f498f05eb505fbd4e94b144c81567569aaa)) ### CI * **release:** 1.8.1-beta.1 [skip ci] ([8f9f96f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8f9f96f7e7ff41d2fff5bbbf18bf4fc85d4f98b3)) * **release:** 1.9.0-beta.1 [skip ci] ([146432d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/146432d476f775510441b062935adc47190141e2)) * **release:** 1.9.0-beta.2 [skip ci] ([5cb5fbf](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5cb5fbf5503eec9b34a6691eb993716cc9a821d6)) --- CHANGELOG.md | 37 +++++++++++++++++++++++++++++++++++++ pyproject.toml | 2 +- 2 files changed, 38 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 63eb6250..60712481 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,40 @@ +## [1.9.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.8.0...v1.9.0) (2024-07-09) + + +### Features + +* add fireworks integration ([df0e310](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/df0e3108299071b849d7e055bd11d72764d24f08)) +* add integration for infos ([3bf5f57](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3bf5f570a8f8e1b037a7ad3c9f583261a1536421)) +* add integrations for markdown files ([2804434](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2804434a9ee12c52ae8956a88b1778a4dd3ec32f)) +* add vertexai integration ([119514b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/119514bdfc2a16dfb8918b0c34ae7cc43a01384c)) +* improve md prompt recognition ([5fe694b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5fe694b6b4545a5091d16110318b992acfca4f58)) + + +### Bug Fixes + +* add test ([3a537ee](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3a537eec6fef1743924a9aa5cef0ba2f8d44bf11)) +* fix pyproject.toml ([7570bf8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7570bf8294e49bc54ec9e296aaadb763873390ca)) + + +### chore + +* **Docker:** fix port number ([afeb81f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/afeb81f77a884799192d79dcac85666190fb1c9d)) +* **CI:** fix pylint workflow ([583c321](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/583c32106e827f50235d8fc69511652fd4b07a35)) +* **rye:** rebuild lockfiles ([27c2dd2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/27c2dd23517a7e4b14fafd00320a8b81f73145dc)) + + +### Docs + +* **roadmap:** fix urls ([14faba4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/14faba4f00dd9f947f8dc5e0b51be49ea684179f)) +* **roadmap:** next steps ([3e644f4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3e644f498f05eb505fbd4e94b144c81567569aaa)) + + +### CI + +* **release:** 1.8.1-beta.1 [skip ci] ([8f9f96f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8f9f96f7e7ff41d2fff5bbbf18bf4fc85d4f98b3)) +* **release:** 1.9.0-beta.1 [skip ci] ([146432d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/146432d476f775510441b062935adc47190141e2)) +* **release:** 1.9.0-beta.2 [skip ci] ([5cb5fbf](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5cb5fbf5503eec9b34a6691eb993716cc9a821d6)) + ## [1.9.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.9.0-beta.1...v1.9.0-beta.2) (2024-07-05) diff --git a/pyproject.toml b/pyproject.toml index 30dad8df..01fb4029 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -2,7 +2,7 @@ name = "scrapegraphai" -version = "1.9.0b2" +version = "1.9.0" description = "A web scraping library based on LangChain which uses LLM and direct graph logic to create scraping pipelines." From c19d83f994a6a817806587dec03c4549cf0932bb Mon Sep 17 00:00:00 2001 From: Marco Vinciguerra <88108002+VinciGit00@users.noreply.github.com> Date: Thu, 11 Jul 2024 18:03:10 +0200 Subject: [PATCH 03/10] Update fetch_node.py --- examples/single_node/fetch_node.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/single_node/fetch_node.py b/examples/single_node/fetch_node.py index d03cb495..ed2de2e0 100644 --- a/examples/single_node/fetch_node.py +++ b/examples/single_node/fetch_node.py @@ -9,7 +9,7 @@ from scrapegraphai.nodes import FetchNode # ************************************************ -robots_node = FetchNode( +fetch_node = FetchNode( input="url | local_dir", output=["doc"], node_config={ @@ -25,6 +25,6 @@ state = { "url": "https://twitter.com/home" } -result = robots_node.execute(state) +result = fetch_node.execute(state) print(result) From 4b65392cca287af84b4c65f41a048450ea6b8ef2 Mon Sep 17 00:00:00 2001 From: salamanderxing Date: Fri, 12 Jul 2024 10:56:25 +0200 Subject: [PATCH 04/10] Resolves #447 --- scrapegraphai/integrations/burr_bridge.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scrapegraphai/integrations/burr_bridge.py b/scrapegraphai/integrations/burr_bridge.py index 019427ef..d1fe566f 100644 --- a/scrapegraphai/integrations/burr_bridge.py +++ b/scrapegraphai/integrations/burr_bridge.py @@ -102,7 +102,7 @@ class BurrBridge: def __init__(self, base_graph, burr_config): self.base_graph = base_graph self.burr_config = burr_config - self.project_name = burr_config.get("project_name", "scrapegraph: {}") + self.project_name = burr_config.get("project_name", "scrapegraph_project") self.app_instance_id = burr_config.get("app_instance_id", "default-instance") self.burr_inputs = burr_config.get("inputs", {}) self.burr_app = None From 316c7debe915380b08553d368d6141b195b84d54 Mon Sep 17 00:00:00 2001 From: semantic-release-bot Date: Fri, 12 Jul 2024 11:49:35 +0000 Subject: [PATCH 05/10] ci(release): 1.9.1 [skip ci] ## [1.9.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.9.0...v1.9.1) (2024-07-12) ### Bug Fixes * solve a burr integration ([881290b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/881290b5066b39c505532656671fbf65f8fc312c)) --- CHANGELOG.md | 7 +++++++ pyproject.toml | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 60712481..541005a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,10 @@ +## [1.9.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.9.0...v1.9.1) (2024-07-12) + + +### Bug Fixes + +* solve a burr integration ([881290b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/881290b5066b39c505532656671fbf65f8fc312c)) + ## [1.9.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.8.0...v1.9.0) (2024-07-09) diff --git a/pyproject.toml b/pyproject.toml index 01fb4029..c17e5bcb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -2,7 +2,7 @@ name = "scrapegraphai" -version = "1.9.0" +version = "1.9.1" description = "A web scraping library based on LangChain which uses LLM and direct graph logic to create scraping pipelines." From 5c6dd8de4da08f09b5dd93c525d14b44778c9659 Mon Sep 17 00:00:00 2001 From: Federico Aguzzi <62149513+f-aguzzi@users.noreply.github.com> Date: Tue, 16 Jul 2024 10:16:44 +0200 Subject: [PATCH 06/10] chore: remove unused workflow This workflow is both unused and incompatible with the pyproject.toml - based build system used in this project. --- .github/workflows/python-app.yml | 39 -------------------------------- 1 file changed, 39 deletions(-) delete mode 100644 .github/workflows/python-app.yml diff --git a/.github/workflows/python-app.yml b/.github/workflows/python-app.yml deleted file mode 100644 index 1168bd9a..00000000 --- a/.github/workflows/python-app.yml +++ /dev/null @@ -1,39 +0,0 @@ -# This workflow will install Python dependencies, run tests and lint with a single version of Python -# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python - -name: Python application - -on: - push: - branches: [ "main" ] - pull_request: - branches: [ "main" ] - -permissions: - contents: read - -jobs: - build: - - runs-on: ubuntu-latest - - steps: - - uses: actions/checkout@v4 - - name: Set up Python 3.10 - uses: actions/setup-python@v3 - with: - python-version: "3.10" - - name: Install dependencies - run: | - python -m pip install --upgrade pip - pip install flake8 pytest - if [ -f requirements.txt ]; then pip install -r requirements.txt; fi - - name: Lint with flake8 - run: | - # stop the build if there are Python syntax errors or undefined names - flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics - # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide - flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics - - name: Test with pytest - run: | - pytest From e471396987bf12beca3a0c91b148de0e0f51d289 Mon Sep 17 00:00:00 2001 From: Byunghyun Kim <94029750+bhyun-kim@users.noreply.github.com> Date: Wed, 17 Jul 2024 20:10:57 -0500 Subject: [PATCH 07/10] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 488c8ed6..560246c4 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # ๐Ÿ•ท๏ธ ScrapeGraphAI: You Only Scrape Once [English](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/README.md) | [ไธญๆ–‡](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/chinese.md) | [ๆ—ฅๆœฌ่ชž](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/japanese.md) -| [์ฝ”๋ฆฌ์•„๋…ธ](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/korean.md) +| [ํ•œ๊ตญ์–ด](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/korean.md) | [ะ ัƒััะบะธะน](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/docs/russian.md) From 3ada4182b57611978cfa8aaf2068177c8c1772ce Mon Sep 17 00:00:00 2001 From: Byunghyun Kim <94029750+bhyun-kim@users.noreply.github.com> Date: Wed, 17 Jul 2024 20:25:01 -0500 Subject: [PATCH 08/10] Update korean.md --- docs/korean.md | 137 ++++++++++++++++++++++++++++++------------------- 1 file changed, 84 insertions(+), 53 deletions(-) diff --git a/docs/korean.md b/docs/korean.md index 40c88e06..7ea1e57e 100644 --- a/docs/korean.md +++ b/docs/korean.md @@ -1,7 +1,7 @@ -# ๐Ÿ•ท๏ธ ScrapeGraphAI: ํ•œ ๋ฒˆ๋งŒ ์Šคํฌ๋ž˜ํ•‘ํ•˜์„ธ์š” +# ๐Ÿ•ท๏ธ ScrapeGraphAI: ํ•œ ๋ฐฉ์— ๋๋‚ด๋Š” ์›น์Šคํฌ๋ž˜ํ•‘ -ScrapeGraphAI๋Š” ์›น ์‚ฌ์ดํŠธ์™€ ๋กœ์ปฌ ๋ฌธ์„œ(XML, HTML, JSON ๋“ฑ)์— ๋Œ€ํ•œ ์Šคํฌ๋ž˜ํ•‘ ํŒŒ์ดํ”„๋ผ์ธ์„ ๋งŒ๋“ค๊ธฐ ์œ„ํ•ด LLM ๋ฐ ์ง์ ‘ ๊ทธ๋ž˜ํ”„ ๋กœ์ง์„ ์‚ฌ์šฉํ•˜๋Š” ํŒŒ์ด์ฌ ์›น ์Šคํฌ๋ž˜ํ•‘ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์ž…๋‹ˆ๋‹ค. +ScrapeGraphAI๋Š” ์›น ์‚ฌ์ดํŠธ์™€ ๋กœ์ปฌ ๋ฌธ์„œ(XML, HTML, JSON ๋“ฑ)์— ๋Œ€ํ•œ ์Šคํฌ๋ž˜ํ•‘ ํŒŒ์ดํ”„๋ผ์ธ์„ ๋งŒ๋“ค๊ธฐ ์œ„ํ•ด LLM ๋ฐ ์ง์ ‘ ๊ทธ๋ž˜ํ”„ ๋กœ์ง์„ ์‚ฌ์šฉํ•˜๋Š” ํŒŒ์ด์ฌ ์›น์Šคํฌ๋ž˜ํ•‘ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์ž…๋‹ˆ๋‹ค. ์ถ”์ถœํ•˜๋ ค๋Š” ์ •๋ณด๋ฅผ ๋งํ•˜๊ธฐ๋งŒ ํ•˜๋ฉด ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๊ฐ€ ์•Œ์•„์„œ ์ฒ˜๋ฆฌํ•ด ์ค๋‹ˆ๋‹ค! @@ -11,41 +11,46 @@ ScrapeGraphAI๋Š” ์›น ์‚ฌ์ดํŠธ์™€ ๋กœ์ปฌ ๋ฌธ์„œ(XML, HTML, JSON ๋“ฑ)์— ๋Œ€ํ•œ ## ๐Ÿš€ ๋น ๋ฅธ ์„ค์น˜ -Scrapegraph-ai์— ๋Œ€ํ•œ ์ฐธ์กฐ ํŽ˜์ด์ง€๋Š” PyPI์˜ ๊ณต์‹ ํŽ˜์ด์ง€์—์„œ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค: pypi. +Scrapegraph-ai์— ๋Œ€ํ•œ ์ฐธ์กฐ ํŽ˜์ด์ง€๋Š” PyPI์˜ ๊ณต์‹ ํŽ˜์ด์ง€์—์„œ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค: [pypi](https://pypi.org/project/scrapegraphai/). -bash -Copia codice +```bash pip install scrapegraphai +``` ์ฐธ๊ณ : ๋‹ค๋ฅธ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์™€์˜ ์ถฉ๋Œ์„ ํ”ผํ•˜๊ธฐ ์œ„ํ•ด ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ฅผ ๊ฐ€์ƒ ํ™˜๊ฒฝ์— ์„ค์น˜ํ•˜๋Š” ๊ฒƒ์ด ์ข‹์Šต๋‹ˆ๋‹ค ๐Ÿฑ ## ๐Ÿ” ๋ฐ๋ชจ ๊ณต์‹ Streamlit ๋ฐ๋ชจ: +[![My Skills](https://skillicons.dev/icons?i=react)](https://scrapegraph-ai-web-dashboard.streamlit.app) Google Colab์„ ์‚ฌ์šฉํ•˜์—ฌ ์›น์—์„œ ์ง์ ‘ ์‚ฌ์šฉํ•ด ๋ณด์„ธ์š”: +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1sEZBonBMGP44CtO6GQTwAlL0BGJXjtfd?usp=sharing) ## ๐Ÿ“– ๋ฌธ์„œ -ScrapeGraphAI์— ๋Œ€ํ•œ ๋ฌธ์„œ๋Š” ์—ฌ๊ธฐ์—์„œ ์ฐพ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. +ScrapeGraphAI์— ๋Œ€ํ•œ ๋ฌธ์„œ๋Š” [์—ฌ๊ธฐ](https://scrapegraph-ai.readthedocs.io/en/latest/)์—์„œ ์ฐพ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. -๋˜ํ•œ Docusaurus๋ฅผ ์—ฌ๊ธฐ์—์„œ ํ™•์ธํ•ด ๋ณด์„ธ์š”. +๋˜ํ•œ Docusaurus๋ฅผ [์—ฌ๊ธฐ](https://scrapegraph-doc.onrender.com/)์—์„œ ํ™•์ธํ•ด ๋ณด์„ธ์š”. ## ๐Ÿ’ป ์‚ฌ์šฉ๋ฒ• -์›น ์‚ฌ์ดํŠธ(๋˜๋Š” ๋กœ์ปฌ ํŒŒ์ผ)์—์„œ ์ •๋ณด๋ฅผ ์ถ”์ถœํ•˜๋Š” ๋ฐ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ์„ธ ๊ฐ€์ง€ ์ฃผ์š” ์Šคํฌ๋ž˜ํ•‘ ํŒŒ์ดํ”„๋ผ์ธ์ด ์žˆ์Šต๋‹ˆ๋‹ค: +์›น์‚ฌ์ดํŠธ(๋˜๋Š” ๋กœ์ปฌ ํŒŒ์ผ)์—์„œ ์ •๋ณด๋ฅผ ์ถ”์ถœํ•˜๊ธฐ ์œ„ํ•ด ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ์—ฌ๋Ÿฌ ํ‘œ์ค€ ์Šคํฌ๋ž˜ํ•‘ ํŒŒ์ดํ”„๋ผ์ธ์ด ์žˆ์Šต๋‹ˆ๋‹ค: +- `SmartScraperGraph`: ์‚ฌ์šฉ์ž ํ”„๋กฌํ”„ํŠธ์™€ ์ž…๋ ฅ ์†Œ์Šค๋งŒ ํ•„์š”๋กœ ํ•˜๋Š” ๋‹จ์ผ ํŽ˜์ด์ง€ ์Šคํฌ๋ž˜ํผ์ž…๋‹ˆ๋‹ค. +- `SearchGraph`: ๊ฒ€์ƒ‰ ์—”์ง„์˜ ์ƒ์œ„ n๊ฐœ ๊ฒ€์ƒ‰ ๊ฒฐ๊ณผ์—์„œ ์ •๋ณด๋ฅผ ์ถ”์ถœํ•˜๋Š” ๋‹ค์ค‘ ํŽ˜์ด์ง€ ์Šคํฌ๋ž˜ํผ์ž…๋‹ˆ๋‹ค. +- `SpeechGraph`: ์›น์‚ฌ์ดํŠธ์—์„œ ์ •๋ณด๋ฅผ ์ถ”์ถœํ•˜๊ณ  ์˜ค๋””์˜ค ํŒŒ์ผ์„ ์ƒ์„ฑํ•˜๋Š” ๋‹จ์ผ ํŽ˜์ด์ง€ ์Šคํฌ๋ž˜ํผ์ž…๋‹ˆ๋‹ค. +- `ScriptCreatorGraph`: ์›น์‚ฌ์ดํŠธ์—์„œ ์ •๋ณด๋ฅผ ์ถ”์ถœํ•˜๊ณ  Python ์Šคํฌ๋ฆฝํŠธ๋ฅผ ์ƒ์„ฑํ•˜๋Š” ๋‹จ์ผ ํŽ˜์ด์ง€ ์Šคํฌ๋ž˜ํผ์ž…๋‹ˆ๋‹ค. -SmartScraperGraph: ์‚ฌ์šฉ์ž ํ”„๋กฌํ”„ํŠธ์™€ ์ž…๋ ฅ ์†Œ์Šค๋งŒ ํ•„์š”ํ•œ ๋‹จ์ผ ํŽ˜์ด์ง€ ์Šคํฌ๋ž˜ํผ; -SearchGraph: ๊ฒ€์ƒ‰ ์—”์ง„์˜ ์ƒ์œ„ n๊ฐœ์˜ ๊ฒ€์ƒ‰ ๊ฒฐ๊ณผ์—์„œ ์ •๋ณด๋ฅผ ์ถ”์ถœํ•˜๋Š” ๋‹ค์ค‘ ํŽ˜์ด์ง€ ์Šคํฌ๋ž˜ํผ; -SpeechGraph: ์›น ์‚ฌ์ดํŠธ์—์„œ ์ •๋ณด๋ฅผ ์ถ”์ถœํ•˜๊ณ  ์˜ค๋””์˜ค ํŒŒ์ผ์„ ์ƒ์„ฑํ•˜๋Š” ๋‹จ์ผ ํŽ˜์ด์ง€ ์Šคํฌ๋ž˜ํผ. -SmartScraperMultiGraph: ๋‹จ์ผ ํ”„๋กฌํ”„ํŠธ๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ์—ฌ๋Ÿฌ ํŽ˜์ด์ง€๋ฅผ ์Šคํฌ๋ž˜ํ•‘ํ•˜๋Š” ์Šคํฌ๋ž˜ํผ -OpenAI, Groq, Azure, Gemini์™€ ๊ฐ™์€ API๋ฅผ ํ†ตํ•ด ๋‹ค์–‘ํ•œ LLM์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์œผ๋ฉฐ, Ollama๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋กœ์ปฌ ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค. +- `SmartScraperMultiGraph`: ๋‹จ์ผ ํ”„๋กฌํ”„ํŠธ์™€ ์†Œ์Šค ๋ชฉ๋ก์„ ์‚ฌ์šฉํ•˜์—ฌ ์—ฌ๋Ÿฌ ํŽ˜์ด์ง€์—์„œ ์ •๋ณด๋ฅผ ์ถ”์ถœํ•˜๋Š” ๋‹ค์ค‘ ํŽ˜์ด์ง€ ์Šคํฌ๋ž˜ํผ์ž…๋‹ˆ๋‹ค. +- `ScriptCreatorMultiGraph`: ๋‹จ์ผ ํ”„๋กฌํ”„ํŠธ์™€ ์†Œ์Šค ๋ชฉ๋ก์„ ์‚ฌ์šฉํ•˜์—ฌ ์—ฌ๋Ÿฌ ํŽ˜์ด์ง€์—์„œ ์ •๋ณด๋ฅผ ์ถ”์ถœํ•˜๋Š” Python ์Šคํฌ๋ฆฝํŠธ๋ฅผ ์ƒ์„ฑํ•˜๋Š” ๋‹ค์ค‘ ํŽ˜์ด์ง€ ์Šคํฌ๋ž˜ํผ์ž…๋‹ˆ๋‹ค. -์‚ฌ๋ก€ 1: ๋กœ์ปฌ ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•˜๋Š” SmartScraper -Ollama๋ฅผ ์„ค์น˜ํ•˜๊ณ  ollama pull ๋ช…๋ น์„ ์‚ฌ์šฉํ•˜์—ฌ ๋ชจ๋ธ์„ ๋‹ค์šด๋กœ๋“œํ•˜์„ธ์š”. +**OpenAI**, **Groq**, **Azure**, **Gemini**์™€ ๊ฐ™์€ API๋ฅผ ํ†ตํ•ด ๋‹ค์–‘ํ•œ LLM์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์œผ๋ฉฐ, **Ollama**๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋กœ์ปฌ ๋ชจ๋ธ๋„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. + +### ์‚ฌ๋ก€ 1: ๋กœ์ปฌ ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•˜๋Š” SmartScraper +[Ollama](https://ollama.com/)๋ฅผ ์„ค์น˜ํ•˜๊ณ  **ollama pull** ๋ช…๋ น์„ ์‚ฌ์šฉํ•˜์—ฌ ๋ชจ๋ธ์„ ๋‹ค์šด๋กœ๋“œํ•˜์„ธ์š”. ```python from scrapegraphai.graphs import SmartScraperGraph @@ -54,19 +59,19 @@ graph_config = { "llm": { "model": "ollama/mistral", "temperature": 0, - "format": "json", # Ollama๋Š” ํ˜•์‹์„ ๋ช…์‹œ์ ์œผ๋กœ ์ง€์ •ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค - "base_url": "http://localhost:11434", # Ollama URL ์„ค์ • + "format": "json", # Ollama needs the format to be specified explicitly + "base_url": "http://localhost:11434", # set Ollama URL }, "embeddings": { "model": "ollama/nomic-embed-text", - "base_url": "http://localhost:11434", # Ollama URL ์„ค์ • + "base_url": "http://localhost:11434", # set Ollama URL }, "verbose": True, } smart_scraper_graph = SmartScraperGraph( - prompt="ํ”„๋กœ์ ํŠธ์™€ ์„ค๋ช…์„ ๋ชจ๋‘ ๋‚˜์—ดํ•˜์„ธ์š”", - # ์ด๋ฏธ ๋‹ค์šด๋กœ๋“œ๋œ HTML ์ฝ”๋“œ๊ฐ€ ์žˆ๋Š” ๋ฌธ์ž์—ด๋„ ํ—ˆ์šฉ + prompt="List me all the projects with their descriptions", + # also accepts a string with the already downloaded HTML code source="https://perinim.github.io/projects", config=graph_config ) @@ -78,15 +83,16 @@ print(result) ์ถœ๋ ฅ์€ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ํ”„๋กœ์ ํŠธ์™€ ์„ค๋ช…์˜ ๋ชฉ๋ก์ด ๋  ๊ฒƒ์ž…๋‹ˆ๋‹ค: ```python -{'projects': [{'title': 'Rotary Pendulum RL', 'description': 'RL ์•Œ๊ณ ๋ฆฌ์ฆ˜์„ ์‚ฌ์šฉํ•˜์—ฌ ์‹ค์ œ ํšŒ์ „ ์ง„์ž๋ฅผ ์ œ์–ดํ•˜๋Š” ์˜คํ”ˆ ์†Œ์Šค ํ”„๋กœ์ ํŠธ'}, {'title': 'DQN Implementation from scratch', 'description': '๊ฐ„๋‹จํ•œ ๋ฐ ์ด์ค‘ ์ง„์ž๋ฅผ ํ›ˆ๋ จํ•˜๊ธฐ ์œ„ํ•œ ๋”ฅ Q-๋„คํŠธ์›Œํฌ ์•Œ๊ณ ๋ฆฌ์ฆ˜ ๊ฐœ๋ฐœ'}, ...]} -์‚ฌ๋ก€ 2: ํ˜ผํ•ฉ ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•˜๋Š” SearchGraph -์šฐ๋ฆฌ๋Š” LLM์— Groq๋ฅผ ์‚ฌ์šฉํ•˜๊ณ , ์ž„๋ฒ ๋”ฉ์— Ollama๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. +{'projects': [{'title': 'Rotary Pendulum RL', 'description': 'Open Source project aimed at controlling a real life rotary pendulum using RL algorithms'}, {'title': 'DQN Implementation from scratch', 'description': 'Developed a Deep Q-Network algorithm to train a simple and double pendulum'}, ...]} ``` +### ์‚ฌ๋ก€ 2: ํ˜ผํ•ฉ ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•˜๋Š” SearchGraph +์šฐ๋ฆฌ๋Š” LLM์— **Groq**๋ฅผ ์‚ฌ์šฉํ•˜๊ณ , ์ž„๋ฒ ๋”ฉ์— **Ollama**๋ฅผ ์‚ฌ์šฉํ•ฉ๋‹ˆ๋‹ค. + ```python from scrapegraphai.graphs import SearchGraph -# ๊ทธ๋ž˜ํ”„ ๊ตฌ์„ฑ ์ •์˜ +# Define the configuration for the graph graph_config = { "llm": { "model": "groq/gemma-7b-it", @@ -95,28 +101,30 @@ graph_config = { }, "embeddings": { "model": "ollama/nomic-embed-text", - "base_url": "http://localhost:11434", # Ollama URL ์ž„์˜ ์„ค์ • + "base_url": "http://localhost:11434", # set ollama URL arbitrarily }, "max_results": 5, } -# SearchGraph ์ธ์Šคํ„ด์Šค ์ƒ์„ฑ +# Create the SearchGraph instance search_graph = SearchGraph( - prompt="Chioggia์˜ ์ „ํ†ต ๋ ˆ์‹œํ”ผ๋ฅผ ๋ชจ๋‘ ๋‚˜์—ดํ•˜์„ธ์š”", + prompt="List me all the traditional recipes from Chioggia", config=graph_config ) -# ๊ทธ๋ž˜ํ”„ ์‹คํ–‰ +# Run the graph result = search_graph.run() print(result) -์ถœ๋ ฅ์€ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ๋ ˆ์‹œํ”ผ ๋ชฉ๋ก์ด ๋  ๊ฒƒ์ž…๋‹ˆ๋‹ค: ``` +์ถœ๋ ฅ์€ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ๋ ˆ์‹œํ”ผ ๋ชฉ๋ก์ด ๋  ๊ฒƒ์ž…๋‹ˆ๋‹ค: + ```python {'recipes': [{'name': 'Sarde in Saรฒre'}, {'name': 'Bigoli in salsa'}, {'name': 'Seppie in umido'}, {'name': 'Moleche frite'}, {'name': 'Risotto alla pescatora'}, {'name': 'Broeto'}, {'name': 'Bibarasse in Cassopipa'}, {'name': 'Risi e bisi'}, {'name': 'Smegiassa Ciosota'}]} -์‚ฌ๋ก€ 3: OpenAI๋ฅผ ์‚ฌ์šฉํ•˜๋Š” SpeechGraph -OpenAI API ํ‚ค์™€ ๋ชจ๋ธ ์ด๋ฆ„๋งŒ ์ „๋‹ฌํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค. ``` +### ์‚ฌ๋ก€ 3: OpenAI๋ฅผ ์‚ฌ์šฉํ•˜๋Š” SpeechGraph + +OpenAI API ํ‚ค์™€ ๋ชจ๋ธ ์ด๋ฆ„๋งŒ ์ „๋‹ฌํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค. ```python from scrapegraphai.graphs import SpeechGraph @@ -135,22 +143,23 @@ graph_config = { } # ************************************************ -# SpeechGraph ์ธ์Šคํ„ด์Šค๋ฅผ ์ƒ์„ฑํ•˜๊ณ  ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค. +# Create the SpeechGraph instance and run it # ************************************************ speech_graph = SpeechGraph( - prompt="ํ”„๋กœ์ ํŠธ์— ๋Œ€ํ•œ ์ž์„ธํ•œ ์˜ค๋””์˜ค ์š”์•ฝ์„ ๋งŒ๋“œ์„ธ์š”.", + prompt="Make a detailed audio summary of the projects.", source="https://perinim.github.io/projects/", config=graph_config, ) result = speech_graph.run() print(result) + ``` ์ถœ๋ ฅ์€ ํŽ˜์ด์ง€์˜ ํ”„๋กœ์ ํŠธ ์š”์•ฝ์ด ํฌํ•จ๋œ ์˜ค๋””์˜ค ํŒŒ์ผ์ด ๋  ๊ฒƒ์ž…๋‹ˆ๋‹ค. -ํ›„์›์‚ฌ +## ์Šคํฐ
@@ -165,46 +174,68 @@ print(result) ๊ธฐ์—ฌ๋ฅผ ํ™˜์˜ํ•˜๋ฉฐ, ๊ฐœ์„  ์‚ฌํ•ญ์„ ๋…ผ์˜ํ•˜๊ณ  ์ œ์•ˆ ์‚ฌํ•ญ์„ ์ฃผ๊ณ ๋ฐ›๊ธฐ ์œ„ํ•ด ์šฐ๋ฆฌ์˜ Discord ์„œ๋ฒ„์— ์ฐธ์—ฌํ•˜์„ธ์š”! -๊ธฐ์—ฌ ๊ฐ€์ด๋“œ๋ผ์ธ์„ ์ฐธ์กฐํ•˜์‹ญ์‹œ์˜ค. +๊ธฐ์—ฌ ๊ฐ€์ด๋“œ๋ผ์ธ์„ ์ฐธ๊ณ ํ•ด์ฃผ์„ธ์š”: [contributing guidelines](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/CONTRIBUTING.md). ## ๐Ÿ“ˆ ๋กœ๋“œ๋งต -ํ”„๋กœ์ ํŠธ ๋กœ๋“œ๋งต์„ ์—ฌ๊ธฐ์—์„œ ํ™•์ธํ•˜์„ธ์š”! ๐Ÿš€ +๋‹ค์Œ ๊ธฐ๋Šฅ๋“ค์„ ์ž‘์—…ํ•˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค! ํ˜‘์—…์— ๊ด€์‹ฌ์ด ์žˆ์œผ์‹œ๋ฉด ํ•ด๋‹น ๊ธฐ๋Šฅ์„ ๋งˆ์šฐ์Šค ์˜ค๋ฅธ์ชฝ ๋ฒ„ํŠผ์œผ๋กœ ํด๋ฆญํ•˜์—ฌ ์ƒˆ ํƒญ์—์„œ PR์„ ์ž‘์„ฑํ•ด์ฃผ์„ธ์š”. ์˜๋ฌธ์‚ฌํ•ญ์ด ์žˆ๊ฑฐ๋‚˜ ๋…ผ์˜ํ•˜๊ณ  ์‹ถ๋‹ค๋ฉด [Discord](https://discord.gg/uJN7TYcpNa)์—์„œ ์ €ํฌ์—๊ฒŒ ์—ฐ๋ฝํ•˜๊ฑฐ๋‚˜ Github์˜ [Discussion](https://github.com/VinciGit00/Scrapegraph-ai/discussions) ํŽ˜์ด์ง€๋ฅผ ์—ด์–ด์ฃผ์„ธ์š”! -๋กœ๋“œ๋งต์„ ๋” ์ธํ„ฐ๋ž™ํ‹ฐ๋ธŒํ•˜๊ฒŒ ์‹œ๊ฐํ™”ํ•˜๊ณ  ์‹ถ์œผ์‹ ๊ฐ€์š”? markdown ๋‚ด์šฉ์„ ํŽธ์ง‘๊ธฐ์— ๋ณต์‚ฌํ•˜์—ฌ markmap ์‹œ๊ฐํ™”๋ฅผ ํ™•์ธํ•˜์„ธ์š”! +```mermaid +%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#5C4B9B', 'edgeLabelBackground':'#ffffff', 'tertiaryColor': '#ffffff', 'primaryBorderColor': '#5C4B9B', 'fontFamily': 'Arial', 'fontSize': '16px', 'textColor': '#5C4B9B' }}}%% +graph LR + A[DeepSearch Graph] --> F[Use Existing Chromium Instances] + F --> B[Page Caching] + B --> C[Screenshot Scraping] + C --> D[Handle Dynamic Content] + D --> E[New Webdrivers] + + style A fill:#ffffff,stroke:#5C4B9B,stroke-width:2px,rx:10,ry:10 + style F fill:#ffffff,stroke:#5C4B9B,stroke-width:2px,rx:10,ry:10 + style B fill:#ffffff,stroke:#5C4B9B,stroke-width:2px,rx:10,ry:10 + style C fill:#ffffff,stroke:#5C4B9B,stroke-width:2px,rx:10,ry:10 + style D fill:#ffffff,stroke:#5C4B9B,stroke-width:2px,rx:10,ry:10 + style E fill:#ffffff,stroke:#5C4B9B,stroke-width:2px,rx:10,ry:10 + + click A href "https://github.com/VinciGit00/Scrapegraph-ai/issues/260" "Open DeepSearch Graph Issue" + click F href "https://github.com/VinciGit00/Scrapegraph-ai/issues/329" "Open Chromium Instances Issue" + click B href "https://github.com/VinciGit00/Scrapegraph-ai/issues/197" "Open Page Caching Issue" + click C href "https://github.com/VinciGit00/Scrapegraph-ai/issues/197" "Open Screenshot Scraping Issue" + click D href "https://github.com/VinciGit00/Scrapegraph-ai/issues/279" "Open Handle Dynamic Content Issue" + click E href "https://github.com/VinciGit00/Scrapegraph-ai/issues/171" "Open New Webdrivers Issue" +``` ## ๏ธ ๊ธฐ์—ฌ์ž๋“ค - - +[![Contributors](https://contrib.rocks/image?repo=VinciGit00/Scrapegraph-ai)](https://github.com/VinciGit00/Scrapegraph-ai/graphs/contributors) ## ๐ŸŽ“ ์ธ์šฉ - ์šฐ๋ฆฌ์˜ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋ฅผ ์—ฐ๊ตฌ ๋ชฉ์ ์œผ๋กœ ์‚ฌ์šฉํ•œ ๊ฒฝ์šฐ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ์ธ์šฉํ•ด ์ฃผ์„ธ์š”: - -text -Copia codice +```text @misc{scrapegraph-ai, author = {Marco Perini, Lorenzo Padoan, Marco Vinciguerra}, title = {Scrapegraph-ai}, year = {2024}, url = {https://github.com/VinciGit00/Scrapegraph-ai}, - note = {๋Œ€๊ทœ๋ชจ ์–ธ์–ด ๋ชจ๋ธ์„ ํ™œ์šฉํ•œ Python ์Šคํฌ๋ ˆ์ดํ•‘ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ} + note = {A Python library for scraping leveraging large language models} } -์ €์ž๋“ค +``` + +## ์ €์ž๋“ค

Authors_logos

-์—ฐ๋ฝ์ฒ˜ -Marco Vinciguerra -Marco Perini -Lorenzo Padoan + +| | ์—ฐ๋ฝ์ฒ˜ | +|--------------------|---------------| +| Marco Vinciguerra | [![Linkedin Badge](https://img.shields.io/badge/-Linkedin-blue?style=flat&logo=Linkedin&logoColor=white)](https://www.linkedin.com/in/marco-vinciguerra-7ba365242/) | +| Marco Perini | [![Linkedin Badge](https://img.shields.io/badge/-Linkedin-blue?style=flat&logo=Linkedin&logoColor=white)](https://www.linkedin.com/in/perinim/) | +| Lorenzo Padoan | [![Linkedin Badge](https://img.shields.io/badge/-Linkedin-blue?style=flat&logo=Linkedin&logoColor=white)](https://www.linkedin.com/in/lorenzo-padoan-4521a2154/) | ## ๐Ÿ“œ ๋ผ์ด์„ ์Šค -ScrapeGraphAI๋Š” MIT License๋กœ ๋ผ์ด์„ ์Šค๊ฐ€ ๋ถ€์—ฌ๋˜์—ˆ์Šต๋‹ˆ๋‹ค. ์ž์„ธํ•œ ๋‚ด์šฉ์€ LICENSE ํŒŒ์ผ์„ ์ฐธ์กฐํ•˜์„ธ์š”. +ScrapeGraphAI๋Š” MIT License๋กœ ๋ฐฐํฌ๋˜์—ˆ์Šต๋‹ˆ. ์ž์„ธํ•œ ๋‚ด์šฉ์€ [LICENSE](https://github.com/VinciGit00/Scrapegraph-ai/blob/main/LICENSE) ํŒŒ์ผ์„ ์ฐธ์กฐํ•˜์„ธ์š”. -๊ฐ์‚ฌ์˜ ๋ง +## ๊ฐ์‚ฌ์˜ ๋ง -ํ”„๋กœ์ ํŠธ์— ๊ธฐ์—ฌํ•œ ๋ชจ๋“  ๋ถ„๋“ค๊ณผ ์˜คํ”ˆ ์†Œ์Šค ์ปค๋ฎค๋‹ˆํ‹ฐ์— ๊ฐ์‚ฌ๋“œ๋ฆฝ๋‹ˆ๋‹ค. -ScrapeGraphAI๋Š” ๋ฐ์ดํ„ฐ ํƒ์ƒ‰ ๋ฐ ์—ฐ๊ตฌ ๋ชฉ์ ์œผ๋กœ๋งŒ ์‚ฌ์šฉ๋˜์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์šฐ๋ฆฌ๋Š” ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์˜ ์˜ค์šฉ์— ๋Œ€ํ•ด ์ฑ…์ž„์„ ์ง€์ง€ ์•Š์Šต๋‹ˆ๋‹ค. \ No newline at end of file +- ํ”„๋กœ์ ํŠธ์— ๊ธฐ์—ฌํ•œ ๋ชจ๋“  ๋ถ„๋“ค๊ณผ ์˜คํ”ˆ ์†Œ์Šค ์ปค๋ฎค๋‹ˆํ‹ฐ์— ๊ฐ์‚ฌ๋“œ๋ฆฝ๋‹ˆ๋‹ค. +- ScrapeGraphAI๋Š” ๋ฐ์ดํ„ฐ ํƒ์ƒ‰ ๋ฐ ์—ฐ๊ตฌ ๋ชฉ์ ์œผ๋กœ๋งŒ ์‚ฌ์šฉ๋˜์–ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์šฐ๋ฆฌ๋Š” ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์˜ ์˜ค์šฉ์— ๋Œ€ํ•ด ์ฑ…์ž„์„ ์ง€์ง€ ์•Š์Šต๋‹ˆ๋‹ค. From b91020ce56e88a75c0eb7544a2d3f61bfc77cb57 Mon Sep 17 00:00:00 2001 From: Kernick Date: Thu, 18 Jul 2024 21:03:55 +0300 Subject: [PATCH 09/10] Update models_tokens.py with gpt-4o-mini OpenAI added a new available model - gpt-4o-mini. Adding this model to the supported models, so it would be possible to use it with the Scrapegraph-ai. --- scrapegraphai/helpers/models_tokens.py | 1 + 1 file changed, 1 insertion(+) diff --git a/scrapegraphai/helpers/models_tokens.py b/scrapegraphai/helpers/models_tokens.py index 0ef10277..d708f7db 100644 --- a/scrapegraphai/helpers/models_tokens.py +++ b/scrapegraphai/helpers/models_tokens.py @@ -16,6 +16,7 @@ models_tokens = { "gpt-4-32k": 32768, "gpt-4-32k-0613": 32768, "gpt-4o": 128000, + "gpt-40-mini":128000, }, "azure": { "gpt-3.5-turbo": 4096, From 4f079cfc7449f574de987ca3fa6c5c80f0c831b3 Mon Sep 17 00:00:00 2001 From: Kernick Date: Thu, 18 Jul 2024 21:06:16 +0300 Subject: [PATCH 10/10] Fixed a typo in the model name --- scrapegraphai/helpers/models_tokens.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scrapegraphai/helpers/models_tokens.py b/scrapegraphai/helpers/models_tokens.py index d708f7db..fb4c1e75 100644 --- a/scrapegraphai/helpers/models_tokens.py +++ b/scrapegraphai/helpers/models_tokens.py @@ -16,7 +16,7 @@ models_tokens = { "gpt-4-32k": 32768, "gpt-4-32k-0613": 32768, "gpt-4o": 128000, - "gpt-40-mini":128000, + "gpt-4o-mini":128000, }, "azure": { "gpt-3.5-turbo": 4096,