CodeCow/Scrapegraph-ai
1
0
Fork 0
mirror of https://github.com/VinciGit00/Scrapegraph-ai.git synced 2026-06-25 21:11:11 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #1 from DPende/refactoring_project

Browse Source 
Refactoring of the project
This commit is contained in:
Marco Vinciguerra 2024-01-27 21:08:08 +01:00 committed by GitHub
commit e3fbd2a137
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
11 changed files with 10 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
.DS_Store vendored
View File

Binary file not shown.

1
.env.example Normal file
View File

@ -0,0 +1 @@
API_KEY="your openai.com api key"

4
.gitignore vendored Normal file
View File

@ -0,0 +1,4 @@
.DS_Store
.DS_Store?
._*
__pycache__/

10
README.md
View File

@ -22,13 +22,13 @@ Follow the following steps:
3. Go to [https://openai.com](https://openai.com/) and login
4. Now you can access to [https://platform.openai.com/docs/overview](https://platform.openai.com/docs/overview)
5. Create a new API key and copy it
![Screenshot 2024-01-26 alle 17.10.10.png](doc/Screenshot_2024-01-26_alle_17.10.10.png)
![Screenshot 2024-01-26 alle 17.10.10.png](docs/generate_api_key/step_1.png)
![Screenshot 2024-01-26 alle 17.10.31.png](doc/Screenshot_2024-01-26_alle_17.10.31.png)
![Screenshot 2024-01-26 alle 17.10.31.png](docs/generate_api_key/step_2.png)
![Screenshot 2024-01-26 alle 17.10.52.png](doc/Screenshot_2024-01-26_alle_17.10.52.png)
![Screenshot 2024-01-26 alle 17.10.52.png](docs/generate_api_key/step_3.png)
![Screenshot 2024-01-26 alle 17.11.10.png](doc/Screenshot_2024-01-26_alle_17.11.10.png)
![Screenshot 2024-01-26 alle 17.11.10.png](docs/generate_api_key/step_4.png)
6. Open the .env file inside main and paste the API key
@ -123,4 +123,4 @@ if __name__ == "__main__":
Developed by
![logo-removebg-preview.png](doc/logo-removebg-preview.png)
![logo-removebg-preview.png](docs/logo_vincios.png)

0
doc/Screenshot_2024-01-26_alle_17.10.10.png → docs/generate_api_key/step_1.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 59 KiB

After

Width:  |  Height:  |  Size: 59 KiB

0
doc/Screenshot_2024-01-26_alle_17.10.31.png → docs/generate_api_key/step_2.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 121 KiB

After

Width:  |  Height:  |  Size: 121 KiB

0
doc/Screenshot_2024-01-26_alle_17.10.52.png → docs/generate_api_key/step_3.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 59 KiB

After

Width:  |  Height:  |  Size: 59 KiB

0
doc/Screenshot_2024-01-26_alle_17.11.10.png → docs/generate_api_key/step_4.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 85 KiB

After

Width:  |  Height:  |  Size: 85 KiB

0
doc/logo-removebg-preview.png → docs/logo_vincios.png
View File

Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 129 KiB

After

Width:  |  Height:  |  Size: 129 KiB

5
getter.py
View File

@ -1,11 +1,9 @@
import requests
from bs4 import BeautifulSoup
HEADERS = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36',
'Accept-Language': 'en-US'}
def get_function(link:str, param = HEADERS) -> str:
"""
It sends a GET request to the specified link with optional headers.
@ -20,9 +18,6 @@ def get_function(link:str, param = HEADERS) -> str:
response = requests.get(url=link, headers=HEADERS)
return str(response.content)
def scraper(link: str, max_char: int) -> str:
"""
Scrapes the HTML text and removes unwanted elements, text, and comments.

1
pydantic_class.py
View File

@ -1,4 +1,3 @@
from langchain_core.pydantic_v1 import BaseModel, Field
class Response(BaseModel):