mirror of https://github.com/VinciGit00/Scrapegraph-ai.git synced 2026-06-15 21:00:30 +08:00

History

VinciGit00 e3d0194dc9 fix: script generator and add new benchmarks		2024-04-30 11:51:04 +02:00
..
inputs	add new benchmarks	2024-04-20 22:49:25 +02:00
.env.example	ADD benchmarks m2 max llama3	2024-04-23 12:28:51 +02:00
benchmark_docker.py	handling the agent model	2024-04-24 13:13:46 +02:00
benchmark_groq.py	fix: script generator and add new benchmarks	2024-04-30 11:51:04 +02:00
benchmark_llama3.py	fix: script generator and add new benchmarks	2024-04-30 11:51:04 +02:00
benchmark_mistral.py	add integration for llama3	2024-04-21 13:11:10 +02:00
benchmark_openai_gpt4.py	ADD benchmarks m2 max llama3	2024-04-23 12:28:51 +02:00
benchmark_openai_gpt35.py	ADD benchmarks m2 max llama3	2024-04-23 12:28:51 +02:00
Readme.md	fix: script generator and add new benchmarks	2024-04-30 11:51:04 +02:00

Readme.md

Local models

The two websites benchmark are:

Example 1: https://perinim.github.io/projects
Example 2: https://www.wired.com (at 17/4/2024)

Both are strored locally as txt file in .txt format because in this way we do not have to think about the internet connection

Hardware	Model	Example 1	Example 2
Macbook 14' m1 pro	Mistral on Ollama with nomic-embed-text	11.60s	26.61s
Macbook m2 max	Mistral on Ollama with nomic-embed-text	8.05s	12.17s
Macbook 14' m1 pro	Llama3 on Ollama with nomic-embed-text	29.87s	35.32s
Macbook m2 max	Llama3 on Ollama with nomic-embed-text	18.36s	78.32s

Note: the examples on Docker are not runned on other devices than the Macbook because the performance are to slow (10 times slower than Ollama). Indeed the results are the following:

Hardware	Example 1	Example 2
Macbook 14' m1 pro	139.89	Too long

Performance on APIs services

Example 1: personal portfolio

URL: https://perinim.github.io/projects Task: List me all the projects with their description.

Name	Execution time (seconds)	total_tokens	prompt_tokens	completion_tokens	successful_requests	total_cost_USD
gpt-3.5-turbo	25.22	445	272	173	1	0.000754
gpt-4-turbo-preview	9.53	449	272	177	1	0.00803
Grooq with nomic-embed-text	1.99	474	284	190	1	0

Example 2: Wired

URL: https://www.wired.com Task: List me all the articles with their description.

Name	Execution time (seconds)	total_tokens	prompt_tokens	completion_tokens	successful_requests	total_cost_USD
gpt-3.5-turbo	25.89	445	272	173	1	0.000754
gpt-4-turbo-preview	64.70	3573	2199	1374	1	0.06321
Grooq with nomic-embed-text	3.82	2459	2192	267	1	0