Commit Graph

17 Commits

Author SHA1 Message Date
Umut CAN
827f7260ad This commit focuses on optimizing the utility modules in the codebase for better performance and maintainability. Key improvements include: - More efficient HTML processing with combined regex operations and optimized tag handling - Enhanced deep copy functionality with better type handling and optimized recursion - Refactored web search with improved error handling and modular helper functions The changes maintain all existing functionality while improving code quality, performance, and maintainability. Documentation and type hints have been enhanced throughout.
Optimize utils modules for better performance and maintainability

- Improve HTML cleanup and minification:
  - Combine regex operations for better performance
  - Add better error handling for HTML processing
  - Optimize tag removal and attribute filtering

- Enhance deep copy functionality:
  - Add special case handling for primitive types
  - Improve type checking and error handling
  - Optimize recursive copying for collections

- Refactor web search functionality:
  - Add input validation and error handling
  - Split search logic into separate helper functions
  - Improve proxy handling and configuration
  - Add better timeout and error management
  - Optimize URL filtering and processing

Technical improvements:
- Better type hints and documentation
- More efficient data structures
- Improved error handling and validation
- Reduced code duplication
- Better separation of concerns

No breaking changes - all existing functionality maintained
2024-10-28 22:40:32 +03:00
Marco Vinciguerra
e017ee21f3 update readme and readibility of the code 2024-10-02 10:07:03 +02:00
Marco Vinciguerra
560f079d4c refactoring of the code 2024-09-28 09:02:20 +02:00
Matteo Vedovati
5b579b323f gode generator v0.1 2024-09-21 11:44:04 +02:00
Marco Vinciguerra
f7ba1f30de refactoring of the code 2024-08-23 11:33:22 +02:00
Marco Vinciguerra
b115f9612f refactoring of code and documentation 2024-08-16 09:01:27 +02:00
Marco Vinciguerra
8b2c266aff refactoring of the code
Co-Authored-By: Matteo Vedovati <68272450+vedovati-matteo@users.noreply.github.com>
2024-08-10 17:44:35 +02:00
Marco Vinciguerra
2804434a9e feat: add integrations for markdown files
Some checks are pending
/ build (3.10) (push) Waiting to run
2024-06-29 13:35:39 +02:00
Marco Perini
283b61fafc docs: better logging 2024-06-13 18:13:47 +02:00
Marco Vinciguerra
5065aa00e1
Merge branch 'pre/beta' into supercoder-327 2024-06-12 12:12:15 +02:00
supercoder-dev
879c94a2b5
Update cleanup_html.py 2024-06-12 14:30:02 +05:30
seyf97
acece72c28
Update cleanup_html.py
Remove redundant lines in Links extraction
2024-06-04 13:49:00 +03:00
Marco Perini
a296927624 feat(omni-scraper): working OmniScraperGraph with images 2024-05-14 13:46:49 +02:00
VinciGit00
dc91719365 Update cleanup_html.py 2024-05-11 10:49:16 +02:00
mayurdb
300fd5d253 Fetch links in the page while parsing html 2024-05-11 09:46:51 +05:30
Marco Perini
864aa91326 feat: revert fetch_node 2024-05-10 15:11:54 +02:00
mayurdb
f8ce3d5916 fix: Augment the information getting fetched from a webpage 2024-05-10 13:28:53 +05:30