This commit enhances the test suite for the FetchNode class by introducing mocking for the execute method using the unittest.mock module.
Changes:
- Imported the patch and MagicMock classes from unittest.mock.
- Decorated each test function with @patch('scrapegraphai.nodes.FetchNode.execute') to mock the execute method.
- Set the return_value of the mocked execute method to a MagicMock instance.
- Added assertions to check if the mocked execute method was called with the expected state dictionary.
- Updated the test functions to use the mocked execute method instead of the actual implementation.
Benefits:
- Improved test reliability by isolating the FetchNode class from external dependencies.
- Faster test execution since external resources (e.g., URLs, files) are not required.
- Better test coverage by testing the execute method's behavior with various input states.
- Increased maintainability by decoupling tests from the implementation details of the execute method.
The functionality of the FetchNode class remains unchanged, but the tests now use mocking to ensure the correct behavior of the execute method without relying on external resources or dependencies.
This commit enhances the test suite for the JSON scraping pipeline by introducing the following improvements:
- Separate configuration from the test code by loading it from a JSON file (config.json)
- Use a parametrized fixture to run the test with multiple configurations automatically
- Read the sample JSON file from a separate inputs directory for better organization
- Add explicit assertions to verify the expected output (list of titles)
- Improve test organization and separation of concerns using fixtures
- Promote better coding practices and make the test suite more extensible
These changes aim to improve the testability, maintainability, and flexibility of the test suite. They make it easier to manage configurations, add or modify test cases, and ensure the robustness of the scraping pipeline. The test suite now follows best practices and is better prepared for future changes and requirements.
the broker has been made fully configurable for anonymity level, admissible locations, scheme and max shape not to waste resources, unlike the original `free-proxy` package.
other options have been explored (e.g., `proxybroker`, `proxybroker2`) due to their built-in proxy server and rotation capabilities, but the former is no longer maintained, and the latter has issue with any python version outside of python 3.9
