Skip to content

Conversation

@kudj
Copy link
Contributor

@kudj kudj commented Nov 7, 2024

Add async HTTP client support

This PR adds an async variant of the HTTP client alongside the existing synchronous implementation.

What's new

  • AsyncHttpClient: New async client built on httpx with the same interface as the sync client

    • Async/await support for non-blocking I/O operations
    • Built-in rate limiting via aiolimiter
    • Same retry logic, error handling, and parameter merging as the sync client
  • Test coverage: Comprehensive async test suite with proper mock handling

  • Documentation: Updated README with async examples and usage patterns

  • Modernization:

    • Migrated from setup.py to pyproject.toml
    • Added uv for dependency management
    • Updated type hints to use modern syntax (dict | None instead of Optional[Dict])
    • Cleaned up imports and removed deprecated patterns

Breaking changes

None - the sync HttpClient API remains unchanged.

@soustruh soustruh force-pushed the feature/async branch 3 times, most recently from de527c3 to 4a4877f Compare October 2, 2025 21:59
@soustruh soustruh self-requested a review October 2, 2025 23:21
@soustruh soustruh force-pushed the feature/async branch 2 times, most recently from cd52e02 to a733f7c Compare October 3, 2025 12:36
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Adds an asynchronous HTTP client alongside the existing synchronous client, migrates packaging to pyproject.toml with uv-based workflows, modernizes type hints, and expands documentation and examples to cover async usage and CI changes.

  • Introduces AsyncHttpClient with retry, backoff, optional rate limiting, and batch processing.
  • Refactors sync client type hints / formatting and replaces setup.py with pyproject.toml plus updated CI workflows.
  • Expands README and examples for async usage (some documentation inconsistencies introduced).

Reviewed Changes

Copilot reviewed 15 out of 18 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
tests/test_async.py New test suite for AsyncHttpClient (params/header/auth/retry handling; lacks coverage for some new behaviors).
src/keboola/http_client/http.py Type hint modernization, import adjustments, formatting; minor logic unchanged except stylistic edits.
src/keboola/http_client/async_client.py New asynchronous client implementation with retries, backoff, rate limiting, and batch processing.
src/keboola/http_client/init.py Exposes AsyncHttpClient at package root.
pyproject.toml New project metadata replacing setup.py; changes project name and dependency management.
flake8.cfg Updated lint configuration (ignore list and limits).
docs/examples/* Added sync and async usage examples, including batch processing and rate limiting demos.
README.md Extended documentation for async client and reorganized sections (contains inaccuracies).
.github/workflows/*.yml Updated CI to use uv, add Python versions, automate version injection and publishing.
setup.py Removed in favor of pyproject.toml.

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.


return url

async def update_auth_header(self, updated_header: dict, overwrite: bool = False):
Copy link

Copilot AI Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Method contains no awaits and performs synchronous mutations only; consider making it a regular def to simplify usage (so callers need not await it) and align with the synchronous client's API.

Suggested change
async def update_auth_header(self, updated_header: dict, overwrite: bool = False):
def update_auth_header(self, updated_header: dict, overwrite: bool = False):

Copilot uses AI. Check for mistakes.
Comment on lines +97 to +100
if overwrite is False:
self._auth_header.update(updated_header)
else:
self._auth_header = updated_header
Copy link

Copilot AI Oct 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Method contains no awaits and performs synchronous mutations only; consider making it a regular def to simplify usage (so callers need not await it) and align with the synchronous client's API.

Copilot uses AI. Check for mistakes.
The issue was that when mock_request didn't have a return value configured and returned a MagicMock object instead, so when response.raise_for_status() was called, the mock returned a coroutine that was never awaited.
Copy link

@ZdenekSrotyr ZdenekSrotyr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, tests look solid and everything reads well.

Copy link
Contributor

@soustruh soustruh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, the updated client was thoroughly tested in many components already, the test coverage is also sufficient 👍

@soustruh soustruh merged commit 7891b0d into main Nov 19, 2025
3 checks passed
@soustruh soustruh deleted the feature/async branch November 19, 2025 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants