usp.web_client package

Submodules

usp.web_client.abstract_client module

Abstract web client class.

class usp.web_client.abstract_client.AbstractWebClient[source]

Bases: object

Abstract web client to be used by the sitemap fetcher.

get(url: str) → usp.web_client.abstract_client.AbstractWebClientResponse[source]

Fetch an URL and return a response.

Method shouldn’t throw exceptions on connection errors (including timeouts); instead, such errors should be reported via Response object.

Parameters:url – URL to fetch.
Returns:Response object.
set_max_response_data_length(max_response_data_length: int) → None[source]

Set the maximum number of bytes that the web client will fetch.

Parameters:max_response_data_length – Maximum number of bytes that the web client will fetch.
class usp.web_client.abstract_client.AbstractWebClientResponse[source]

Bases: object

Abstract response.

class usp.web_client.abstract_client.AbstractWebClientSuccessResponse[source]

Bases: usp.web_client.abstract_client.AbstractWebClientResponse

Successful response.

header(case_insensitive_name: str) → Optional[str][source]

Return HTTP header value for a given case-insensitive name, or None if such header wasn’t set.

Parameters:case_insensitive_name – HTTP header’s name, e.g. “Content-Type”.
Returns:HTTP header’s value, or None if it was unset.
raw_data() → bytes[source]

Return encoded raw data of the response.

Returns:Encoded raw data of the response.
status_code() → int[source]

Return HTTP status code of the response.

Returns:HTTP status code of the response, e.g. 200.
status_message() → str[source]

Return HTTP status message of the response.

Returns:HTTP status message of the response, e.g. “OK”.
usp.web_client.abstract_client.RETRYABLE_HTTP_STATUS_CODES = {400, 408, 429, 499, 500, 502, 503, 504, 509, 520, 521, 522, 523, 524, 525, 526, 527, 530, 598}

HTTP status codes on which a request should be retried.

class usp.web_client.abstract_client.WebClientErrorResponse(message: str, retryable: bool)[source]

Bases: usp.web_client.abstract_client.AbstractWebClientResponse

Error response.

message() → str[source]

Return message describing what went wrong.

Returns:Message describing what went wrong.
retryable() → bool[source]

Return True if request should be retried.

Returns:True if request should be retried.

usp.web_client.requests_client module

requests-based implementation of web client class.

class usp.web_client.requests_client.RequestsWebClient[source]

Bases: usp.web_client.abstract_client.AbstractWebClient

requests-based web client to be used by the sitemap fetcher.

get(url: str) → usp.web_client.abstract_client.AbstractWebClientResponse[source]

Fetch an URL and return a response.

Method shouldn’t throw exceptions on connection errors (including timeouts); instead, such errors should be reported via Response object.

Parameters:url – URL to fetch.
Returns:Response object.
set_max_response_data_length(max_response_data_length: int) → None[source]

Set the maximum number of bytes that the web client will fetch.

Parameters:max_response_data_length – Maximum number of bytes that the web client will fetch.
set_timeout(timeout: int) → None[source]

Set HTTP request timeout.

class usp.web_client.requests_client.RequestsWebClientErrorResponse(message: str, retryable: bool)[source]

Bases: usp.web_client.abstract_client.WebClientErrorResponse

requests-based error response.

class usp.web_client.requests_client.RequestsWebClientSuccessResponse(requests_response: requests.models.Response, max_response_data_length: Optional[int] = None)[source]

Bases: usp.web_client.abstract_client.AbstractWebClientSuccessResponse

requests-based successful response.

header(case_insensitive_name: str) → Optional[str][source]

Return HTTP header value for a given case-insensitive name, or None if such header wasn’t set.

Parameters:case_insensitive_name – HTTP header’s name, e.g. “Content-Type”.
Returns:HTTP header’s value, or None if it was unset.
raw_data() → bytes[source]

Return encoded raw data of the response.

Returns:Encoded raw data of the response.
status_code() → int[source]

Return HTTP status code of the response.

Returns:HTTP status code of the response, e.g. 200.
status_message() → str[source]

Return HTTP status message of the response.

Returns:HTTP status message of the response, e.g. “OK”.

Module contents