usp.web_client package¶
Submodules¶
usp.web_client.abstract_client module¶
Abstract web client class.
-
class
usp.web_client.abstract_client.
AbstractWebClient
[source]¶ Bases:
object
Abstract web client to be used by the sitemap fetcher.
-
get
(url: str) → usp.web_client.abstract_client.AbstractWebClientResponse[source]¶ Fetch an URL and return a response.
Method shouldn’t throw exceptions on connection errors (including timeouts); instead, such errors should be reported via Response object.
Parameters: url – URL to fetch. Returns: Response object.
-
-
class
usp.web_client.abstract_client.
AbstractWebClientResponse
[source]¶ Bases:
object
Abstract response.
-
class
usp.web_client.abstract_client.
AbstractWebClientSuccessResponse
[source]¶ Bases:
usp.web_client.abstract_client.AbstractWebClientResponse
Successful response.
-
header
(case_insensitive_name: str) → Optional[str][source]¶ Return HTTP header value for a given case-insensitive name, or None if such header wasn’t set.
Parameters: case_insensitive_name – HTTP header’s name, e.g. “Content-Type”. Returns: HTTP header’s value, or None if it was unset.
-
raw_data
() → bytes[source]¶ Return encoded raw data of the response.
Returns: Encoded raw data of the response.
-
-
usp.web_client.abstract_client.
RETRYABLE_HTTP_STATUS_CODES
= {400, 408, 429, 499, 500, 502, 503, 504, 509, 520, 521, 522, 523, 524, 525, 526, 527, 530, 598}¶ HTTP status codes on which a request should be retried.
-
class
usp.web_client.abstract_client.
WebClientErrorResponse
(message: str, retryable: bool)[source]¶ Bases:
usp.web_client.abstract_client.AbstractWebClientResponse
Error response.
usp.web_client.requests_client module¶
requests-based implementation of web client class.
-
class
usp.web_client.requests_client.
RequestsWebClient
[source]¶ Bases:
usp.web_client.abstract_client.AbstractWebClient
requests-based web client to be used by the sitemap fetcher.
-
get
(url: str) → usp.web_client.abstract_client.AbstractWebClientResponse[source]¶ Fetch an URL and return a response.
Method shouldn’t throw exceptions on connection errors (including timeouts); instead, such errors should be reported via Response object.
Parameters: url – URL to fetch. Returns: Response object.
-
-
class
usp.web_client.requests_client.
RequestsWebClientErrorResponse
(message: str, retryable: bool)[source]¶ Bases:
usp.web_client.abstract_client.WebClientErrorResponse
requests-based error response.
-
class
usp.web_client.requests_client.
RequestsWebClientSuccessResponse
(requests_response: requests.models.Response, max_response_data_length: Optional[int] = None)[source]¶ Bases:
usp.web_client.abstract_client.AbstractWebClientSuccessResponse
requests-based successful response.
-
header
(case_insensitive_name: str) → Optional[str][source]¶ Return HTTP header value for a given case-insensitive name, or None if such header wasn’t set.
Parameters: case_insensitive_name – HTTP header’s name, e.g. “Content-Type”. Returns: HTTP header’s value, or None if it was unset.
-
raw_data
() → bytes[source]¶ Return encoded raw data of the response.
Returns: Encoded raw data of the response.
-