File: //usr/lib/python3/dist-packages/landscape/lib/__pycache__/fetch.cpython-38.pyc
U
���c� � @ s� d dl Z d dlZd dlZd dlmZ d dlmZ d dlmZ d dl m
Z
m Z G dd� de �Z
G dd � d e
�ZG d
d � d e
�Zd d
i ddddd dddf dd�Zdd� Zd dd�Zd dd�Zd!dd�Zdd � Zed kr�eejd d� � dS )"� N)� OptionParser)� DeferredList)�
deferToThread)� iteritems�
networkStringc @ s e Zd ZdS )�
FetchErrorN)�__name__�
__module__� __qualname__� r r �5/usr/lib/python3/dist-packages/landscape/lib/fetch.pyr s r c @ s$ e Zd Zdd� Zdd� Zdd� ZdS )�
HTTPCodeErrorc C s || _ || _d S �N)� http_code�body)�selfr r r r r �__init__ s zHTTPCodeError.__init__c C s
d| j S )Nz Server returned HTTP code %d�r �r r r r �__str__ s zHTTPCodeError.__str__c C s
d| j S )Nz <HTTPCodeError http_code=%d>r r r r r �__repr__ s zHTTPCodeError.__repr__N)r r r
r r r r r r r r
s r
c @ s0 e Zd Zdd� Zdd� Zdd� Zedd� �Zd S )
� PyCurlErrorc C s || _ || _d S r )�
error_code�_message)r r �messager r r r s zPyCurlError.__init__c C s d| j | jf S )Nz Error %d: %s�r r r r r r r # s zPyCurlError.__str__c C s d| j | jf S )Nz <PyCurlError args=(%d, '%s')>r r r r r r & s �zPyCurlError.__repr__c C s | j S r )r r r r r r * s zPyCurlError.messageN)r r r
r r r �propertyr r r r r r s
r F� � iX Tc
C s ddl } t|t�s |�d�}t�|�}
t�� }|dkr>| �� }|�| jt t
| ��� |r�|�| j d� |r�|�| j t
|�� |�| j|
j� |r�| �d�r�|�| jt |�� |r�|�| jdd� tt|��D �� |r�|�| jd� | r�|�| jd � |
dk �r|�| jt |
�� | dk �r,|�| jt | �� |�| jd
� |�| j|� |�| jd � |�| j |� |�| j d � |�| j |j � |�| j d� |�| j!d � z |�"� W n< | j#k
�r� } zt$|j%d |j%d ��W 5 d}~X Y nX |�&� }|�'| j(�}|d k�rt)||��|S )
aC Retrieve a URL and return the content.
@param url: The url to be fetched.
@param post: If true, the POST method will be used (defaults to GET).
@param data: Data to be sent to the server as the POST content.
@param headers: Dictionary of header => value entries to be used on the
request.
@param curl: A pycurl.Curl instance to use. If not provided, one will be
created.
@param cainfo: Path to the file with CA certificates.
@param insecure: If true, perform curl using insecure option which will
not attempt to verify authenticity of the peer's certificate. (Used
during autodiscovery)
@param follow: If True, follow HTTP redirects (default True).
@param user_agent: The user-agent to set in the request.
@param proxy: The proxy url to use for the request.
r Nzutf-8Tzhttps:c S s g | ] }d | �qS )z%s: %sr )�.0Zpairr r r �
<listcomp>\ s zfetch.<locals>.<listcomp>F� � s gzip,deflate�� )*�pycurl�
isinstance�bytes�encode�io�BytesIOZCurlZsetoptZURLr �strZPOSTZ
POSTFIELDSIZE�lenZ READFUNCTION�read�
startswithZCAINFOZ
HTTPHEADER�sortedr ZSSL_VERIFYPEERZFOLLOWLOCATIONZ USERAGENTZPROXYZ MAXREDIRSZCONNECTTIMEOUTZLOW_SPEED_LIMITZLOW_SPEED_TIMEZNOSIGNALZ
WRITEFUNCTION�writeZDNS_CACHE_TIMEOUT�ENCODINGZperform�errorr �args�getvalueZgetinfoZ HTTP_CODEr
)�url�post�dataZheaders�cainfoZcurlZconnect_timeoutZ
total_timeoutZinsecureZfollowZ
user_agent�proxyr$ �output�input�er r r r r �fetch/ sX
�
(
r< c O s t tf| �|�S )z]Retrieve a URL asynchronously.
@return: A C{Deferred} resulting in the URL content.
)r r< )r2 �kwargsr r r � fetch_async� s r> c K sR g }| D ]:}t |f|�}|r(|�||� |r8|�||� |�|� qt|ddd�S )a9
Retrieve a list of URLs asynchronously.
@param callback: Optionally, a function that will be fired one time for
each successful URL, and will be passed its content and the URL itself.
@param errback: Optionally, a function that will be fired one time for each
failing URL, and will be passed the failure and the URL itself.
@return: A C{DeferredList} whose callback chain will be fired as soon as
all downloads have terminated. If an error occurs, the errback chain
of the C{DeferredList} will be fired immediatly.
T)ZfireOnOneErrbackZ
consumeErrors)r> Z addCallbackZ
addErrback�appendr )�urls�callback�errbackr= Zresultsr4 �resultr r r �fetch_many_async� s rD c C s. | � d��d�d }|dk r*tj�||�}|S )z�Return the last component of the given C{url}.
@param url: The URL to get the filename from.
@param directory: Optionally a path to prepend to the returned filename.
@note: Any trailing slash in the C{url} will be removed
�/���N)�rstrip�split�os�path�join)r4 � directory�filenamer r r �url_to_filename� s rN c s. � fdd�}�fdd�}t | f||d�|��S )aJ
Retrieve a list of URLs and save their content as files in a directory.
@param urls: The list URLs to fetch.
@param directory: The directory to save the files to, the name of the file
will equal the last fragment of the URL.
@param logger: Optional function to be used to log errors for failed URLs.
c s, t |� d�}t|d�}|�| � |�� d S )N�rL �wb)rN �openr/ �close)r6 r4 rM �fdrO r r r/ � s
z fetch_to_files.<locals>.writec s � r� d|t | j�f � | S )Nz Couldn't fetch file from %s (%s))r* �value)Zfailurer4 )�loggerr r � log_error� s �z!fetch_to_files.<locals>.log_error)rA rB )rD )r@ rL rU r= r/ rV r )rL rU r �fetch_to_files� s
rW c C sZ t � }|jddd� |jddd� |�d� |�| �\}\}tt||j|j|jd�� d S ) Nz--post�
store_true)�actionz--datar )�defaultz--cainfo)r5 r6 r7 )r Z
add_option�
parse_args�printr< r5 r6 r7 )r2 �parserZoptionsr4 r r r �test� s
�r^ �__main__r! )NN)N)N)rI �sysr( Zoptparser Ztwisted.internet.deferr Ztwisted.internet.threadsr Ztwisted.python.compatr r � Exceptionr r
r r< r> rD rN rW r^ r �argvr r r r �<module> s0
�
R