File: //usr/lib/python3/dist-packages/pip/_internal/index/__pycache__/collector.cpython-310.pyc
o
@%Ne�S � @ s� d Z ddlZddlZddlZddlZddlZddlZddlZddlZ ddl
Z ddl Z ddl
mZ ddlmZ ddlmZmZmZmZmZmZmZmZmZmZm Z m Z ddl m Z m Z ddl!m"Z" ddl#m$Z$m%Z% dd l&m'Z' dd
l(m)Z) dd l*m+Z+ dd l,m-Z- dd
l.m/Z/ ddl0m1Z1 ddl2m3Z3 ddl4m5Z5m6Z6 ddl7m8Z8 ddl9m:Z:m;Z;m<Z< er�ddlm=Z= ne>Z=e�?e@�ZAe jBjCjDZEeeFeFf ZGdeFdeeF fdd�ZHG dd� deI�ZJde"ddfd d �ZKG d d � d eI�ZLdeFd e-ddfd!d"�ZMdeFd e-de"fd#d$�ZNd%eGdeeF fd&d'�ZOd(eEd)eFdeFfd*d+�ZPd,eFdeFfd-d.�ZQd,eFdeFfd/d0�ZRe�Sd1ejT�ZUd2eFd3eVdeFfd4d5�ZWdeFdeFfd6d7�ZXd8eeFeeF f d)eFd9eFdee) fd:d;�ZYG d<d=� d=�ZZG d>d?� d?e=�Z[d@e[de[fdAdB�Z\dCdDdee) fdEdF�Z]e\dCdDdGeVdee) fdHdI��Z^G dJdD� dD�Z_G dKdL� dLe�Z` d]dMe)dNe eFeIf dOeedP ddfdQdR�Zad^de"dTeVde_fdUdV�Zb d]dMe)d ee- dedD fdWdX�ZcG dYdZ� dZe�ZdG d[d\� d\�ZedS )_zO
The main purpose of this module is to expose LinkCollector.collect_sources().
� N)�
HTMLParser)�Values) �
TYPE_CHECKING�Any�Callable�Dict�Iterable�List�MutableMapping�
NamedTuple�Optional�Sequence�Tuple�Union)�html5lib�requests)�Response)�
RetryError�SSLError)�NetworkConnectionError)�Link)� SearchScope)�
PipSession)�raise_for_status)�
deprecated)�is_archive_file)�pairwise�redact_auth_from_url)�vcs� )�CandidatesFromPage�
LinkSource� build_source)�Protocol�url�returnc C s6 t jD ]}| �� �|�r| t|� dv r| S qdS )zgLook for VCS schemes in the URL.
Returns the matched VCS scheme, or None if there's no match.
z+:N)r �schemes�lower�
startswith�len)r$ �scheme� r+ �?/usr/lib/python3/dist-packages/pip/_internal/index/collector.py�_match_vcs_scheme; s
�r- c s* e Zd Zdededdf� fdd� Z� ZS )�_NotHTML� content_type� request_descr% Nc s t � �||� || _|| _d S �N)�super�__init__r/ r0 )�selfr/ r0 �� __class__r+ r, r3 G s
z_NotHTML.__init__)�__name__�
__module__� __qualname__�strr3 �
__classcell__r+ r+ r5 r, r. F s "r. �responsec C s. | j �dd�}|�� �d�st|| jj��dS )z�Check the Content-Type header to ensure the response contains HTML.
Raises `_NotHTML` if the content type is not text/html.
� Content-Type� � text/htmlN)�headers�getr' r( r. �request�method)r< r/ r+ r+ r, �_ensure_html_headerM s �rD c @ s e Zd ZdS )�_NotHTTPN)r7 r8 r9 r+ r+ r+ r, rE W s rE �sessionc C sF t j�| �\}}}}}|dvrt� �|j| dd�}t|� t|� dS )z�Send a HEAD request to the URL, and ensure the response contains HTML.
Raises `_NotHTTP` if the URL is not available for a HEAD request, or
`_NotHTML` if the content type is not text/html.
> �http�httpsT)�allow_redirectsN)�urllib�parse�urlsplitrE �headr rD )r$ rF r* �netloc�path�query�fragment�respr+ r+ r, �_ensure_html_response[ s rS c C sR t t| �j�r
t| |d� t�dt| �� |j| ddd�d�}t|� t |� |S )a Access an HTML page with GET, and return the response.
This consists of three parts:
1. If the URL looks suspiciously like an archive, send a HEAD first to
check the Content-Type is HTML, to avoid downloading a large file.
Raise `_NotHTTP` if the content type cannot be determined, or
`_NotHTML` if it is not HTML.
2. Actually perform the request. Raise HTTP exceptions on network failures.
3. Check the Content-Type header to make sure we got HTML, and raise
`_NotHTML` otherwise.
�rF zGetting page %sr? z max-age=0)�Acceptz
Cache-Control)r@ )
r r �filenamerS �logger�debugr rA r rD )r$ rF rR r+ r+ r, �_get_html_responsek s
��rY r@ c C s2 | rd| v rt �| d �\}}d|v r|d S dS )z=Determine if we have any encoding information in our headers.r= �charsetN)�cgi� parse_header)r@ r/ �paramsr+ r+ r, �_get_encoding_from_headers� s
r^ �document�page_urlc C s. | � d�D ]}|�d�}|dur| S q|S )a$ Determine the HTML document's base URL.
This looks for a ``<base>`` tag in the HTML document. If present, its href
attribute denotes the base URL of anchor tags in the document. If there is
no such tag (or if it does not have a valid href attribute), the HTML
file's URL is used as the base URL.
:param document: An HTML document representation. The current
implementation expects the result of ``html5lib.parse()``.
:param page_url: The URL of the HTML document.
TODO: Remove when `html5lib` is dropped.
z.//base�hrefN)�findallrA )r_ r` �basera r+ r+ r, �_determine_base_url� s
�rd �partc C � t j�t j�| ��S )zP
Clean a "part" of a URL path (i.e. after splitting on "@" characters).
)rJ rK �quote�unquote�re r+ r+ r, �_clean_url_path_part� s rj c C rf )z�
Clean the first part of a URL path that corresponds to a local
filesystem path (i.e. the first part after splitting on "@" characters).
)rJ rB � pathname2url� url2pathnameri r+ r+ r, �_clean_file_url_path� s
rm z(@|%2F)rO �
is_local_pathc C s^ |rt }nt}t�| �}g }tt�|dg��D ]\}}|�||�� |�|�� � qd� |�S )z*
Clean the path portion of a URL.
r> )
rm rj �_reserved_chars_re�splitr � itertools�chain�append�upper�join)rO rn �
clean_func�parts�
cleaned_parts�to_clean�reservedr+ r+ r, �_clean_url_path� s
r{ c C s6 t j�| �}|j }t|j|d�}t j�|j|d��S )z�
Make sure a link is fully quoted.
For example, if ' ' occurs in the URL, it will be replaced with "%20",
and without double-quoting other characters.
)rn )rO )rJ rK �urlparserN r{ rO �
urlunparse�_replace)r$ �resultrn rO r+ r+ r, � _clean_link� s r� �element_attribs�base_urlc C sL | � d�}|s dS ttj�||��}| � d�}| � d�}t||||d�}|S )zW
Convert an anchor element's attributes in a simple repository page to a Link.
ra Nzdata-requires-pythonz data-yanked)�
comes_from�requires_python�
yanked_reason)rA r� rJ rK �urljoinr )r� r` r� ra r$ � pyrequirer� �linkr+ r+ r, �_create_link_from_element� s
�r� c @ s6 e Zd Zd dd�Zdedefdd �Zdefd
d �ZdS )
�CacheablePageContent�page�HTMLPager% Nc C s |j sJ �|| _d S r1 )�cache_link_parsingr� )r4 r� r+ r+ r, r3 s
z CacheablePageContent.__init__�otherc C s t |t| ��o| jj|jjkS r1 )�
isinstance�typer� r$ )r4 r� r+ r+ r, �__eq__ s zCacheablePageContent.__eq__c C s t | jj�S r1 )�hashr� r$ �r4 r+ r+ r, �__hash__ s z CacheablePageContent.__hash__)r� r� r% N) r7 r8 r9 r3 �object�boolr� �intr� r+ r+ r+ r, r� s
r� c @ s& e Zd Zdddedee fdd�ZdS )�
ParseLinksr� r� �use_deprecated_html5libr% c C s d S r1 r+ )r4 r� r� r+ r+ r, �__call__ s zParseLinks.__call__N)r7 r8 r9 r� r r r� r+ r+ r+ r, r� s ���r� �fnc sX t jdd�dtdtdtt f� fdd� ��t �� �dd dtdtt f� �fd
d � �}|S ) z�
Given a function that parses an Iterable[Link] from an HTMLPage, cache the
function's result (keyed by CacheablePageContent), unless the HTMLPage
`page` has `page.cache_link_parsing == False`.
N)�maxsize�cacheable_pager� r% c s t � | j|��S r1 )�listr� )r� r� )r� r+ r, �wrapper- s z'with_cached_html_pages.<locals>.wrapperr� r� c s" | j r
�t| �|�S t� | |��S r1 )r� r� r� )r� r� �r� r� r+ r, �wrapper_wrapper3 s z/with_cached_html_pages.<locals>.wrapper_wrapper)� functools� lru_cacher� r� r r �wraps)r� r� r+ r� r, �with_cached_html_pages& s
���"r� r� r� c c sZ � t j| j| jdd�}| j}t||�}|�d�D ]}t|j||d�}|du r'q|V qdS )z~
Parse an HTML document, and yield its anchor elements as Link objects.
TODO: Remove when `html5lib` is dropped.
F)�transport_encoding�namespaceHTMLElementsz.//a�r` r� N) r rK �content�encodingr$ rd rb r� �attrib)r� r_ r$ r� �anchorr� r+ r+ r, �_parse_links_html5lib< s$ ��
��r� r� c
c s� � | j pd}|s.d�|�}| jdt|�� }|�|��� dkr.td| j� d�dddd� d }|r9t| �E dH dS t � }|�
| j�|�� | j}|j pL|}|j D ]}t
|||d
�} | du r^qP| V qPdS ) zP
Parse an HTML document, and yield its anchor elements as Link objects.
zutf-8z<!doctype html>Nz The HTML index page being used (z�) is not a proper HTML 5 document. This is in violation of PEP 503 which requires these pages to be well-formed HTML 5 documents. Please reach out to the owners of this index page, and ask them to update this index page to a valid HTML 5 document.z22.2iI* )�reason� replacement�gone_in�issueTr� )r� �encoder� r) �decoder' r r$ r� �HTMLLinkParser�feedr� �anchorsr� )
r� r� r� �expected_doctype� actual_start�parserr$ r� r� r� r+ r+ r, � parse_linksU s< �
�
��r� c @ sD e Zd ZdZ d
dedee dededdf
d d
�Zdefd d �Z dS )r� z'Represents one page, along with its URLTr� r� r$ r� r% Nc C s || _ || _|| _|| _dS )am
:param encoding: the encoding to decode the given content.
:param url: the URL from which the HTML was downloaded.
:param cache_link_parsing: whether links parsed from this page's url
should be cached. PyPI index urls should
have this set to False, for example.
N)r� r� r$ r� )r4 r� r� r$ r� r+ r+ r, r3 � s
zHTMLPage.__init__c C s
t | j�S r1 )r r$ r� r+ r+ r, �__str__� s
zHTMLPage.__str__�T)
r7 r8 r9 �__doc__�bytesr r: r� r3 r� r+ r+ r+ r, r� � s �����
�c s� e Zd ZdZdededdf� fdd� Zdeddfd d
�Zd ed ee ee
e f ddfd
d�Z d ee ee
e f de
e fdd�Z ddd�Z
� ZS )r� zf
HTMLParser that keeps the first base HREF and a list of all anchor
elements' attributes.
�args�kwargsr% Nc s( t � j|i |�� d| _d | _g | _d S )NF)r2 r3 �
_seen_declr� r� )r4 r� r� r5 r+ r, r3 � s
zHTMLLinkParser.__init__�declc C s |� � dkr
| �� d| _d S )Nz doctype htmlT)r' � _raise_errorr� )r4 r� r+ r+ r, � handle_decl� s
zHTMLLinkParser.handle_decl�tag�attrsc C s` | j s| �� |dkr | jd u r | �|�}|d ur || _d S d S |dkr.| j�t|�� d S d S )Nrc �a)r� r� r� �get_hrefr� rs �dict)r4 r� r� ra r+ r+ r, �handle_starttag� s
��z HTMLLinkParser.handle_starttagc C s" |D ] \}}|dkr| S qd S )Nra r+ )r4 r� �name�valuer+ r+ r, r� � s
�zHTMLLinkParser.get_hrefc C s t d��)Nz�HTML doctype missing or incorrect. Expected <!DOCTYPE html>.
If you believe this error to be incorrect, try passing the command line option --use-deprecated=html5lib and please leave a comment on the pip issue at https://github.com/pypa/pip/issues/10825.)�
ValueErrorr� r+ r+ r, r� � s �zHTMLLinkParser._raise_error)r% N)r7 r8 r9 r� r r3 r: r� r r r r� r� r� r; r+ r+ r5 r, r� � s && r� r� r� �meth).Nc C s |d u rt j}|d| |� d S )Nz%Could not fetch URL %s: %s - skipping)rW rX )r� r� r� r+ r+ r, �_handle_get_page_fail� s r� Tr� c C s t | j�}t| j|| j|d�S )N)r� r$ r� )r^ r@ r� r� r$ )r< r� r� r+ r+ r, �_make_html_page� s
�r� c
C s |d u rt d��| j�dd�d }t|�}|r t�d|| � d S tj�|�\}}}}}}|dkrPt j
� tj �
|��rP|�d�sC|d7 }tj�|d�}t�d |� zt||d
�}W n� tyh t�d | � Y d S ty� } zt�d | |j|j� W Y d }~d S d }~w ty� } z t| |� W Y d }~d S d }~w ty� } z t| |� W Y d }~d S d }~w ty� } zd
} | t|�7 } t| | tjd� W Y d }~d S d }~w t j y� } zt| d|� �� W Y d }~d S d }~w t j y� t| d� Y d S w t || j d�S )Nz?_get_html_page() missing 1 required keyword argument: 'session'�#r r zICannot look at %s URL %s because it does not support lookup as web pages.�file�/z
index.htmlz# file: URL is directory, getting %srT z`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.ziSkipping page %s because the %s request got Content-Type: %s.The only supported Content-Type is text/htmlz4There was a problem confirming the ssl certificate: )r� zconnection error: z timed out)r� )!� TypeErrorr$ rp r- rW �warningrJ rK r| �osrO �isdirrB rl �endswithr� rX rY rE r. r0 r/ r r� r r r: �infor �ConnectionError�Timeoutr� r� )
r� rF r$ �
vcs_schemer* �_rO rR �excr� r+ r+ r, �_get_html_page� sv ��
�����
�� �� ���� �r� c @ s. e Zd ZU eee ed< eee ed< dS )�CollectedSources�
find_links�
index_urlsN)r7 r8 r9 r
r r! �__annotations__r+ r+ r+ r, r� s
r� c
@ s� e Zd ZdZdededdfdd�Ze dded ed
e dd fd d ��Z
e de e
fd
d��Zdedee fdd�Zde
dedefdd�ZdS )�
LinkCollectorz�
Responsible for collecting Link objects from all configured locations,
making network requests as needed.
The class's main method is its collect_sources() method.
rF � search_scoper% Nc C s || _ || _d S r1 )r� rF )r4 rF r� r+ r+ r, r3 + s
zLinkCollector.__init__F�options�suppress_no_indexc C s` |j g|j }|jr |s t�dd�dd� |D ��� g }|jp g }tj||d�}t ||d�}|S )z�
:param session: The Session to use to make requests.
:param suppress_no_index: Whether to ignore the --no-index option
when constructing the SearchScope object.
zIgnoring indexes: %s�,c s s � | ]}t |�V qd S r1 )r )�.0r$ r+ r+ r, � <genexpr>C s � z'LinkCollector.create.<locals>.<genexpr>�r� r� )rF r� )
� index_url�extra_index_urls�no_indexrW rX ru r� r �creater� )�clsrF r� r� r� r� r� �link_collectorr+ r+ r, r� 3 s"
�
��zLinkCollector.createc C s | j jS r1 )r� r� r� r+ r+ r, r� T s zLinkCollector.find_links�locationc C s t || jd�S )z>
Fetch an HTML page containing package links.
rT )r� rF )r4 r� r+ r+ r, �
fetch_pageX s zLinkCollector.fetch_page� project_name�candidates_from_pagec s� t �� �fdd��j�|�D ���� }t �� �fdd��jD ���� }t�tj �rIdd� t
� ||�D �}t |�� d|� d�g| }t�
d�|�� tt|�t|�d �S )
Nc 3 �& � | ]}t |� �jjd d d�V qdS )F�r� �page_validator�
expand_dirr� N�r" rF �is_secure_origin�r� �loc�r� r4 r+ r, r� d � � ��
�z0LinkCollector.collect_sources.<locals>.<genexpr>c 3 r� )Tr� Nr� r� r� r+ r, r� n r� c S s* g | ]}|d ur|j d urd|j � ��qS )Nz* )r� )r� �sr+ r+ r, �
<listcomp>z s
�z1LinkCollector.collect_sources.<locals>.<listcomp>z' location(s) to search for versions of �:�
r� )� collections� OrderedDictr� �get_index_urls_locations�valuesr� rW � isEnabledFor�logging�DEBUGrq rr r) rX ru r� r� )r4 r� r� �index_url_sources�find_links_sources�linesr+ r� r, �collect_sources^ s2
� �
� �
�
����z LinkCollector.collect_sources)F)r7 r8 r9 r� r r r3 � classmethodr r� r� �propertyr r: r� r r r� r� r r� r r+ r+ r+ r, r� " s< ��
����� � ���r� r1 r� )fr� r[ r r� rq r r� �re� urllib.parserJ �urllib.request�xml.etree.ElementTree�xml� html.parserr �optparser �typingr r r r r r r
r r r
r r � pip._vendorr r �pip._vendor.requestsr � pip._vendor.requests.exceptionsr r �pip._internal.exceptionsr �pip._internal.models.linkr �!pip._internal.models.search_scoper � pip._internal.network.sessionr �pip._internal.network.utilsr � pip._internal.utils.deprecationr � pip._internal.utils.filetypesr �pip._internal.utils.miscr r �pip._internal.vcsr �sourcesr r! r" r# r� � getLoggerr7 rW �etree� ElementTree�Element� HTMLElementr: �ResponseHeadersr- � Exceptionr. rD rE rS rY r^ rd rj rm �compile�
IGNORECASEro r� r{ r� r� r� r� r� r� r� r� r� r� r� r� r� r� r+ r+ r+ r, �<module> s� 8
2 ���
� /.��
�
�
�
���
�?