o

���g{:�@
s�dZd
dl
mZ
d
dlZd
dlZd
dlmZmZmZm	Z	m
Z
mZmZ
d
dl
mZ
d
dl
mZ
d
dl
mZ
d
dl
mZ
erHd
d	lmZ
d
d
lmZ
Gdd�d�Ze�ZGd
d�d�ZGdd�d�Ze�Zdg
ZdS)a�Make approximate assertions as "expectations" on test results.

This module is designed to be used within test cases decorated with the
`@pytest.mark.decorator` decorator
It allows you to log scores about a test case and optionally make assertions that log as
"expectation" feedback to LangSmith.

Example usage:
    .. code-block:: python

        import pytest
        from langsmith import expect


        @pytest.mark.langsmith
        def test_output_semantically_close():
            response = oai_client.chat.completions.create(
                model="gpt-3.5-turbo",
                messages=[
                    {"role": "system", "content": "You are a helpful assistant."},
                    {"role": "user", "content": "Say hello!"},
                ],
            )
            response_txt = response.choices[0].message.content
            # Intended usage
            expect.embedding_distance(
                prediction=response_txt,
                reference="Hello!",
            ).to_be_less_than(0.9)

            # Score the test case
            matcher = expect.edit_distance(
                prediction=response_txt,
                reference="Hello!",
            )
            # Apply an assertion and log 'expectation' feedback to LangSmith
            matcher.to_be_less_than(1)

            # You can also directly make assertions on values directly
            expect.value(response_txt).to_contain("Hello!")
            # Or using a custom check
            expect.value(response_txt).against(lambda x: "Hello" in x)

            # You can even use this for basic metric logging within tests

            expect.score(0.8)
            expect.score(0.7, key="similarity").to_be_greater_than(0.7)
�)
�annotationsN)�
TYPE_CHECKING�Any�Callable�Literal�Optional�Union�overload�
�client)
�run_helpers)
�	run_trees)
�utils)
�EditDistanceConfig)
�EmbeddingConfigc@
s$eZ
dZd
Zd
dd�Zddd�Zd	S)�_NULL_SENTRYz�A sentinel singleton class used to distinguish omitted keyword arguments
    from those passed in with the value None (which may have different behavior).
    �return�Literal[False]c


C
�d
S)NF��
�selfrr�</usr/local/lib/python3.10/dist-packages/langsmith/_expect.py�__bool__P�
z_NULL_SENTRY.__bool__�strc


C
r)N�	NOT_GIVENrrrrr�__repr__Srz_NULL_SENTRY.__repr__N)rr)rr)�__name__�
__module__�__qualname__�__doc__rrrrrrrKs

rc@
s�eZ
dZd
Z		d2d3d
d�Zd4d5dd�Zd6dd�Zd7dd�Zd7dd�Zd8d"d#�Z	d9d:d&d'�Z
d7d(d)�Zd;d*d+�Zd<d,d-�Z
d=d0d1�ZdS)>�_Matcherz4A class for making assertions on expectation values.Nr�Optional[ls_client.Client]�keyr�valuer�	_executor�,Optional[ls_utils.ContextThreadPoolExecutor]�run_id�
Optional[str]cC
sF|
|_||_
||_|ptjd
d�
|_t��}|r|j|_	dS||_	dS)N��
�max_workers)
�_clientr$r%�ls_utils�ContextThreadPoolExecutorr&�rh�get_current_run_tree�trace_id�_run_id)rrr$r%r&r(�rtrrr�__init__]s




z_Matcher.__init__�score�int�messager�NonecC
s<t�
�s|jst��|_|jj|jj|jd
|
|d�
dSdS)N�expectation)r(r$r6�comment)	r.�test_tracking_is_disabledr-r4�get_cached_clientr&�submit�create_feedbackr3)rr6r8rrr�_submit_feedbackls










��z_Matcher._submit_feedback�	condition�bool�method_namec
C
s^z|
sJ|�
�
|jd
d|j
�d|��d�
WdSty.
}
z|�dt|�
�
|d�d}~w
w)N�
z	Success: �
.)
r8r
)r@r$�AssertionError�repr)rrAr8rC�
errr�_assertxs

"



��z_Matcher._assert�floatc	C
s.|�|j
|
kd
|j�d|
�d|j
��d�
dS)z�Assert that the expectation value is less than the given value.

        Args:
            value: The value to compare against.

        Raises:
            AssertionError: If the expectation value is not less than the given value.
        �	Expected z to be less than �
, but got �to_be_less_thanN�rIr%r$�rr%rrrrM�s
	


�z_Matcher.to_be_less_thanc	C
s.|�|j
|
kd
|j�d|
�d|j
��d�
dS)a
Assert that the expectation value is greater than the given value.

        Args:
            value: The value to compare against.

        Raises:
            AssertionError: If the expectation value is not
            greater than the given value.
        rKz to be greater than rL�to_be_greater_thanNrNrOrrrrP��




�z_Matcher.to_be_greater_than�	min_value�	max_valuecC
sD|�|
|j
ko|kn
d
|j�d|
�d|�d|j
��d�
dS)aJ
Assert that the expectation value is between the given min and max values.

        Args:
            min_value: The minimum value (exclusive).
            max_value: The maximum value (exclusive).

        Raises:
            AssertionError: If the expectation value
                is not between the given min and max.
        rKz to be between z and rL�
to_be_betweenNrN)rrRrSrrrrT�s


��z_Matcher.to_be_between��	precisionc	C
s:|�t
|j|�t
|
|�kd
|j�d|
�d|j��d�
dS)ak
Assert that the expectation value is approximately equal to the given value.

        Args:
            value: The value to compare against.
            precision: The number of decimal places to round to for comparison.

        Raises:
            AssertionError: If the rounded expectation value
                does not equal the rounded given value.
        rKz to be approximately rL�to_be_approximatelyN)rI�roundr%r$)rr%rVrrrrW�s



�z_Matcher.to_be_approximatelyc	C
s.|�|j
|
kd
|j�d|
�d|j
��d�
dS)a
Assert that the expectation value equals the given value.

        Args:
            value: The value to compare against.

        Raises:
            AssertionError: If the expectation value does
                not exactly equal the given value.
        rKz to be equal to rL�to_equalNrNrOrrrrY�rQz_Matcher.to_equalc

C
s(|�|j
d
ud|j�d|j
��d�
d
S)z�Assert that the expectation value is None.

        Raises:
            AssertionError: If the expectation value is not None.
        NrKz to be None, but got �
to_be_nonerNrrrrrZ�s



�z_Matcher.to_be_nonecC
s(|�|
|j
vd
|j�d|
�d�d�
dS)z�Assert that the expectation value contains the given value.

        Args:
            value: The value to check for containment.

        Raises:
            AssertionError: If the expectation value does not contain the given value.
        rKz to contain z, but it does not�
to_containNrNrOrrrr[�s
	


�z_Matcher.to_contain�funcrcC
s0t�
|
�
}|�|
|j�
d
|�d|j��d�
dS)z�Assert the expectation value against a custom function.

        Args:
            func: A custom function that takes the expectation value as input.

        Raises:
            AssertionError: If the custom function returns False.
        z
Assertion z failed for �againstN)�inspect�	signaturerIr%r$)rr\�func_signaturerrrr]�s
	



�z_Matcher.against)NN)
rr#r$rr%rr&r'r(r)�
N)r6r7r8r)rr9)rArBr8rrCrrr9)r%rJrr9)rRrJrSrJrr9)
rU)r%rJrVr7rr9)rr9)r%rrr9)r\rrr9)rrr r!r5r@rIrMrPrTrWrYrZr[r]rrrrr"Zs

�






r"c@
s�eZ
dZd
Zdd�
d.dd�Zdd�
d/dd�Zdd�
d0dd�Zd1dd�Zdddd�d2d!d"�Ze	d1d#d$��
Z
e	d3d&d$��
Z
edfd4d)d$�Z
d5d,d-�ZdS)6�_Expectz1A class for setting expectations on test results.Nr
rr#c

C
s*|
|_t
jd
d�
|_tj|jjdd�
dS)Nr*r+T)
�wait)r-r.r/�executor�atexit�register�shutdown�rrrrrr5
s


z_Expect.__init__�
�config�
predictionr�	referencerj�Optional[EmbeddingConfig]rr"c
		C
s�d
dlm
}
|p	i}|�d�
rdn
d}||d�
}|j|
|d�}||jd�}|�d	||d
|�d|j��d��
t|jd	||jd
�S)aCompute the embedding distance between the prediction and reference.

        This logs the embedding distance to LangSmith and returns a `_Matcher` instance
        for making assertions on the distance value.

        By default, this uses the OpenAI API for computing embeddings.

        Args:
            prediction: The predicted string to compare.
            reference: The reference string to compare against.
            config: Optional configuration for the embedding distance evaluator.
                Supported options:
                - `encoder`: A custom encoder function to encode the list of input
                     strings to embeddings. Defaults to the OpenAI API.
                - `metric`: The distance metric to use for comparison.
                    Supported values: "cosine", "euclidean", "manhattan",
                    "chebyshev", "hamming".

        Returns:
            A `_Matcher` instance for the embedding distance value.


        Examples:
            >>> expect.embedding_distance(
            ...     prediction="hello",
            ...     reference="hi",
            ... ).to_be_less_than(1.0)
        r
)
�EmbeddingDistance�encoder�custom�openairi�rkrl)ro�metric�embedding_distance�Using z
, Metric: �r6�source_infor;�
r&)	�'langsmith._internal._embedding_distancern�get�evaluate�distancer@r"r-rd)	rrkrlrjrn�encoder_func�	evaluatorr6�src_inforrrrt
s #








��
�z_Expect.embedding_distance�Optional[EditDistanceConfig]c

	C
s�d
dlm
}
|p	i}|�d�
pd}|�dd�}||d�
}|j|
|d�}||d	�}	|�d
||	d|�d|��d
��
t|jd
||jd�S)a
Compute the string distance between the prediction and reference.

        This logs the string distance (Damerau-Levenshtein) to LangSmith and returns
        a `_Matcher` instance for making assertions on the distance value.

        This depends on the `rapidfuzz` package for string distance computation.

        Args:
            prediction: The predicted string to compare.
            reference: The reference string to compare against.
            config: Optional configuration for the string distance evaluator.
                Supported options:
                - `metric`: The distance metric to use for comparison.
                    Supported values: "damerau_levenshtein", "levenshtein",
                    "jaro", "jaro_winkler", "hamming", "indel".
                - `normalize_score`: Whether to normalize the score between 0 and 1.

        Returns:
            A `_Matcher` instance for the string distance value.

        Examples:
            >>> expect.edit_distance("hello", "helo").to_be_less_than(1)
        r
)
�EditDistancers�damerau_levenshtein�normalize_scoreTrirr)rs�	normalize�
edit_distanceruz
, Normalize: rvrx)�"langsmith._internal._edit_distancer�rzr{r@r"r-rd)
rrkrlrjr�rsr�r~r6rrrrr�=
s(










��



�z_Expect.edit_distancer%rcC
st|j
d
|
|jd�S)a"
Create a `_Matcher` instance for making assertions on the given value.

        Args:
            value: The value to make assertions on.

        Returns:
            A `_Matcher` instance for the given value.

        Examples:
           >>> expect.value(10).to_be_less_than(20)
        r%rx)r"r-rdrOrrrr%r
sz
_Expect.valuer6)r$�
source_run_idr;�Union[float, int, bool]r$r��Optional[ls_client.ID_TYPE]r;r)cC
s.|�||
d
di
||d��
t
|j||
|jd�S)a
Log a numeric score to LangSmith.

        Args:
            score: The score value to log.
            key: The key to use for logging the score. Defaults to "score".

        Examples:
            >>> expect.score(0.8)  # doctest: +ELLIPSIS
            <langsmith._expect._Matcher object at ...>

            >>> expect.score(0.8, key="similarity").to_be_greater_than(0.7)
        �methodzexpect.score)r6rwr�r;rx)r@r"r-rd)rr6r$r�r;rrrr6�
s



��	z
_Expect.scorec
C
�dSrarrOrrr�__call__�
rz_Expect.__call__�ls_client.Clientc



C
r�rarrhrrrr��
r�
Optional[Any]�Union[_Expect, _Matcher]cC
s t|d
�
}|
t
u
r|�|
�
S|S)Nr
)rbrr%)rr%r�expectedrrrr��
s




�results�dictcC
sTt�
�}|r	|jn
d}t��s(|jst��|_|jj	|jj
f
||
d
�|�
�

dSdS)N)r(r$)r0r1r2r.r<r-r4r=rdr>r?)rr$r��current_runr(rrrr@�
s







�
�

��z_Expect._submit_feedback)rr#)rkrrlrrjrmrr")rkrrlrrjr�rr")r%rrr")
r6r�r$rr�r�r;r)rr")rr�rrb)rr#r%r�rr�)r$rr�r�)
rrr r!r5rtr�r%r6r	r�rr@rrrrrb�s(

�;�
5

�!

�rb�expect) r!�
__future__rrer^�typingrrrrrrr	�	langsmithr�	ls_clientrr0r
r4rr.r�rryrrrr"rbr��__all__rrrr�<module>
s(1
$






&?