HEX
Server: Apache/2.4.52 (Ubuntu)
System: Linux spn-python 5.15.0-89-generic #99-Ubuntu SMP Mon Oct 30 20:42:41 UTC 2023 x86_64
User: arjun (1000)
PHP: 8.1.2-1ubuntu2.20
Disabled: NONE
$ pwd: /usr/local/lib/python3.10/dist-packages/langsmith/testing/__pycache__/
Upload Files
File: //usr/local/lib/python3.10/dist-packages/langsmith/testing/__pycache__/_internal.cpython-310.pyc
o

���gp��@s�ddlmZddlZddlZddlZddlZddlZddlZddlZddl	Z	ddl
Z
ddlZddlZddl
Z
ddlZddlmZddlmZddlmZmZmZmZmZmZmZmZmZmZddlmZddl m!Z"ddl m#Z$dd	l m%Z&dd
l m'Z'dd
l m'Z(ddl m)Z*ddl m+Z,dd
l-m.Z.ddl/m0Z0ddl1m2Z2z
ddl3Z3e3j4j5Z6Wne7y�Gdd�de5�Z6Ynwe	�8e9�Z:ed�Z;ed�Z<edxdd��Z=eddddd�dyd#d��Z=dzd'd�Z=d{d)d*�Z>d|d+d,�Z?d}d/d0�Z@d~d3d4�ZAdd:d;�ZBd�d=d>�ZCed?eeDd@�ZEd�dBdC�ZFGdDd<�d<�ZGGdEdF�dF�ZHejIeeHdGddH�ZJGdIdJ�dJedKdL�ZKd�dOdP�ZLd�dSdT�ZMd�dXdY�ZNd�dZd[�ZOe=ZPd�d]d^�ZQd�d`da�ZRd�dcdd�ZS	d�ddde�d�dmdn�ZTejUdodp�d�dsdt��ZVd�dvdw�ZWdS)��)�annotationsN)�Future)�Path)
�Any�Callable�	Generator�Optional�Sequence�Tuple�TypeVar�Union�cast�overload)�	TypedDict)�client)�env)�run_helpers)�	run_trees)�schemas)�utils)�_orjson)�
dumps_json)�ID_TYPEc@seZdZdS)�
SkipExceptionN)�__name__�
__module__�__qualname__�rr�F/usr/local/lib/python3.10/dist-packages/langsmith/testing/_internal.pyr2sr�T�U�funcr�returncC�dS�Nr�r!rrr�test=sr&��id�output_keysr�test_suite_namer(�Optional[uuid.UUID]r)�Optional[Sequence[str]]r�Optional[ls_client.Client]r*�
Optional[str]�Callable[[Callable], Callable]cCr#r$rr'rrrr&Cs�argsr�kwargscs�t|�dd�|�dd�|�dd�|�dd�t�|�dd��d��|r,t�d|�����t����r7t�	d	�d��fd
d�}|rMt
|d�rM||d�S|S)a�Trace a pytest test case in LangSmith.

    This decorator is used to trace a pytest test to LangSmith. It ensures
    that the necessary example data is created and associated with the test function.
    The decorated function will be executed as a test case, and the results will be
    recorded and reported by LangSmith.

    Args:
        - id (Optional[uuid.UUID]): A unique identifier for the test case. If not
            provided, an ID will be generated based on the test function's module
            and name.
        - output_keys (Optional[Sequence[str]]): A list of keys to be considered as
            the output keys for the test case. These keys will be extracted from the
            test function's inputs and stored as the expected outputs.
        - client (Optional[ls_client.Client]): An instance of the LangSmith client
            to be used for communication with the LangSmith service. If not provided,
            a default client will be used.
        - test_suite_name (Optional[str]): The name of the test suite to which the
            test case belongs. If not provided, the test suite name will be determined
            based on the environment or the package name.

    Returns:
        Callable: The decorated test function.

    Environment:
        - LANGSMITH_TEST_CACHE: If set, API calls will be cached to disk to
            save time and costs during testing. Recommended to commit the
            cache files to your repository for faster CI/CD runs.
            Requires the 'langsmith[vcr]' package to be installed.
        - LANGSMITH_TEST_TRACKING: Set this variable to the path of a directory
            to enable caching of test results. This is useful for re-running tests
             without re-executing the code. Requires the 'langsmith[vcr]' package.

    Example:
        For basic usage, simply decorate a test function with `@pytest.mark.langsmith`.
        Under the hood this will call the `test` method:

        .. code-block:: python

            import pytest


            # Equivalently can decorate with `test` directly:
            # from langsmith import test
            # @test
            @pytest.mark.langsmith
            def test_addition():
                assert 3 + 4 == 7


        Any code that is traced (such as those traced using `@traceable`
        or `wrap_*` functions) will be traced within the test case for
        improved visibility and debugging.

        .. code-block:: python

            import pytest
            from langsmith import traceable


            @traceable
            def generate_numbers():
                return 3, 4


            @pytest.mark.langsmith
            def test_nested():
                # Traced code will be included in the test case
                a, b = generate_numbers()
                assert a + b == 7

        LLM calls are expensive! Cache requests by setting
        `LANGSMITH_TEST_CACHE=path/to/cache`. Check in these files to speed up
        CI/CD pipelines, so your results only change when your prompt or requested
        model changes.

        Note that this will require that you install langsmith with the `vcr` extra:

        `pip install -U "langsmith[vcr]"`

        Caching is faster if you install libyaml. See
        https://vcrpy.readthedocs.io/en/latest/installation.html#speed for more details.

        .. code-block:: python

            # os.environ["LANGSMITH_TEST_CACHE"] = "tests/cassettes"
            import openai
            import pytest
            from langsmith import wrappers

            oai_client = wrappers.wrap_openai(openai.Client())


            @pytest.mark.langsmith
            def test_openai_says_hello():
                # Traced code will be included in the test case
                response = oai_client.chat.completions.create(
                    model="gpt-3.5-turbo",
                    messages=[
                        {"role": "system", "content": "You are a helpful assistant."},
                        {"role": "user", "content": "Say hello!"},
                    ],
                )
                assert "hello" in response.choices[0].message.content.lower()

        LLMs are stochastic. Naive assertions are flakey. You can use langsmith's
        `expect` to score and make approximate assertions on your results.

        .. code-block:: python

            import pytest
            from langsmith import expect


            @pytest.mark.langsmith
            def test_output_semantically_close():
                response = oai_client.chat.completions.create(
                    model="gpt-3.5-turbo",
                    messages=[
                        {"role": "system", "content": "You are a helpful assistant."},
                        {"role": "user", "content": "Say hello!"},
                    ],
                )
                # The embedding_distance call logs the embedding distance to LangSmith
                expect.embedding_distance(
                    prediction=response.choices[0].message.content,
                    reference="Hello!",
                    # The following optional assertion logs a
                    # pass/fail score to LangSmith
                    # and raises an AssertionError if the assertion fails.
                ).to_be_less_than(1.0)
                # Compute damerau_levenshtein distance
                expect.edit_distance(
                    prediction=response.choices[0].message.content,
                    reference="Hello!",
                    # And then log a pass/fail score to LangSmith
                ).to_be_less_than(1.0)

        The `@test` decorator works natively with pytest fixtures.
        The values will populate the "inputs" of the corresponding example in LangSmith.

        .. code-block:: python

            import pytest


            @pytest.fixture
            def some_input():
                return "Some input"


            @pytest.mark.langsmith
            def test_with_fixture(some_input: str):
                assert "input" in some_input

        You can still use pytest.parametrize() as usual to run multiple test cases
        using the same test function.

        .. code-block:: python

            import pytest


            @pytest.mark.langsmith(output_keys=["expected"])
            @pytest.mark.parametrize(
                "a, b, expected",
                [
                    (1, 2, 3),
                    (3, 4, 7),
                ],
            )
            def test_addition_with_multiple_inputs(a: int, b: int, expected: int):
                assert a + b == expected

        By default, each test case will be assigned a consistent, unique identifier
        based on the function name and module. You can also provide a custom identifier
        using the `id` argument:

        .. code-block:: python

            import pytest
            import uuid

            example_id = uuid.uuid4()


            @pytest.mark.langsmith(id=str(example_id))
            def test_multiplication():
                assert 3 * 4 == 12

        By default, all test inputs are saved as "inputs" to a dataset.
        You can specify the `output_keys` argument to persist those keys
        within the dataset's "outputs" fields.

        .. code-block:: python

            import pytest


            @pytest.fixture
            def expected_output():
                return "input"


            @pytest.mark.langsmith(output_keys=["expected_output"])
            def test_with_expected_output(some_input: str, expected_output: str):
                assert expected_output in some_input


        To run these tests, use the pytest CLI. Or directly run the test functions.

        .. code-block:: python

            test_output_semantically_close()
            test_addition()
            test_nested()
            test_with_fixture("Some input")
            test_with_expected_output("Some input", "Some")
            test_multiplication()
            test_openai_says_hello()
            test_addition_with_multiple_inputs(1, 2, 3)
    r(Nr)rr*�cache)r(r)rr*r2zUnexpected keyword arguments: zLLANGSMITH_TEST_TRACKING is set to 'false'. Skipping LangSmith test tracking.r!rr"csVt���rt���dd�d
���fdd��}|St���dd�d
���fdd	��}|S)N)�request�	test_argsrr3�test_kwargsc�sF��r
�|i|��IdHSt�g|�Rd|i|�d�i��IdHdS�N�pytest_request�langtest_extra)�
_arun_test�r3r4r5��disable_trackingr!r8rr�
async_wrapper?s������z.test.<locals>.decorator.<locals>.async_wrappercs8�r	�|i|��St�g|�Rd|i|�d�i��dSr6)�	_run_testr:r;rr�wrapperOs�����z(test.<locals>.decorator.<locals>.wrapper)r4rr3rr5r)�inspect�iscoroutinefunction�	functools�wraps)r!r=r?�r<r8r%r�	decorator<s
�
ztest.<locals>.decoratorr�r!rr"r)�_UTExtra�pop�ls_utils�
get_cache_dir�warnings�warn�keys�test_tracking_is_disabled�logger�info�callable)r0r1rErrDrr&Ms&`



��!�strcCs�tj�d�r"tj�d�r"|tjd}tt�tj	|�j
dd��}ntt��j
dd��}tj�d�r9tjd}nt�
d�p?d}|�d|��}|S)N�PYTEST_XDIST_TESTRUNUID�xdist��LANGSMITH_EXPERIMENTF�TestSuiteResult�:)�os�environ�get�	importlib�util�	find_specrR�uuid�uuid5�
NAMESPACE_DNS�hex�uuid4rI�get_tracer_project)r*�id_name�id_�prefix�namerrr�_get_experiment_namefsricCslt�d�}|r	|St��d}zt�|�}|r |�d|j��WSWt
d��ty5t�	d�Yt
d��w)N�
TEST_SUITE�	repo_name�.z3Could not determine test suite name from file path.z9Please set the LANGSMITH_TEST_SUITE environment variable.)rI�get_env_var�ls_env�get_git_infor@�	getmoduler�
BaseExceptionrO�debug�
ValueError)r!r*rk�modrrr�_get_test_suite_namexs

���ru�ls_client.Client�ls_schemas.DatasetcCsx|j|d�r|j|d�St���d�pd}d}|r |d|��7}z|j||ddid�WStjy;|j|d�YSw)	N)�dataset_name�
remote_url�z
Test suitez for �__ls_runner�pytest)rx�description�metadata)�has_dataset�read_datasetrnror[�create_datasetrI�LangSmithConflictError)rr*�repor}rrr�_get_test_suite�s��r��
test_suite�ls_schemas.TracerSessioncCsRt|j�}z|j||jdt���d�dd�d�WStjy(|j	|d�YSw)NzTest Suite Results.�revision_idr|)r�r{)�reference_dataset_idr}r~)�project_name)
rirh�create_projectr(rn�get_langchain_env_var_metadatar[rIr��read_project)rr��experiment_namerrr�_start_experiment�s
����r��inputs�Optional[dict]�suite_id�	uuid.UUID�Tuple[uuid.UUID, str]cCs�zttt�|���t����}Wnty|j}Ynw|�|�d|j��}t	|d�r<t
dd�|jD��r<|t|�7}t
�t
j|�|tt|��d�fS)Nz::�
pytestmarkcss�|]}|jdkVqdS)�parametrizeN�rh)�.0�mrrr�	<genexpr>�s�

�z"_get_example_id.<locals>.<genexpr>)rRrr@�getfile�relative_to�cwdrsrr�hasattr�anyr��
_stringifyr_r`ra�len)r!r�r��	file_path�
identifierrrr�_get_example_id�s 
��"r��_LangSmithTestSuitecCs�t��pi}|��|��}|jj}|jj|ji|�|t�	��
d�dd��d�|r>|ddur>|jj||d|d��d�|rV|ddurX|jj||d	|d��d�dSdSdS)
Nr�r|)�dataset_versionr�r{)r~�commitzgit:commit:)�
dataset_id�as_of�tag�branchzgit:branch:)rnro�shutdown�get_dataset_version�_datasetr(r�update_project�
experiment_idr�r[�update_dataset_tag)r��git_infor�r�rrr�
_end_tests�s6���	�
��r��VT)�bound�valuescCs |dur|St�|�}t�|�Sr$)�	ls_client�_dumps_jsonr�loads)r��btsrrr�_serde_example_values�s

r�c@s�eZdZUdZded<e��ZdDd
d�Ze	dd
��Z
e	dd��Ze	dd��Ze
	dEdFdd��Ze	dd��Zdd�Z				dGdHd'd(�ZdId+d,�Zdddddd-�dJd2d3�Z		dKdLd8d9�ZdMd;d<�Zd=d>�Z		dKdNd@dA�ZdOdBdC�ZdS)Pr�Nr��
_instancesrr-�
experimentr��datasetrwcCs<|pt��|_||_||_|j|_t��|_	t
�t|�dSr$)
�rt�get_cached_clientr�_experimentr��modified_at�_dataset_versionrI�ContextThreadPoolExecutor�	_executor�atexit�registerr�)�selfrr�r�rrr�__init__�s
z_LangSmithTestSuite.__init__cC�|jjSr$)r�r(�r�rrrr(��z_LangSmithTestSuite.idcCr�r$)r�r(r�rrrr��r�z!_LangSmithTestSuite.experiment_idcC�|jSr$)r�r�rrrr�sz_LangSmithTestSuite.experimentr!rr*r.r"cCs�|pt��}|pt|�}|j�&|jsi|_||jvr.t||�}t||�}||||�|j|<Wd�n1s8wY|j|Sr$)r�r�ru�_lockr�r�r�)�clsrr!r*r�r�rrr�	from_tests


��
z_LangSmithTestSuite.from_testcCr�r$)r�rhr�rrrrhr�z_LangSmithTestSuite.namecCr�r$)r�r�rrrr�sz'_LangSmithTestSuite.get_dataset_versionF�run_idr��error�skipped�bool�
pytest_pluginr�
pytest_nodeid�NonecCsR|rd}d}n|rd}d}nd}d}|r|r|�|d|i�|j�|j||�dS)Nr�r�failed��passed�status)�update_process_statusr��submit�_submit_result)r�r�r�r�r�r��scorer�rrr�
submit_resultsz!_LangSmithTestSuite.submit_resultr��
Optional[int]cCs|jj|d|d�dS)N�pass)�keyr��r�create_feedback)r�r�r�rrrr�2sz"_LangSmithTestSuite._submit_result)r��outputsr~r�r��
example_idr�r�r~c	CsP|r|r||d�}dd�|��D�}|�||�|r|��n|}t|�}t|�}z	|jj|d�}WntjyJ|jj||||j	||j
jd�}Yn;w|durT||jksp|dur]||j
ksp|durf||jkspt|j�t|j	�kr�|jj|j	||||j	d�|jj|j	d�}|jdur�|j|_dS|jr�|jr�|j|jkr�|j|_dSdSdSdS)N)r��reference_outputscSsi|]\}}|dur||�qSr$r)r��k�vrrr�
<dictcomp>Asz4_LangSmithTestSuite.sync_example.<locals>.<dictcomp>)r�)r�r�r�r�r~�
created_at)r�r�r�r~r�)�itemsr��copyr�r�read_examplerI�LangSmithNotFoundError�create_exampler(r��
start_timer�r�r~rRr��update_exampler�r�)	r�r�r�r�r~r�r��update�examplerrr�sync_example5sR


���
���z _LangSmithTestSuite.sync_exampler�feedback�Union[dict, list]r1cKsvt|t�r|n|g}|D],}|r*|r*d|vr|dn|d}|�|d|d|ii�|jj|jf||d�|��qdS)Nr��valuer�r�)r�r�)�
isinstance�listr�r�r��_create_feedback)r�r�r�r�r�r1�fb�valrrr�_submit_feedbackis�����z$_LangSmithTestSuite._submit_feedback�dictcKs|jj|fi|�|��dSr$r�)r�r�r�r1rrrr�|sz$_LangSmithTestSuite._create_feedbackcCs|j��dSr$)r�r�r�rrrr�sz_LangSmithTestSuite.shutdownrc	Cs|jj|j||||||d�S)N)�run_treer�r�r�r�r�)r�r��_end_run�r�rr�r�r�r�r�rrr�end_run�s	�z_LangSmithTestSuite.end_runcCs*|j||j|d�|j|d�|��dS)N�r�r�)r�)r�r��end�patchrrrrr�sz_LangSmithTestSuite._end_run)rr-r�r�r�rwr$)rr-r!rr*r.r"r�)NFNN)r�r�r�r.r�r�r�rr�rr"r�)r�r�r�r�r"r�)
r�r�r�r�r�r�r~r�r"r�)NN)
r�rr�r�r�rr�rr1r)r�rr�r�r1rr"r�)r"r�r"r�)rrrr��__annotations__�	threading�RLockr�r��propertyr(r�r��classmethodr�rhr�r�r�r�r�r�r�rrrrrrr��sJ




�
�
�8�
	�c@s�eZdZ				d,d-dd�Zddd�d.dd�Zd/dd�Zd0dd�Zd1dd�Zd2dd�Z		d3d4d$d%�Z	d5d&d'�Z
d5d(d)�Zd6d*d+�ZdS)7�	_TestCaseNr�r�r�r�r�r�rr�r�r�r�r"r�cCst||_||_||_||_||_d|_||_||_|r4|r6|�|j	j
|�|r+|�|�|r8|�|�dSdSdSdSr$)
r�r�r�r�r��_logged_reference_outputsr�r��add_process_to_test_suiter�rh�
log_inputs�log_reference_outputs)r�r�r�r�r�r�r�r�rrrr��s$
�
�z_TestCase.__init__rr�cCs |jj|j|||j|jd�dS)N)r�r�r�r�)r�r�r�r�r�)r�r�r�rrrr��s
�z_TestCase.sync_exampler1cOs*|jj|ii|�t|j|jd����dS)N)r�r�)r�r�r�r�r�)r�r0r1rrr�submit_feedback�s����
�z_TestCase.submit_feedbackr�cC�,|jr|jr|j�|jd|i�dSdSdS)Nr��r�r�r�)r�r�rrrr��

��z_TestCase.log_inputscCr)Nr�r)r�r�rrr�log_outputs�rz_TestCase.log_outputscCs2||_|jr|jr|j�|jd|i�dSdSdS)Nr�)rr�r�r�)r�r�rrrr�s
��z_TestCase.log_reference_outputsFr�r.r�r�cCs|jj|j|||j|jd�S)N)r�r�r�r�)r�r�r�r�r�)r�r�r�rrr�submit_test_result�s�z_TestCase.submit_test_resultcC�0|jr|jr|j�|jdt��i�dSdSdS)Nr��r�r�r��timer�rrrr���
��z_TestCase.start_timecCr)N�end_timerr�rrrr�rz_TestCase.end_timecCs>|dus
t|t�s
d|i}|jj||j||j|j|jd�dS)N�output)r�r�r�)r�r�r�rr�rr�r�)r�rr�rrrrs
�z_TestCase.end_run)NNNN)r�r�r�r�r�r�r�rr�rr�r�r�r�r"r�)r�r�r�r�r"r�)r1r�r�r�r"r��r�r�r"r��r�r�r"r�)NF)r�r.r�r�r"r�r)r�rr"r�)
rrrr�r�rrrrrr�rrrrrrr
�s$��



	�

r
�
_TEST_CASE)�defaultc@s6eZdZUded<ded<ded<ded<ded	<d
S)rGr-rr+r(r,r)r.r*r2N)rrrrrrrrrGs
rGF)�total�sig�inspect.SignaturecCsBt|dd�pd}t|dd�pd}|rd|����}|�|�|��S)Nrrz�__doc__z - )�getattr�strip)r!r$rhr}rrr�_get_test_reprs
r)r7r8c	Os|dpt��}|d}t�|�}tj|g|�Ri|��pd}d}	|r:i}	|s-d}
t|
��|D]
}|�|d�|	|<q/t�	|||�
d��}t|||j�\}
}|dpR|
}
|r\|j
j�d�nd}|rd|jjnd}|r{tt|jj�dt|j�|j|jj<t||
t��||	||d�}|S)	Nrr)zU'output_keys' should only be specified when marked test function has input arguments.r*r(�langsmith_output_pluginz/compare?selectedSessions=)r�r�r�r�r�)r�r�r@�	signature�rh�_get_inputs_safersrHr�r�r[r�r(�config�
pluginmanager�
get_plugin�node�nodeidr
rRr��urlr��test_suite_urlsrhr
r_rc)r!r7r8r0r1rr)r+r�r��msgr�r�r��example_namer�r��	test_caserrr�_create_test_case%sP
��������	r8r4r5r�c		s t�g��Ri��||d����t�������fdd�}|dr0t|d��jj�d�}nd}t��}i|dp<i��jjj	t
�j�d��}tjd	ii|�d|i����/t
j|�jjjgd��|�Wd�n1sqwYWd�dSWd�dS1s�wYdS)
N�r7r8csR���tjt�dd��j�j�j�jjt	fdd��`}zRz	��i���}Wn<t	yF}z�j
t|�dd���|dt|�i�|�d}~wt
ya}z�j
t|�d���|d�|�d}~ww��||�W���n���wWd�n1s}wYz��
�WdSt
y�}zt�d	�j�d
|���WYd}~dSd}~ww�Nr�TestF)rhr��reference_example_idr�r��exceptions_to_handle�_end_on_exitT)r�r��skipped_reason)r�z%Failed to create feedback for run_id z:
�r�r,�tracer'r�r�r�r�rhrr�reprrrqrrO�warning�r�result�e�r!r4r7r5rr�_testhsF
��������z_run_test.<locals>._testr2�.yamlr~�r�r<��ignore_hostsr�r8r!�setrr�r(r,�get_tracing_contextr�rhrRr��tracing_contextrI�with_optional_cacher�api_url�	r!r7r8r4r5rH�
cache_path�current_contextr~rrGrr>Xs@���
�
 
������P�r>c		�s(�t�g��Ri��||d����t�������fdd�}|dr1t|d��jj�d�}nd}t��}i|dp=i��jjj	t
�j�d��}tjd	ii|�d|i����2t
j|�jjjgd��|�IdHWd�n1suwYWd�dSWd�dS1s�wYdS)
Nr9c�sZ����tjt�dd��j�j�j�jjt	fdd��c}zUz��i���IdH}Wn<t	yJ}z�j
t|�dd���|dt|�i�|�d}~wt
ye}z�j
t|�d���|d�|�d}~ww��||�W���n���wWd�n1s�wYz��
�WdSt
y�}zt�d	�j�d
|���WYd}~dSd}~wwr:r@rDrGrrrH�sH�
��������z_arun_test.<locals>._testr2rIr~rJrKrrMrSrrGrr9�sB����
�
 
������P�r9r�cC�Rt��rt�d�dSt��}t��}|r|sd}t|��|�	|�|�
|�dS)a'Log run inputs from within a pytest test run.

    .. warning::

        This API is in beta and might change in future versions.

    Should only be used in pytest tests decorated with @pytest.mark.langsmith.

    Args:
        inputs: Inputs to log.

    Example:
        .. code-block:: python

            from langsmith import testing as t


            @pytest.mark.langsmith
            def test_foo() -> None:
                x = 0
                y = 1
                t.log_inputs({"x": x, "y": y})
                assert foo(x, y) == 2
    z?LANGSMITH_TEST_TRACKING is set to 'false'. Skipping log_inputs.Nz�log_inputs should only be called within a pytest test decorated with @pytest.mark.langsmith, and with tracing enabled (by setting the LANGSMITH_TRACING environment variable to 'true').)rIrNrOrPr,�get_current_run_treer!r[rs�
add_inputsr)r�rr7r5rrrr�s��
rr�cCrV)aJLog run outputs from within a pytest test run.

    .. warning::

        This API is in beta and might change in future versions.

    Should only be used in pytest tests decorated with @pytest.mark.langsmith.

    Args:
        outputs: Outputs to log.

    Example:
        .. code-block:: python

            from langsmith import testing as t


            @pytest.mark.langsmith
            def test_foo() -> None:
                x = 0
                y = 1
                result = foo(x, y)
                t.log_outputs({"foo": result})
                assert result == 2
    z@LANGSMITH_TEST_TRACKING is set to 'false'. Skipping log_outputs.Nz�log_outputs should only be called within a pytest test decorated with @pytest.mark.langsmith, and with tracing enabled (by setting the LANGSMITH_TRACING environment variable to 'true').)rIrNrOrPr,rWr!r[rs�add_outputsr)r�rr7r5rrrrs��
rr�cCs<t��rt�d�dSt��}|sd}t|��|�|�dS)asLog example reference outputs from within a pytest test run.

    .. warning::

        This API is in beta and might change in future versions.

    Should only be used in pytest tests decorated with @pytest.mark.langsmith.

    Args:
        outputs: Reference outputs to log.

    Example:
        .. code-block:: python

            from langsmith import testing


            @pytest.mark.langsmith
            def test_foo() -> None:
                x = 0
                y = 1
                expected = 2
                testing.log_reference_outputs({"foo": expected})
                assert foo(x, y) == expected
    zJLANGSMITH_TEST_TRACKING is set to 'false'. Skipping log_reference_outputs.Nzglog_reference_outputs should only be called within a pytest test decorated with @pytest.mark.langsmith.)rIrNrOrPr!r[rsr)r�r7r5rrrr;s��r)r�r�r��!Optional[Union[dict, list[dict]]]r�r��!Optional[Union[int, bool, float]]r��&Optional[Union[str, int, float, bool]]c	Kst��rt�d�dS|rt|||f�rd}t|��|s$|s$d}t|��|r;d|i}|dur2||d<|dur:||d<n	t��}t�	�}|rH|sNd}t|��|j
d	krr|j�	d
�rr|jd
}|�t
|t�rg|nd|i�|j|d<n|j}|j|ttttf|�fi|��dS)
a/Log run feedback from within a pytest test run.

    .. warning::

        This API is in beta and might change in future versions.

    Should only be used in pytest tests decorated with @pytest.mark.langsmith.

    Args:
        key: Feedback name.
        score: Numerical feedback value.
        value: Categorical feedback value
        kwargs: Any other Client.create_feedback args.

    Example:
        .. code-block:: python

            import pytest
            from langsmith import testing as t


            @pytest.mark.langsmith
            def test_foo() -> None:
                x = 0
                y = 1
                expected = 2
                result = foo(x, y)
                t.log_feedback(key="right_type", score=isinstance(result, int))
                assert result == expected
    �ALANGSMITH_TEST_TRACKING is set to 'false'. Skipping log_feedback.NzGMust specify one of 'feedback' and ('key', 'score', 'value'), not both.zDMust specify at least one of 'feedback' or ('key', 'score', value').r�r�r�z�log_feedback should only be called within a pytest test decorated with @pytest.mark.langsmith, and with tracing enabled (by setting the LANGSMITH_TRACING environment variable to 'true').�
evaluators�reference_run_idr��
source_run_id)rIrNrOrPr�rsr,rWr!r[�session_namer~rYr�r�r(�trace_idrr
rr�)	r�r�r�r�r1r5rr7r�rrr�log_feedbackesF'����
�&rc�Feedbackr�rh�2Generator[Optional[run_trees.RunTree], None, None]ccs��t��rt�d�dVdSt��}t��}|r|s!d}t|��|j	j
j|j|j
d�}tj||jdd|d��}|VWd�dS1sFwYdS)a$	Trace the computation of a pytest run feedback as its own run.

    .. warning::

        This API is in beta and might change in future versions.

    Args:
        name: Feedback run name. Defaults to "Feedback".

    Example:
        .. code-block:: python

            import openai
            import pytest

            from langsmith import testing as t
            from langsmith import wrappers

            oai_client = wrappers.wrap_openai(openai.Client())


            @pytest.mark.langsmith
            def test_openai_says_hello():
                # Traced code will be included in the test case
                text = "Say hello!"
                response = oai_client.chat.completions.create(
                    model="gpt-4o-mini",
                    messages=[
                        {"role": "system", "content": "You are a helpful assistant."},
                        {"role": "user", "content": text},
                    ],
                )
                t.log_inputs({"text": text})
                t.log_outputs({"response": response.choices[0].message.content})
                t.log_reference_outputs({"response": "hello!"})

                # Use this context manager to trace any steps used for generating evaluation
                # feedback separately from the main application logic
                with t.trace_feedback():
                    grade = oai_client.chat.completions.create(
                        model="gpt-4o-mini",
                        messages=[
                            {
                                "role": "system",
                                "content": "Return 1 if 'hello' is in the user message and 0 otherwise.",
                            },
                            {
                                "role": "user",
                                "content": response.choices[0].message.content,
                            },
                        ],
                    )
                    # Make sure to log relevant feedback within the context for the
                    # trace to be associated with this feedback.
                    t.log_feedback(
                        key="llm_judge", score=float(grade.choices[0].message.content)
                    )

                assert "hello" in response.choices[0].message.content.lower()
    r]Nz�trace_feedback should only be called within a pytest test decorated with @pytest.mark.langsmith, and with tracing enabled (by setting the LANGSMITH_TRACING environment variable to 'true').)r�r<r_�ignorer^)rhr��parentr�r~)rIrNrOrPr,rWr!r[rsr�r�rhr�r(rAr�)rh�
parent_runr7r5r~rrrr�trace_feedback�s6�@����"�ri�xcCs0z
t|�jddd�WStyt|�YSw)Nzutf-8�surrogateescape)�errors)r�decode�	ExceptionrR)rjrrrr�s
�r�rF)
r(r+r)r,rr-r*r.r"r/)r0rr1rr"r)r*rRr"rR)r!rr"rR)rrvr*rRr"rw)rrvr�rwr"r�)r!rr�r�r�r�r"r�)r�r�)r�r�r"r�)r!rr$r%r"rR)r!rr0rr7rr8rGr1rr"r
)r!rr4rr7rr8rGr5rr"r�rrr r$)r�rZr�rRr�r[r�r\r1rr"r�)rhrRr"re)rjrr"rR)X�
__future__rr��
contextlib�contextvars�datetimerBr\r@�loggingrYr	rr_rK�concurrent.futuresr�pathlibr�typingrrrrr	r
rrr
r�typing_extensionsr�	langsmithrr�rrnrr,rr�r�
ls_schemasrrI�langsmith._internalr�langsmith._internal._serder�langsmith.clientrr|�skiprnr�ImportError�	getLoggerrrOrr r&rirur�r�r�r�r�r�r�r�r
�
ContextVarr!rGr)r8r>r9�unitrrrrc�contextmanagerrir�rrrr�<module>s�0
�
�
	






?m


3
DE

+
,+��Q�]
@ Telegram