File: //usr/local/lib/python3.10/dist-packages/langsmith/__pycache__/client.cpython-310.pyc
o
���g�] � @ s6 U d Z ddlmZ ddlZddlZddlmZ ddlZddl Z ddl
Z
ddl Z ddl Z ddl
Z
ddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlmZ ddl m Z ddl m Z ddlm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0 ddl1m2Z3 ddl4Z4dd l4m5Z6 dd
l7m8Z9 dd l:m;Z;m<Z< dd l=m>Z> dd
l?m@Z@ ddlAZAddlAmBZC ddlAmDZE ddlAmFZG ddlHmIZI ddlJmKZK ddlJmLZM ddlNmOZO ddlPmQZQ ddlRmSZSmTZTmUZUmVZV ddlWmXZXmYZYmZZZ ddl[m\Z\m]Z]m^Z^m_Z_m`Z`maZambZbmcZc ddldmeZf ddlgmhZh zddlimjZj W n ek�yD G d d � d �ZjY nw e �r|ddllZmdd lnmoZo ddlAmDZD dd lpmqZr dd lsmtZtmuZumvZv dd!lwmxZxmyZymzZzm{Z{m|Z|m}Z}m~Z~mZ e��e��Z�e��d"�Z�d#Z�d$Z�d%e�d&< d'ee>�j�v Z� ( )dxdyd3d4�Z�dzd7d8�Z�e/ej�e�f Z�e)d9 Z�d{d;d<�Z�d|d@dA�Z�d}dDdE�Z�d~dGdH�Z�ddLdM�Z�d�d�dRdS�Z�ej<d�dUdV��Z�ej<dWdX�d�d[dV��Z�d\dX�d�d]dV�Z�e
j�d^d_�d`da� �Z�G dbdc� dce6j��Z�G ddde� de�Z� d�d�dkdl�Z� d�d�dmdn�Z�d�drds�Z�d�dvdw�Z�dS )�a� Client for interacting with the LangSmith API.
Use the client to customize API keys / workspace connections, SSL certs,
etc. for tracing.
Also used to create, read, update, and delete LangSmith resources
such as runs (~trace spans), datasets, examples (~records),
feedback (~metrics), projects (tracer sessions/groups), etc.
For detailed API documentation, visit: https://docs.smith.langchain.com/.
� )� annotationsN)� signature)�Path)�
PriorityQueue)�
TYPE_CHECKING�Any�
AsyncIterable�Callable� DefaultDict�Dict�Iterable�Iterator�List�Literal�Mapping�Optional�Sequence�Tuple�Type�Union�cast)�parse)�adapters)� multipart)� TypeGuard�overload)�PoolKey)�Retry)�env)�schemas)�utils)�_orjson)�TracingQueueItem)�tracing_control_thread_func)� warn_beta)�CompressedTraces)� _AUTO_SCALE_UP_NTHREADS_LIMIT�_BLOCKSIZE_BYTES� _BOUNDARY�_SIZE_LIMIT_BYTES)�
MultipartPart�MultipartPartsAndContext� join_multipart_parts_and_context)�SerializedFeedbackOperation�SerializedRunOperation�#combine_serialized_queue_operations�$compress_multipart_parts_and_context�serialize_feedback_dict�serialize_run_dict�<serialized_feedback_operation_to_multipart_parts_and_context�7serialized_run_operation_to_multipart_parts_and_context)�
dumps_json)�AttachmentInfo)�ZoneInfoc @ s e Zd ZdZdS )r7 zIntroduced in python 3.9.N)�__name__�
__module__� __qualname__�__doc__� r< r< �;/usr/local/lib/python3.10/dist-packages/langsmith/client.pyr7 n s r7 )�Runnable�� evaluator)� AEVALUATOR_T� ATARGET_T�AsyncExperimentResults)�COMPARATIVE_EVALUATOR_T�DATA_T� EVALUATOR_T� EXPERIMENT_T�SUMMARY_EVALUATOR_T�TARGET_T� ComparativeExperimentResults�ExperimentResultszurllib3.connectionpool� x-api-keyr< ztuple[Dict, ...]� EMPTY_SEQ�
key_blocksize� �dataset� url_or_token�Union[str, uuid.UUID]�api_url�str� num_parts�int�kind�return�Tuple[str, str]c C s� zt | tj�s t�| �r|t| �fW S W n ty Y nw t�t| ��}|j�d�}t |�|kr=|| }t
|dd� n t � d|� d| � ���|j
dkrSd}||fS |j
dkrZd }||fS )
z*Parse a public dataset URL or share token.�/z token parts)�varzInvalid public z URL: zsmith.langchain.com� https://api.smith.langchain.comzbeta.smith.langchain.comz$https://beta.api.smith.langchain.com)�
isinstance�uuid�UUIDrT �
ValueError� urllib_parse�urlparse�path�split�len�_as_uuid�ls_utils�LangSmithUserError�netloc)rQ rS rU rW �
parsed_url�
path_parts�
token_uuidr< r< r= �_parse_token_or_url� s& � �
�rm �url�boolc C s8 zt �| �j�d�d }|�d�W S ty Y dS w )z�Check if the URL is langchain hosted.
Args:
url (str): The URL to check.
Returns:
bool: True if the URL is langchain hosted, False otherwise.
�:r z
langchain.comF)ra �urlsplitri rd �endswith� Exception)rn ri r< r< r= �_is_langchain_hosted� s �rt )�tool�chain�llm� retriever� embedding�prompt�parserr c C sX t dg d�ddddd�} tj�d�}ttt|�d���d k}|r$d
| d < tj d i | ��S )
z�Get the default retry configuration.
If urllib3 version is 1.26 or greater, retry on all methods.
Returns:
Retry: The default retry configuration.
� )i� i� i� � i� � �?FT)�total�status_forcelist�backoff_factor�raise_on_redirect�raise_on_status�respect_retry_after_header�urllib3�.)� � N�allowed_methodsr< )
�dict� importlib�metadata�version�tuple�maprV rd rg �LangSmithRetry)� retry_params�urllib3_version�use_allowed_methodsr< r< r= �_default_retry_config� s �
r� �session�requests.Session�Nonec C s t �d� | �� dS )z\Close the session.
Args:
session (requests.Session): The session to close.
zClosing Client.sessionN)�logger�debug�close)r� r< r< r= �
close_session� s
r� �api_key�
Optional[str]c C s& |st | �rt�dtj� dS dS dS )a Verify API key is provided if url not localhost.
Args:
api_url (str): The API URL.
api_key (Optional[str]): The API key.
Returns:
None
Raises:
LangSmithUserError: If the API key is not provided when using the hosted service.
z8API key must be provided when using hosted LangSmith APIN)rt �warnings�warnrg � LangSmithMissingAPIKeyWarning�rS r� r< r< r= �_validate_api_key_if_hosted� s ��r� � float | Nonec C sB t �d�} | du r dS t| �}|dk s|dkr t �d|� ���|S )zeGet the tracing sampling rate.
Returns:
Optional[float]: The tracing sampling rate.
�TRACING_SAMPLING_RATENr r� zELANGSMITH_TRACING_SAMPLING_RATE must be between 0 and 1 if set. Got: )rg � get_env_var�floatrh )�sampling_rate_str�
sampling_rater< r< r= �_get_tracing_sampling_rate s
��r� �_write_api_urls�Optional[Dict[str, str]]�Dict[str, str]c C s� | p
t �t�dd��} i }| �� D ]/\}}|�� }|s t�d��|�� �d��d��d�}|�� �d��d�}t ||� |||< q|S )N�LANGSMITH_RUNS_ENDPOINTSz{}zFLangSmith runs API URL within LANGSMITH_RUNS_ENDPOINTS cannot be empty�"�'rZ )
�json�loads�os�getenv�items�striprg rh �rstripr� )r� �processed_write_api_urlsrn r� �
processed_url�processed_api_keyr< r< r= �_get_write_api_urls s
��
r� �value�ID_TYPEr[ � uuid.UUIDc
C sV zt | tj�s
t�| �W S | W S ty* } z|pd}t�|� d| � ��|�d }~ww )Nr� z* must be a valid UUID or UUID string. Got )r] r^ r_ r` rg rh )r� r[ �er< r< r= rf ( s ����rf � Optional[Union[str, uuid.UUID]]c C � d S �Nr< )r� r< r< r= � _ensure_uuid2 s r� T�� accept_nullr� �Optional[uuid.UUID]c C r� r� r< �r� r� r< r< r= r� 6 s Fc C s | d u r |rd S t �� S t| �S r� )r^ �uuid4rf r� r< r< r= r� < s
r� ��maxsizec C s t �| �}|j�d�d }|S )Nrp r )ra rb ri rd )rn rj �hostr< r< r= �
_parse_urlD s
r� c sH e Zd Zg d�Zejejejejdfd� fd
d�
Zd� fdd� Z � Z
S )�_LangSmithHttpAdapter)� max_retries�config�_pool_connections�
_pool_maxsize� _pool_block�
_blocksizei @ �pool_connectionsrV � pool_maxsizer� �Union[Retry, int, None]�
pool_blockro � blocksizerX r� c s || _ t� �||||� d S r� )r� �super�__init__)�selfr� r� r� r� r� �� __class__r< r= r� U s z _LangSmithHttpAdapter.__init__Fc s& t r| j|d< t� j|||fi |��S )Nr� )�URLLIB3_SUPPORTS_BLOCKSIZEr� r� �init_poolmanager)r� � connectionsr� �block� pool_kwargsr� r< r= r� ` s
z&_LangSmithHttpAdapter.init_poolmanager) r� rV r� rV r� r� r� ro r� rV rX r� �F) r8 r9 r: � __attrs__�requests_adapters�DEFAULT_POOLSIZE�DEFAULT_RETRIES�DEFAULT_POOLBLOCKr� r� �
__classcell__r< r< r� r= r� K s � r� c @ s�
e Zd ZdZg d�Z �d�dddddddddddd� �d�d d �Z�d�d d!�Z�d�d"d#�Ze�d�d$d%��Z e�d�d&d'��Z
e�d�d)d*��Z e�d�d,d-��Z �d�d/d0�Z
�d�d3d4�Zdd5dddd6d7��d�dHdI�ZddJ��d�dOdP�ZddQdRdS��d�dXdY�ZdejjdZ��d�dedf�Zddejjdg��d�djdk�Z l l�dАd�drds�Ze�d�dudv��Zdldw��d�d{d|�Zdddld}��d�d�d��Z�d�d�d��Z �d�d�d��Z �d�d�d��Z �d�d�d��Z �d�dld���d�d�d��Z �d�d�d��Z!�d�d�d��Z" �d�dldld���d�d�d��Z#d�d���d�d�d��Z$d�d���d�d�d��Z%ddddddddddld��
�d�d�d��Z&�d�d�d��Z'�d�d�d�dÄZ(�d�d�dńZ)�d�d�dȄZ* l�d�d�d�d˄Z+ddddddddddddddddd̜�d�d�d�Z,ddddddddddddddddd��d�d�d�Z-ddd��d�d�d�Z.dd��d�d�d��Z/�d�d�d��Z0�d�d�d��Z1�d�d�d��Z2 �dĐd�d�d��Z3 �dĐd�d �d�Z4 �d�d�d��d�d�d�Z5 �d�d�d��d�d�d �Z6�d�d
�d �Z7�d�d �d
�Z8d�d��d�d�d�Z9dddd�d��d��d�d�Z:dddldd�d��d��d �d �Z;ddddd�d!��d��d"�d#�Z<�d��d%�d&�Z=�d��d(�d)�Z>e?�@�d*�dddl�d+��d��d.�d/��ZAd�d0��d��d1�d2�ZBdd�d*��d��d3�d4�ZC �d��d��d9�d:�ZDe?�@d�ddd��d��d;�d<��ZEdejjdddd�d=��d �dC�dD�ZFdd�dE��d�dF�dG�ZGe?�@�dE�dd�dE��d�dH�dI��ZH �d�d�d��d�dN�dO�ZI �d�d�d��d�dQ�dR�ZJdddddd�dS��d�dW�dX�ZKe?�@�dY�dd�dY��d�dZ�d[��ZLdd�dY��d�d_�d`�ZMdddd�da��d�dd�de�ZNdddd�df��d �dh�di�ZOdd�dj��d
�dm�dn�ZP�d �do�dp�ZQe?�@�dY� �d �d
�du�dv��ZRe?�@�dY� �d �d�d{�d|��ZS �d�d�d}�d~�ZT l l�dАd�d��d��ZUddl�d���d�d��d��ZVddl�d���d�d��d��ZWddl�d���d�d��d��ZXddddddd�d���d�d��d��ZYe?�@�dY� �d��d�d��d���ZZd�d���d�d��d��Z[ �d�d�ddddl�d���d�d��d��Z\e]�d��d���d�d��d���Z^e]d�d���d�d��d���Z_dddddd�d���d �d��d��Z`dddddd�d���d �dŐdƄZa�d �dǐdȄZb�d �dɐdʄZcddd�d˜�d �dΐdτZddddl�dМ�d!�dԐdՄZe�d"�dאdZf�d#�dܐd݄Zgd�dޜ�d$�d�d�Zhdddl�d��d%�d�d�Zi �dd�d��d&�d�d�Zjdddl�d��d'�d��d��Zkdddddejljmddd�d�ddddd�d���d(�d �d �Zndddd�d
��d)�d�d�Zo�d*�d�d�Zpdddd�d��d+�d�d�Zq�d,�d�d�Zr �d�dddd�d��d-�d �d �Zsddd�d ��d.�d#�d$�Ztdd�d%��d/�d*�d+�Zud�d,��d0�d.�d/�Zvdddd�d0��d1�d3�d4�Zwdd�d5��d2�d8�d9�Zx�d3�d:�d;�Zyd�d<��d4�d=�d>�Zz�d5�d?�d@�Z{�d6�dA�dB�Z|�d7�dC�dD�Z}�d8�dG�dH�Z~ddddd�dI��d9�dM�dN�Zd�dOddddldd�dP��d:�dZ�d[�Z�d�dOddddldd�dP��d:�d\�d]�Z��d;�d_�d`�Z��d<�dc�dd�Z�� ��d=�d>�df�dg�Z��d?�dk�dl�Z��d@�dm�dn�Z��dA�do�dp�Z��dB�dq�dr�Z��dB�ds�dt�Z��du�d�ddlej�j��dvd�dw��dC�d�d��Z��dD�d��d��Z�ddddl�d���dE�d��d��Z�d�d���dF�d��d��Z�ddddd�d���dG�d��d��Z��dH�d��d��Z�dl�d���dI�d��d��Z�d�d�dl�d���dJ�d��d��Z�dl�d���dK�d��d��Z�d�d�dddd�d���dL�d��d��Z��d�d��d��Z�e� � � 5 �dM�dN�d��d���Z�e� � � 5 �dM�dO�d��d���Z� � � 5 �dM�dP�d��d��Z� � � 5 �dM�dQ�ddÄZ�dS (R �Clientz.Client for interacting with the LangSmith API.)� __weakref__rS r� � retry_config�
timeout_msr� �_get_data_type_cached�_web_url�
_tenant_id�tracing_sample_rate�_filtered_post_uuids�
tracing_queue� _anonymizer� _hide_inputs�
_hide_outputs�_infor� � _settings�_manual_cleanup� _pyo3_client�compressed_traces�_data_available_event�_futuresNT) r� r� r� �web_urlr� �auto_batch_tracing�
anonymizer� hide_inputs� hide_outputs�info�api_urlsrS r� r� r� �Optional[Retry]r� �%Optional[Union[int, Tuple[int, int]]]r� r� �Optional[requests.Session]r� ro r� � Optional[Callable[[dict], dict]]r �-Optional[Union[Callable[[dict], dict], bool]]r r �/Optional[Union[dict, ls_schemas.LangSmithInfo]]r r� rX r� c
C s� |r | r t �d��t�d�st�d�r t�d�r t �d��t� | _t� | _t| �| _ | j r=t
t | j ��| _ | j | j | _
nt �|�| _ t �|�| _
t| j | j
� | j | j
i| _ |p[t� | _t|t�rf||fn|pid| _|| _d| _|ru|nt�� }
|
| _| du s�t| tj �r�| ntj di | ��| _ t � | t | j� t!�"t |
� d| _#d| _$d| _%|r�t&� | _'t(j)t*t �+| �fd��,� nd| _'t-| jt.t/d d
�}| j�0d |� | j�0d |� t1j2d
d�| j3�| _4|| _5| dur�| nt �6d�dk| _7|
dur�|
nt �6d�dk| _8d| _9t �6d�du�red}zddl:}W n t;�y. } zt<�=dt>|�� W Y d}~nd}~ww |�red}d}d}d }z|�?| j | j
||||�| _9W n t@�yd } zt<�=dt>|�� W Y d}~nd}~ww d| _Ad| _BdS )a� Initialize a Client instance.
Args:
api_url (Optional[str]): URL for the LangSmith API. Defaults to the LANGCHAIN_ENDPOINT
environment variable or https://api.smith.langchain.com if not set.
api_key (Optional[str]): API key for the LangSmith API. Defaults to the LANGCHAIN_API_KEY
environment variable.
retry_config (Optional[Retry]): Retry configuration for the HTTPAdapter.
timeout_ms (Optional[Union[int, Tuple[int, int]]]): Timeout for the HTTPAdapter. Can also be a 2-tuple of
(connect timeout, read timeout) to set them separately.
web_url (Optional[str]): URL for the LangSmith web app. Default is auto-inferred from
the ENDPOINT.
session (Optional[requests.Session]): The session to use for requests. If None, a new session will be
created.
auto_batch_tracing (bool, default=True): Whether to automatically batch tracing.
anonymizer (Optional[Callable[[dict], dict]]): A function applied for masking serialized run inputs and outputs,
before sending to the API.
hide_inputs (Optional[Union[Callable[[dict], dict], bool]]): Whether to hide run inputs when tracing with this client.
If True, hides the entire inputs. If a function, applied to
all run inputs when creating runs.
hide_outputs (Optional[Union[Callable[[dict], dict], bool]]): Whether to hide run outputs when tracing with this client.
If True, hides the entire outputs. If a function, applied to
all run outputs when creating runs.
info (Optional[ls_schemas.LangSmithInfo]): The information about the LangSmith API.
If not provided, it will be fetched from the API.
api_urls (Optional[Dict[str, str]]): A dictionary of write API URLs and their corresponding API keys.
Useful for multi-tenant setups. Data is only read from the first
URL in the dictionary. However, ONLY Runs are written (POST and PATCH)
to all URLs in the dictionary. Feedback, sessions, datasets, examples,
annotation queues and evaluation results are only written to the first.
Raises:
LangSmithUserError: If the API key is not provided when using the hosted service.
If both api_url and api_urls are provided.
z-You cannot provide both api_url and api_urls.�LANGSMITH_ENDPOINT�LANGCHAIN_ENDPOINTr� z]You cannot provide both LANGSMITH_ENDPOINT / LANGCHAIN_ENDPOINT and LANGSMITH_RUNS_ENDPOINTS.)i' i�_ N)�target�argsr� )r� r� r� zhttp://zhttps://�
r� � HIDE_INPUTS�true� HIDE_OUTPUTS�USE_PYO3_CLIENTr zaFailed to import `langsmith_pyo3` when PyO3 client was requested, falling back to Python impl: %si@B �d �� z|Failed to instantiate `langsmith_pyo3.BlockingTracingClient` when PyO3 client was requested, falling back to Python impl: %sFr< )Crg rh r� r� r� r� �setr� r� r� �next�iterrS r� � get_api_url� get_api_keyr� r� r� r] rV r� r� r� �requests�Sessionr� �
ls_schemas�
LangSmithInfor� �weakref�finalizer� �atexit�registerr� r� r� r r� � threading�Thread� _tracing_control_thread_func�ref�startr� r' r& �mount� functools� lru_cache�_get_data_typer� r� r� r� r� r� �langsmith_pyo3� ImportErrorr� �warning�repr�BlockingTracingClientrs r� r� )r� rS r� r� r� r� r� r� r� r r r r �session_�adapterr+ r� �queue_capacity�
batch_size�batch_timeout_millis�worker_threadsr< r< r= r� � s� 3������
����
��
�� �� � ���
����
zClient.__init__rT c C s | j }d|� d�S )z�Return an HTML representation of the instance with a link to the URL.
Returns:
str: The HTML representation of the instance.
z <a href="z6", target="_blank" rel="noopener">LangSmith Client</a>)� _host_url)r� �linkr< r< r= � _repr_html_5 s zClient._repr_html_c C s d| j � d�S )z�Return a string representation of the instance with a link to the URL.
Returns:
str: The string representation of the instance.
zClient (API URL: �))rS �r� r< r< r= �__repr__> s zClient.__repr__c C s
t | j�S r� )r� rS r: r< r< r= �_hostF �
z Client._hostc C s t �| j| j�S )zThe web host url.)rg � get_host_urlr� rS r: r< r< r= r6 J s zClient._host_urlr� c C s& dt j� �dd�}| jr| j|t< |S )z|Get the headers for the API request.
Returns:
Dict[str, str]: The headers for the API request.
z
langsmith-py/�application/json)z
User-Agent�Accept)� langsmith� __version__r� � X_API_KEY)r� �headersr< r< r= �_headersO s
�
zClient._headers�ls_schemas.LangSmithInfoc
C s� | j du rXz+| jddddi| jd d | jd d fd �}t�|� tjd i |�� ��| _ W | j S tyW } zt �
d
| j � d t |�� �� t�� | _ W Y d}~| j S d}~ww | j S )
z�Get the information about the LangSmith API.
Returns:
ls_schemas.LangSmithInfo: The information about the LangSmith API, or None if the API is
not available.
N�GETz/infor@ r? r r r� )rD �timeoutzFailed to get info from �: r< )
r� �request_with_retriesr� rg �raise_for_status_with_textr r r� �
BaseExceptionr� r- rS r. )r� �responser� r< r< r= r ^ s(
�
����z Client.info� ls_schemas.LangSmithSettingsc C s< | j du r| �dd�}t�|� tjdi |�� ��| _ | j S )zzGet the settings for the current tenant.
Returns:
dict: The settings for the current tenant.
NrG z /settingsr< )r� rJ rg rK r �LangSmithSettingsr� )r� rM r< r< r= �
_get_settingsw s
zClient._get_settings�content_length�
Optional[int]c C sf |d u s | j d u r d S ttj| j �}|j}|sd S |�d�}|d u r$d S ||kr1d|� d|� d�S d S )N�size_limit_byteszThe content length of z) bytes exceeds the maximum size limit of z bytes.)r� r r r �batch_ingest_config�get)r� rQ r �bic�
size_limitr< r< r= �_content_above_size� s
��zClient._content_above_sizer� � )�request_kwargs�stop_after_attempt�retry_on� to_ignore�handle_response�_context�method�0Literal['GET', 'POST', 'PUT', 'PATCH', 'DELETE']�pathnamerZ �Optional[Mapping]r[ rV r\ �'Optional[Sequence[Type[BaseException]]]r] r^ �1Optional[Callable[[requests.Response, int], Any]]r_ �kwargsr �requests.Responsec K sl |pi }d| j d d | j d d fi|�| �di | j�|�di ��| �di ��i�}|dkrDd|v rDd|vrD|d �d �sDd
|d d < t�� tjt| j�d �g}
g |pUd �tjtj tj
f�R } g |pdd �R } d
}
t |�D �]�}�z!z8t� t
|
�� | jj||�d�s�| j| n|fddi|��}
W d
� n1 s�w Y t�|
� |
W W S tjjy� } z#t�d|� |d |kr�� d| t�� d }t�|� W Y d
}~W qmd
}~w tj�y� } z�|
d
u�r�|d
u�r|d |k �r||
|d �}|�rW Y d
}~W qm|
j dk�r t�
d|� d|� dt |�� |� ���|
j dk�r1t� d|� |� d|� ���|
j dk�rFt� d|� d t |�� |� ���|
j d k�r[t� d |� d t |�� |� ���|
j d k�rpt� d |� d t |�� |� ���|
j d!k�r�t�!d"|� d t |�� |� ���t�"d#|� d|� dt |�� ���t�#d#|� d|� dt |�� ���d
}~w tj$�y6 } z�| jd$k�r�d%nd&}z t%|j�r�t|jj&�d'��nd(�}| �'|�}|�r�|}W n t(�y� d
}Y nw |j�r�|jj&�d)��p�d(nd(}|d
d*� |d+d
� }}d,t)dt*|�d- � }|� |� |� �}t�d.|� d|� d|� dt |�� d/|� d0|� |� �
�|�d
}~w t+�y� } zPt,|j-�}t*|�dk�rN|d nd(}|�.d1d2�}|�r{d3�/t|d �g|g d4d5� t*|�dk�rt|dd
� ng D � �} n|} t�"d#|� d|� d| � |� ��|�d
}~ww | �y� } z|
d
u�r�t�d|� |
W Y d
}~ S W Y d
}~qmd
}~w tj �y |d |k�rʂ |
d
u�r�z t0|
j&�d6d7��} W n t+�y� } zt�1d8t |�� d9} W Y d
}~nd
}~ww | d| t�� } t�| � Y qm | �y) |d |k�r� d| t�� d }t�|� Y qmw t�"d#|� d|� d:���);a� Send a request with retries.
Args:
method (str): The HTTP request method.
pathname (str): The pathname of the request URL. Will be appended to the API URL.
request_kwargs (Mapping): Additional request parameters.
stop_after_attempt (int, default=1): The number of attempts to make.
retry_on (Optional[Sequence[Type[BaseException]]]): The exceptions to retry on. In addition to:
[LangSmithConnectionError, LangSmithAPIError].
to_ignore (Optional[Sequence[Type[BaseException]]]): The exceptions to ignore / pass on.
handle_response (Optional[Callable[[requests.Response, int], Any]]): A function to handle the response and return whether to continue retrying.
_context (str, default=""): The context of the request.
**kwargs (Any): Additional keyword arguments to pass to the request.
Returns:
requests.Response: The response object.
Raises:
LangSmithAPIError: If a server error occurs.
LangSmithUserError: If the request fails.
LangSmithConnectionError: If a connection error occurs.
LangSmithError: If the request fails.
rH r r r� rD rG �data�files� Content-Typer? )r� r< N�http�streamFzPassing on exception %srO r~ i� z Server error caused failure to � z in LangSmith API. r} z(Client took too long to send request to i� zRate limit exceeded for z. i� zAuthentication failed for i� zResource not found for i� z
Conflict for z
Failed to r\ z'Please confirm your LANGCHAIN_ENDPOINT.z(Please confirm your internet connection.zContent-LengthrY rL � ����*� z#Connection error caused failure to z
Content-Length: z
API Key: r� zsession (project)�
c S � g | ]}t |��qS r< �rT )�.0�argr< r< r= �
<listcomp>L � z/Client.request_with_retries.<locals>.<listcomp>z retry-after�30z Invalid retry-after header: %s� z in LangSmith API.)2r� rE rU rg �FilterLangSmithRetry�FilterPoolFullWarningrT r< �LangSmithConnectionError�LangSmithRequestTimeout�LangSmithAPIError�range� filter_logs�_urllib3_loggerr� �request�
startswithrS rK r �
exceptions� ReadTimeoutr� r� �random�time�sleep� HTTPError� status_coder. �LangSmithRateLimitError�LangSmithAuthError�LangSmithNotFoundError�LangSmithConflictError�LangSmithErrorrh �ConnectionErrorrV rD rX r` �maxre rs �listr
�replace�joinr� r- ) r� r` rb rZ r[ r\ r] r^ r_ rf �logging_filters� retry_on_�
to_ignore_rM �idxr� �
sleep_time�should_continue�recommendationrQ �size_recr� �prefix�suffix�filler�masked_api_keyr
�msg�emsg� retry_afterr< r< r= rJ � s� % ����
�
�� � ���� ���� �
�
���� ��� �� �� �� ������ ����
��� ���������
�&�������
��
���
� �zClient.request_with_retries��paramsrc r� �Optional[dict]�Iterator[dict]c c s� � |r|� � ni }|�dd�}|�dd�|d< ||d< | jd||d�}|�� }|s,dS |E dH t|�|d k r;dS |t|�7 }q) z�Get a paginated list of items.
Args:
path (str): The path of the request URL.
params (Optional[dict]): The query parameters.
Yields:
The items in the paginated list.
�offsetr �limitr TrG r� N)�copyrU rJ r� re )r� rc r� �params_r� rM r� r< r< r= �_get_paginated_listt s& � �
�zClient._get_paginated_list�POST�runs)�body�request_method�data_keyr� r� �Literal['GET', 'POST']r� c c s� � |r|� � ni } | j||dt|�id�}|�� }|s dS |�|�s%dS || E dH |�d�}|s5dS |�d�s<dS |d |d< q
)a� Get a cursor paginated list of items.
Args:
path (str): The path of the request URL.
body (Optional[dict]): The query body.
request_method (Literal["GET", "POST"], default="POST"): The HTTP request method.
data_key (str, default="runs"): The key in the response body that contains the items.
Yields:
The items in the paginated list.
Trh �rZ N�cursorsr �cursor)r� rJ � _dumps_jsonr� rU ) r� rc r� r� r� r� rM �
response_bodyr� r< r< r= �_get_cursor_paginated_list� s, ���
�z!Client._get_cursor_paginated_list)� description� data_type�df� pd.DataFrame�name�
input_keys�
Sequence[str]� output_keysr� r� � Optional[ls_schemas.DataType]�ls_schemas.Datasetc C s: t �� }|j|dd� |�d� | jd|f|||||d�S )az Upload a dataframe as individual examples to the LangSmith API.
Args:
df (pd.DataFrame): The dataframe to upload.
name (str): The name of the dataset.
input_keys (Sequence[str]): The input keys.
output_keys (Sequence[str]): The output keys.
description (Optional[str]): The description of the dataset.
data_type (Optional[DataType]): The data type of the dataset.
Returns:
Dataset: The uploaded dataset.
Raises:
ValueError: If the csv_file is not a string or tuple.
Examples:
.. code-block:: python
from langsmith import Client
import os
import pandas as pd
client = Client()
df = pd.read_parquet("path/to/your/myfile.parquet")
input_keys = ["column1", "column2"] # replace with your input column names
output_keys = ["output1", "output2"] # replace with your output column names
dataset = client.upload_dataframe(
df=df,
input_keys=input_keys,
output_keys=output_keys,
name="My Parquet Dataset",
description="Dataset created from a parquet file",
data_type="kv", # The default
)
F)�indexr zdata.csv)r� r� r� r� r� )�io�BytesIO�to_csv�seek�
upload_csv)r� r� r� r� r� r� r� �csv_filer< r< r= �upload_dataframe� s /
�zClient.upload_dataframe)r� r� r� r� �"Union[str, Tuple[str, io.BytesIO]]c
C s8 ||d�}|r ||d< |r||d< |rt �|�|d< tt�� �|d< t|t�rJt|d��}d|i} | jdd || d
�}
W d � n1 sDw Y nt|t�r[| jdd |d|id
�}
nt d ��t �
|
� |
� � } d
| v r�d| d
v r�t|t�ry|n|d } | � d�d } t d| � d���t
jdi | �| j| �� d���S )a� Upload a CSV file to the LangSmith API.
Args:
csv_file (Union[str, Tuple[str, io.BytesIO]]): The CSV file to upload. If a string, it should be the path
If a tuple, it should be a tuple containing the filename
and a BytesIO object.
input_keys (Sequence[str]): The input keys.
output_keys (Sequence[str]): The output keys.
name (Optional[str]): The name of the dataset.
description (Optional[str]): The description of the dataset.
data_type (Optional[ls_schemas.DataType]): The data type of the dataset.
Returns:
Dataset: The uploaded dataset.
Raises:
ValueError: If the csv_file is not a string or tuple.
Examples:
.. code-block:: python
from langsmith import Client
import os
client = Client()
csv_file = "path/to/your/myfile.csv"
input_keys = ["column1", "column2"] # replace with your input column names
output_keys = ["output1", "output2"] # replace with your output column names
dataset = client.upload_csv(
csv_file=csv_file,
input_keys=input_keys,
output_keys=output_keys,
name="My CSV Dataset",
description="Dataset created from a CSV file",
data_type="kv", # The default
)
)r� r� r� r� r� �id�rb�filer� z/datasets/upload)rh ri Nz"csv_file must be a string or tuple�detailzalready existsr rZ ����Dataset z already exists�r6 r� r< )rg �get_enum_valuerT r^ r� r] �openrJ r� r` rK r� rd r �Datasetr6 �_get_optional_tenant_id)
r� r� r� r� r� r� r� rh �f�file_rM �result� file_namer< r< r= r� � sR 1�
� ��
�
�
�zClient.upload_csvF�run�3Union[ls_schemas.Run, dict, ls_schemas.RunLikeDict]�updater� r� c C s@ t |d�rtt|d��r|�� }ntt|�}d|vr!t�� |d< nt|d t�r1t� |d �|d< d|v rO|d durO|rFt
� |d �|d< | � |d �|d< d|v rm|d durm|rdt
� |d �|d< | �
|d �|d< |s~|�d�s~tj�tjj�|d< d|v r�|�d�d vr�|�dd� |S |�d�r�|d �d
d� |S ) a� Transform the given run object into a dictionary representation.
Args:
run (Union[ls_schemas.Run, dict]): The run object to transform.
update (Optional[bool]): Whether the payload is for an "update" event.
copy (Optional[bool]): Whether to deepcopy run inputs/outputs.
Returns:
dict: The transformed run object as a dictionary.
r� r� �inputsN�outputs�
start_time�
serialized�run_type)rw rz �graph)�hasattr�callable�getattrr� r r^ r� r] rT r_ rg � deepish_copy�_hide_run_inputs�_hide_run_outputsrU �datetime�now�timezone�utc�pop)r� r� r� r� �
run_creater< r< r= �_run_transformR s0
�zClient._run_transform�Sequence[dict]c st t �� }| D ]1}tt|�di ��}|�di �}i |�|�|d< |�di �� t �� }� �� fdd�|�� D �� qd S )N�extra�runtimer� c s i | ] \}}|� vr||�qS r< r< �ru �k�v�r� r< r= �
<dictcomp>� � z.Client._insert_runtime_env.<locals>.<dictcomp>)�ls_env�get_runtime_environmentr r� �
setdefault� get_langchain_env_var_metadatar� r� )r� � runtime_envr� � run_extrar� �langchain_metadatar< r� r= �_insert_runtime_env� s ��zClient._insert_runtime_env��patch�Iterable[dict]r �
list[dict]c C s� | j d u r t|�S |r+g }|D ]}t|d �}|| jvr"|�|� q| j�|� q|S g }|D ]*}|d |�d�krB|�d�| jvsIt�� | j k rO|�|� q/| j�t|d �� q/|S )Nr� �trace_id) r� r� rf r� �append�removerU r� �add)r� r� r �sampledr� �run_idr< r< r= �_filter_for_sampling� s$
zClient._filter_for_sampling)� project_name� revision_id� dangerously_allow_filesystemr� �Dict[str, Any]r� �
RUN_TYPE_Tr r r c
K s� |p |� dt�� �}i |�||||d��}| �|g�s dS |dur)||d d d< | j|dd�}| �|g� |�d �durX|d �� D ]} t| t �rWt| d
t
�rW|sWt d ��qC|�d �dur�|�d
�dur�| j durs| j �
|� dS | jdur�| jdu r�t d��td|�}
t|
�\} } | jj� t| | jt� | j jd
7 _| j�� W d � n1 s�w Y tt| �� �� dS | jdur�td|�}
| j�t|d
|
�� dS | � |� dS | � |� dS )a� Persist a run to the LangSmith API.
Args:
name (str): The name of the run.
inputs (Dict[str, Any]): The input values for the run.
run_type (str): The type of the run, such as tool, chain, llm, retriever,
embedding, prompt, or parser.
project_name (Optional[str]): The project name of the run.
revision_id (Optional[Union[UUID, str]]): The revision ID of the run.
**kwargs (Any): Additional keyword arguments.
Returns:
None
Raises:
LangSmithUserError: If the API key is not provided when using the hosted service.
Examples:
.. code-block:: python
from langsmith import Client
import datetime
from uuid import uuid4
client = Client()
run_id = uuid4()
client.create_run(
id=run_id,
project_name=project_name,
name="test_run",
run_type="llm",
inputs={"prompt": "hello world"},
outputs={"generation": "hi there"},
start_time=datetime.datetime.now(datetime.timezone.utc),
end_time=datetime.datetime.now(datetime.timezone.utc),
hide_inputs=True,
hide_outputs=True,
)
� session_name)r r� r� r� Nr� r� r F�r� � attachmentsr� �UMust set dangerously_allow_filesystem=True to allow passing in Paths for attachments.r � dotted_order�@Run compression is enabled but threading event is not configured�post) r� rg �get_tracer_projectr
r� r rU �valuesr] r� r r` r� �
create_runr� r� r2 r4 �lockr0 r( � trace_countr � _close_filesr� r� �putr" � _create_run)
r� r� r� r� r r r rf r� �
attachment�
serialized_op�multipart_form� opened_filesr< r< r= r � sz 2��� � ����
�
��
� �
�zClient.create_runr� c C sP | j �� D ] \}}i | j�t|i�}| jd|� d�t|�|d�tjfd� qd S )Nr� �/runs�rh rD )rZ r] )r� r� rE rC rJ r� rg r� )r� r� rS r� rD r< r< r= r! $ s ���zClient._create_runc C �D | j du ri S | jrt�t|��}| �|�S | j du r |S | � |�S �NTF)r� r� r! r� r� )r� r� � json_inputsr< r< r= r� 1 �
zClient._hide_run_inputsr� c C r( r) )r� r� r! r� r� )r� r� � json_outputsr< r< r= r� ; r+ zClient._hide_run_outputs�ops� List[SerializedRunOperation]c
C s g g d�}|D ]_}t |t�rSt�|j�}|jr t�|j�|d< |jr*t�|j�|d< |jr5t�|j�|d< |j r=t
� d� ||j �
d|j� d|j� �t�|�f� qt |t�r^t
� d� qt
�d t|�� q| j}|jpni �d
�pst}t�t�}t�t�}d } tt t d d d
g�D ]Y}
t� ||
�} | r�| d kr�| t | d d � |kr�| j t�|�d|
� dd�!||
�� �d� d } |�"� |�"� | �#� \} }
| t |
�7 } ||
�
t�|
�� ||
�
| � | s�q�| �rd�!dd� |�$� D ��}| j t�|�d| d� d S d S )N)r r r� r� �eventszBAttachments are not supported when use_multipart_endpoint is Falseztrace=z,id=z;Feedback operations are not supported in non-multipart modez&Unknown item type in tracing queue: %srS r r r r� rr rI �; )r_ c s s( � | ]\}}|� d d� |�� �V qdS )rI r0 N)r� r� r< r< r= � <genexpr>� s �& z/Client._batch_ingest_run_ops.<locals>.<genexpr>)%r] r. r! r� �_noner� �Fragmentr� r/ r r� r- � operationr r r� �dumpsr- �error�typer rT rU r) � collections� defaultdictr� r r r �dequere �_post_batch_ingest_runsr� �clear�popleftr� )r� r- �ids_and_partial_body�op� curr_dictr rS � body_chunks� context_ids� body_size�key�
body_deque�curr_id� curr_body�contextr< r< r= �_batch_ingest_run_opsE st ��
�
�
�
��
� ��
��z Client._batch_ingest_run_ops)� pre_sampled�create�GOptional[Sequence[Union[ls_schemas.Run, ls_schemas.RunLikeDict, Dict]]]rJ c s� |s|sdS � fdd�|pt D �}� fdd�|pt D �}|D ]}|�d�r*|�d�s/t�d��q |D ]}|�d�r>|�d�sCt�d��q2|sR� �|�}� j|dd �}|sX|sXdS � �|| � ttt t t
t � d
d � |D �d d � |D �����}� �
|� dS )
aq Batch ingest/upsert multiple runs in the Langsmith system.
Args:
create (Optional[Sequence[Union[Run, RunLikeDict]]]):
A sequence of `Run` objects or equivalent dictionaries representing
runs to be created / posted.
update (Optional[Sequence[Union[Run, RunLikeDict]]]):
A sequence of `Run` objects or equivalent dictionaries representing
runs that have already been created and should be updated / patched.
pre_sampled (bool, default=False): Whether the runs have already been subject
to sampling, and therefore should not be sampled again.
Defaults to False.
Raises:
LangsmithAPIError: If there is an error in the API request.
Returns:
None
Note:
- The run objects MUST contain the dotted_order and trace_id fields
to be accepted by the API.
Examples:
.. code-block:: python
from langsmith import Client
import datetime
from uuid import uuid4
client = Client()
_session = "__test_batch_ingest_runs"
trace_id = uuid4()
trace_id_2 = uuid4()
run_id_2 = uuid4()
current_time = datetime.datetime.now(datetime.timezone.utc).strftime(
"%Y%m%dT%H%M%S%fZ"
)
later_time = (
datetime.datetime.now(datetime.timezone.utc) + timedelta(seconds=1)
).strftime("%Y%m%dT%H%M%S%fZ")
runs_to_create = [
{
"id": str(trace_id),
"session_name": _session,
"name": "run 1",
"run_type": "chain",
"dotted_order": f"{current_time}{str(trace_id)}",
"trace_id": str(trace_id),
"inputs": {"input1": 1, "input2": 2},
"outputs": {"output1": 3, "output2": 4},
},
{
"id": str(trace_id_2),
"session_name": _session,
"name": "run 3",
"run_type": "chain",
"dotted_order": f"{current_time}{str(trace_id_2)}",
"trace_id": str(trace_id_2),
"inputs": {"input1": 1, "input2": 2},
"error": "error",
},
{
"id": str(run_id_2),
"session_name": _session,
"name": "run 2",
"run_type": "chain",
"dotted_order": f"{current_time}{str(trace_id)}."
f"{later_time}{str(run_id_2)}",
"trace_id": str(trace_id),
"parent_run_id": str(trace_id),
"inputs": {"input1": 5, "input2": 6},
},
]
runs_to_update = [
{
"id": str(run_id_2),
"dotted_order": f"{current_time}{str(trace_id)}."
f"{later_time}{str(run_id_2)}",
"trace_id": str(trace_id),
"parent_run_id": str(trace_id),
"outputs": {"output1": 4, "output2": 5},
},
]
client.batch_ingest_runs(create=runs_to_create, update=runs_to_update)
Nc � g | ] }� j |d d��qS )Fr �r� �ru r� r: r< r= rw � � �z,Client.batch_ingest_runs.<locals>.<listcomp>c s g | ]
}� j |d dd��qS )TF)r� r� rN rO r: r< r= rw � � ��r r z:Batch ingest requires trace_id and dotted_order to be set.Tr c s � � | ]}t d |�V qdS �r N�r2 rO r< r< r= r1 � � z+Client.batch_ingest_runs.<locals>.<genexpr>c s rR �r NrT rO r< r< r= r1 rU )rM rU rg rh r
r r r r. r/ r� � itertoolsrv rI )r� rK r� rJ � create_dicts� update_dictsr� �serialized_opsr< r: r= �batch_ingest_runs� sL b
�
�����
���� zClient.batch_ingest_runs�bytesc C s� | j �� D ]_\}}z | jd|� d�|i | j�t|i�d�tjfd|d� W q tyd } z3zt� t
|�|�}d� |�� � }t
�d|� �� W n tyY t
�dt|�� �� Y nw W Y d }~qd }~ww d S )Nr� z /runs/batchr' r| )rZ r] r[ r_ rY z Failed to batch ingest runs: )r� r� rJ rE rC rg r� rs � traceback�format_exception_onlyr7 r� r� r� r- r. )r� r� r_ rS r� r� �exc_desc_lines�exc_descr< r< r= r; s6 ���
� ����z Client._post_batch_ingest_runs�@list[Union[SerializedRunOperation, SerializedFeedbackOperation]]c
C s� g }i }|D ]-}t |t�r t|�\}}|�|� |�|� qt |t�r+|�t|�� qt�dt |�� qt
|�}|rTz| � |� W t t
|�� �� d S t t
|�� �� w d S )Nz+Unknown operation type in tracing queue: %s)r] r. r4 r r� r- r3 r� r6 r7 r, �_send_multipart_reqr r� r )r� r- �parts�opened_files_dictr? �partr% �
acc_multipartr< r< r= �_multipart_ingest_ops2 s&
�
� &�z Client._multipart_ingest_ops)rJ r c s� |s|sdS � fdd�|pt D �}� fdd�|pt D �}|r3|D ]}|�d�r,|�d�s1t�d��q ~|rJ|D ]}|�d�rC|�d�sHt�d��q7~|r�|r�d d
� |D �}g } |D ]#}|d |v rw|�� D ]\}
} | duru| ||d |
< qeqY| �|� qY~| }|s�� �|�}� j|d d
�}|s�|s�dS � �|� � �|� tt t
� dd� |D �dd� |D ����} | D ]$}
t |
t
�r�|
jr�|
j�� D ]}t |t�r�t |d t�r�|s�td��q�q�� �| � dS )a� Batch ingest/upsert multiple runs in the Langsmith system.
Args:
create (Optional[Sequence[Union[ls_schemas.Run, RunLikeDict]]]):
A sequence of `Run` objects or equivalent dictionaries representing
runs to be created / posted.
update (Optional[Sequence[Union[ls_schemas.Run, RunLikeDict]]]):
A sequence of `Run` objects or equivalent dictionaries representing
runs that have already been created and should be updated / patched.
pre_sampled (bool, default=False): Whether the runs have already been subject
to sampling, and therefore should not be sampled again.
Defaults to False.
Raises:
LangsmithAPIError: If there is an error in the API request.
Returns:
None
Note:
- The run objects MUST contain the dotted_order and trace_id fields
to be accepted by the API.
Examples:
.. code-block:: python
from langsmith import Client
import datetime
from uuid import uuid4
client = Client()
_session = "__test_batch_ingest_runs"
trace_id = uuid4()
trace_id_2 = uuid4()
run_id_2 = uuid4()
current_time = datetime.datetime.now(datetime.timezone.utc).strftime(
"%Y%m%dT%H%M%S%fZ"
)
later_time = (
datetime.datetime.now(datetime.timezone.utc) + timedelta(seconds=1)
).strftime("%Y%m%dT%H%M%S%fZ")
runs_to_create = [
{
"id": str(trace_id),
"session_name": _session,
"name": "run 1",
"run_type": "chain",
"dotted_order": f"{current_time}{str(trace_id)}",
"trace_id": str(trace_id),
"inputs": {"input1": 1, "input2": 2},
"outputs": {"output1": 3, "output2": 4},
},
{
"id": str(trace_id_2),
"session_name": _session,
"name": "run 3",
"run_type": "chain",
"dotted_order": f"{current_time}{str(trace_id_2)}",
"trace_id": str(trace_id_2),
"inputs": {"input1": 1, "input2": 2},
"error": "error",
},
{
"id": str(run_id_2),
"session_name": _session,
"name": "run 2",
"run_type": "chain",
"dotted_order": f"{current_time}{str(trace_id)}."
f"{later_time}{str(run_id_2)}",
"trace_id": str(trace_id),
"parent_run_id": str(trace_id),
"inputs": {"input1": 5, "input2": 6},
},
]
runs_to_update = [
{
"id": str(run_id_2),
"dotted_order": f"{current_time}{str(trace_id)}."
f"{later_time}{str(run_id_2)}",
"trace_id": str(trace_id),
"parent_run_id": str(trace_id),
"outputs": {"output1": 4, "output2": 5},
},
]
client.multipart_ingest(create=runs_to_create, update=runs_to_update)
Nc s g | ]}� � |��qS r< rN rO r: r< r= rw � � z+Client.multipart_ingest.<locals>.<listcomp>c rM )T)r� rN rO r: r< r= rw � rP r r zNMultipart ingest requires trace_id and dotted_order to be set in create dicts.zNMultipart ingest requires trace_id and dotted_order to be set in update dicts.c S s i | ]}|d |�qS �r� r< rO r< r< r= r� � rh z+Client.multipart_ingest.<locals>.<dictcomp>r� Tr c s rR rS rT rO r< r< r= r1 � rU z*Client.multipart_ingest.<locals>.<genexpr>c s rR rV rT rO r< r< r= r1 � rU r� r )rM rU rg rh r� r r
r r/ r� rW rv r] r. r r r� r r` rg )r� rK r� rJ r rX rY r� � create_by_id�standalone_updatesr� r� rZ r? r" r< r: r= �multipart_ingestK s� c
����� ��
��� � ����zClient.multipart_ingestr| ��attempts�accr+ rn c
C s� |j }|j}| j�� D ]�\}}td|d �D ]�}z0tj|td�}|jdkr*|� � } n|} | j
d|� d�| i | j �t |d|j
i�d�d|d� W nv tjyS Y nl tjtjtjfy} }
z||krmt�d |
� �� nW Y d }
~
qW Y d }
~
qd }
~
w ty� } z6zt�t| �| �} d
�| ��� }
t�d |
� �� W n ty� t�d t| �� �� Y nw W Y d } ~ d S d } ~ ww q d S ) Nr� ��boundary� -1r� �/runs/multipartrj r' �rZ r[ r_ z!Failed to multipart ingest runs: rY ) rc rH r� r� r� �rqtb_multipart�MultipartEncoderr( re � to_stringrJ rE rC � content_typerg r� r} r~ r r� r- rs r] r^ r7 r� r� r. )r� ro rn rc r_ rS r� r� �encoderrh �excr� r_ r` r< r< r= rb � sb
���� � �� �����zClient._send_multipart_req� data_stream�
io.BytesIO�compressed_traces_info�Optional[Tuple[int, int]]c
C s� d}| j �� D ]�\}}|�d� td|d �D ]�}z3i | j�|dt� �d|r,t|d �nd|r5t|d �ndd��}| jd|� d�||d �d|d
� W nv tj yW Y nl tj
tj tj fy� } z||krqt
�d | � �� nW Y d } ~ qW Y d } ~ qd } ~ w ty� }
z6zt�t|
�|
�} d�| ��� } t
�d | � �� W n ty� t
�d t|
�� �� Y nw W Y d }
~
d S d }
~
ww qd S )
zASend a zstd-compressed multipart form data stream to the backend.rY r r� z multipart/form-data; boundary=�zstd)z X-API-KEYrj zContent-EncodingzX-Pre-Compressed-SizezX-Post-Compressed-Sizer� rs r' rt z,Failed to send compressed multipart ingest: N)r� r� r� r� rE r( rT rJ rg r� r} r~ r r� r- rs r] r^ r7 r� r� r. )
r� r{ r} rn r_ rS r� r� rD rz r� r_ r` r< r< r= � _send_compressed_multipart_req( sv
������
�� ��� ������z%Client._send_compressed_multipart_req)
r� �end_timer6 r� r� r/ r� �tagsr r r r� r� �Optional[datetime.datetime]r6 �Optional[Dict]r/ �Optional[Sequence[dict]]r� r� �Optional[List[str]]r � Optional[ls_schemas.Attachments]c
K sX t |d�|| �dd�| �dd�| �dd�| || �dd�| �dd�d� }
|
rD|
�� D ]\}}t|t�r?t|d t�r?| s?td
��q)|
|
d < | jdusN| jduoY|
d duoY|
d du}| j |
gd d
�sddS |duro|�
� |
d< n t j � t j
j��
� |
d< |dur�||
d< |dur�| �|�|
d< |dur�|s�t�|�}| �|�|
d< |dur�||
d< |
d r�| �|
g� | jdur�| j�|
� dS |�r%td|
d�}| jdu�rt|�\}}| jj�% | jdu r�td��t|| jt� | j j d 7 _ | j� � W d � n 1 �sw Y t t |� � �� dS | jdu�r#| j�!t"|
d |�� dS dS | �#|
� dS )a Update a run in the LangSmith API.
Args:
run_id (Union[UUID, str]): The ID of the run to update.
name (Optional[str]): The name of the run.
end_time (Optional[datetime.datetime]): The end time of the run.
error (Optional[str]): The error message of the run.
inputs (Optional[Dict]): The input values for the run.
outputs (Optional[Dict]): The output values for the run.
events (Optional[Sequence[dict]]): The events for the run.
extra (Optional[Dict]): The extra information for the run.
tags (Optional[List[str]]): The tags for the run.
attachments (Optional[Dict[str, Attachment]]): A dictionary of attachments to add to the run. The keys are the attachment names,
and the values are Attachment objects containing the data and mime type.
**kwargs (Any): Kwargs are ignored.
Returns:
None
Examples:
.. code-block:: python
from langsmith import Client
import datetime
from uuid import uuid4
client = Client()
project_name = "__test_update_run"
start_time = datetime.datetime.now()
revision_id = uuid4()
run: dict = dict(
id=uuid4(),
name="test_run",
run_type="llm",
inputs={"text": "hello world"},
project_name=project_name,
api_url=os.getenv("LANGCHAIN_ENDPOINT"),
start_time=start_time,
extra={"extra": "extra"},
revision_id=revision_id,
)
# Create the run
client.create_run(**run)
run["outputs"] = {"output": ["Hi"]}
run["extra"]["foo"] = "bar"
run["name"] = "test_run_updated"
# Update the run
client.update_run(run["id"], **run)
r r N�
parent_run_idr �
session_idr ) r� r� r r� r r� r� r� r r� r r Tr r� r6 r� r� r/ r� r )r4 �payloadr )$rf r� r� r] r� r r` r� r� r
� isoformatr� r� r� r� r� rg r� r� r r� �
update_runr2 r4 r r� r0 r( r r r r� r r r" � _update_run)r� r r� r� r6 r� r� r/ r� r� r r rf rh �_r" �
use_multipartr# r$ r% r< r< r= r� n s� B
� � ����
�
�
��
�� � ��zClient.update_run�
run_updatec C sR | j �� D ]!\}}i | j�t|i�}| jd|� d|d � �t|�|d�d� qd S )N�PATCH�/runs/r� r' r� )r� r� rE rC rJ r� )r� r� rS r� rD r< r< r= r� � s �����zClient._update_runc C s� | j du rdS | jdu rtd��ddlm}m} || ddd�\}}|durKd}z|j| j|||d�}| j�|� W n t yJ | j|||d� Y nw | jr^t
� | j�\}}| j� |� dS dS )z3Force flush the currently buffered compressed runs.Nz>Run compression is enabled but request pool futures is not setr )�HTTP_REQUEST_THREAD_POOL�'_tracing_thread_drain_compressed_bufferr� )rW rS rm )
r� r� r` �&langsmith._internal._background_threadr� r� �submitr� r
� RuntimeError�cf�wait�difference_update) r� rn r� r� �final_data_streamr} �future�doner� r< r< r= �flush_compressed_traces s<
����
���z Client.flush_compressed_tracesc C s2 | j dur | �� dS | jdur| j�� dS dS )z;Flush either queue or compressed buffer, depending on mode.N)r� r� r� r� r: r< r< r= �flush9 s
�z Client.flush�ls_schemas.Runc C s� | j |jd�}t�t�}i }t|dd� d�D ] }|jdu r't�d|j � d���||j �
|� |||j < q|� |j g �|_ |�
� D ] \}}||| _ qA|S )z�Load child runs for a given run.
Args:
run (Run): The run to load child runs for.
Returns:
Run: The run with loaded child runs.
Raises:
LangSmithError: If a child run has no parent.
ri c S s | j S r� )r )�rr< r< r= �<lambda>S s z)Client._load_child_runs.<locals>.<lambda>)rD Nz
Child run z has no parent)� list_runs�
child_run_idsr8 r9 r� �sortedr� rg r� r� r r� �
child_runsr� )r� r� r� �treemapr� � child_runr �childrenr< r< r= �_load_child_runs@ s" �
�
zClient._load_child_runs�load_child_runsc C sb | � ddt|d�� ��}t|�� dd�}tjd d|i|�� �d| ji��}|r/|jr/| �|�}|S )
a< Read a run from the LangSmith API.
Args:
run_id (Union[UUID, str]):
The ID of the run to read.
load_child_runs (bool, default=False):
Whether to load nested child runs.
Returns:
Run: The run read from the LangSmith API.
Examples:
.. code-block:: python
from langsmith import Client
# Existing run
run_id = "your-run-id"
client = Client()
stored_run = client.read_run(run_id)
rG r� r �s3_urls��attachments_keyr r6 Nr< ) rJ rf �/_convert_stored_attachments_to_attachments_dictr� r �Runr6 r� r� )r� r r� rM r r� r< r< r= �read_run^ s �����
zClient.read_run)�
project_idr r� r �reference_example_id�query�filter� trace_filter� tree_filter�is_rootr� r� r6 �run_ids�selectr� r� �+Optional[Union[ID_TYPE, Sequence[ID_TYPE]]]�#Optional[Union[str, Sequence[str]]]r �Optional[ID_TYPE]r� r� r� r� r� r� �Optional[bool]r� r� r� �Optional[Sequence[ID_TYPE]]r� �Optional[Sequence[str]]r� �Iterator[ls_schemas.Run]c + s2 � g }t |tjtf�r|�|� n
t |t�r|�|� |dur3t |t�r'|g}|�� fdd�|D �� g d�}|p:|}|r?|nd||rF|gnd|||| |
| | rS| �� nd|
|||d�|�}dd� |�� D �}t � j
d|d ��D ]'\}}t |d
d �}t j
dd |i|�d
� ji��V |dur�|d |kr� dS qodS )a� List runs from the LangSmith API.
Args:
project_id (Optional[Union[UUID, str], Sequence[Union[UUID, str]]]):
The ID(s) of the project to filter by.
project_name (Optional[Union[str, Sequence[str]]]): The name(s) of the project to filter by.
run_type (Optional[str]): The type of the runs to filter by.
trace_id (Optional[Union[UUID, str]]): The ID of the trace to filter by.
reference_example_id (Optional[Union[UUID, str]]): The ID of the reference example to filter by.
query (Optional[str]): The query string to filter by.
filter (Optional[str]): The filter string to filter by.
trace_filter (Optional[str]): Filter to apply to the ROOT run in the trace tree. This is meant to
be used in conjunction with the regular `filter` parameter to let you
filter runs by attributes of the root run within a trace.
tree_filter (Optional[str]): Filter to apply to OTHER runs in the trace tree, including
sibling and child runs. This is meant to be used in conjunction with
the regular `filter` parameter to let you filter runs by attributes
of any run within a trace.
is_root (Optional[bool]): Whether to filter by root runs.
parent_run_id (Optional[Union[UUID, str]]):
The ID of the parent run to filter by.
start_time (Optional[datetime.datetime]):
The start time to filter by.
error (Optional[bool]): Whether to filter by error status.
run_ids (Optional[Sequence[Union[UUID, str]]]):
The IDs of the runs to filter by.
select (Optional[Sequence[str]]): The fields to select.
limit (Optional[int]): The maximum number of runs to return.
**kwargs (Any): Additional keyword arguments.
Yields:
The runs.
Examples:
.. code-block:: python
# List all runs in a project
project_runs = client.list_runs(project_name="<your_project>")
# List LLM and Chat runs in the last 24 hours
todays_llm_runs = client.list_runs(
project_name="<your_project>",
start_time=datetime.now() - timedelta(days=1),
run_type="llm",
)
# List root traces in a project
root_runs = client.list_runs(project_name="<your_project>", is_root=1)
# List runs without errors
correct_runs = client.list_runs(project_name="<your_project>", error=False)
# List runs and only return their inputs/outputs (to speed up the query)
input_output_runs = client.list_runs(
project_name="<your_project>", select=["inputs", "outputs"]
)
# List runs by run ID
run_ids = [
"a36092d2-4ad5-4fb4-9c0d-0dba9a2ed836",
"9398e6be-964f-4aa4-8ae9-ad78cd4b7074",
]
selected_runs = client.list_runs(id=run_ids)
# List all "chain" type runs that took more than 10 seconds and had
# `total_tokens` greater than 5000
chain_runs = client.list_runs(
project_name="<your_project>",
filter='and(eq(run_type, "chain"), gt(latency, 10), gt(total_tokens, 5000))',
)
# List all runs called "extractor" whose root of the trace was assigned feedback "user_score" score of 1
good_extractor_runs = client.list_runs(
project_name="<your_project>",
filter='eq(name, "extractor")',
trace_filter='and(eq(feedback_key, "user_score"), eq(feedback_score, 1))',
)
# List all runs that started after a specific timestamp and either have "error" not equal to null or a "Correctness" feedback score equal to 0
complex_runs = client.list_runs(
project_name="<your_project>",
filter='and(gt(start_time, "2023-07-15T12:34:56Z"), or(neq(error, null), and(eq(feedback_key, "Correctness"), eq(feedback_score, 0.0))))',
)
# List all runs where `tags` include "experimental" or "beta" and `latency` is greater than 2 seconds
tagged_runs = client.list_runs(
project_name="<your_project>",
filter='and(or(has(tags, "experimental"), has(tags, "beta")), gt(latency, 2))',
)
Nc s g | ] }� j |d �j�qS ��r )� read_projectr� �ru r� r: r< r= rw � s z$Client.list_runs.<locals>.<listcomp>) �app_pathr� �completion_cost�completion_tokensr r� r6 r/ r� �feedback_stats�first_token_timer� r� r� r� r� �parent_run_ids� prompt_cost�
prompt_tokensr� r� r� r� �statusr� �
total_cost� total_tokensr )r� r� �reference_exampler� r� r� r� r� �
parent_runr� r6 r� �tracer� c S � i | ] \}}|d ur||�qS r� r< r� r< r< r= r� / r� z$Client.list_runs.<locals>.<dictcomp>� /runs/query�r� r� r� r r6 r� r< )r] r^ r_ rT r r� �extendr� r� � enumerater� r� r r� r6 )r� r� r r� r r� r� r� r� r� r� r� r� r6 r� r� r� rf � project_ids�default_select�
body_query�ir� r r< r: r= r� � sb �o
�
�� ����
���zClient.list_runs)r� r� r� r� �
project_namesr� �reference_example_idsr� r� r6 r� r� r� r� r� �data_source_typer� �Optional[List[ID_TYPE]]r� r� r� r� r� r� c s� ddl m}m} |p g }|r:|� �!� � �fdd�|D �}||�D ]
}|�|�� j� q W d � n1 s5w Y |||||||| |
| | |
|||d�}dd� |�� D �}�jd d
d t|�id �}t �
|� |� � S )
a� Get aggregate statistics over queried runs.
Takes in similar query parameters to `list_runs` and returns statistics
based on the runs that match the query.
Args:
id (Optional[List[Union[UUID, str]]]): List of run IDs to filter by.
trace (Optional[Union[UUID, str]]): Trace ID to filter by.
parent_run (Optional[Union[UUID, str]]): Parent run ID to filter by.
run_type (Optional[str]): Run type to filter by.
project_names (Optional[List[str]]): List of project names to filter by.
project_ids (Optional[List[Union[UUID, str]]]): List of project IDs to filter by.
reference_example_ids (Optional[List[Union[UUID, str]]]): List of reference example IDs to filter by.
start_time (Optional[str]): Start time to filter by.
end_time (Optional[str]): End time to filter by.
error (Optional[bool]): Filter by error status.
query (Optional[str]): Query string to filter by.
filter (Optional[str]): Filter string to apply.
trace_filter (Optional[str]): Trace filter string to apply.
tree_filter (Optional[str]): Tree filter string to apply.
is_root (Optional[bool]): Filter by root run status.
data_source_type (Optional[str]): Data source type to filter by.
Returns:
Dict[str, Any]: A dictionary containing the run statistics.
r ��ThreadPoolExecutor� as_completedc s g | ]
}� j �j|d ��qS r� )r� r� r� ��executorr� r< r= rw p rQ z(Client.get_run_stats.<locals>.<listcomp>N)r� r� r� r� r� r� r� r� r6 r� r� r� r� r� r� c S r� r� r< r� r< r< r= r� � r� z(Client.get_run_stats.<locals>.<dictcomp>r� z /runs/statsrh r� ) �concurrent.futuresr� r� r r� r� r� rJ r� rg rK r� )r� r� r� r� r� r� r� r� r� r� r6 r� r� r� r� r� r� r� r� �futuresr� r� rM r< r� r= �
get_run_stats= sH . � � ����
zClient.get_run_stats)r r� �ls_schemas.RunBasec C s� t |dd� }r n.t |dd� }r| j|d�j}n |dur |}n|dur,| j|d�j}n t�� }| j|d�j}t|d�}| j� d| �� � d|� d|j� d�S ) a� Get the URL for a run.
Not recommended for use within your agent runtime.
More for use interacting with runs after the fact
for data analysis or ETL workloads.
Args:
run (RunBase): The run.
project_name (Optional[str]): The name of the project.
project_id (Optional[Union[UUID, str]]): The ID of the project.
Returns:
str: The URL for the run.
r� Nr r� z/o/z /projects/p/z/r/z
?poll=true)r� r� r� rg r rf r6 �_get_tenant_id)r� r� r r� r� r � session_id_r< r< r= � get_run_url� s
��zClient.get_run_url)�share_idr� c C sh t |d�}t|�|ptt�� �d�}| jdd|� d�| j|d�}t�|� |�� d }| j � d|� d �S )
a: Get a share link for a run.
Args:
run_id (Union[UUID, str]): The ID of the run to share.
share_id (Optional[Union[UUID, str]]): Custom share ID.
If not provided, a random UUID will be generated.
Returns:
str: The URL of the shared run.
r )r � share_token�PUTr� �/share�rD r� r� �/public/�/r)
rf rT r^ r� rJ rE rg rK r� r6 )r� r r� �run_id_rh rM r� r< r< r= � share_run� s
�
�
zClient.share_runc C �. | j ddt|d�� d�| jd�}t�|� dS )z�Delete share link for a run.
Args:
run_id (Union[UUID, str]): The ID of the run to unshare.
Returns:
None
�DELETEr� r r� �rD N�rJ rf rE rg rK )r� r rM r< r< r= � unshare_run� � �zClient.unshare_runc C s\ | j ddt|d�� d�| jd�}t�|� |�� }|du s!d|vr#dS | j� d|d � d �S )
z�Retrieve the shared link for a specific run.
Args:
run_id (Union[UUID, str]): The ID of the run.
Returns:
Optional[str]: The shared link for the run, or None if the link is not
available.
rG r� r r� r� Nr� r� r� )rJ rf rE rg rK r� r6 )r� r rM r� r< r< r= �read_run_shared_link� s
�
zClient.read_run_shared_linkc C s | � t|d��}|duS )z�Get share state for a run.
Args:
run_id (Union[UUID, str]): The ID of the run.
Returns:
bool: True if the run is shared, False otherwise.
r N)r� rf )r� r r7 r< r< r= �
run_is_shared� s zClient.run_is_sharedr� �Union[ID_TYPE, str]c C st t |ddd�\}}d|� d�}|dur |dt|d�� �7 }| jd || jd
�}t�|� tjd i |�� �d | j i��S )
aQ Get shared runs.
Args:
share_token (Union[UUID, str]): The share token or URL of the shared run.
run_id (Optional[Union[UUID, str]]): The ID of the specific run to retrieve.
If not provided, the full shared run will be returned.
Returns:
Run: The shared run.
rY r� �rW r� �/runNrZ r rG r� r6 r< )
rm rf rJ rE rg rK r r� r� r6 )r� r� r r� rl rc rM r< r< r= �read_shared_run
s
�
zClient.read_shared_runc c s` � |rd|ini }t |ddd�\}}| jd|� d�|d�D ]}tjd
i |�d| ji��V q d S ) a Get shared runs.
Args:
share_token (Union[UUID, str]): The share token or URL of the shared run.
run_ids (Optional[List[str]]): A list of run IDs to filter the results by.
Yields:
A shared run.
r� rY r� r� r� r� r� r6 Nr< )rm r� r r� r6 )r� r� r� r� r� rl r� r< r< r= �list_shared_runs
s �
� �zClient.list_shared_runs�� dataset_name�
dataset_idr � ls_schemas.DatasetShareSchemac C s� |du r |du r t d��|du r| j|d�j}| jddt|d�� d�| jd�}t�|� |�� }t t
j i |�d | j � d
t|d d �� d
�i��S )a Retrieve the shared schema of a dataset.
Args:
dataset_id (Optional[Union[UUID, str]]): The ID of the dataset.
Either `dataset_id` or `dataset_name` must be given.
dataset_name (Optional[str]): The name of the dataset.
Either `dataset_id` or `dataset_name` must be given.
Returns:
ls_schemas.DatasetShareSchema: The shared schema of the dataset.
Raises:
ValueError: If neither `dataset_id` nor `dataset_name` is given.
N�/Either dataset_id or dataset_name must be givenr rG �
/datasets/r r� r� rn r� r� zresponse.share_token�/d)
r` � read_datasetr� rJ rf rE rg rK r� r r �DatasetShareSchemar6 )r� r r rM �dr< r< r= �read_dataset_shared_schema2
s* �
�
���z!Client.read_dataset_shared_schemac C s� |du r |du r t d��|du r| j|d�j}dt|�i}| jddt|d�� d�| j|d�}t�|� |� � }t
t j i |�d | j
� d
|d � d �i��S )
a
Get a share link for a dataset.
Args:
dataset_id (Optional[Union[UUID, str]]): The ID of the dataset.
Either `dataset_id` or `dataset_name` must be given.
dataset_name (Optional[str]): The name of the dataset.
Either `dataset_id` or `dataset_name` must be given.
Returns:
ls_schemas.DatasetShareSchema: The shared schema of the dataset.
Raises:
ValueError: If neither `dataset_id` nor `dataset_name` is given.
Nr r r r� r r� r� rn r� r� r )r` r
r� rT rJ rf rE rg rK r� r r r r6 )r� r r rh rM r r< r< r= �
share_datasetZ
s$ ��
�zClient.share_datasetc C r� )z�Delete share link for a dataset.
Args:
dataset_id (Union[UUID, str]): The ID of the dataset to unshare.
Returns:
None
r� r r r� r� Nr� )r� r rM r< r< r= �unshare_dataset�
r� zClient.unshare_datasetc C s\ t || j�\}}| jdd|� d�| jd�}t�|� tjdi |�� �| j d|� d�d���S ) z�Get shared datasets.
Args:
share_token (Union[UUID, str]): The share token or URL of the shared dataset.
Returns:
Dataset: The shared dataset.
rG r� � /datasetsr� r )r6 � _public_pathNr< )
rm rS rJ rE rg rK r r� r� r6 )r� r� r� rl rM r< r< r= �read_shared_dataset�
s
�
�
�zClient.read_shared_dataset�� example_idsr �List[ls_schemas.Example]c s` i }|durdd� |D �|d< � j ddt|d�� d�� j|d �}t�|� � fd
d�|�� D �S ) aR Get shared examples.
Args:
share_token (Union[UUID, str]): The share token or URL of the shared dataset.
example_ids (Optional[List[UUID, str]], optional): The IDs of the examples to filter by. Defaults to None.
Returns:
List[ls_schemas.Example]: The list of shared examples.
Nc S rs r< rt )ru r� r< r< r= rw �
rx z/Client.list_shared_examples.<locals>.<listcomp>r� rG r� r� � /examples�rD r� c s& g | ]}t jdi |�d � ji���qS )r6 r< )r �Exampler6 )ru rP r: r< r= rw �
s ��)rJ rf rE rg rK r� )r� r� r r� rM r< r: r= �list_shared_examples�
s �
�zClient.list_shared_examples)r� r� �
name_containsr� �dataset_share_tokenr �(Iterator[ls_schemas.TracerSessionResult]c
c sv � |||d�}t |d�}t| jd|� d�|d��D ] \}} tjd i | �d| ji��V |dur8|d |kr8 dS qdS )
a< List shared projects.
Args:
dataset_share_token (str): The share token of the dataset.
project_ids (Optional[List[Union[UUID, str]]]): List of project IDs to filter the results, by default None.
name (Optional[str]): Name of the project to filter the results, by default None.
name_contains (Optional[str]): Substring to search for in project names, by default None.
limit (Optional[int]): Maximum number of projects to return, by default None.
Yields:
The shared projects.
)r� r� r r r� z/datasets/sessionsr� r6 Nr� r< )rf r� r� r �TracerSessionResultr6 )
r� r r� r� r r� r� r� r� �projectr< r< r= �list_shared_projects�
s �
� � ��zClient.list_shared_projects)r� r� �upsert�
project_extra�reference_dataset_idr� r r! r" �ls_schemas.TracerSessionc C s� | j � d�}|}|ri |pi �d|i�}|||tt�� �d�} i }
|r'd|
d< |dur/|| d< | jd|i | j�d d
i�t| �d �} t�| � t j
d
i | � � �d | j i��S )a� Create a project on the LangSmith API.
Args:
project_name (str): The name of the project.
project_extra (Optional[dict]): Additional project information.
metadata (Optional[dict]): Additional metadata to associate with the project.
description (Optional[str]): The description of the project.
upsert (bool, default=False): Whether to update the project if it already exists.
reference_dataset_id (Optional[Union[UUID, str]): The ID of the reference dataset to associate with the project.
Returns:
TracerSession: The created project.
� /sessionsr� )r� r� r� r� Tr Nr" r� rj r? �rD rh r6 r< )
rS rT r^ r� rJ rE r� rg rK r �
TracerSessionr� r6 ) r� r r� r� r r! r" �endpointr� r� r� rM r< r< r= �create_project�
s,
��
zClient.create_project)r� r� r� r! r� c C s� | j � dt|d�� �}|}|ri |pi �d|i�}||||r"|�� ndd�} | jd|i | j�ddi�t| �d �}
t�|
� tj d i |
�
� �d
| j i��S ) a, Update a LangSmith project.
Args:
project_id (Union[UUID, str]):
The ID of the project to update.
name (Optional[str]):
The new name to give the project. This is only valid if the project
has been assigned an end_time, meaning it has been completed/closed.
description (Optional[str]):
The new description to give the project.
metadata (Optional[dict]):
Additional metadata to associate with the project.
project_extra (Optional[dict]):
Additional project information.
end_time (Optional[datetime.datetime]):
The time the project was completed.
Returns:
TracerSession: The updated project.
�
/sessions/r� r� N)r� r� r� r� r� rj r? r% r6 r< ) rS rf r� rJ rE r� rg rK r r& r� r6 ) r� r� r� r� r� r! r� r' r� r� rM r< r< r= �update_project s" ��
zClient.update_projectr� c
C s� | j d ur| j S z6| jddddid�}|�� }t|t�r9t|�dkr<tjd i |d �d| ji��}|j | _ | j W S W d S W d S t
y[ } zt j dt
|�d d
� W Y d }~d S d }~ww ) NrG r$ r� r� r� r r6 z*Failed to get tenant ID from LangSmith: %sT��exc_infor< )r� rJ r� r] r� re r r r6 � tenant_idrs r� r� r. )r� rM r� �tracer_sessionr� r< r< r= r� G s4
����
�
�
���z Client._get_optional_tenant_idr� c C s | � � }|d u r
t�d��|S )NzNo tenant ID found)r� rg r� )r� r- r< r< r= r� [ s
zClient._get_tenant_id)r� r )r� r �
include_statsr/ � ls_schemas.TracerSessionResultc C s� d}ddi}|dur|dt |d�� �7 }n
|dur ||d< ntd��||d < | jd
||d �}|�� }t|t�rUt|�d krFt�d
|� d���t j
di |d �d| j i��S t j
di |�� �d| j i��S )a� Read a project from the LangSmith API.
Args:
project_id (Optional[str]):
The ID of the project to read.
project_name (Optional[str]): The name of the project to read.
Only one of project_id or project_name may be given.
include_stats (bool, default=False):
Whether to include a project's aggregate statistics in the response.
Returns:
TracerSessionResult: The project.
r$ r� r� NrZ r� r� �'Must provide project_name or project_idr/ rG r� r zProject �
not foundr6 r< ) rf r` rJ r� r] r� re rg r� r r r6 )r� r� r r/ rc r� rM r� r< r< r= r� a s*
� ��zClient.read_project�r� c C s* z | j |d� W dS tjy Y dS w )a$ Check if a project exists.
Args:
project_name (str):
The name of the project to check for.
project_id (Optional[str]):
The ID of the project to check for.
Returns:
bool: Whether the project exists.
r� FT)r� rg r� )r� r r� r< r< r= � has_project� s ��zClient.has_projectc s� t jdtdd� ddlm}m} ddl}� j||dg d�d �}g }g }� fd
d �} d }
d} |� ��} g }
|D ]x}d
|jidd� |j �
� D ��dd� |j pLi �
� D ��|j r\|j |j
�� nd|j|jd��}|jru|�dd� |j�
� D �� |jr|�|j� n
t�d|j� d�� t|�|
dkr�|| | |
� }r�|
�| �| |�� | |
7 } |�|� q5|| d� r�|
�| �| || d� �� W d � n1 s�w Y |�|��d
�}dd� ||
�D �}|r�|�|��d
�}|j|ddd�}|�|j dd��S )a� Read the record-level information from an experiment into a Pandas DF.
Note: this will fetch whatever data exists in the DB. Results are not
immediately available in the DB upon evaluation run completion.
Args:
project_id (Optional[Union[UUID, str]]): The ID of the project.
project_name (Optional[str]): The name of the project.
Returns:
pd.DataFrame: A dataframe containing the test results.
z%Function get_test_results is in beta.rO )�
stacklevelr r� NT)r� r� r� r� r6 r� r� r� )r� r r� r� c s � j | d�}dd� |D �S )Nr c S s. g | ]}d |j idd� |jpi �� D ���qS )�
example_idc S � i | ]
\}}d |� �|�qS )z
reference.r< r� r< r< r= r� � � zNClient.get_test_results.<locals>.fetch_examples.<locals>.<listcomp>.<dictcomp>)r� r� r� �ru �exampler< r< r= rw � s ����zCClient.get_test_results.<locals>.fetch_examples.<locals>.<listcomp>)�
list_examples)�batch�examplesr: r< r= �fetch_examples� s �z/Client.get_test_results.<locals>.fetch_examples�2 r6 c S r7 )zinput.r< r� r< r< r= r� � r8 z+Client.get_test_results.<locals>.<dictcomp>c S r7 )zoutputs.r< r� r< r< r= r� � r8 )�execution_timer6 r� c S s" i | ]
\}}d |� �|� d��qS )z feedback.�avg)rU r� r< r< r= r� � s ��zRun z has no reference example ID.c S s g | ] }|� � D ]}|�qqS r< )r� )ru r� �outputr< r< r= rw � s
��z+Client.get_test_results.<locals>.<listcomp>)�
left_index� right_index�records)�orient) r� r� � UserWarningr� r� r� �pandasr� r� r� r� r� r� r� �
total_secondsr6 r� r� r� r r� r- re r� � DataFrame� set_index�merge�json_normalize�to_dict)r� r� r r� r� �pdr� �resultsr r> r3 r� r� r� r� �rowr< � result_df�example_outputs�
example_dfr< r: r= �get_test_results� sr ��
����� �� � �$�zClient.get_test_results�reference_dataset_name�reference_free�Optional[Dict[str, Any]]�"Iterator[ls_schemas.TracerSession]c c s � d|dur t |d�ndi} |dur|| d< |dur || d< |dur&|| d< |dur7|dur2td��|| d< n|durF| j|d �j}|| d< |durN|| d
< |durYt�|�| d < t| jd | d
��D ] \}
} tj di | �d| j
i��V |dur�|
d |kr� dS qbdS )a| List projects from the LangSmith API.
Args:
project_ids (Optional[List[Union[UUID, str]]]):
A list of project IDs to filter by, by default None
name (Optional[str]):
The name of the project to filter by, by default None
name_contains (Optional[str]):
A string to search for in the project name, by default None
reference_dataset_id (Optional[List[Union[UUID, str]]]):
A dataset ID to filter by, by default None
reference_dataset_name (Optional[str]):
The name of the reference dataset to filter by, by default None
reference_free (Optional[bool]):
Whether to filter for only projects not associated with a dataset.
limit (Optional[int]):
The maximum number of projects to return, by default None
metadata (Optional[Dict[str, Any]]):
Metadata to filter by.
Yields:
The projects.
Raises:
ValueError: If both reference_dataset_id and reference_dataset_name are given.
r� Nr r� r� r zGOnly one of reference_dataset_id or reference_dataset_name may be given�reference_datasetr rW r� r$ r� r6 r� r< ) �minr` r
r� r� r5 r� r� r r& r6 ) r� r� r� r r" rV rW r� r� r� r� r r< r< r= �
list_projects sD �&��
�� � ��zClient.list_projectsc C sX |durt | j|d�j�}n|du rtd��| jddt|d�� �| jd�}t�|� dS )am Delete a project from LangSmith.
Args:
project_name (Optional[str]):
The name of the project to delete.
project_id (Optional[str]):
The ID of the project to delete.
Returns:
None
Raises:
ValueError: If neither project_name or project_id is provided.
Nr� r1 r� r) r� r� ) rT r� r� r` rJ rf rE rg rK )r� r r� rM r< r< r= �delete_projectJ s �zClient.delete_project)r� r� �
inputs_schema�outputs_schema�transformationsr� �ls_schemas.DataTyper^ r_ r` �0Optional[List[ls_schemas.DatasetTransformation]]c
C s� ||j ||r
d|indd�}|dur||d< |dur ||d< |dur&||d< | jddi | j�d d
i�t�|�d �} t�| � tjd
i | � � �| j
| � � d ���S )a� Create a dataset in the LangSmith API.
Args:
dataset_name (str):
The name of the dataset.
description (Optional[str]):
The description of the dataset.
data_type (DataType, default=DataType.kv):
The data type of the dataset.
inputs_schema (Optional[Dict[str, Any]]):
The schema definition for the inputs of the dataset.
outputs_schema (Optional[Dict[str, Any]]):
The schema definition for the outputs of the dataset.
transformations (Optional[List[DatasetTransformation]]):
A list of transformations to apply to the dataset.
metadata (Optional[dict]):
Additional metadata to associate with the dataset.
Returns:
Dataset: The created dataset.
Raises:
requests.HTTPError: If the request to create the dataset fails.
r� N)r� r� r` r� r� �inputs_schema_definition�outputs_schema_definitionr� r rj r? r% r� r< ) r� rJ rE r! r5 rg rK r r� r� r6 r� )
r� r r� r� r^ r_ r` r� rP rM r< r< r= �create_datasetg s0 $��
�
�zClient.create_dataset�r r c C s, z
| j ||d� W dS tjy Y dS w )aG Check whether a dataset exists in your tenant.
Args:
dataset_name (Optional[str]):
The name of the dataset to check.
dataset_id (Optional[Union[UUID, str]]):
The ID of the dataset to check.
Returns:
bool: Whether the dataset exists.
rf TF)r
rg r� )r� r r r< r< r= � has_dataset� s �zClient.has_datasetc C s� d}ddi}|dur|dt |d�� �7 }n
|dur ||d< ntd��| jd ||d
�}|�� }t|t�rTt|�d krBt�d |� d
���t j
di |d �| j | � � d���S t j
di |�| j | � � d���S )a1 Read a dataset from the LangSmith API.
Args:
dataset_name (Optional[str]):
The name of the dataset to read.
dataset_id (Optional[Union[UUID, str]]):
The ID of the dataset to read.
Returns:
Dataset: The dataset.
r r� r� NrZ r r� �'Must provide dataset_name or dataset_idrG r� r r� r2 r� r< )
rf r` rJ r� r] r� re rg r� r r� r6 r� )r� r r rc r� rM r� r< r< r= r
� s<
�
��
��
�zClient.read_dataset� from_version� Union[str, datetime.datetime]�
to_version�ls_schemas.DatasetDiffInfoc C s� |du r|du r t d��| j|d�j}t|d�}| jdd|� d�| jt|tj�r,|�� n|t|tj�r7|�� n|d�d �}t �
|� t j d
i |�
� ��S ) as Get the difference between two versions of a dataset.
Args:
dataset_id (Optional[Union[UUID, str]]):
The ID of the dataset.
dataset_name (Optional[str]):
The name of the dataset.
from_version (Union[str, datetime.datetime]):
The starting version for the diff.
to_version (Union[str, datetime.datetime]):
The ending version for the diff.
Returns:
DatasetDiffInfo: The difference between the two versions of the dataset.
Examples:
.. code-block:: python
# Get the difference between two tagged versions of a dataset
from_version = "prod"
to_version = "dev"
diff = client.diff_dataset_versions(
dataset_name="my-dataset",
from_version=from_version,
to_version=to_version,
)
# Get the difference between two timestamped versions of a dataset
from_version = datetime.datetime(2024, 1, 1)
to_version = datetime.datetime(2024, 2, 1)
diff = client.diff_dataset_versions(
dataset_name="my-dataset",
from_version=from_version,
to_version=to_version,
)
N�&Must provide either dataset name or IDr r rG r z/versions/diff)ri rk r r< )r` r
r� rf rJ rE r] r� r� rg rK r �DatasetDiffInfor� )r� r r ri rk �dsidrM r< r< r= �diff_dataset_versions� s& ,
�
���
z Client.diff_dataset_versionsr� c C sj d}|durn|dur| j |d�j}ntd��| �d|� dt|d�� d��}d d
� |j�� �d �D �}|S ) a� Download a dataset in OpenAI Jsonl format and load it as a list of dicts.
Args:
dataset_id (Optional[Union[UUID, str]]):
The ID of the dataset to download.
dataset_name (Optional[str]):
The name of the dataset to download.
Returns:
list[dict]: The dataset loaded as a list of dicts.
Raises:
ValueError: If neither dataset_id nor dataset_name is provided.
r Nr rh rG rZ r z
/openai_ftc S s g | ]}t �|��qS r< )r� r� )ru �liner< r< r= rw R
rh z9Client.read_dataset_openai_finetuning.<locals>.<listcomp>rr )r
r� r` rJ rf �textr� rd )r� r r rc rM rP r< r< r= � read_dataset_openai_finetuning3
s �z%Client.read_dataset_openai_finetuning)� dataset_idsr� r �dataset_name_containsr� r� rt ru � Iterator[ls_schemas.Dataset]c
c s� � d|dur t |d�ndi}|dur||d< |dur ||d< |dur&||d< |dur.||d< |dur9t�|�|d< t| jd |d
��D ]"\}} tjd
i | �| j| �� d ���V |durd|d |krd dS qBdS )a List the datasets on the LangSmith API.
Args:
dataset_ids (Optional[List[Union[UUID, str]]]):
A list of dataset IDs to filter the results by.
data_type (Optional[str]):
The data type of the datasets to filter the results by.
dataset_name (Optional[str]):
The name of the dataset to filter the results by.
dataset_name_contains (Optional[str]):
A substring to search for in the dataset names.
metadata (Optional[Dict[str, Any]]):
A dictionary of metadata to filter the results by.
limit (Optional[int]):
The maximum number of datasets to return.
Yields:
The datasets.
r� Nr r� r� r� r r� r r� r� r� r< ) r[ r� r5 r� r� r r� r6 r� )
r� rt r� r ru r� r� r� r� rP r< r< r= �
list_datasetsU
s4 � � �� ���zClient.list_datasets)r r c C sR |dur | j |d�j}|du rtd��| jddt|d�� �| jd�}t�|� dS )a& Delete a dataset from the LangSmith API.
Args:
dataset_id (Optional[Union[UUID, str]]):
The ID of the dataset to delete.
dataset_name (Optional[str]):
The name of the dataset to delete.
Returns:
None
Nr rm r� r r r� )r
r� r` rJ rf rE rg rK )r� r r rM r< r< r= �delete_dataset�
s �zClient.delete_dataset�as_of�datetime.datetime�tagc C s` |dur | j |d�j}|du rtd��| jddt|d�� d�| j|�� |d�d �}t�|� dS )
a� Update the tags of a dataset.
If the tag is already assigned to a different version of this dataset,
the tag will be moved to the new version. The as_of parameter is used to
determine which version of the dataset to apply the new tags to.
It must be an exact version of the dataset to succeed. You can
use the read_dataset_version method to find the exact version
to apply the tags to.
Args:
dataset_id (Optional[Union[UUID, str]]):
The ID of the dataset to update.
dataset_name (Optional[str]):
The name of the dataset to update.
as_of (datetime.datetime):
The timestamp of the dataset to apply the new tags to.
tag (str):
The new tag to apply to the dataset.
Returns:
None
Examples:
.. code-block:: python
dataset_name = "my-dataset"
# Get the version of a dataset <= a given timestamp
dataset_version = client.read_dataset_version(
dataset_name=dataset_name, as_of=datetime.datetime(2024, 1, 1)
)
# Assign that version a new tag
client.update_dataset_tags(
dataset_name="my-dataset",
as_of=dataset_version.as_of,
tag="prod",
)
Nr rm r� r r z/tags�ry r{ r� ) r
r� r` rJ rf rE r� rg rK �r� r r ry r{ rM r< r< r= �update_dataset_tag�
s -�� zClient.update_dataset_tag)r r �searchr� r �#Iterator[ls_schemas.DatasetVersion]c c s� � |du r | j |d�j}||durt|d�ndd�}t| jdt|d�� d�|d��D ]\}}tjd
i |��V |durD|d |krD dS q*dS ) as List dataset versions.
Args:
dataset_id (Optional[Union[UUID, str]]): The ID of the dataset.
dataset_name (Optional[str]): The name of the dataset.
search (Optional[str]): The search query.
limit (Optional[int]): The maximum number of versions to return.
Yields:
The dataset versions.
Nr r )r r� r r z /versionsr� r� r< )r
r� r[ r� r� rf r �DatasetVersion)r� r r r r� r� r� r� r< r< r= �list_dataset_versions�
s" ��� ���z Client.list_dataset_versions)r r ry r{ �ls_schemas.DatasetVersionc C sn |du r | j |d�j}|r|s|du r|du rtd��| jddt|d�� d�||d�d �}tjd
i |�� ��S ) a[ Get dataset version by as_of or exact tag.
Ues this to resolve the nearest version to a given timestamp or for a given tag.
Args:
dataset_id (Optional[ID_TYPE]): The ID of the dataset.
dataset_name (Optional[str]): The name of the dataset.
as_of (Optional[datetime.datetime]): The timestamp of the dataset
to retrieve.
tag (Optional[str]): The tag of the dataset to retrieve.
Returns:
DatasetVersion: The dataset version.
Examples:
.. code-block:: python
# Get the latest version of a dataset
client.read_dataset_version(dataset_name="my-dataset", tag="latest")
# Get the version of a dataset <= a given timestamp
client.read_dataset_version(
dataset_name="my-dataset",
as_of=datetime.datetime(2024, 1, 1),
)
# Get the version of a dataset with a specific tag
client.read_dataset_version(dataset_name="my-dataset", tag="prod")
Nr z/Exactly one of as_of and tag must be specified.rG r r z/versionr| r� r< )r
r� r` rJ rf r r� r� r} r< r< r= �read_dataset_version s &�zClient.read_dataset_version)�source_api_urlr � token_or_urlr� c
C s |p| j }t||�\}}t|dd�}|�|�}|p|j}z| j|d�}t�d|� d�� |W S tj y8 Y nw zKt
|� |��}| j ||j
|jpLtjj|j|j|jd�}z| jdd� |D �d d� |D �|jd
� W W ~|S ty� } z t�d |� d �� | �d
} ~ ww ~w )a� Clone a public dataset to your own langsmith tenant.
This operation is idempotent. If you already have a dataset with the given name,
this function will do nothing.
Args:
token_or_url (str): The token of the public dataset to clone.
source_api_url (Optional[str]): The URL of the langsmith server where the data is hosted.
Defaults to the API URL of your current client.
dataset_name (Optional[str]): The name of the dataset to create in your tenant.
Defaults to the name of the public dataset.
Returns:
Dataset: The cloned dataset.
� placeholderr� r r� z) already exists in your tenant. Skipping.)r r� r� r^ r_ r` c S � g | ]}|j �qS r< )r� �ru r� r< r< r= rw n � z/Client.clone_public_dataset.<locals>.<listcomp>c S r� r< )r� r� r< r< r= rw o r� )r� r� r z)An error occurred while creating dataset z . You should delete it manually.N)rS rm r� r r� r
r� r rg r� r� r re r� r� r �DataType�kvr^ r_ r` �create_examplesr� rL r6 )
r� r� r� r rl �
source_client�dsr= rP r� r< r< r= �clone_public_dataset7 sV
�
�� �
��
��� zClient.clone_public_datasetc C s | j |d�}|jS )N�r )r
r� )r� r rP r< r< r= r* � s zClient._get_data_typerz �
generation�
created_at�ls_schemas.Examplec C s | j d|id|i|||d�S )aF Add an example (row) to an LLM-type dataset.
Args:
prompt (str):
The input prompt for the example.
generation (Optional[str]):
The output generation for the example.
dataset_id (Optional[Union[UUID, str]]):
The ID of the dataset.
dataset_name (Optional[str]):
The name of the dataset.
created_at (Optional[datetime.datetime]):
The creation timestamp of the example.
Returns:
Example: The created example
�inputrB �r� r� r r r� )�create_example)r� rz r� r r r� r< r< r= �create_llm_example� s �zClient.create_llm_example�messages�:List[Union[Mapping[str, Any], ls_schemas.BaseMessageLike]]� generations�>Optional[Union[Mapping[str, Any], ls_schemas.BaseMessageLike]]c C s� g }|D ] }t �|�r|�t �ttj|��� q|�tt|�� qd}|dur;t �|�r6t �ttj|��}ntt|�}| jd|i|durHd|ind|||d�S )a� Add an example (row) to a Chat-type dataset.
Args:
messages (List[Union[Mapping[str, Any], BaseMessageLike]]):
The input messages for the example.
generations (Optional[Union[Mapping[str, Any], BaseMessageLike]]):
The output messages for the example.
dataset_id (Optional[Union[UUID, str]]):
The ID of the dataset.
dataset_name (Optional[str]):
The name of the dataset.
created_at (Optional[datetime.datetime]):
The creation timestamp of the example.
Returns:
Example: The created example
Nr� rB r� ) rg �is_base_message_liker �convert_langchain_messager r �BaseMessageLiker� r� ) r� r� r� r r r� � final_input�message�final_generationsr< r< r= �create_chat_example� s.
��
�
�zClient.create_chat_examplec
C s� |du r
| j |d�j}d}| �|�}|tjjkrm|jdkr&td|j� d���zt� |j
�}W n tyA td|j� d|j
� ���w d|i}|j sLd}n�zt� |j �} W n tyg td |j� d
|j � ���w d | i}nk|tjj
kr�|jdkr�td|j� d ���z
dt�|j
�i}W n ty� td
|j� d|j
� ���w |j s�d}n3z
d t�|j �i}W n( ty� td|j� d
|j � ���w |tjjkr�|j
}|j }ntd|� d���| j|||||d�S )a� Add an example (row) to a dataset from a run.
Args:
run (Run): The run to create an example from.
dataset_id (Optional[Union[UUID, str]]): The ID of the dataset.
dataset_name (Optional[str]): The name of the dataset.
created_at (Optional[datetime.datetime]): The creation timestamp of the example.
Returns:
Example: The created example
Nr rw z Run type z+ is not supported for dataset of type 'LLM'z2Error converting LLM run inputs to prompt for run z
with inputs r� z7Error converting LLM run outputs to generation for run z with outputs rB z, is not supported for dataset of type 'chat'z9Error converting LLM run inputs to chat messages for run z=Error converting LLM run outputs to chat generations for run z
Dataset type z not recognized.r� )r
r� r� r r� rw r� r` rg �get_prompt_from_inputsr� r� � get_llm_generation_from_outputs�chat�get_messages_from_inputs�#get_message_generation_from_outputsr� r� )
r� r� r r r� � dataset_typerz r� r� r� r< r< r= �create_example_from_run� s�
� ���� ����
� ������ ���� �z Client.create_example_from_runr= ��Union[List[ls_schemas.ExampleUploadWithAttachments] | List[ls_schemas.ExampleUpsertWithAttachments] | List[ls_schemas.ExampleUpdateWithAttachments],]�include_dataset_id�/tuple[Any, bytes, Dict[str, io.BufferedReader]]c C s� g }i }|rt |d tj�std��|d j}|D �]}t |tj�s2t |tj�s2t |tj�s2td��|jd ur=t|j�}ntt �
� �}t |tj�rLd } n|j } i |rVd|ini �| d ur`d| ini �}
|j d urm|j |
d< |j
d urw|j
|
d< t|
�} |�|� d | di ff� t|j�} |�|� d �d | di ff� |jr�t|j�}
|�|� d
�d |
di ff� |j�r|j�� D ]U\}\}}t |t�r�|r�tj�|�}t|d �}||t|�tt �
� � < |�|� d |� �d ||� d
|� �i ff� q�td��|�|� d |� �d ||� d
t|�� �i ff� q�t |tj��r-|j�r-t|j�}|�|� d�d |di ff� qtj |t d�}|jdk�r@|� � }n|}|||fS )Nr zWThe examples must be of type ExampleUpsertWithAttachments if include_dataset_id is TruezyThe examples must be of type ExampleUploadWithAttachments or ExampleUpsertWithAttachments or ExampleUpdateWithAttachmentsr r� r� rd r? z.inputsz.outputsr� z .attachment.z ; length=zMdangerously_allow_filesystem must be True to upload files from the filesystemz.attachments_operationsrp rr ) r] r � ExampleUpsertWithAttachmentsr` r � ExampleUploadWithAttachments� ExampleUpdateWithAttachmentsr� rT r^ r� r� r� rd r� r r� r� r r� r r� rc �getsizer� re �attachments_operationsru rv r( rw )r� r= r� r rc rd r r: r6 r� � example_body�valb�inputsb�outputsbr� � mime_type�attachment_data� file_sizer� �attachments_operationsbry rh r< r< r= �_prepare_multipart_data8 s�
�
�
�
��
��
���
���
���
�� ��� � ���
��
����
z Client._prepare_multipart_data)�updatesr r� �7Optional[List[ls_schemas.ExampleUpdateWithAttachments]]�!ls_schemas.UpsertExamplesResponsec
C � | j jpi �dd�std��|du rg }| j|d|d�\}}}z)| jdd|� d�|i | j�d |ji�d
�d �}t� |� W t
t |� � �� |�
� S t
t |� � �� w ) aJ Update examples using multipart.
Args:
dataset_id (Union[UUID, str]): The ID of the dataset to update.
updates (Optional[List[ExampleUpdateWithAttachments]]): The updates to apply to the examples.
Raises:
ValueError: If the multipart examples endpoint is not enabled.
�"dataset_examples_multipart_enabledF�qYour LangSmith version does not allow using the multipart examples endpoint, please update to the latest version.N�r� r r� �/v1/platform/datasets/r rj r' r� �r �instance_flagsrU r` r� rJ rE rx rg rK r r� r r� )r� r r� r ry rh rd rM r< r< r= �update_examples_multipart� s: �� �
���� �z Client.update_examples_multipart)�uploadsr r� �7Optional[List[ls_schemas.ExampleUploadWithAttachments]]c
C r� ) a� Upload examples using multipart.
Args:
dataset_id (Union[UUID, str]): The ID of the dataset to upload to.
uploads (Optional[List[ExampleUploadWithAttachments]]): The examples to upload.
Returns:
ls_schemas.UpsertExamplesResponse: The count and ids of the successfully uploaded examples
Raises:
ValueError: If the multipart examples endpoint is not enabled.
r� Fr� Nr� r� r� r rj r' r� r� )r� r r� r ry rh rd rM r< r< r= �upload_examples_multipart s: �� �
���� �z Client.upload_examples_multipart)�upsertsr r� �7Optional[List[ls_schemas.ExampleUpsertWithAttachments]]c
C s� | j jpi �dd�std��|du rg }| j|d|d�\}}}z%| jdd|i | j�d |ji�d
�d �}t� |� W t
t |� � �� |�
� S t
t |� � �� w ) z�Upsert examples.
.. deprecated:: 0.1.0
This method is deprecated. Use :func:`langsmith.upload_examples_multipart` instead.
�examples_multipart_enabledFr� NTr� r� z /v1/platform/examples/multipartrj r' r� r� )r� r� r ry rh rd rM r< r< r= �upsert_examples_multipart4 s: �� ����� �z Client.upsert_examples_multipart)r� r� �splits�source_run_ids�idsr r �Sequence[Mapping[str, Any]]�/Optional[Sequence[Optional[Mapping[str, Any]]]]r� �-Optional[Sequence[Optional[str | List[str]]]]r� �%Optional[Sequence[Optional[ID_TYPE]]]r� c
s$ � du r |du r t d��� du r| j|d�j� |||||d�}
t|�} |
�� D ] \} }
|
durEt|
�| krEt d| � dt|
�� d| � d���q'� fd d
�t||pUdgt|� |p]dgt|� |pedgt|� |pmdgt|� |pudgt|� �D �}| jd d i | j�d
di�t|�d�}t �
|� dS )a� Create examples in a dataset.
Args:
inputs (Sequence[Mapping[str, Any]]):
The input values for the examples.
outputs (Optional[Sequence[Optional[Mapping[str, Any]]]]):
The output values for the examples.
metadata (Optional[Sequence[Optional[Mapping[str, Any]]]]):
The metadata for the examples.
splits (Optional[Sequence[Optional[str | List[str]]]]):
The splits for the examples, which are divisions
of your dataset such as 'train', 'test', or 'validation'.
source_run_ids (Optional[Sequence[Optional[Union[UUID, str]]]]):
The IDs of the source runs associated with the examples.
ids (Optional[Sequence[Union[UUID, str]]]):
The IDs of the examples.
dataset_id (Optional[Union[UUID, str]]):
The ID of the dataset to create the examples in.
dataset_name (Optional[str]):
The name of the dataset to create the examples in.
**kwargs: Any: Additional keyword arguments are ignored.
Raises:
ValueError: If neither dataset_id nor dataset_name is provided.
Returns:
None
Nz3Either dataset_id or dataset_name must be provided.r )r� r� r� r� r� �
Length of � (z#) does not match length of inputs (r9 c
s8 g | ]\}}}}}}||� |||pt t�� �|d ��qS ))r� r� r r� rd r� �
source_run_id)rT r^ r� )ru �in_�out_� metadata_�split_�id_�source_run_id_r� r< r= rw � s
���z*Client.create_examples.<locals>.<listcomp>r� �/examples/bulkrj r? r% ) r` r
r� re r� �ziprJ rE r� rg rK )r� r� r� r� r� r� r� r r rf �
sequence_args� input_len�arg_name� arg_valuer= rM r< r� r= r� _ sH )����
���zClient.create_examples�Mapping[str, Any]�Optional[Mapping[str, Any]]rd �Optional[str | List[str]]r6 r� c
C s� |du r | j |d�j}|||||| d�}
|r |�� |
d< |p#tt�� �|
d< | jddi | j�dd i�td
d � |
� � D ��d �} t
� | � | � � } t
jdi | �| j| �� d
���S )ak Create a dataset example in the LangSmith API.
Examples are rows in a dataset, containing the inputs
and expected outputs (or other reference information)
for a model or chain.
Args:
inputs (Mapping[str, Any]):
The input values for the example.
dataset_id (Optional[Union[UUID, str]]):
The ID of the dataset to create the example in.
dataset_name (Optional[str]):
The name of the dataset to create the example in.
created_at (Optional[datetime.datetime]):
The creation timestamp of the example.
outputs (Optional[Mapping[str, Any]]):
The output values for the example.
metadata (Optional[Mapping[str, Any]]):
The metadata for the example.
split (Optional[str | List[str]]):
The splits for the example, which are divisions
of your dataset such as 'train', 'test', or 'validation'.
example_id (Optional[Union[UUID, str]]):
The ID of the example to create. If not provided, a new
example will be created.
source_run_id (Optional[Union[UUID, str]]):
The ID of the source run associated with this example.
Returns:
Example: The created example.
Nr )r� r� r r� rd r� r� r� r� r rj r? c S r� r� r< r� r< r< r= r� � r� z)Client.create_example.<locals>.<dictcomp>r% r� r< )r
r� r� rT r^ r� rJ rE r� r� rg rK r� r r r6 r� )
r� r� r r r� r� r� rd r6 r� rh rM r� r< r< r= r� � s4 ,� �
�
�zClient.create_example)ry c C sp | j ddt|d�� �d|r|�� ndid�}|�� }t|dd�}tjd i d d
� |�� D ��|| j| � � d ���S )
a� Read an example from the LangSmith API.
Args:
example_id (Union[UUID, str]): The ID of the example to read.
as_of (Optional[datetime.datetime]): The dataset version tag OR
timestamp to retrieve the example as of.
Response examples will only be those that were present at the time
of the tagged (or timestamped) version.
Returns:
Example: The example.
rG �
/examples/r6 ry Nr� �attachment_urlsr� c S � i | ] \}}|d kr||�qS �r� r< r� r< r< r= r� r� z'Client.read_example.<locals>.<dictcomp>�r r6 r� r< )
rJ rf r� r� r� r r r� r6 r� )r� r6 ry rM r: r r< r< r= � read_example s" ����
�zClient.read_exampler )r� r� r� r� �include_attachments�'Optional[Union[datetime.datetime, str]]�inline_s3_urlsr� r� � Iterator[ls_schemas.Example]c k s � i | �||t |tj�r|�� n||||dur t|d�nd|
d��}
| dur,t| �|
d< |dur5||
d< n|durE| j|d�j}||
d< n | rNg d�|
d< t| jd |
d
��D ]0\}}t |d d �}t
j di d
d� |� � D ��|| j
| �� d���V |dur�|d |kr� dS qWdS )a� Retrieve the example rows of the specified dataset.
Args:
dataset_id (Optional[Union[UUID, str]]): The ID of the dataset to filter by.
Defaults to None.
dataset_name (Optional[str]): The name of the dataset to filter by.
Defaults to None.
example_ids (Optional[Sequence[Union[UUID, str]]): The IDs of the examples to filter by.
Defaults to None.
as_of (Optional[Union[datetime.datetime, str]]): The dataset version tag OR
timestamp to retrieve the examples as of.
Response examples will only be those that were present at the time
of the tagged (or timestamped) version.
splits (Optional[Sequence[str]]): A list of dataset splits, which are
divisions of your dataset such as 'train', 'test', or 'validation'.
Returns examples only from the specified splits.
inline_s3_urls (bool, default=True): Whether to inline S3 URLs.
Defaults to True.
offset (int, default=0): The offset to start from. Defaults to 0.
limit (Optional[int]): The maximum number of examples to return.
metadata (Optional[dict]): A dictionary of metadata to filter by.
filter (Optional[str]): A structured filter string to apply to
the examples.
include_attachments (bool, default=False): Whether to include the
attachments in the response. Defaults to False.
**kwargs (Any): Additional keyword arguments are ignored.
Yields:
The examples.
Examples:
List all examples for a dataset:
.. code-block:: python
from langsmith import Client
client = Client()
# By Dataset ID
examples = client.list_examples(
dataset_id="c9ace0d8-a82c-4b6c-13d2-83401d68e9ab"
)
# By Dataset Name
examples = client.list_examples(dataset_name="My Test Dataset")
List examples by id
.. code-block:: python
example_ids = [
"734fc6a0-c187-4266-9721-90b7a025751a",
"d6b4c1b9-6160-4d63-9b61-b034c585074f",
"4d31df4e-f9c3-4a6e-8b6c-65701c2fed13",
]
examples = client.list_examples(example_ids=example_ids)
List examples by metadata
.. code-block:: python
examples = client.list_examples(
dataset_name=dataset_name, metadata={"foo": "bar"}
)
List examples by structured filter
.. code-block:: python
examples = client.list_examples(
dataset_name=dataset_name,
filter='and(not(has(metadata, \'{"foo": "bar"}\')), exists(metadata, "tenant_id"))',
)
Nr )r� r� ry r� r� r� r� r� rP r )r� r� r� r� r r� r� r� c S r� r� r< r� r< r< r= r� � r� z(Client.list_examples.<locals>.<dictcomp>r� r� r< )r] r� r� r[ r� r
r� r� r� r� r r r� r6 r� )r� r r r ry r� r� r� r� r� r� r� rf r� r� r: r r< r< r= r; $ sN �Z��
��� ���zClient.list_examples�latest)r{ c K sB t |d�}| jdd|� d�| jt�d|i|��d�}t�|� dS )ag Enable dataset indexing. Examples are indexed by their inputs.
This enables searching for similar examples by inputs with
``client.similar_examples()``.
Args:
dataset_id (Union[UUID, str]): The ID of the dataset to index.
tag (Optional[str]): The version of the dataset to index. If 'latest'
then any updates to the dataset (additions, updates, deletions of
examples) will be reflected in the index.
**kwargs (Any): Additional keyword arguments to pass as part of request body.
Returns:
None
r r� r z/indexr{ r% N)rf rJ rE r� r5 rg rK )r� r r{ rf �respr< r< r= �
index_dataset� s
�zClient.index_dataset)r� � List[ls_schemas.ExampleSearch]c
K s� t |d�}||d�|�}|dur||d< | jdd|� d�| jt�|�d�}t�|� g }|�� d D ]} |�tj d
i | �d|i��� q2|S ) a� Retrieve the dataset examples whose inputs best match the current inputs.
**Note**: Must have few-shot indexing enabled for the dataset. See
`client.index_dataset()`.
Args:
inputs (dict): The inputs to use as a search query. Must match the dataset
input schema. Must be JSON serializable.
limit (int): The maximum number of examples to return.
dataset_id (Union[UUID, str]): The ID of the dataset to search over.
filter (Optional[str]): A filter string to apply to the search results. Uses
the same syntax as the `filter` parameter in `list_runs()`. Only a subset
of operations are supported. Defaults to None.
For example, you can use ``and(eq(metadata.some_tag, 'some_value'), neq(metadata.env, 'dev'))``
to filter only examples where some_tag has some_value, and the environment is not dev.
**kwargs: Additional keyword arguments to pass as part of request body.
Returns:
list[ExampleSearch]: List of ExampleSearch objects.
Examples:
.. code-block:: python
from langsmith import Client
client = Client()
client.similar_examples(
{"question": "When would i use the runnable generator"},
limit=3,
dataset_id="...",
)
.. code-block:: python
[
ExampleSearch(
inputs={
"question": "How do I cache a Chat model? What caches can I use?"
},
outputs={
"answer": "You can use LangChain's caching layer for Chat Models. This can save you money by reducing the number of API calls you make to the LLM provider, if you're often requesting the same completion multiple times, and speed up your application.\n\nfrom langchain.cache import InMemoryCache\nlangchain.llm_cache = InMemoryCache()\n\n# The first time, it is not yet in cache, so it should take longer\nllm.predict('Tell me a joke')\n\nYou can also use SQLite Cache which uses a SQLite database:\n\nrm .langchain.db\n\nfrom langchain.cache import SQLiteCache\nlangchain.llm_cache = SQLiteCache(database_path=\".langchain.db\")\n\n# The first time, it is not yet in cache, so it should take longer\nllm.predict('Tell me a joke') \n"
},
metadata=None,
id=UUID("b2ddd1c4-dff6-49ae-8544-f48e39053398"),
dataset_id=UUID("01b6ce0f-bfb6-4f48-bbb8-f19272135d40"),
),
ExampleSearch(
inputs={"question": "What's a runnable lambda?"},
outputs={
"answer": "A runnable lambda is an object that implements LangChain's `Runnable` interface and runs a callbale (i.e., a function). Note the function must accept a single argument."
},
metadata=None,
id=UUID("f94104a7-2434-4ba7-8293-6a283f4860b4"),
dataset_id=UUID("01b6ce0f-bfb6-4f48-bbb8-f19272135d40"),
),
ExampleSearch(
inputs={"question": "Show me how to use RecursiveURLLoader"},
outputs={
"answer": 'The RecursiveURLLoader comes from the langchain.document_loaders.recursive_url_loader module. Here\'s an example of how to use it:\n\nfrom langchain.document_loaders.recursive_url_loader import RecursiveUrlLoader\n\n# Create an instance of RecursiveUrlLoader with the URL you want to load\nloader = RecursiveUrlLoader(url="https://example.com")\n\n# Load all child links from the URL page\nchild_links = loader.load()\n\n# Print the child links\nfor link in child_links:\n print(link)\n\nMake sure to replace "https://example.com" with the actual URL you want to load. The load() method returns a list of child links found on the URL page. You can iterate over this list to access each child link.'
},
metadata=None,
id=UUID("0308ea70-a803-4181-a37d-39e95f138f8c"),
dataset_id=UUID("01b6ce0f-bfb6-4f48-bbb8-f19272135d40"),
),
]
r )r� r� Nr� r� r z/searchr% r= r< )
rf rJ rE r� r5 rg rK r r �
ExampleSearch)
r� r� r� r r� rf �reqr� r= �exr< r< r= �similar_examples� s&
O��
�
zClient.similar_examples)r� r� r� rd r r� r� �*Optional[ls_schemas.AttachmentsOperations]c
C s� |dur| j jp i �dd�std��t||||||d�}dd� |�� D �}| jdd t|d
�� �i | j�d d i�t d
d� |�� D ��d�} t
� | � | � � S )aT Update a specific example.
Args:
example_id (Union[UUID, str]):
The ID of the example to update.
inputs (Optional[Dict[str, Any]]):
The input values to update.
outputs (Optional[Mapping[str, Any]]):
The output values to update.
metadata (Optional[Dict]):
The metadata to update.
split (Optional[str | List[str]]):
The dataset split to update, such as
'train', 'test', or 'validation'.
dataset_id (Optional[Union[UUID, str]]):
The ID of the dataset to update.
attachments_operations (Optional[AttachmentsOperations]):
The attachments operations to perform.
Returns:
Dict[str, Any]: The updated example.
Nr� F�kYour LangSmith version does not allow using the attachment operations, please update to the latest version.)r� r� r r� rd r� c S r� r� r< r� r< r< r= r� Z r� z)Client.update_example.<locals>.<dictcomp>r� r� r6 rj r? c S r� r� r< r� r< r< r= r� _ r� r% )
r r� rU r` r� r� rJ rf rE r� rg rK r� )
r� r6 r� r� r� rd r r� r: rM r< r< r= �update_example* s0 ! ����
zClient.update_example�r� r� r� r� rt r� �Sequence[ID_TYPE]�,Optional[Sequence[Optional[Dict[str, Any]]]]�"Optional[Sequence[Optional[Dict]]]�>Optional[Sequence[Optional[ls_schemas.AttachmentsOperations]]]c C s6 |dur| j jp i �dd�std��||||||d�}t|�} |�� D ] \}
} | durAt| �| krAtd|
� dt| �� d| � d ���q#d
d � t||pOdgt|� |pWdgt|� |p_dgt|� |pgdgt|� |podgt|� |pwdgt|� �D �} | jd d
i | j�ddi�t dd � | D ��d�}
t
� |
� |
� � S )at Update multiple examples.
Args:
example_ids (Sequence[Union[UUID, str]]):
The IDs of the examples to update.
inputs (Optional[Sequence[Optional[Dict[str, Any]]]):
The input values for the examples.
outputs (Optional[Sequence[Optional[Mapping[str, Any]]]]):
The output values for the examples.
metadata (Optional[Sequence[Optional[Mapping[str, Any]]]]):
The metadata for the examples.
splits (Optional[Sequence[Optional[str | List[str]]]]):
The splits for the examples, which are divisions
of your dataset such as 'train', 'test', or 'validation'.
dataset_ids (Optional[Sequence[Optional[Union[UUID, str]]]]):
The IDs of the datasets to move the examples to.
attachments_operations (Optional[Sequence[Optional[ls_schemas.AttachmentsOperations]]):
The operations to perform on the attachments.
Returns:
Dict[str, Any]: The response from the server (specifies the number of examples updated).
Nr� Fr� r� r� r� z%) does not match length of examples (r9 c
S s. g | ]\}}}}}}}|||||||d ��qS ))r� r� r� r r� rd r� r< )ru r� r� r� r� r� � dataset_id_�attachments_operations_r< r< r= rw � s
���z*Client.update_examples.<locals>.<listcomp>r� r� rj r? c S s g | ] }d d� |� � D ��qS )c S r� r� r< r� r< r< r= r� � r� z5Client.update_examples.<locals>.<listcomp>.<dictcomp>)r� r9 r< r< r= rw � s ��r% )
r r� rU r` re r� r� rJ rE r� rg rK r� )r� r r� r� r� r� rt r� r� � examples_lenr� r� r= rM r< r< r= �update_examplesd s\ # ��� ���
�����
zClient.update_examplesc C �, | j ddt|d�� �| jd�}t�|� dS )z�Delete an example by ID.
Args:
example_id (Union[UUID, str]):
The ID of the example to delete.
Returns:
None
r� r� r6 r� Nr� )r� r6 rM r< r< r= �delete_example� �
�zClient.delete_examplec C sD | j ddi | j�ddi�tddd� t|�D �i�d�}t�|� d S )
z�Delete multiple examples by ID.
Parameters
----------
example_ids : Sequence[ID_TYPE]
The IDs of the examples to delete.
r� r rj r? r� c S �& g | ]\}}t t|d |� d����qS �z example_ids[�]�rT rf �ru r� r� r< r< r= rw � � ��z*Client.delete_examples.<locals>.<listcomp>r% N)rJ rE r� r� rg rK )r� r rM r< r< r= �delete_examples� s ����
zClient.delete_examples)r r ry �'Optional[Union[str, datetime.datetime]]� List[str]c C s~ |du r|du r t d��| j|d�j}i }|dur't|tj�r#|�� n||d< | jddt|d�� d�|d �}t� |� |�
� S )
a� Get the splits for a dataset.
Args:
dataset_id (Optional[Union[UUID, str]]): The ID of the dataset.
dataset_name (Optional[str]): The name of the dataset.
as_of (Optional[Union[str, datetime.datetime]]): The version
of the dataset to retrieve splits for. Can be a timestamp or a
string tag. Defaults to "latest".
Returns:
List[str]: The names of this dataset's splits.
N� Must provide dataset name or IDr ry rG r r �/splitsr� ) r` r
r� r] r� r� rJ rf rg rK r� )r� r r ry r� rM r< r< r= �list_dataset_splits� s ��
zClient.list_dataset_splits)r r r �
split_name�
List[ID_TYPE]r c C sl |du r|du r t d��| j|d�j}|dd� t|�D �|d�}| jddt|d �� d
�|d �}t�|� dS ) aY Update the splits for a dataset.
Args:
dataset_id (Optional[Union[UUID, str]]): The ID of the dataset to update.
dataset_name (Optional[str]): The name of the dataset to update.
split_name (str): The name of the split to update.
example_ids (List[Union[UUID, str]]): The IDs of the examples to add to or
remove from the split.
remove (Optional[bool]): If True, remove the examples from the split.
If False, add the examples to the split. Defaults to False.
Returns:
None
Nr r c S r r r r r< r< r= rw , r
z0Client.update_dataset_splits.<locals>.<listcomp>)r r= r r� r r r �r� )r` r
r� r� rJ rf rg rK )r� r r r r r rh rM r< r< r= �update_dataset_splits s �� �z Client.update_dataset_splits�9Union[ls_schemas.Run, ls_schemas.RunBase, str, uuid.UUID]c C s2 t |ttjf�r| j||d�}|S ttj|�}|S )aJ Resolve the run ID.
Args:
run (Union[Run, RunBase, str, UUID]):
The run to resolve.
load_child_runs (bool):
Whether to load child runs.
Returns:
Run: The resolved run.
Raises:
TypeError: If the run type is invalid.
�r� )r] rT r^ r_ r� r r r� )r� r� r� �run_r< r< r= �_resolve_run_id8 s
�zClient._resolve_run_idr: �5Union[ls_schemas.Example, str, uuid.UUID, dict, None]� Optional[ls_schemas.Example]c C s� t |ttjf�r| �|�}|S t |tj�r|}|S t |t�r0tjdi |�| j| � � d���}|S |j
dur=| �|j
�}|S d}|S )a+ Resolve the example ID.
Args:
example (Optional[Union[Example, str, UUID, dict]]):
The example to resolve.
run (Run):
The run associated with the example.
Returns:
Optional[Example]: The resolved example.
r� Nr< ) r] rT r^ r_ r� r r r� r6 r� r� )r� r: r� �reference_example_r< r< r= �_resolve_example_idQ s&
�
��
�
� �zClient._resolve_example_id)�fn_namerP �JUnion[ls_evaluator.EvaluationResult, ls_evaluator.EvaluationResults, dict]r �#List[ls_evaluator.EvaluationResult]c s� ddl m� d��fdd� � dd d
�}t|�j�r |g}|S ||�r/� fdd�|d D �}|S t|t�r>� tt|��g}|S tdt|�� d���)Nr r? �
single_result�*Union[ls_evaluator.EvaluationResult, dict]rX � ls_evaluator.EvaluationResultc s. t | t�r�jdi � | �d�d�| ���S | S )N� reasoning)rD �commentr< )r] r� �EvaluationResultrU )r# )r � ls_evaluatorr< r= � _cast_result{ s
���z1Client._select_eval_results.<locals>._cast_resultrP r �)TypeGuard[ls_evaluator.EvaluationResults]c S s t | t�od| v S )NrP )r] r� )rP r< r< r= �_is_eval_results� s z5Client._select_eval_results.<locals>._is_eval_resultsc s g | ]}� |��qS r< r< )ru r� )r* r< r= rw � rx z/Client._select_eval_results.<locals>.<listcomp>z!Invalid evaluation results type: z.. Must be EvaluationResult, EvaluationResults.)r# r$ rX r% )rP r rX r+ )�langsmith.evaluationr@ r] r( r� r r` r7 )r� rP r r, �results_r< )r* r r) r= �_select_eval_resultsq s
�
���zClient._select_eval_results)� source_infor� r� r@ �ls_evaluator.RunEvaluatorr0 r� �9Optional[Union[ls_schemas.Example, str, dict, uuid.UUID]]r% c
C s@ | j ||d�}| �||�}|j||d�}| j|||d�} | d S )a! Evaluate a run.
Args:
run (Union[Run, RunBase, str, UUID]):
The run to evaluate.
evaluator (RunEvaluator):
The evaluator to use.
source_info (Optional[Dict[str, Any]]):
Additional information about the source of the evaluation to log
as feedback metadata.
reference_example (Optional[Union[Example, str, dict, UUID]]):
The example to use as a reference for the evaluation.
If not provided, the run's reference example will be used.
load_child_runs (bool, default=False):
Whether to load child runs when resolving the run ID.
Returns:
Feedback: The feedback object created by the evaluation.
r �r: �r0 r )r r � evaluate_run�_log_evaluation_feedback�
r� r� r@ r0 r� r� r r �evaluator_responserP r< r< r= r5 � s ��zClient.evaluate_run)� _executorr8 �Optional[ls_schemas.Run]r9 � Optional[cf.ThreadPoolExecutor]c s� �� |�}� �fdd�}|D ]F}|pi } |jr i |j�| �} d }
|jr'|j}
n|d ur.|j}
||
|j|j|j|j|j| |j t
t t j
|j�t jj||j|rP|jnd d�
q|S )Nc s0 � r� j �jfi | �� d S �jdi | �� d S )Nr< )r� �create_feedback)rf �r9 r� r< r= �_submit_feedback� s z9Client._log_evaluation_feedback.<locals>._submit_feedback)
r rD �scorer� r' �
correctionr0 r� �feedback_config�feedback_source_typer� r� r )r/ �evaluator_info�
target_run_idr� rD r? r� r' r@ r� r r r �FeedbackConfigrA �FeedbackSourceType�MODELr� r ) r� r8 r� r0 r� r9 rP r> �res� source_info_r� r< r= r= r6 � s:
� �z Client._log_evaluation_feedback�%Union[ls_schemas.Run, str, uuid.UUID]c
� sH �| j ||d�}| �||�}|j||d�I dH }| j|||d�} | d S )a7 Evaluate a run asynchronously.
Args:
run (Union[Run, str, UUID]):
The run to evaluate.
evaluator (RunEvaluator):
The evaluator to use.
source_info (Optional[Dict[str, Any]]):
Additional information about the source of the evaluation to log
as feedback metadata.
reference_example (Optional[Union[Example, str, dict, UUID]]):
The example to use as a reference for the evaluation.
If not provided, the run's reference example will be used.
load_child_runs (bool, default=False)
Whether to load child runs when resolving the run ID.
Returns:
EvaluationResult: The evaluation result object created by the evaluation.
r r3 Nr4 r )r r �
aevaluate_runr6 r7 r< r< r= rK � s � ��zClient.aevaluate_runr )r? r� r@ r' r0 rB r� � feedback_idrA r[ r� �comparative_experiment_id�feedback_group_idr� r rD r? � Union[float, int, bool, None]r� �Union[str, dict, None]r@ �Union[dict, None]r' �Union[str, None]rB �)Union[ls_schemas.FeedbackSourceType, str]rL rA �#Optional[ls_schemas.FeedbackConfig]rM rN �ls_schemas.Feedbackc
K sX |du r |
du r t d��|dur|
durt d��|r%t�dt|�� �t� �zot|tj�s2t�|�}|tjjkr?tj |d�}n|tjj
krLtj |d�}nt d|� ���|j dur[|j ni |_ | durpd|j vrpdt
| �i|j d< |j r�d|j v r�|j d }t|d �r�t|�r�|�� }d|v r�t
t|j d d d
��|d< ||j d< tjd'i d t|
��dt|d d
��dt|d d
��d|�d|�d|�d|�d|�d|�dtj�tjj��dtj�tjj��d| �dt|
d d
��dt|d d
��dt|d d
��d|��}| jjp�i �d d �}|�ru| jj�rut � | jjd ��ru| j du�s | j du�ru|j du�rut!|�}| j du�rbt"|�}| j j#� t$|| j t%� | j j&d 7 _&| j'�rQ| j'�(� W d � n 1 �s\w Y n)| j du�rt| j �)t*t
|j+�|�� nt,|jd d ��}| j-d!d"d#|i| t j.fd$� tj/d'i |�� ��W S t0�y� } z t1j2d%d d&� |�d}~ww )(a Create a feedback in the LangSmith API.
Args:
run_id (Optional[Union[UUID, str]]):
The ID of the run to provide feedback for. Either the run_id OR
the project_id must be provided.
key (str):
The name of the metric or 'aspect' this feedback is about.
score (Optional[Union[float, int, bool]]):
The score to rate this run on the metric or aspect.
value (Optional[Union[float, int, bool, str, dict]]):
The display value or non-numeric value for this feedback.
correction (Optional[dict]):
The proper ground truth for this run.
comment (Optional[str]):
A comment about this feedback, such as a justification for the score or
chain-of-thought trajectory for an LLM judge.
source_info (Optional[Dict[str, Any]]):
Information about the source of this feedback.
feedback_source_type (Union[FeedbackSourceType, str]):
The type of feedback source, such as model (for model-generated feedback)
or API.
source_run_id (Optional[Union[UUID, str]]):
The ID of the run that generated this feedback, if a "model" type.
feedback_id (Optional[Union[UUID, str]]):
The ID of the feedback to create. If not provided, a random UUID will be
generated.
feedback_config (Optional[FeedbackConfig]):
The configuration specifying how to interpret feedback with this key.
Examples include continuous (with min/max bounds), categorical,
or freeform.
stop_after_attempt (int, default=10):
The number of times to retry the request before giving up.
project_id (Optional[Union[UUID, str]]):
The ID of the project_id to provide feedback on. One - and only one - of
this and run_id must be provided.
comparative_experiment_id (Optional[Union[UUID, str]]):
If this feedback was logged as a part of a comparative experiment, this
associates the feedback with that experiment.
feedback_group_id (Optional[Union[UUID, str]]):
When logging preferences, ranking runs, or other comparative feedback,
this is used to group feedback together.
extra (Optional[Dict]):
Metadata for the feedback.
trace_id (Optional[Union[UUID, str]]):
The trace ID of the run to provide feedback for. Enables batch ingestion.
**kwargs (Any):
Additional keyword arguments.
Returns:
Feedback: The created feedback object.
Nz-One of run_id and project_id must be providedz2Only one of run_id and project_id must be providedzLThe following arguments are no longer used in the create_feedback endpoint: r� z Unknown feedback source type �__runr r� z+feedback_source.metadata['__run']['run_id']r� Tr� r rD r? r� r@ r' �feedback_sourcer� � modified_atrA r� rM rN r� �use_multipart_endpointFz0.8.10r� )� exclude_noner� � /feedbackrh )rZ r[ r\ zError creating feedbackr+ r< )3r` r� r� r� �DeprecationWarningr] r rF �API�APIFeedbackSourcerG �ModelFeedbackSourcer� rT r� r� r� rf �FeedbackCreater� r� r� r� r� r rT rU r� rg �is_version_greater_or_equalr� r� r r1 r3 r r0 r( r r� r r r" r� r� rJ r� �Feedbackrs r� r6 )r� r rD r? r� r@ r' r0 rB r� rL rA r[ r� rM rN r� r rf rW � _run_meta�feedbackr� r# r$ �feedback_blockr� r< r< r= r< s� L�� �
� �
��
� � ���� �
� � �
�� ��� �� ���� ��
�
� �� ���� ��zClient.create_feedback)r? r� r@ r' �(Union[float, int, bool, str, dict, None]c C s� i }|dur
||d< |dur||d< |dur||d< |dur"||d< | j ddt|d�� �i | j�d d
i�t|�d �}t�|� dS ) aj Update a feedback in the LangSmith API.
Args:
feedback_id (Union[UUID, str]):
The ID of the feedback to update.
score (Optional[Union[float, int, bool]]):
The score to update the feedback with.
value (Optional[Union[float, int, bool, str, dict]]):
The value to update the feedback with.
correction (Optional[dict]):
The correction to update the feedback with.
comment (Optional[str]):
The comment to update the feedback with.
Returns:
None
Nr? r� r@ r' r� �
/feedback/rL rj r? r% )rJ rf rE r� rg rK )r� rL r? r� r@ r' �feedback_updaterM r< r< r= �update_feedback� s �zClient.update_feedbackc C s, | � ddt|d�� ��}tjdi |�� ��S )z�Read a feedback from the LangSmith API.
Args:
feedback_id (Union[UUID, str]):
The ID of the feedback to read.
Returns:
Feedback: The feedback.
rG rg rL Nr< )rJ rf r rb r� �r� rL rM r< r< r= �
read_feedback s
�zClient.read_feedback)r� � feedback_keyrB r� rl �1Optional[Sequence[ls_schemas.FeedbackSourceType]]� Iterator[ls_schemas.Feedback]c k s� � ||dur t |d�ndd�|�}|dur||d< |dur!||d< t| jd|d��D ]\}}tjd i |��V |durD|d |krD dS q*dS )
a� List the feedback objects on the LangSmith API.
Args:
run_ids (Optional[Sequence[Union[UUID, str]]]):
The IDs of the runs to filter by.
feedback_key (Optional[Sequence[str]]):
The feedback key(s) to filter by. Examples: 'correctness'
The query performs a union of all feedback keys.
feedback_source_type (Optional[Sequence[FeedbackSourceType]]):
The type of feedback source, such as model or API.
limit (Optional[int]):
The maximum number of feedback to return.
**kwargs (Any):
Additional keyword arguments.
Yields:
The feedback objects.
Nr )r� r� rD �sourcer[ r� r� r< )r[ r� r� r rb ) r� r� rl rB r� rf r� r� rd r< r< r= �
list_feedback s$ � �� ���zClient.list_feedbackc C r )z�Delete a feedback by ID.
Args:
feedback_id (Union[UUID, str]):
The ID of the feedback to delete.
Returns:
None
r� rg rL r� Nr� rj r< r< r= �delete_feedbackD r zClient.delete_feedback)r� r@ r' r� rR c
C sh t || jdd�\}}|| jkrtd|� ���| jddt|�� �t|||||d��| jd�} t�| � dS ) aX Create feedback from a presigned token or URL.
Args:
token_or_url (Union[str, uuid.UUID]): The token or URL from which to create
feedback.
score (Optional[Union[float, int, bool]]): The score of the feedback.
Defaults to None.
value (Optional[Union[float, int, bool, str, dict]]): The value of the
feedback. Defaults to None.
correction (Optional[dict]): The correction of the feedback.
Defaults to None.
comment (Optional[str]): The comment of the feedback. Defaults
to None.
metadata (Optional[dict]): Additional metadata for the feedback.
Defaults to None.
Raises:
ValueError: If the source API URL is invalid.
Returns:
None
r� )rU zInvalid source API URL. r� z/feedback/tokens/)r? r� r@ r' r� r' N) rm rS r` rJ rf r� rE rg rK )
r� r� r? r� r@ r' r� r� rl rM r< r< r= �create_feedback_from_tokenU s&
�
��
�z!Client.create_feedback_from_token)�
expirationrA rL rs �0Optional[datetime.datetime | datetime.timedelta]� ls_schemas.FeedbackIngestTokenc C s� ||||p
t t�� �d�}|du r tjdddd�|d< n0t|tj�r)|�� |d< n#t|tj�rCtj|j |j
d |j
d d d�|d< n t d
t |�� ���| j
d d t|�d
�}t�|� tjdi |�� ��S )a� Create a pre-signed URL to send feedback data to.
This is useful for giving browser-based clients a way to upload
feedback data directly to LangSmith without accessing the
API key.
Args:
run_id (Union[UUID, str]):
The ID of the run.
feedback_key (str):
The key of the feedback to create.
expiration (Optional[datetime.datetime | datetime.timedelta]): The expiration time of the pre-signed URL.
Either a datetime or a timedelta offset from now.
Default to 3 hours.
feedback_config (Optional[FeedbackConfig]):
If creating a feedback_key for the first time,
this defines how the metric should be interpreted,
such as a continuous score (w/ optional bounds),
or distribution over categorical values.
feedback_id (Optional[Union[UUID, str]): The ID of the feedback to create. If not provided, a new
feedback will be created.
Returns:
FeedbackIngestToken: The pre-signed URL for uploading feedback data.
)r rl rA r� Nr r| ��days�hours�minutes�
expires_in�
expires_at� �< �Unknown expiration type: r� �/feedback/tokens)rh r< )rT r^ r� r �TimeDeltaInputr] r� r� � timedeltarw �secondsr` r7 rJ r� rg rK �FeedbackIngestTokenr� )r� r rl rs rA rL r� rM r< r< r= � create_presigned_feedback_token� s6 #� � ��
z&Client.create_presigned_feedback_token)rs �feedback_configs�
feedback_keysr� �7Optional[Sequence[Optional[ls_schemas.FeedbackConfig]]]�(Sequence[ls_schemas.FeedbackIngestToken]c s^ |durt |�t |�krtd��|sdgt |� }d\��|du r*tjdddd��n,t|tj�r5|�� �n!t|tj�rMtj|j|j d |j d d d��n td t
|�� ���t ���fd
d �t ||�D ��� d� �fdd� �g }t
jt �j�d��,���fdd ��j�� D �}t
�|�D ]}|�� }|�dd � |D �� q�W d � |S 1 s�w Y |S )a Create a pre-signed URL to send feedback data to.
This is useful for giving browser-based clients a way to upload
feedback data directly to LangSmith without accessing the
API key.
Args:
run_id (Union[UUID, str]):
The ID of the run.
feedback_keys (Sequence[str]):
The key of the feedback to create.
expiration (Optional[datetime.datetime | datetime.timedelta]): The expiration time of the pre-signed URL.
Either a datetime or a timedelta offset from now.
Default to 3 hours.
feedback_configs (Optional[Sequence[Optional[FeedbackConfig]]]):
If creating a feedback_key for the first time,
this defines how the metric should be interpreted,
such as a continuous score (w/ optional bounds),
or distribution over categorical values.
Returns:
Sequence[FeedbackIngestToken]: The pre-signed URL for uploading feedback data.
NzBThe length of feedback_keys and feedback_configs must be the same.�NNr r| rv r| r} r~ c s g | ] \}}�||�� d ��qS ))r rl rA rz r{ r< )ru rl rA )r{ rz r r< r= rw s ���z;Client.create_presigned_feedback_tokens.<locals>.<listcomp>rS rT r� r� rX r� c sB �j d| � d�� i �j�t|p�ji�d�d�}t�|� |�� S )Nr� r r' r� )rJ rE rC r� rg rK r� )rS r� rM )r� r� r< r= r� s �
���
z4Client.create_presigned_feedback_tokens.<locals>.req)� max_workersc s g | ] \}}� � �||��qS r< )r� )ru rS r� )r� r� r< r= rw ' s ��c S s g | ]
}t jd i |���qS )r< )r r� )ru re r< r< r= rw . r8 )rS rT r� r� rX r� )re r` r r� r] r� r� r� rw r� r7 r� r� r� r� r� r� r� r� r� ) r� r r� rs r� �tokens�futs�futrM r< )r� r� r{ rz r� r r� r= � create_presigned_feedback_tokens� sZ "��
���� � ��
�
�
z'Client.create_presigned_feedback_tokens)r� �(Iterator[ls_schemas.FeedbackIngestToken]c c sp � t |d�|durt|d�ndd�}t| jd|d��D ]\}}tjdi |��V |dur5|d |kr5 dS qdS ) a
List the feedback ingest tokens for a run.
Args:
run_id (Union[UUID, str]): The ID of the run to filter by.
limit (Optional[int]): The maximum number of tokens to return.
Yields:
The feedback ingest tokens.
r Nr )r r� r r� r� r< )rf r[ r� r� r r� )r� r r� r� r� �tokenr< r< r= � list_presigned_feedback_tokens2 s �� ���z%Client.list_presigned_feedback_tokens)� queue_idsr� r r� r� �$Iterator[ls_schemas.AnnotationQueue]c c s� � |durdd� t |�D �nd|||durt|d�ndd�}t | jd|d��D ]\}}tjd i |��V |durA|d |krA dS q'dS )
a List the annotation queues on the LangSmith API.
Args:
queue_ids (Optional[List[Union[UUID, str]]]):
The IDs of the queues to filter by.
name (Optional[str]):
The name of the queue to filter by.
name_contains (Optional[str]):
The substring that the queue name should contain.
limit (Optional[int]):
The maximum number of queues to return.
Yields:
The annotation queues.
Nc S s" g | ]
\}}t |d |� d���qS )z
queue_ids[r
)rf r r< r< r= rw g s " z1Client.list_annotation_queues.<locals>.<listcomp>r )r� r� r r� �/annotation-queuesr� r� r< )r� r[ r� r �AnnotationQueue)r� r� r� r r� r� r� �queuer< r< r= �list_annotation_queuesN s$ ���
����z Client.list_annotation_queues)r� �queue_idr� �ls_schemas.AnnotationQueuec C s` |||dur
t |�nt t�� �d�}| jdddd� |�� D �d�}t�|� tjdi |� � ��S ) a� Create an annotation queue on the LangSmith API.
Args:
name (str):
The name of the annotation queue.
description (Optional[str]):
The description of the annotation queue.
queue_id (Optional[Union[UUID, str]]):
The ID of the annotation queue.
Returns:
AnnotationQueue: The created annotation queue object.
N)r� r� r� r� r� c S r� r� r< r� r< r< r= r� � r� z2Client.create_annotation_queue.<locals>.<dictcomp>r r< )
rT r^ r� rJ r� rg rK r r� r� )r� r� r� r� r� rM r< r< r= �create_annotation_queuex s ��
�z Client.create_annotation_queuec C s t | j|gd��S )z�Read an annotation queue with the specified queue ID.
Args:
queue_id (Union[UUID, str]): The ID of the annotation queue to read.
Returns:
AnnotationQueue: The annotation queue object.
)r� )r r� )r� r� r< r< r= �read_annotation_queue� s
z Client.read_annotation_queue)r� c C s0 | j ddt|d�� �||d�d�}t�|� dS )a} Update an annotation queue with the specified queue_id.
Args:
queue_id (Union[UUID, str]): The ID of the annotation queue to update.
name (str): The new name for the annotation queue.
description (Optional[str]): The new description for the
annotation queue. Defaults to None.
Returns:
None
r� �/annotation-queues/r� )r� r� r N�rJ rf rg rK )r� r� r� r� rM r< r< r= �update_annotation_queue� s ��z Client.update_annotation_queuec C s4 | j ddt|d�� �ddi| j�d�}t�|� dS )z�Delete an annotation queue with the specified queue ID.
Args:
queue_id (Union[UUID, str]): The ID of the annotation queue to delete.
Returns:
None
r� r� r� r@ r? r� Nr� )r� r� rM r< r< r= �delete_annotation_queue� s �z Client.delete_annotation_queuec C s: | j ddt|d�� d�dd� t|�D �d�}t�|� dS ) a7 Add runs to an annotation queue with the specified queue ID.
Args:
queue_id (Union[UUID, str]): The ID of the annotation queue.
run_ids (List[Union[UUID, str]]): The IDs of the runs to be added to the annotation
queue.
Returns:
None
r� r� r� r& c S r )zrun_ids[r
r r r< r< r= rw � s & z7Client.add_runs_to_annotation_queue.<locals>.<listcomp>r N)rJ rf r� rg rK )r� r� r� rM r< r< r= � add_runs_to_annotation_queue� s
�z#Client.add_runs_to_annotation_queuec C s2 | � ddt|d�� dt|d�� ��}t�|� dS )a? Delete a run from an annotation queue with the specified queue ID and run ID.
Args:
queue_id (Union[UUID, str]): The ID of the annotation queue.
run_id (Union[UUID, str]): The ID of the run to be added to the annotation
queue.
Returns:
None
r� r� r� r� r Nr� )r� r� r rM r< r< r= � delete_run_from_annotation_queue� s
�z'Client.delete_run_from_annotation_queuer� �%ls_schemas.RunWithAnnotationQueueInfoc C sL dt |d�� d�}| jd|� d|� �| jd�}t�|� tjdi |�� ��S ) a� Get a run from an annotation queue at the specified index.
Args:
queue_id (Union[UUID, str]): The ID of the annotation queue.
index (int): The index of the run to retrieve.
Returns:
RunWithAnnotationQueueInfo: The run at the specified index.
Raises:
LangSmithNotFoundError: If the run is not found at the given index.
LangSmithError: For other API-related errors.
r� r� r rG rZ r� Nr< )rf rJ rE rg rK r �RunWithAnnotationQueueInfor� )r� r� r� �base_urlrM r< r< r= � get_run_from_annotation_queue� s �
z$Client.get_run_from_annotation_queue)rZ r� r� r� r� � experimentsrZ � ls_schemas.ComparativeExperimentc C s� |st d��|du r| j|d d�j}|st d��|p tt�� �|||||p-tj�tjj �i d�}|dur;||d d< t
d d
� |� � D ��} | j d d d
| id�}
t
�|
� |
�� } tjdi | ��S )a� Create a comparative experiment on the LangSmith API.
These experiments compare 2 or more experiment results over a shared dataset.
Args:
name (str): The name of the comparative experiment.
experiments (Sequence[Union[UUID, str]]): The IDs of the experiments to compare.
reference_dataset (Optional[Union[UUID, str]]): The ID of the dataset these experiments are compared on.
description (Optional[str]): The description of the comparative experiment.
created_at (Optional[datetime.datetime]): The creation time of the comparative experiment.
metadata (Optional[Dict[str, Any]]): Additional metadata for the comparative experiment.
id (Optional[Union[UUID, str]]): The ID of the comparative experiment.
Returns:
ComparativeExperiment: The created comparative experiment object.
z$At least one experiment is required.Nr r3 z A reference dataset is required.)r� r� �experiment_idsr" r� r� r� r� r� c S s i | ]\}}||�qS r< r< r� r< r< r= r� > rh z8Client.create_comparative_experiment.<locals>.<dictcomp>r� z/datasets/comparativerh r� r< )r` r� r" rT r^ r� r� r� r� r� r� r� rJ rg rK r� r �ComparativeExperiment) r� r� r� rZ r� r� r� r� r� �serrM �
response_dr< r< r= � create_comparative_experiment s< ��� ��
z$Client.create_comparative_experimentrn )�
evaluation�concurrency_levelr �project_metadata�dataset_version�verbose� input_mapperr �llm_or_chain_factoryr� �
Optional[Any]r� r� r� r� r� � Optional[Callable[[Dict], Any]]c
� sb �t �dt� zddlm} W n ty td��w | d||| |||||| |
|d� | ��I dH S )z�Asynchronously run the Chain or language model on a dataset.
.. deprecated:: 0.1.0
This method is deprecated. Use :func:`langsmith.aevaluate` instead.
z}The `arun_on_dataset` method is deprecated and will be removed in a future version.Please use the `aevaluate` method instead.r )�arun_on_datasetzlThe client.arun_on_dataset function requires the langchainpackage to run.
Install with pip install langchain) r r� �clientr� r� r r� r� r� r r� Nr< )r� r� r\ �langchain.smithr� r, )
r� r r� r� r� r r� r� r� r� r rf �_arun_on_datasetr< r< r= r� J s6 �� ��� �zClient.arun_on_datasetc
K sZ t �dt� zddlm} W n ty td��w | d|||| ||||| |
|d� | ��S )z�Run the Chain or language model on a dataset.
.. deprecated:: 0.1.0
This method is deprecated. Use :func:`langsmith.aevaluate` instead.
z{The `run_on_dataset` method is deprecated and will be removed in a future version.Please use the `evaluate` method instead.r )�run_on_datasetzkThe client.run_on_dataset function requires the langchainpackage to run.
Install with pip install langchain) r r� r� r� r� r r� r� r� r r� Nr< )r� r� r\ r� r� r, )
r� r r� r� r� r r� r� r� r� r rf �_run_on_datasetr< r< r= r� z s4 � ��� �zClient.run_on_dataset�ownerc C s | � � }|dkp |j|kS )z�Check if the current workspace has the same handle as owner.
Args:
owner (str): The owner to check against.
Returns:
bool: True if the current tenant is the owner, False otherwise.
�-)rP �
tenant_handle)r� r� �settingsr< r< r= �_current_tenant_is_owner� s z Client._current_tenant_is_owner�action�ls_utils.LangSmithUserErrorc C s" t �d|� d| �� j� d|� ��S )NzCannot z% for another tenant.
Current tenant: z,
Requested tenant: )rg rh rP r� )r� r� r� r< r< r= �_owner_conflict_error� s ���z Client._owner_conflict_error�prompt_owner_and_namec C s> | j dd|� d�||d�d�}|�� d }|r |d d S d S )
a� Get the latest commit hash for a prompt.
Args:
prompt_owner_and_name (str): The owner and name of the prompt.
limit (int, default=1): The maximum number of commits to fetch. Defaults to 1.
offset (int, default=0): The number of commits to skip. Defaults to 0.
Returns:
Optional[str]: The latest commit hash, or None if no commits are found.
rG � /commits/rZ )r� r� r� �commitsr � commit_hashN)rJ r� )r� r� r� r� rM r� r< r< r= �_get_latest_commit_hash� s
� z Client._get_latest_commit_hash�prompt_identifier�like�Dict[str, int]c C s@ t �|�\}}}| jdd|� d|� �d|id�}|�� |�� S )a� Like or unlike a prompt.
Args:
prompt_identifier (str): The identifier of the prompt.
like (bool): True to like the prompt, False to unlike it.
Returns:
A dictionary with the key 'likes' and the count of likes as the value.
Raises:
requests.exceptions.HTTPError: If the prompt is not found or
another error occurs.
r� z/likes/rZ r� r )rg �parse_prompt_identifierrJ �raise_for_statusr� )r� r� r� r� � prompt_namer� rM r< r< r= �_like_or_unlike_prompt� s �z Client._like_or_unlike_promptc C sl t �|�\}}}| �|�s | j� d|� d|� d|dd� � �S | �� }| j� d|� d|dd� � d|j� �S )z�Get a URL for a prompt.
Args:
prompt_identifier (str): The identifier of the prompt.
Returns:
str: The URL for the prompt.
z/hub/rZ rp N� z /prompts/z?organizationId=)rg r� r� r6 rP r� )r� r� r� r� r� r� r< r< r= �_get_prompt_url� s
�
$ ��zClient._get_prompt_urlc C s | � |�}|r dS dS )z�Check if a prompt exists.
Args:
prompt_identifier (str): The identifier of the prompt.
Returns:
bool: True if the prompt exists, False otherwise.
TF)�
get_prompt)r� r� rz r< r< r= �_prompt_exists s
zClient._prompt_existsc C � | j |dd�S )z�Like a prompt.
Args:
prompt_identifier (str): The identifier of the prompt.
Returns:
Dict[str, int]: A dictionary with the key 'likes' and the count of likes as the value.
T�r� �r� �r� r� r< r< r= � like_prompt �
zClient.like_promptc C r� )z�Unlike a prompt.
Args:
prompt_identifier (str): The identifier of the prompt.
Returns:
Dict[str, int]: A dictionary with the key 'likes' and the count of likes as the value.
Fr� r� r� r< r< r= �
unlike_prompt r� zClient.unlike_promptr �desc)r� r� � is_public� is_archived�
sort_field�sort_directionr� r� r� r� �ls_schemas.PromptSortFieldr� �Literal['desc', 'asc']� ls_schemas.ListPromptsResponsec
C s^ |||rdn|dur dnd|rdnd||||rdndd�}| j dd|d�} tjdi | �� ��S ) a# List prompts with pagination.
Args:
limit (int, default=100): The maximum number of prompts to return. Defaults to 100.
offset (int, default=0): The number of prompts to skip. Defaults to 0.
is_public (Optional[bool]): Filter prompts by if they are public.
is_archived (Optional[bool]): Filter prompts by if they are archived.
sort_field (PromptSortField): The field to sort by.
Defaults to "updated_at".
sort_direction (Literal["desc", "asc"], default="desc"): The order to sort by.
Defaults to "desc".
query (Optional[str]): Filter prompts by a search query.
Returns:
ListPromptsResponse: A response object containing
the list of prompts.
r N�false)r� r� r� r� r� r� r� � match_prefixrG �/repos/r� r< )rJ r �ListPromptsResponser� )
r� r� r� r� r� r� r� r� r� rM r< r< r= � list_prompts( s
�
zClient.list_prompts�Optional[ls_schemas.Prompt]c C sZ t �|�\}}}z| �dd|� d|� ��}tjdi |�� d ��W S t jy, Y dS w )a� Get a specific prompt by its identifier.
Args:
prompt_identifier (str): The identifier of the prompt.
The identifier should be in the format "prompt_name" or "owner/prompt_name".
Returns:
Optional[Prompt]: The prompt object.
Raises:
requests.exceptions.HTTPError: If the prompt is not found or
another error occurs.
rG r� rZ �repoNr< )rg r� rJ r �Promptr� r� �r� r� r� r� r� rM r< r< r= r� T s �zClient.get_prompt�r� �readmer� r� r� �ls_schemas.Promptc C s� | � � }|r|jst�d��t�|�\}}} | j|d�s"| �d|��||p&d|p)d|p,g |d�}
| jdd|
d�} | �� t j
d i | � � d ��S ) a Create a new prompt.
Does not attach prompt object, just creates an empty prompt.
Args:
prompt_identifier (str): The identifier of the prompt.
The identifier should be in the formatof owner/name:hash, name:hash, owner/name, or name
description (Optional[str]): A description of the prompt.
readme (Optional[str]): A readme for the prompt.
tags (Optional[Sequence[str]]): A list of tags for the prompt.
is_public (bool): Whether the prompt should be public. Defaults to False.
Returns:
Prompt: The created prompt object.
Raises:
ValueError: If the current tenant is not the owner.
HTTPError: If the server request fails.
��Cannot create a public prompt without first
creating a LangChain Hub handle. You can add a handle by creating a public prompt at:
https://smith.langchain.com/prompts)r� zcreate a promptrY )� repo_handler� r� r� r� r� r� r r� Nr< ) rP r� rg rh r� r� r� rJ r� r r� r� ) r� r� r� r� r� r� r� r� r� r� r� rM r< r< r= �
create_prompti s"
� �zClient.create_prompt��parent_commit_hash�objectr� c C s� | � |�s
t�d��zddlm} W n ty td��w ||�}t�|�}t�|�\}}} |� d|� �}
|dks>|du rC| � |
�}||d�} | j
d d
|
� �| d �} | �� d d
}
| � |
� d|
� ��S )a� Create a commit for an existing prompt.
Args:
prompt_identifier (str): The identifier of the prompt.
object (Any): The LangChain object to commit.
parent_commit_hash (Optional[str]): The hash of the parent commit.
Defaults to latest commit.
Returns:
str: The url of the prompt commit.
Raises:
HTTPError: If the server request fails.
ValueError: If the prompt does not exist.
z0Prompt does not exist, you must create it first.r )r5 zvThe client.create_commit function requires the langchain_corepackage to run.
Install with `pip install langchain_core`rZ r� N)�
parent_commit�manifestr� r� r �commitr� rp ) r� rg r� �langchain_core.load.dumpr5 r, r� r� r� r� rJ r� )r� r� r� r� r5 � json_object�
manifest_dictr� r� r� r� � request_dictrM r� r< r< r= �
create_commit� s.
� ��
�zClient.create_commit)r� r� r� r� r� c
C s� | � � }|r
|js
td��i }|dur||d< |dur ||d< |dur'||d< |dur/||d< |dur7||d< t�|�\} }
} | jdd | � d
|
� �|d �} | �� | �� S ) a$ Update a prompt's metadata.
To update the content of a prompt, use push_prompt or create_commit instead.
Args:
prompt_identifier (str): The identifier of the prompt to update.
description (Optional[str]): New description for the prompt.
readme (Optional[str]): New readme for the prompt.
tags (Optional[Sequence[str]]): New list of tags for the prompt.
is_public (Optional[bool]): New public status for the prompt.
is_archived (Optional[bool]): New archived status for the prompt.
Returns:
Dict[str, Any]: The updated prompt data as returned by the server.
Raises:
ValueError: If the prompt_identifier is empty.
HTTPError: If the server request fails.
r� Nr� r� r� r� r� r� r� rZ r )rP r� r` rg r� rJ r� r� )
r� r� r� r� r� r� r� r� r� r� r� r� rM r< r< r= �
update_prompt� s,
��zClient.update_promptc C sJ t �|�\}}}| �|�s| �d|��| �dd|� d|� ��}|�� dS )a6 Delete a prompt.
Args:
prompt_identifier (str): The identifier of the prompt to delete.
Returns:
bool: True if the prompt was successfully deleted, False otherwise.
Raises:
ValueError: If the current tenant is not the owner of the prompt.
zdelete a promptr� r� rZ N)rg r� r� r� rJ r� r� r< r< r= �
delete_prompt
s
zClient.delete_prompt��
include_modelr �ls_schemas.PromptCommitc
C sX t �|�\}}}| �dd|� d|� d|� |rdnd� ��}tjdi ||d�|�� ���S ) a Pull a prompt object from the LangSmith API.
Args:
prompt_identifier (str): The identifier of the prompt.
Returns:
PromptCommit: The prompt object.
Raises:
ValueError: If no commits are found for the prompt.
rG r� rZ z?include_model=truerY �r� r� Nr< )rg r� rJ r � PromptCommitr� )r� r� r r� r� r� rM r< r< r= �pull_prompt_commit s
�
���zClient.pull_prompt_commit�r� r� r �'Iterator[ls_schemas.ListedPromptCommit]c c s� � t �|�\}}}|durtd|�n|||d�}d} ||d< | jdd|� d |� d �|d
�}
|
�� } | d } | d }
| s>dS | D ] }|durM| |krM dS tjdi ||d
�|���V | d7 } q@|t| �7 }||
krldS q)a List commits for a given prompt.
Args:
prompt_identifier (str): The identifier of the prompt in the format 'owner/repo_name'.
limit (Optional[int]): The maximum number of commits to return. If None, returns all commits. Defaults to None.
offset (int, default=0): The number of commits to skip before starting to return results. Defaults to 0.
include_model (bool, default=False): Whether to include the model information in the commit data. Defaults to False.
Yields:
A ListedPromptCommit object for each commit.
Note:
This method uses pagination to retrieve commits. It will make multiple API calls if necessary to retrieve all commits
or up to the specified limit.
Nr r r Tr� rG r� rZ r� r� r r r� r< )rg r� r[ rJ r� r �ListedPromptCommitre )r� r� r� r� r r� r� r� r� r� rM �valr� r �itr< r< r= �list_prompt_commits? s< ��� �
�zClient.list_prompt_commitsc C sF z(ddl m} ddlm} ddlm} ddlm} ddlm } ddl
m }m } W n t
y3 t
d��w zdd lm}
W n t
yL tjd
d � �}
Y nw | j||d �} |
� � |t�| j��} W d
� n1 sjw Y t| |�st| | �r�t| j|�r�t| |�r�| nt| | �r�t| j|�r�| jnd
}
|
d
u r�t�d��|
jd
u r�i |
_|
j�| j | j | j d�� |�r!t| | ��r!t| j|��r!t | j �dk�r!t| j!|��s!t| j!|��rt| j!j"|��rt#| | j| j!j"B �}t |j �dk�r
|j d }| | j|j$di i | j!j%���|j!�} | S |} | S t| j!|��r | j| j!B } | S | S )a� Pull a prompt and return it as a LangChain PromptTemplate.
This method requires `langchain_core`.
Args:
prompt_identifier (str): The identifier of the prompt.
include_model (Optional[bool], default=False): Whether to include the model information in the prompt data.
Returns:
Any: The prompt object in the specified format.
r )�BaseLanguageModel)r� )�BaseOutputParser)�BasePromptTemplate)�StructuredPrompt)�RunnableBinding�RunnableSequenceztThe client.pull_prompt function requires the langchain_corepackage to run.
Install with `pip install langchain_core`)� suppress_langchain_beta_warningc s s � d V d S r� r< r< r< r< r= r � s �
z;Client.pull_prompt.<locals>.suppress_langchain_beta_warningr� Nz-Prompt object is not a valid prompt template.)� lc_hub_owner� lc_hub_repo�lc_hub_commit_hashrO r| r� r< )&�#langchain_core.language_models.baser �langchain_core.load.loadr� � langchain_core.output_parsersr �langchain_core.promptsr
�!langchain_core.prompts.structuredr � langchain_core.runnables.baser r r, �langchain_core._apir �
contextlib�contextmanagerr r� r5 r� r] �firstrg r� r� r� r� r� r� re �steps�last�boundr �bindrf )r� r� r r r� r r
r r r r �
prompt_objectrz �prompt_template�seq� rebound_llmr< r< r= � pull_promptw s� �� �� ���
���
��
�
����
� �
�
�� �zClient.pull_prompt)r� r� r� r� r� r� c C s� | � |�r tdd� ||||fD ��r | j|||||d� n| j||dur&|nd|||d� |du r7| j|d�S | j|||d�}|S ) a� Push a prompt to the LangSmith API.
Can be used to update prompt metadata or prompt content.
If the prompt does not exist, it will be created.
If the prompt exists, it will be updated.
Args:
prompt_identifier (str): The identifier of the prompt.
object (Optional[Any]): The LangChain object to push.
parent_commit_hash (str): The parent commit hash.
Defaults to "latest".
is_public (Optional[bool]): Whether the prompt should be public.
If None (default), the current visibility status is maintained for existing prompts.
For new prompts, None defaults to private.
Set to True to make public, or False to make private.
description (Optional[str]): A description of the prompt.
Defaults to an empty string.
readme (Optional[str]): A readme for the prompt.
Defaults to an empty string.
tags (Optional[Sequence[str]]): A list of tags for the prompt.
Defaults to an empty list.
Returns:
str: The URL of the prompt.
c s s � | ]}|d uV qd S r� r< )ru �paramr< r< r= r1 s �
�z%Client.push_prompt.<locals>.<genexpr>r� NF)r� r� r� r� )r� r� )r� �anyr� r� r� r� ) r� r� r� r� r� r� r� r� rn r< r< r= � push_prompt� s6
&
���� �zClient.push_promptc C s
d| _ dS )z2Manually trigger cleanup of the background thread.TN)r� r: r< r< r= �cleanup r= zClient.cleanuprh �Optional[DATA_T]�
evaluators� Optional[Sequence[EVALUATOR_T]]�summary_evaluators�'Optional[Sequence[SUMMARY_EVALUATOR_T]]�experiment_prefix�max_concurrency�num_repetitions�blocking�
experiment�Optional[EXPERIMENT_T]�upload_resultsr �'Union[TARGET_T, Runnable, EXPERIMENT_T]rK c
K r� r� r< �r� r rh r- r/ r� r1 r� r2 r3 r4 r5 r7 rf r< r< r= �evaluate# � zClient.evaluate�+Optional[Sequence[COMPARATIVE_EVALUATOR_T]]�(Union[Tuple[EXPERIMENT_T, EXPERIMENT_T]]rJ c
K r� r� r< r9 r< r< r= r: 6 r; �IOptional[Union[Sequence[EVALUATOR_T], Sequence[COMPARATIVE_EVALUATOR_T]]]�JUnion[TARGET_T, Runnable, EXPERIMENT_T, Tuple[EXPERIMENT_T, EXPERIMENT_T]]�6Union[ExperimentResults, ComparativeExperimentResults]c
K s6 ddl m} ||f|||||||| | |
| | d� |
��S )a�# Evaluate a target system on a given dataset.
Args:
target (Union[TARGET_T, Runnable, EXPERIMENT_T, Tuple[EXPERIMENT_T, EXPERIMENT_T]]):
The target system or experiment(s) to evaluate. Can be a function
that takes a dict and returns a dict, a langchain Runnable, an
existing experiment ID, or a two-tuple of experiment IDs.
data (DATA_T): The dataset to evaluate on. Can be a dataset name, a list of
examples, or a generator of examples.
evaluators (Optional[Union[Sequence[EVALUATOR_T], Sequence[COMPARATIVE_EVALUATOR_T]]]):
A list of evaluators to run on each example. The evaluator signature
depends on the target type. Default to None.
summary_evaluators (Optional[Sequence[SUMMARY_EVALUATOR_T]]): A list of summary
evaluators to run on the entire dataset. Should not be specified if
comparing two existing experiments. Defaults to None.
metadata (Optional[dict]): Metadata to attach to the experiment.
Defaults to None.
experiment_prefix (Optional[str]): A prefix to provide for your experiment name.
Defaults to None.
description (Optional[str]): A free-form text description for the experiment.
max_concurrency (Optional[int], default=0): The maximum number of concurrent
evaluations to run. If None then no limit is set. If 0 then no concurrency.
Defaults to 0.
blocking (bool, default=True): Whether to block until the evaluation is complete.
Defaults to True.
num_repetitions (int, default=1): The number of times to run the evaluation.
Each item in the dataset will be run and evaluated this many times.
Defaults to 1.
experiment (Optional[EXPERIMENT_T]): An existing experiment to
extend. If provided, experiment_prefix is ignored. For advanced
usage only. Should not be specified if target is an existing experiment or
two-tuple fo experiments.
upload_results (bool, default=True): Whether to upload the results to LangSmith.
Defaults to True.
**kwargs (Any): Additional keyword arguments to pass to the evaluator.
Returns:
ExperimentResults: If target is a function, Runnable, or existing experiment.
ComparativeExperimentResults: If target is a two-tuple of existing experiments.
Examples:
Prepare the dataset:
.. code-block:: python
from langsmith import Client
client = Client()
dataset = client.clone_public_dataset(
"https://smith.langchain.com/public/419dcab2-1d66-4b94-8901-0357ead390df/d"
)
dataset_name = "Evaluate Examples"
Basic usage:
.. code-block:: python
def accuracy(outputs: dict, reference_outputs: dict) -> dict:
# Row-level evaluator for accuracy.
pred = outputs["response"]
expected = reference_outputs["answer"]
return {"score": expected.lower() == pred.lower()}
.. code-block:: python
def precision(outputs: list[dict], reference_outputs: list[dict]) -> dict:
# Experiment-level evaluator for precision.
# TP / (TP + FP)
predictions = [out["response"].lower() for out in outputs]
expected = [ref["answer"].lower() for ref in reference_outputs]
# yes and no are the only possible answers
tp = sum([p == e for p, e in zip(predictions, expected) if p == "yes"])
fp = sum([p == "yes" and e == "no" for p, e in zip(predictions, expected)])
return {"score": tp / (tp + fp)}
def predict(inputs: dict) -> dict:
# This can be any function or just an API call to your app.
return {"response": "Yes"}
results = client.evaluate(
predict,
data=dataset_name,
evaluators=[accuracy],
summary_evaluators=[precision],
experiment_prefix="My Experiment",
description="Evaluating the accuracy of a simple prediction model.",
metadata={
"my-prompt-version": "abcd-1234",
},
)
Evaluating over only a subset of the examples
.. code-block:: python
experiment_name = results.experiment_name
examples = client.list_examples(dataset_name=dataset_name, limit=5)
results = client.evaluate(
predict,
data=examples,
evaluators=[accuracy],
summary_evaluators=[precision],
experiment_prefix="My Experiment",
description="Just testing a subset synchronously.",
)
Streaming each prediction to more easily + eagerly debug.
.. code-block:: python
results = client.evaluate(
predict,
data=dataset_name,
evaluators=[accuracy],
summary_evaluators=[precision],
description="I don't even have to block!",
blocking=False,
)
for i, result in enumerate(results): # doctest: +ELLIPSIS
pass
Using the `evaluate` API with an off-the-shelf LangChain evaluator:
.. code-block:: python
from langsmith.evaluation import LangChainStringEvaluator
from langchain.chat_models import init_chat_model
def prepare_criteria_data(run: Run, example: Example):
return {
"prediction": run.outputs["output"],
"reference": example.outputs["answer"],
"input": str(example.inputs),
}
results = client.evaluate(
predict,
data=dataset_name,
evaluators=[
accuracy,
LangChainStringEvaluator("embedding_distance"),
LangChainStringEvaluator(
"labeled_criteria",
config={
"criteria": {
"usefulness": "The prediction is useful if it is correct"
" and/or asks a useful followup question."
},
"llm": init_chat_model("gpt-4o"),
},
prepare_data=prepare_criteria_data,
),
],
description="Evaluating with off-the-shelf LangChain evaluators.",
summary_evaluators=[precision],
)
View the evaluation results for experiment:...
Evaluating a LangChain object:
.. code-block:: python
from langchain_core.runnables import chain as as_runnable
@as_runnable
def nested_predict(inputs):
return {"response": "Yes"}
@as_runnable
def lc_predict(inputs):
return nested_predict.invoke(inputs)
results = client.evaluate(
lc_predict,
data=dataset_name,
evaluators=[accuracy],
description="This time we're evaluating a LangChain object.",
summary_evaluators=[precision],
)
Comparative evaluation:
.. code-block:: python
results = client.evaluate(
# The target is a tuple of the experiment IDs to compare
target=(
"12345678-1234-1234-1234-123456789012",
"98765432-1234-1234-1234-123456789012",
),
evaluators=[accuracy],
summary_evaluators=[precision],
)
Evaluate an existing experiment:
.. code-block:: python
results = client.evaluate(
# The target is the ID of the experiment we are evaluating
target="12345678-1234-1234-1234-123456789012",
evaluators=[accuracy],
summary_evaluators=[precision],
)
.. versionadded:: 0.2.0
r )r: � rh r- r/ r� r1 r� r2 r3 r� r4 r5 r7 )� langsmith.evaluation._runnerr: )r� r rh r- r/ r� r1 r� r2 r3 r4 r5 r7 rf � evaluate_r< r< r= r: I s( m���NUnion[DATA_T, AsyncIterable[schemas.Example], Iterable[schemas.Example], None]�4Optional[Sequence[Union[EVALUATOR_T, AEVALUATOR_T]]]�6Optional[Union[schemas.TracerSession, str, uuid.UUID]]�VUnion[ATARGET_T, AsyncIterable[dict], Runnable, str, uuid.UUID, schemas.TracerSession]rC c
� s> �ddl m} ||f|||||||| | |
| | d� |
��I dH S )at Evaluate an async target system on a given dataset.
Args:
target (Union[ATARGET_T, AsyncIterable[dict], Runnable, str, uuid.UUID, TracerSession]):
The target system or experiment(s) to evaluate. Can be an async function
that takes a dict and returns a dict, a langchain Runnable, an
existing experiment ID, or a two-tuple of experiment IDs.
data (Union[DATA_T, AsyncIterable[Example]]): The dataset to evaluate on. Can be a dataset name, a list of
examples, an async generator of examples, or an async iterable of examples.
evaluators (Optional[Sequence[EVALUATOR_T]]): A list of evaluators to run
on each example. Defaults to None.
summary_evaluators (Optional[Sequence[SUMMARY_EVALUATOR_T]]): A list of summary
evaluators to run on the entire dataset. Defaults to None.
metadata (Optional[dict]): Metadata to attach to the experiment.
Defaults to None.
experiment_prefix (Optional[str]): A prefix to provide for your experiment name.
Defaults to None.
description (Optional[str]): A description of the experiment.
max_concurrency (Optional[int], default=0): The maximum number of concurrent
evaluations to run. If None then no limit is set. If 0 then no concurrency.
Defaults to 0.
num_repetitions (int, default=1): The number of times to run the evaluation.
Each item in the dataset will be run and evaluated this many times.
Defaults to 1.
blocking (bool, default=True): Whether to block until the evaluation is complete.
Defaults to True.
experiment (Optional[TracerSession]): An existing experiment to
extend. If provided, experiment_prefix is ignored. For advanced
usage only.
upload_results (bool, default=True): Whether to upload the results to LangSmith.
Defaults to True.
**kwargs (Any): Additional keyword arguments to pass to the evaluator.
Returns:
AsyncIterator[ExperimentResultRow]: An async iterator over the experiment results.
Environment:
- LANGSMITH_TEST_CACHE: If set, API calls will be cached to disk to save time and
cost during testing. Recommended to commit the cache files to your repository
for faster CI/CD runs.
Requires the 'langsmith[vcr]' package to be installed.
Examples:
Prepare the dataset:
.. code-block:: python
import asyncio
from langsmith import Client
client = Client()
dataset = client.clone_public_dataset(
"https://smith.langchain.com/public/419dcab2-1d66-4b94-8901-0357ead390df/d"
)
dataset_name = "Evaluate Examples"
Basic usage:
.. code-block:: python
def accuracy(outputs: dict, reference_outputs: dict) -> dict:
# Row-level evaluator for accuracy.
pred = outputs["resposen"]
expected = reference_outputs["answer"]
return {"score": expected.lower() == pred.lower()}
def precision(outputs: list[dict], reference_outputs: list[dict]) -> dict:
# Experiment-level evaluator for precision.
# TP / (TP + FP)
predictions = [out["response"].lower() for out in outputs]
expected = [ref["answer"].lower() for ref in reference_outputs]
# yes and no are the only possible answers
tp = sum([p == e for p, e in zip(predictions, expected) if p == "yes"])
fp = sum([p == "yes" and e == "no" for p, e in zip(predictions, expected)])
return {"score": tp / (tp + fp)}
async def apredict(inputs: dict) -> dict:
# This can be any async function or just an API call to your app.
await asyncio.sleep(0.1)
return {"response": "Yes"}
results = asyncio.run(
client.aevaluate(
apredict,
data=dataset_name,
evaluators=[accuracy],
summary_evaluators=[precision],
experiment_prefix="My Experiment",
description="Evaluate the accuracy of the model asynchronously.",
metadata={
"my-prompt-version": "abcd-1234",
},
)
)
Evaluating over only a subset of the examples using an async generator:
.. code-block:: python
async def example_generator():
examples = client.list_examples(dataset_name=dataset_name, limit=5)
for example in examples:
yield example
results = asyncio.run(
client.aevaluate(
apredict,
data=example_generator(),
evaluators=[accuracy],
summary_evaluators=[precision],
experiment_prefix="My Subset Experiment",
description="Evaluate a subset of examples asynchronously.",
)
)
Streaming each prediction to more easily + eagerly debug.
.. code-block:: python
results = asyncio.run(
client.aevaluate(
apredict,
data=dataset_name,
evaluators=[accuracy],
summary_evaluators=[precision],
experiment_prefix="My Streaming Experiment",
description="Streaming predictions for debugging.",
blocking=False,
)
)
async def aenumerate(iterable):
async for elem in iterable:
print(elem)
asyncio.run(aenumerate(results))
Running without concurrency:
.. code-block:: python
results = asyncio.run(
client.aevaluate(
apredict,
data=dataset_name,
evaluators=[accuracy],
summary_evaluators=[precision],
experiment_prefix="My Experiment Without Concurrency",
description="This was run without concurrency.",
max_concurrency=0,
)
)
Using Async evaluators:
.. code-block:: python
async def helpfulness(outputs: dict) -> dict:
# Row-level evaluator for helpfulness.
await asyncio.sleep(5) # Replace with your LLM API call
return {"score": outputs["output"] == "Yes"}
results = asyncio.run(
client.aevaluate(
apredict,
data=dataset_name,
evaluators=[helpfulness],
summary_evaluators=[precision],
experiment_prefix="My Helpful Experiment",
description="Applying async evaluators example.",
)
)
Evaluate an existing experiment:
.. code-block:: python
results = asyncio.run(
client.aevaluate(
# The target is the ID of the experiment we are evaluating
target="419dcab2-1d66-4b94-8901-0357ead390df",
evaluators=[accuracy, helpfulness],
summary_evaluators=[precision],
)
)
.. versionadded:: 0.2.0
r )� aevaluaterA N)� langsmith.evaluation._arunnerrH )r� r rh r- r/ r� r1 r� r2 r3 r4 r5 r7 rf �
aevaluate_r< r< r= rH J s* � _�� �zClient.aevaluater� )rS r� r� r� r� r r� r r� r� r� r r� ro r� r r r r r r r r r� rX r� )rX rT )rX r� )rX rF )rX rN )rQ rR rX r� )r` ra rb rT rZ rc r[ rV r\ rd r] rd r^ re r_ rT rf r rX rg )rc rT r� r� rX r� )
rc rT r� r� r� r� r� rT rX r� )r� r� r� rT r� r� r� r� r� r� r� r� rX r� )r� r� r� r� r� r� r� r� r� r� r� r� rX r� )FF)r� r� r� ro r� ro rX r� )r� r� rX r� )r� r r ro rX r )r� rT r� r r� r r r� r r� r ro rf r rX r� )r� r� )r� r� )r� r� )r- r. rX r� r� )rK rL r� rL rJ ro rX r� )r� r\ r_ rT )r- ra rX r� )
rK rL r� rL rJ ro r ro rX r� )ro r+ rn rV )r{ r| r} r~ rn rV )r r� r� r� r� r� r6 r� r� r� r� r� r/ r� r� r� r� r� r r� r ro rf r rX r� )r� r� rX r� )r| )rn rV rX r� )rX r� )r� r� rX r� r� )r r� r� ro rX r� )$r� r� r r� r� r� r r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r6 r� r� r� r� r� r� rR rf r rX r� )"r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r6 r� r� r� r� r� r� r� r� r� r� r� r� r� rX r )r� r� r r� r� r� rX rT )r r� r� r� rX rT )r r� rX r� )r r� rX r� )r r� rX ro )r� r� r r� rX r� )r� r� r� r� rX r� )r r� r r� rX r )r r� rX r� )r� rT rX r� )r� rT r r� rX r ) r rT r� r� r� r� r r� r� rR rX r )r rT r� r� r� r� r ro r! r� r" r� rX r# )r� r� r� r� r� r� r� r� r! r� r� r� rX r# )rX r� )rX r� )r� r� r r� r/ ro rX r0 )r rT r� r� rX ro )r� r� r r� rX r� )NNNNNNNN)r� r� r� r� r r� r" r� rV r� rW r� r� rR r� rX rX rY )r r� r� r� rX r� )r rT r� r� r� ra r^ rX r_ rX r` rb r� r� rX r� )r r� r r� rX ro )r r� r r� rX r� )
r r� r r� ri rj rk rj rX rl )r r� r r� rX r� )rt r� r� r� r r� ru r� r� rX r� rR rX rv )r r� r r� rX r� )
r r� r r� ry rz r{ rT rX r� )
r r� r r� r r� r� rR rX r� )
r r� r r� ry r� r{ r� rX r� )r� rT r� r� r r� rX r� )r r� rX ra )NNNN) rz rT r� r� r r� r r� r� r� rX r� ) r� r� r� r� r r� r r� r� r� rX r� )NNN)
r� r� r r� r r� r� r� rX r� )r= r� r� ro r ro rX r� )r r� r� r� r ro rX r� )r r� r� r� r ro rX r� )r� r� r ro rX r� )r� r� r� r� r� r� r� r� r� r� r� r� r r� r r� rf r rX r� )r� r� r r� r r� r� r� r� r� r� r� rd r� r6 r� r� r� rX r� )r6 r� ry r� rX r� )NNNNNT)r r� r r� r r� ry r� r� r� r� ro r� rV r� rR r� r� r� r� r� ro rf r rX r� )r r� r{ rT rf r rX r� ) r� r� r� rV r r� r� r� rf r rX r� )r6 r� r� rX r� r� r� r� rd r� r r� r� r� rX r )r r� r� r� r� r� r� r� r� r� rt r� r� r rX r )r6 r� rX r� )r r� rX r� )r r� r r� ry r rX r ) r r� r r� r rT r r r ro rX r� )r� r r� ro rX r� )r: r r� r� rX r )rP r! r r� rX r" ) r� r r@ r1 r0 rX r� r2 r� ro rX r% ) r8 r! r� r: r0 rX r� r� r9 r; rX r" ) r� rJ r@ r1 r0 rX r� r2 r� ro rX r% )&r r� rD rT r? rO r� rP r@ rQ r' rR r0 rX rB rS r� r� rL r� rA rT r[ rV r� r� rM r� rN r� r� r� r r� rf r rX rU ) rL r� r? rO r� rf r@ rQ r' rR rX r� )rL r� rX rU ) r� r� rl r� rB rm r� rR rf r rX rn )rL r� rX r� )r� rR r? rO r� rf r@ rQ r' rR r� r� rX r� ) r r� rl rT rs rt rA rT rL r� rX ru )
r r� r� r� rs rt r� r� rX r� )r r� r� rR rX r� )
r� r� r� r� r r� r� rR rX r� )r� rT r� r� r� r� rX r� )r� r� rX r� )r� r� r� rT r� r� rX r� )r� r� rX r� )r� r� r� r rX r� )r� r� r r� rX r� )r� r� r� rV rX r� )r� rT r� r� rZ r� r� r� r� r� r� rX r� r� rX r� )r rT r� r r� r� r� rV r r� r� rX r� r� r� ro r� r� r r� rf r rX r )r� rT rX ro )r� rT r� rT rX r� )r� r )r� rT r� rV r� rV rX r� )r� rT r� ro rX r� )r� rT rX rT )r� rT rX ro )r� rT rX r� )r� rV r� rV r� r� r� r� r� r� r� r� r� r� rX r� )r� rT rX r� ) r� rT r� r� r� r� r� r� r� ro rX r� )r� rT r� r r� r� rX rT )r� rT r� r� r� r� r� r� r� r� r� r� rX r )r� rT rX r� )r� rT r r� rX r )
r� rT r� rR r� rV r ro rX r )r� rT r r� rX r )r� rT r� r� r� rT r� r� r� r� r� r� r� r� rX rT ) NNNNNNr r� TNT) rh r, r- r. r/ r0 r� r� r1 r� r� r� r2 rR r3 rV r4 ro r5 r6 r7 ro r r8 rf r rX rK ) rh r, r- r< r/ r0 r� r� r1 r� r� r� r2 rR r3 rV r4 ro r5 r6 r7 ro r r= rf r rX rJ ) rh r, r- r> r/ r0 r� r� r1 r� r� r� r2 rR r3 rV r4 ro r5 r6 r7 ro r r? rf r rX r@ ) rh rD r- rE r/ r0 r� r� r1 r� r� r� r2 rR r3 rV r4 ro r5 rF r7 ro r rG rf r rX rC )�r8 r9 r: r; � __slots__r� r8 r; �propertyr< r6 rE r rP rX rJ r� r� r r� r� r� r� r� � staticmethodr r
r r! r� r� rI r[ r; rg rl rb r� r� r� r� r� r� r� r� r� r� r� r� r� r� r r r
r r r r r r( r* r� r� rg �xor_argsr� r4 rU r\ r] re rg r
rp rs rw rx r~ r� r� r� r* r� r� r� r� r� r� r� r� r� r� r; r$ r� r� r� r r r r r r r r/ r5 r6 rK rF r] r< ri rk rp rq rr r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� �PromptSortField�
updated_atr� r� r� r� r� r� r r
r' r* r+ r r: rH r< r< r< r= r� g sD �� 2
� a�$�0�A�]�.�$� w
I� � � � /4�J� - �(� >�\�&
����*��(� �%�3�1
�*��h�E� �C�
�1��G��%�
5� �?�&�5�J
�
!�:�`� �4�5�/�
Z�H�%� � � �g�>�^�(�)&�,�1� �5�1� C�+�+��;�F�j� �.�#
� �@�5�2
� �,�:�9�:�#�9�g�E��� �r� r� r � model_kwargsrX r� c
C s� zddl m} W n ty td��w |� }|pi }|�dd�}z |j| fd|i|��W S tyA } zt�d|� ���d}~ww )a% Convert a prompt to OpenAI format.
Requires the `langchain_openai` package to be installed.
Args:
messages (Any): The messages to convert.
model_kwargs (Optional[Dict[str, Any]]): Model configuration arguments including
`stop` and any other required arguments. Defaults to None.
Returns:
dict: The prompt in OpenAI format.
Raises:
ImportError: If the `langchain_openai` package is not installed.
ls_utils.LangSmithError: If there is an error during the conversion process.
r )�
ChatOpenAIz�The convert_prompt_to_openai_format function requires the langchain_openaipackage to run.
Install with `pip install langchain_openai`�stopNz#Error converting to OpenAI format: )�langchain_openairR r, r� �_get_request_payloadrs rg r� )r� rQ rR �openairS r� r< r< r= � convert_prompt_to_openai_format< s �� ��rW c
C s� zddl m} W n ty td��w |pi }|�dd�}|�dd�}|�dd�}|d |||d �|��}z|j| |d
�W S tyQ } zt�d |� ���d}~ww )
a Convert a prompt to Anthropic format.
Requires the `langchain_anthropic` package to be installed.
Args:
messages (Any): The messages to convert.
model_kwargs (Optional[Dict[str, Any]]):
Model configuration arguments including `model_name` and `stop`.
Defaults to None.
Returns:
dict: The prompt in Anthropic format.
r )�
ChatAnthropicz�The convert_prompt_to_anthropic_format function requires the langchain_anthropic package to run.
Install with `pip install langchain_anthropic`�
model_namezclaude-3-haiku-20240307rS NrH )rY rH rS )rS z&Error converting to Anthropic format: r< )�langchain_anthropicrX r, r� rU rs rg r� )r� rQ rX rY rS rH � anthropicr� r< r< r= �"convert_prompt_to_anthropic_formatc s, �� ����r\ rh r� �dict[str, AttachmentInfo]c C s~ i }|| v r=| | r=| | � � D ],\}}tj|d dd�}|�� t�|j�}tdi |d ||�d�d���||�d�< q|S ) zOConvert attachments from the backend database format to the user facing format.�
presigned_urlT)rl r� )r^ �readerr� z attachment.Nr< ) r� r rU r� r� r� �contentr6 � removeprefix)rh r� �attachments_dictrD r� rM r_ r< r< r= r� � s ��r� ri �List[io.BufferedReader]c C s: | D ]}z|� � W q ty t�d|j� Y qw dS )z2Close all opened files used in multipart requests.zCould not close file: %sN)r� rs r� r� r� )ri r� r< r< r= r � s ��r )rO rP )
rQ rR rS rT rU rV rW rT rX rY )rn rT rX ro )rX r )r� r� rX r� )rS rT r� r� rX r� )rX r� )r� r� rX r� r� )r� r� r[ r� rX r� )r� r� rX r� )r� r� r� ro rX r� )r� r� r� ro )r� r rQ rX rX r� )rh r� r� rT rX r] )ri rc rX r� )�r; �
__future__r r r8 r� r� r� r r� r( r� �importlib.metadatar� rW r� �loggingr� r� r" r� r] �typingr^ r� r �inspectr �pathlibr r� r r r r r r
r r r
r r r r r r r r r �urllibr ra r r r� �requests_toolbeltr ru �typing_extensionsr r �urllib3.poolmanagerr � urllib3.utilr rA r r� r r r rg �langsmith._internalr! r� r" r# r$ �#langsmith._internal._beta_decoratorr$ �&langsmith._internal._compressed_tracesr% � langsmith._internal._constantsr& r' r( r) � langsmith._internal._multipartr* r+ r, � langsmith._internal._operationsr- r. r/ r0 r1 r2 r3 r4 �langsmith._internal._serder5 r� �langsmith.schemasr6 �zoneinfor7 r, rH rO �langchain_core.runnablesr> r- r@ r) rI rA rB rC rB rD rE rF rG rH rI rJ rK � getLoggerr8 r� r� rC rM �__annotations__�
parametersr� rm rt r_ rT r� r r� r� r� r� r� rf r� r) r� � HTTPAdapterr� r� rW r\ r� r r< r< r< r= �<module> s* L (
� (
�
�
�
�) �
)