o +i̒@s>dZddlmZddlZddlZddlZddlmZddlm Z m Z ddl m Z m Z mZmZmZmZddlmZddlmZdd lmZzdd lmZmZmZmZWneygdd lmZmZmZmZYnwddlZdd l m!Z!dd l"m#Z#m$Z$m%Z%m&Z&e'e(Z)Gd ddeZ*GdddeddZ+GdddeZ,GdddeddZ-GdddZ.ee,e-e/fZ0GdddeZ1ee1e/fZ2Gddde.Z3d;dd Z4d!Z5dd.d/Z9d?d1d2Z:d@d6d7Z;ee e ej&e ej%gee,e-ffe edS)Bz?This module contains the evaluator classes for evaluating runs.) annotationsN)abstractmethod) AwaitableSequence)AnyCallableLiteralOptionalUnioncast) TypedDict run_helpers)schemas) BaseModelFieldValidationError validator)wraps) SCORE_TYPE VALUE_TYPEExampleRunc@s$eZdZUdZded< ded<dS)Categoryz$A category for categorical feedback.Optional[Union[float, int]]valuestrlabelN__name__ __module__ __qualname____doc____annotations__r$r$[/var/www/html/psymed-ai/venv/lib/python3.10/site-packages/langsmith/evaluation/evaluator.pyr/s rc@s8eZdZUdZded< ded< ded< ded<d S) FeedbackConfigzkConfiguration to define a type of feedback. Applied on on the first creation of a `feedback_key`. z0Literal['continuous', 'categorical', 'freeform']typerminmaxz%Optional[list[Union[Category, dict]]] categoriesNrr$r$r$r%r&8s  r&F)totalc@seZdZUdZded< dZded< dZded< dZd ed < dZd ed < e e d Z ded< dZ ded< dZ ded< dZded< dZd ed< GdddZedddddZdS)EvaluationResultzEvaluation result.rkeyNrscorerrz Optional[str]commentzOptional[dict] correction)default_factorydictevaluator_infoz%Optional[Union[FeedbackConfig, dict]]feedback_configOptional[Union[uuid.UUID, str]] source_run_id target_run_idextrac@seZdZdZdZdS)zEvaluationResult.ConfigzPydantic model configuration.FN)rr r!r" allow_extrar$r$r$r%Configbsr:T)precCs6d|vs |ddurt|ttfrtd||S)z$Check that the value is not numeric.r.NzJNumeric values should be provided in the 'score' field, not 'value'. Got: ) isinstanceintfloatloggerwarning)clsvvaluesr$r$r%check_value_non_numericgsz(EvaluationResult.check_value_non_numeric)rr r!r"r#r.rr/r0rr2r3r4r6r7r8r:rrDr$r$r$r%r,Gs2          r,c@seZdZUdZded<dS)EvaluationResultszqBatch evaluation results. This makes it easy for your evaluator to return multiple metrics at once. zlist[EvaluationResult]resultsNrr$r$r$r%rEvs rEc@s4eZdZdZe  ddd d Z  ddd dZdS) RunEvaluatorzEvaluator interface class.NrunrexampleOptional[Example]evaluator_run_idOptional[uuid.UUID]return*Union[EvaluationResult, EvaluationResults]cCsdS)zEvaluate an example.Nr$)selfrHrIrKr$r$r% evaluate_runszRunEvaluator.evaluate_runcs4tfdd}td|IdHS)z#Evaluate an example asynchronously.csBtjdiWdS1swYdS)Nr$)rhtracing_contextrPr$current_contextrKrIrHrOr$r%_run_with_contexts $z5RunEvaluator.aevaluate_run.._run_with_contextN)rQget_tracing_contextasyncioget_running_looprun_in_executor)rOrHrIrKrUr$rSr% aevaluate_runszRunEvaluator.aevaluate_runNNrHrrIrJrKrLrMrN)rr r!r"rrPrZr$r$r$r%rGs rGc@s@eZdZUdZded< ded< dZded< dZd ed <dS) ComparisonEvaluationResultzFeedback scores for the results of comparative evaluations. These are generated by functions that compare two or more runs, returning a ranking or other feedback. rr-z'dict[Union[uuid.UUID, str], SCORE_TYPE]scoresNr5r6z6Optional[Union[str, dict[Union[uuid.UUID, str], str]]]r/)rr r!r"r#r6r/r$r$r$r%r]s   r]cseZdZdZ d-d.ddZ d/d0ddZd1ddZd2ddZed3ddZ  d4d5d$d%Z  d4d6fd&d' Z d-d7d(d)Z d8d+d,Z ZS)9DynamicRunEvaluatoraA dynamic evaluator that wraps a function and transforms it into a `RunEvaluator`. This class is designed to be used with the `@run_evaluator` decorator, allowing functions that take a `Run` and an optional `Example` as arguments, and return an `EvaluationResult` or `EvaluationResults`, to be used as instances of `RunEvaluator`. Attributes: func (Callable): The function that is wrapped by this evaluator. NfuncXCallable[[Run, Optional[Example]], Union[_RUNNABLE_OUTPUT, Awaitable[_RUNNABLE_OUTPUT]]]afuncIOptional[Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]]cst|\}|rt|\}d fdd }t||ddlm}|dur4|j||d |_t|d d |_t |rR|durAt d |j||d |_t|d d |_dS|jt t t ttgtf||d |_t|d d |_dS)zInitialize the `DynamicRunEvaluator` with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns a dict or `ComparisonEvaluationResult`. inputsr2rMc,dur|S|d|d\}}}|S)NrHrIgetrd_ traced_inputsprepare_inputsr$r%process_inputs  z4DynamicRunEvaluator.__init__..process_inputsrr Nrmrr_Func was provided as a coroutine function, but afunc was also provided. If providing both, func should be a regular function to avoid ambiguity.rdr2rMr2)_normalize_evaluator_funcr langsmithrensure_traceablerbgetattr_nameinspectiscoroutinefunction TypeErrorr rrr r_RUNNABLE_OUTPUTr`rOr`rbrmrr$rkr%__init__s2     zDynamicRunEvaluator.__init__FresultUnion[EvaluationResult, dict]r6 uuid.UUID allow_no_keyboolrMr,c sttr js |_Sz2stddvr"|r"|jd<tfdddDr4tdtd id|iWStyS}ztd|d}~ww) NziExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got empty result: r-c3s|]}|vVqdSNr$).0kr}r$r% z@DynamicRunEvaluator._coerce_evaluation_result..)r.rr/zrExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score' or categorical 'value'; got r6z[Expected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got r$)r<r,r6 ValueErrorrvallr)rOr}r6rer$rr%_coerce_evaluation_results<   z-DynamicRunEvaluator._coerce_evaluation_resultrFUnion[dict, EvaluationResults]rNcsPd|vr|}fdd|dD|d<tdi|Sjtt|ddS)NrFcsg|] }j|dqS))r6)r)rrrOr6r$r% !s zBDynamicRunEvaluator._coerce_evaluation_results..T)r6rr$)copyrErr r2)rOrFr6cpr$rr%_coerce_evaluation_resultss   z.DynamicRunEvaluator._coerce_evaluation_resultsMUnion[EvaluationResult, EvaluationResults, dict, str, int, bool, float, list]cCs.t|tr |js ||_|St|}|||Sr)r<r,r6_format_evaluator_resultr)rOr}r6r$r$r%_format_result+s  z"DynamicRunEvaluator._format_resultcC t|dSzCheck if the evaluator function is asynchronous. Returns: bool: `True` if the evaluator function is asynchronous, `False` otherwise. rbhasattrrOr$r$r%is_async9 zDynamicRunEvaluator.is_asyncrHrrIrJrKrLcCst|dst}|rtd||||S|dur"t}d|j i}t |ddr4t |j |d<|j ||||dd}|||S) aEvaluate a run using the wrapped function. This method directly invokes the wrapped function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. r`tCannot call `evaluate_run` on an async run evaluator from within an running event loop. Use `aevaluate_run` instead.Nr7 session_id experimentrun_idmetadatalangsmith_extra)rrWget_event_loop is_running RuntimeErrorrun_until_completerZuuiduuid4idrurrr`r)rOrHrIrK running_looprr}r$r$r%rPBs$    z DynamicRunEvaluator.evaluate_runcs|t|dst||IdHS|durt}d|ji}t|ddr*t|j|d<|j ||||ddIdH}| ||S)aEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rbNr7rrrr) rsuperrZrrrrurrrbr)rOrHrIrKrr} __class__r$r%rZhs    z!DynamicRunEvaluator.aevaluate_runcC |||S)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. )rP)rOrHrIr$r$r%__call__ zDynamicRunEvaluator.__call__rcCd|jdS))Represent the DynamicRunEvaluator object.zrvrr$r$r%__repr__zDynamicRunEvaluator.__repr__r)r`rarbrc)F)r}r~r6rrrrMr,)rFrr6rrMrN)r}rr6rrMrNrMrr[r\)rHrrIrJrKrL)rHrrIrJrMrNrMr)rr r!r"r|rrrpropertyrrPrZrr __classcell__r$r$rr%r_s& >     )" r_r`racCt|S)zmCreate a run evaluator from a function. Decorator that transforms a function into a `RunEvaluator`. )r_r`r$r$r% run_evaluators ri'objrcCs,t|}t|tkr|dtdd}|S)Nz...))reprlen_MAXSIZE)rsr$r$r% _maxsize_reprs rc@sxeZdZdZ d$d%ddZed&d d Z d$d'ddZ d$d'ddZ d$d'ddZ d(ddZ e d)ddZ d*d"d#Z dS)+DynamicComparisonRunEvaluatorz4Compare predictions (as traces) from 2 or more runs.Nr`fCallable[[Sequence[Run], Optional[Example]], Union[_COMPARISON_OUTPUT, Awaitable[_COMPARISON_OUTPUT]]]rbUOptional[Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]]cst|\}|rt|\}d fdd }t||ddlm}|dur4|j||d |_t|d d |_t |rR|durAt d |j||d |_t|d d |_dS|jt t t tttgtf||d |_t|d d |_dS)zInitialize the `DynamicRunEvaluator` with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns an `EvaluationResult` or `EvaluationResults`. rdr2rMcre)NrunsrIrfrhrkr$r%rmrnz>DynamicComparisonRunEvaluator.__init__..process_inputsrr Nrorr_rprq)$_normalize_comparison_evaluator_funcrrsrrtrbrurvrwrxryr rrrr r_COMPARISON_OUTPUTr`r{r$rkr%r|s@      z&DynamicComparisonRunEvaluator.__init__rMrcCrrrrr$r$r%rrz&DynamicComparisonRunEvaluator.is_asyncr Sequence[Run]rIrJr]cCsjt|dst}|rtd||||St}| |}|j ||||dd}| |||S)zCompare runs to score preferences. Args: runs: A list of runs to compare. example: An optional example to be used in the evaluation. r`rrtagsr) rrWrrrr acompare_runsrr _get_tagsr`_format_results)rOrrIrr6rr}r$r$r% compare_runss"   z*DynamicComparisonRunEvaluator.compare_runscsTt|ds |||St}||}|j||||ddIdH}||||S)aEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: runs (Run): The runs to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. rbrrN)rrrrrrbr)rOrrIr6rr}r$r$r%r s    z+DynamicComparisonRunEvaluator.acompare_runscCr)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. )r)rOrrIr$r$r%r<rz&DynamicComparisonRunEvaluator.__call__rcCr)rzhzADynamicComparisonRunEvaluator._format_results..)r^r-r6r-zXExpected 'dict', 'list' or 'ComparisonEvaluationResult' result object. Received: result=r6zExpected a dictionary with a 'key' and dictionary of scores mappingrun IDs to numeric scores, or ComparisonEvaluationResult object, got r$) r<r]r6listziprvr2rr)rOr}r6rmsgrr$r$r%r\s@     z-DynamicComparisonRunEvaluator._format_resultsr)r`rrbrr)rrrIrJrMr]r)rrrMr)r}rr6rrrrMr])rr r!r"r|rrrrrr staticmethodrrr$r$r$r%rs  @      rrrMcCr)z.Create a comaprison evaluator from a function.)rrr$r$r%comparison_evaluatorsrrtuple[Union[Callable[[Run, Optional[Example]], _RUNNABLE_OUTPUT], Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]], Optional[Callable[..., dict]]]c:dtddjD}ddjD|r6tfdd|Ds@tfdd|Ddkr@d d }t|tfd d|DrR|d d gkrVdfStr|dfdd dfdd }tdrtt dn|j |_ |fSdfdd dfdd }tdrt dn|j |_ |fS)N)rHrIrdoutputsreference_outputs attachmentscS g|] \}}|j|jkr|qSr$kind VAR_KEYWORDrpnamepr$r$r%r z-_normalize_evaluator_func..cS"g|] \}}|jtjjur|qSr$defaultrw Parameteremptyrr$r$r%r c3 |] }|vp |vVqdSrr$rrargs_with_defaultssupported_argsr$r%r z,_normalize_evaluator_func..cg|]}|vr|qSr$r$rarr$r%rrUInvalid evaluator function. Must have at least one argument. Supported arguments are . Please see https://docs.smith.langchain.com/evaluation/how_to_guides/evaluation/evaluate_llm_application#use-custom-evaluatorsc3rrr$rrr$r%rrrHrIrrJrMtuple[list, dict, dict]c|||r|jni|jp i|r|jpini|r|jpinid}i}g}i}jD]/\}}||vrY|j|j|jfvrC|||n||||<|dvrSt ||n||||<q*|||fSN)rHrIrdrrr)rHrI rdrr parametersitemsrPOSITIONAL_OR_KEYWORDPOSITIONAL_ONLYrrrHrIarg_mapkwargsargsrj param_nameparamsigr$r%_prepare_inputs8      z2_normalize_evaluator_func.._prepare_inputsrzc&||\}}}|i|IdHSrr$rHrIr rrirr`r$r%awrapperz+_normalize_evaluator_func..awrapperrcrrrrr r$r%rrc||\}}}|i|Srr$rrr$r%wrapper sz*_normalize_evaluator_func..wrapper)rHrrIrJrMr)rHrrIrJrMrz rw signaturerrrrrrxrrurr`all_argsrrrr$rrr`r rr%rrsP    rrtuple[Union[Callable[[Sequence[Run], Optional[Example]], _COMPARISON_OUTPUT], Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]], Optional[Callable[..., dict]]]cr)NrrIrdrrcSrr$rrr$r$r%r rz8_normalize_comparison_evaluator_func..cSrr$rrr$r$r%r!rc3rrr$rrr$r%r'rz7_normalize_comparison_evaluator_func..crr$r$rrr$r%r*rrrrc3rrr$rrr$r%r5rrrIrrJrMrc|||r|jnidd|D|r|jpinid}i}g}i}jD]/\}}||vrS|j|j|jfvr=|||n||||<|dvrMt||n||||<q$|||fS)NcSg|]}|jpiqSr$rrrHr$r$r%rFQ_normalize_comparison_evaluator_func.._prepare_inputs..rrrI rdrrrrrrrrrrIrrr rjr r r r$r%r?2      z=_normalize_comparison_evaluator_func.._prepare_inputsrcrrr$rrIr rrirr$r%r]rz6_normalize_comparison_evaluator_func..awrapperrcr)NcSrr$r r!r$r$r%rsr"r#rr$r%r&r r$r%rlr'crrr$r(rr$r%rsz5_normalize_comparison_evaluator_func..wrapper)rrrIrJrMr)rrrIrJrMrrrr$rr%rsP    rr};Union[EvaluationResults, dict, str, int, bool, float, list]Union[EvaluationResults, dict]cCst|tttfrd|i}|S|std|t|tr3tdd|Ds-td|dd|i}|St|tr>d|i}|St|trF |Std |) Nr.zdExpected a non-empty dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got css|]}t|tVqdSr)r<r2)rxr$r$r%rsz+_format_evaluator_result..z8Expected a list of dicts or EvaluationResults. Received .rFrzZExpected a dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got ) r<rr>r=rrrrr2rr$r$r%rs6    rSUMMARY_EVALUATOR_TcsdtddjD}ddjD|r6tfdd|DsJtfdd|DdkrJd d }|rF|d |d 7}t|tfd d|Dr[|d dgkr]Sdfdd }tdrrtd|_ |S|j |_ |S)NrexamplesrdrrcSsg|]\}}|qSr$r$rr$r$r%rsz0_normalize_summary_evaluator..cSrr$rrr$r$r%rrc3rrr$rrr$r%rrz/_normalize_summary_evaluator..crr$r$rrr$r%rrrrr,z Received arguments c3s|]}|vVqdSrr$r)rr$r%rrrr/Sequence[schemas.Run]Sequence[schemas.Example]rMrNcs||dd|Ddd|Ddd|Dd}i}g}jD]\}}||vr?|j|j|jfvr9|||q ||||<q |i|}t|trN|St|S)NcSsg|]}|jqSr$)rdrrIr$r$r%rszA_normalize_summary_evaluator..wrapper..cSrr$r r!r$r$r%rr"cSrr$r r2r$r$r%rr"r.) rrrrrrr<r,r)rr/rrr r r r})r`r r$r%rs*     z-_normalize_summary_evaluator..wrapperr)rr0r/r1rMrN) rwrrrrrrrrur)r`rrrr$)rr`r rr%_normalize_summary_evaluators> r3)r`ra)rr)r`rrMr)r`rrMr)r`rrMr)r}r)rMr*)r`rrMr-)?r" __future__rrWrwrabcrcollections.abcrrtypingrrrr r r typing_extensionsr rsrrQr pydantic.v1rrrr ImportErrorpydanticlogging functoolsrlangsmith.schemasrrrr getLoggerrr?rr&r,rErGr2rzr]rr_rrrrrrrrrrr-r3r$r$r$r%sn         /   i  M