ŀi<h UdZddlmZddlmZddlZddd Zejejd d gejejd Z ejdejdejddejdejdejddejdejdejdejddejd ejdejdejd!d"d#$d%ejd&ejdejdejd!d"ej j"d'(d%ejd)ejdejdejdd*ejd+ejdejdejd!d"d,d-ej$d./dejd0ejdejd12d3ejd4ejdejd12ejd!d"d,d-ej$d52d6d7i8d3ejd9ejd:ejd;ejdej$d?d@eAd%ejdBejd:ejd;ejd>ej$d?dCdDeEdFejdGejd:ejd;ejd>ej$d?dHdIdCdDeJdFejdKejd:ejd;ejdej$d?dCdDeLdFg ZeejedM<ej,eedNeOZdPZej6j9dQdRSdTZej6j9dQdRSdUZej6j9dQdRSej6j>dVZ y)WzTests for tunings.tune().)types) pytest_helperNv1beta1z?https://us-central1-autopush-aiplatform.sandbox.googleapis.com/) api_versionbase_urlbleuztest prompt template)nameprompt_templategs://sararob_test/output_uri_prefixgcs_destination)metrics output_configtest_dataset_gcs_urigemini-2.5-flashzVgs://cloud-samples-data/ai-platform/generative_ai/gemini-1_5/text/sft_train_data.jsonlgcs_uri) base_modeltraining_datasetz1gcs_uri parameter is not supported in Gemini API.)r parametersexception_if_mldevtest_tune_pretuned_modelzFprojects/801452371447/locations/us-central1/models/9030969596621881344Vgs://cloud-samples-data/ai-platform/generative_ai/gemini-2_0/text/sft_train_data.jsonlzis not supported in Gemini API+test_tune_pretuned_model_with_checkpoint_id3)pre_tuned_model_checkpoint_idrrconfigtest_tune_simple_dpozModel display namePREFERENCE_TUNING)tuned_model_display_name epoch_countmethodz)parameter is not supported in Gemini API.test_tune_dpo_with_betag?)r&r'r(beta*test_non_pretuned_model_with_checkpoint_idzis not supported in Gemini API.#test_dataset_gcs_uri_all_parameters?ADAPTER_SIZE_ONEz[gs://cloud-samples-data/ai-platform/generative_ai/gemini-1_5/text/sft_validation_data.jsonl)r&r'learning_rate_multiplier adapter_sizevalidation_dataset$test_dataset_vertex_dataset_resourcezHprojects/801452371447/locations/us-central1/datasets/5779918772206829568)vertex_dataset_resourcezAvertex_dataset_resource parameter is not supported in Gemini API.,test_dataset_dataset_resource_all_parameterszHprojects/801452371447/locations/us-central1/datasets/1168232753779441664 testlabelkeytestlabelvalue)r&r'r/r0r1labelstest_tune_distillationz#meta/llama3_1@llama-3.1-8b-instructzEgs://nathreya-oss-tuning-sdk-test/distillation-openai-opposites.jsonl DISTILLATIONzdeepseek-ai/deepseek-v3.1-maaszIgs://nathreya-oss-tuning-sdk-test/distillation-val-openai-opposites.jsonlzDgs://nathreya-oss-tuning-sdk-test/ayushagra-distillation-test-folder)r(base_teacher_modelr'r1 output_uri http_optionstest_tune_oss_sftzigs://nathreya-oss-tuning-sdk-test/ayushagra-distillation-test-folder/postprocess/node-0/checkpoints/finalz=gs://nathreya-oss-tuning-sdk-test/ayushagra-distillation-test)r'r1custom_base_modelr<r=znot supported in Gemini APItest_tune_oss_sft_hyperparamsgMb0?TUNING_MODE_FULL)r'r1 learning_rate tuning_moder?r<r=test_tune_oss_distillation)r(r;r'r1r?r<r= test_tablez tunings.tune)fileglobals_for_file test_methodrE)pytest_asynciozconfig.getoption('--private')zSkipping in pre-public tests)reasonc t|jjr|jjdt j dt j ddddt jdt  }|jtjjk(sJ|jtk(sJy y zTests tuning with eval config.rrrtuning job with eval configr$r-r.[gs://cloud-samples-data/ai-platform/generative_ai/gemini-2_0/text/sft_validation_data.jsonlr&r'r/r0r1evaluation_configr!N) _api_clientvertexaituningstune genai_types TuningDatasetCreateTuningJobConfigTuningValidationDatasetrPstateJobStateJOB_STATE_PENDINGclient tuning_jobs h/var/www/html/content-pipeline/venv/lib/python3.12/site-packages/google/genai/tests/tunings/test_tune.pytest_eval_configr`s   $$#"00:RS  . .#@#&)(@@s . % J   {33EE EE E  ' '+< << <#!c r|jjr tjtjddddtjdtjdgtj tj d  tjd d }|jjdtjdtjdd ddtjd|}|jtjjk(sJyy)z&Tests tuning with eval config metrics.zprompt-relevancezXHow well does the response address the prompt?: PROMPT: {request} RESPONSE: {response} Tz:You are a cat. Make all evaluations from this perspective.)r r return_raw_outputjudge_model_system_instructionr )r rouge_1r rrr$z test-model)sampling_countautorater_model)rrautorater_configrrrrMr-r.rNrOr!N)rQrRrUEvaluationConfigMetric OutputConfigGcsDestinationAutoraterConfigrSrTrVrWrXrYrZr[)r]rPr^s r_test_eval_config_with_metricsrns#    !22   ' |"&/k     F +   I .  "..'66"6 %44( !*$$#"00:RS  . .#@#&)(@@s . % J   {33EE EE EI!rac zK|jjr|jjj dt j dt jddddt jdt  d {}|jt jjk(sJy y 7/wrL) rQrRaiorSrTrUrVrWrXrPrYrZr[r\s r_test_eval_config_asyncrqLs   zz))..#"00:RS  . .#@#&)(@@s . /  J   {33EE EE E!! sBB; B9 0B;)!__doc__rrUrpytestVERTEX_HTTP_OPTIONSrirjrkrlrP TestTableItemCreateTuningJobParametersrVrW TuningMethodr%rXrElist__annotations__setup__file__globals pytestmarkpytest_pluginsmarkskipifr`rnasynciorqrar_rsb" $ Q /+.. 8NO ++**2 222   M #8;88)6[66p O  M '8;88_6[66p <  M :8;88_6[66p5;44.1  <  M #8;88)6[66p5;44)=*  G M &8;88)6[66p5;44)="//AA   G  M 98;88)6[66p5;44.1  =  M 28;88)6[66p5;44)=),/#F;#F#Fy$   O#& M 38;88)6[66(r _  M ;8;88)6[66(r5;44)=),/#F;#F#F,v$'(89   _%( M %8;88<6[66_5;44%#C#F;#F#Fg$b0   G%( M 8;88<6[66_5;44#F;#F#Fg$#NZ0  9#& M ,8;88<6[66_5;44#F;#F#Fg$%.#NZ0   "9'* M )8;88<6[66_5;44%#C#F;#F#Fg$#NZ0   "9'wP1 D,, -Pd!] Y  %# )= =.# )&F &FR# )F  Fra