testing framework for transformRequest+Response

[knjiang]

This PR adds testing framework for transformRequest and transformResponse. This is specifically an offline testing framework where we test lingua transformations against saved snapshots.

We use the existing cases and WASM bindings generated from previous PR transform_request, transform_response and validate_*_json to verify valid transforms and no regressions.

High level, my mental model is:

• coverage-report ensures internal consistency between the Universal model & all the providers
• transforms ensures external compatibility with OpenAPI schema validations during every transform and using the actual SDK post-transform.

Diagram:

  Phase 1: Capture (one-time, requires API keys)                                                                                
                                                                                                                                
  getCaseForProvider(caseName, source)                                                                                          
             │                                                                                                                  
             ▼                                                                                                                  
  ┌─────────────────────────────────────┐                                                                                       
  │   transformAndValidateRequest()     │                                                                                       
  │   • transform_request() ────────────┼──► validate_*_request()                                                               
  └──────────────────┬──────────────────┘                                                                                       
                     │                                                                                                          
                     ▼                                                                                                          
  ┌─────────────────────────────────────┐                                                                                       
  │   callProvider(target, request)     │                                                                                       
  │   • openai.chat.completions.create()│                                                                                       
  │   • anthropic.messages.create()     │                                                                                       
  └──────────────────┬──────────────────┘                                                                                       
                     │                                                                                                          
                     ▼                                                                                                          
           validate_*_response()                                                                                                
                     │                                                                                                          
                     ▼                                                                                                          
           writeFileSync(path, response)                                                                                        
           transforms/{src}_to_{tgt}/{case}.json                                                                                
                                                                                                                                
  Phase 2: Test (CI, no API calls)                                                                                              
                                                                                                                                
  getCaseForProvider(caseName, source)                                                                                          
             │                                                                                                                  
             ▼                                                                                                                  
  ┌─────────────────────────────────────┐                                                                                       
  │   transformAndValidateRequest()     │                                                                                       
  │   • transform_request() ────────────┼──► validate_*_request()                                                               
  └──────────────────┬──────────────────┘                                                                                       
                     │                                                                                                          
                     ▼                                                                                                          
             toMatchSnapshot("request")                                                                                         
                     │                                                                                                          
                     ▼                                                                                                          
  ┌─────────────────────────────────────┐                                                                                       
  │   loadAndValidateResponse()         │                                                                                       
  │   • readFileSync(path) ─────────────┼──► validate_*_response()                                                              
  └──────────────────┬──────────────────┘                                                                                       
                     │                                                                                                          
                     ▼                                                                                                          
  ┌─────────────────────────────────────┐                                                                                       
  │   transformResponseData()           │                                                                                       
  │   • transform_response() ───────────┼──► validate_*_response()                                                              
  └──────────────────┬──────────────────┘                                                                                       
                     │                                                                                                          
                     ▼                                                                                                          
             toMatchSnapshot("response")                                                                                        
                                                                                                                                

[knjiang]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• testing framework for transformRequest+Response #72 👈 (View in Graphite)
• lingua-wasm bindings for request/response #69
• Add universal param configs #61 : 1 other dependent PR (#70 )
• add anthropic messages parameter test cases #59
• add chat completion parameter test cases #58
• add openai responses parameter test cases #54
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[knjiang]

only error i found so far, this seems to be valid since we define in our case max_tokens = 100 with high reasoning.

idk if we want to throw our own error or something. https://github.com/braintrustdata/lingua/blob/main/payloads/cases/simple.ts#L146

[knjiang]

caught this from the validate_response_json js binding. responses require input_token_details and output_token_details

[knjiang]

temporary, i didn't want to explode the diff so doing this here.

[knjiang]

anthropic_to_chatcompletions means anthropic payload using a chat completions model - we save the actual chat completion response payload so we don't have to incur the LLM cost.