KasadaraAI: Optimizing Test Case Generation through Parameter Tuning

1. Executive Summary

KasadaraAI, powered by the Vertex AI Generative AI model “gemini-1.5-pro“, has been developed to automate and enhance the process of generating comprehensive test cases. This case study examines the effects of various parameter adjustments on the quality and diversity of generated test cases, providing insights into optimizing the system for different testing scenarios. 

2. Objectives

  1. Analyze the impact of different parameter configurations on test case generation.
  2. Identify optimal parameter settings for various testing requirements.
  3. Provide recommendations for fine-tuning KasadaraAI to meet specific testing needs.

3. Methodology

By conducting a series of experiments by adjusting key parameters of the KasadaraAI system and analyzing the resulting test cases. The parameters studied include: 

  • – max_output_tokens 
  • – temperature 
  • – top_p 
  • – top_k 
  • – presence_penalty
  • – frequency_penalty 
  • – seed 
  • – candidate_count

4. Current Configuration

The baseline configuration of KasadaraAI is: 

generation_config = { 
    “max_output_tokens“: 8192, 
    “temperature”: 0.2, 
    “top_p“: 0.8, 
    “top_k“: 40 
} 

Parameter Descriptions

1. max_output_tokens (8192): Defines the maximum length of the generated response, allowing for detailed, comprehensive test cases. 

2. temperature (0.2): Controls the randomness of the output. The low value produces more focused and deterministic results. 

3. top_p (0.8): Implements nucleus sampling, considering tokens with cumulative probability up to 0.8, balancing between focused output and allowing some variation. 

4. top_k (40): Limits token selection to the top 40 most likely options at each step, maintaining relevance and coherence in generated test cases. 

5. Investigation Results

Investigation 1: Increasing Diversity

Modified Configuration: 

generation_config = { 
    “max_output_tokens“: 8192, 
    “temperature”: 0.4, 
    “top_p“: 0.9, 
    “top_k“: 50, 
    “presence_penalty“: 0.6, 
    “frequency_penalty“: 0.3 
} 

Results: This configuration led to more diverse test cases, covering a wider range of scenarios and edge cases. However, it occasionally produced less relevant test cases. 

Investigation 2: Enhancing Consistency

Modified Configuration:

generation_config = { 
    “max_output_tokens“: 8192, 
    “temperature”: 0.1, 
    “top_p“: 0.75, 
    “top_k“: 30, 
    “seed”: 42 
} 

Results: This setup generated highly consistent and focused test cases, ideal for regression testing and core functionality verification.

Investigation 3: Balancing Creativity and Consistency

Modified Configuration:

generation_config = { 
    “max_output_tokens“: 8192, 
    “temperature”: 0.3, 
    “top_p“: 0.85, 
    “top_k“: 45, 
    “presence_penalty“: 0.3, 
    “frequency_penalty“: 0.2 
} 

Results: This configuration provided a good balance between diverse scenarios and consistent, relevant test cases.

6. Key Findings

  1. Temperature: Higher values (0.3-0.4) increased test case diversity, while lower values (0.1-0.2) enhanced consistency.
  2. Top_p and Top_k: Increasing these values broadened the range of generated scenarios but required careful balancing to maintain relevance.
  3. Presence and Frequency Penalties: These parameters were effective in reducing repetition and encouraging the exploration of diverse test scenarios.
  4. Seed: Using a fixed seed value ensured reproducibility, crucial for debugging and consistent test suite generation.

7. Recommendations

1. For Exploratory Testing:

  • – Increase temperature (0.4-0.5) 

  • – Increase top_p (0.9-0.95) 

  •  – Implement moderate presence and frequency penalties 

2. For Regression Testing:

  • – Keep temperature low (0.1-0.2) 

  • – Decrease top_p (0.7-0.8) 

  • – Use a fixed seed for reproducibility 

3. For General Test Suite Generation:

  • – Use moderate settings (temperature: 0.3, top_p: 0.85) 

  • – Adjust top_k based on desired variety (40-50) 

  • – Implement light presence and frequency penalties

4. For Comprehensive Test Suites:

  • – Increase max_output_tokens if needed 

  • – Use candidate_count > 1 to generate multiple sets 

  • – Combine results from multiple parameter configurations 

8. Conclusion

The study demonstrates that fine-tuning KasadaraAIs parameters can significantly impact the quality and nature of generated test cases. By adjusting these parameters, testers can tailor the output to meet specific testing objectives, from exploratory testing to rigorous regression suites. 

By leveraging these insights, testing teams can maximize the effectiveness of KasadaraAI, leading to more comprehensive test coverage and improved software quality.