Introduction

HumaLab SDK

Python SDK for HumaLab - A platform for adaptive AI validation.

Overview

HumaLab SDK provides a comprehensive toolkit for creating, managing, and evaluating AI validation scenarios. It enables researchers and developers to:

• Define adaptive scenarios with probabilistic distributions
• Track metrics and statistics across multiple episodes
• Manage assets and resources for simulation environments
• Upload and analyze results to the HumaLab platform

Installation

Install HumaLab SDK using pip:

pip install humalab

Quick Start

Here's a simple example to get you started:

import humalab as hl
 
# Initialize HumaLab
with hl.init(api_key="your_api_key",
             project="my_project",
             name="my_first_run",
             scenario={
                 "param1": "${uniform(0, 10)}",
                 "param2": "${gaussian(5, 1)}"
             }) as run:
 
    # Create an episode
    with run.create_episode() as episode:
        # Your validation logic here
        # ...

Key Concepts

Scenarios

Scenarios define the configuration space for your AI validation tasks using probability distributions. They allow you to create adaptive test cases that explore different parameter combinations.

Runs

A Run represents a complete validation experiment containing multiple episodes. It tracks all metrics and manages the lifecycle of your validation task.

Episodes

Episodes are individual instances of a scenario with specific parameter values. Each episode represents one execution of your validation logic.

Metrics

HumaLab supports various types of metrics for tracking validation results:

• Standard metrics for time-series data
• Scenario statistics for distribution analysis
• Summary metrics for aggregated results
• Code artifacts for version tracking

Next Steps

• Scenarios Guide - Learn how to define adaptive scenarios
• Runs & Episodes - Understand the run lifecycle
• Metrics - Track and analyze your results
• API Reference - Detailed API documentation