👩‍💻 API Reference

experiment_results_manager

ExperimentRun

Example usage:

import experiment_results_manager as erm

# Create an experiment run
er = erm.ExperimentRun(
    experiment_id="my_experiment",
    variant_id="main"
)

# Log relevant data
er.log_param("objective", "rmse")
er.log_metric("rmse", "0.9")
er.log_figure(mpl_fig, "ROC Curve")
er.log_text("lorem ipsum...", "text")
er.log_artifact(
    pickle.dumps(model),
    artifact_id="model",
    filename="model.pickle"
)

# Display the run
html = erm.compare_runs(er)
# Or compare side by side with other runs
html = erm.compare_runs(er, er2, er3)
display(HTML(html))

# Save the run to access later
saved_path = erm.save_run_to_registry(er, "s3://erm-registry")

init

__init__(
    experiment_id: str,
    variant_id: str = "main",
    run_id: Optional[str] = None,
    timestamp_utc: Optional[datetime] = None,
    features: Optional[List[str]] = None,
    params: Optional[
        Dict[str, Union[str, int, float]]
    ] = None,
    metrics: Optional[
        Dict[str, Union[str, int, float]]
    ] = None,
    dicts: Optional[Dict[str, Dict[str, Any]]] = None,
    artifacts: Optional[Dict[str, Artifact]] = None,
) -> None

Initializes a new ExperimentRun object with the given experiment_id, variant_id, run_id, and timestamp_utc. If timestamp_utc is not provided, it defaults to the current UTC datetime. If run_id is not provided, it is generated from the timestamp_utc using the format "%Y_%m_%d__%H_%M_%S".

log_artifact

log_artifact(
    src_path_or_bytes: Union[str, bytes],
    artifact_id: str,
    filename: Union[str, None] = None,
    artifact_type: ArtifactType = ArtifactType.BINARY,
) -> None

Logs an artifact to the experiment run. :param str|bytes src_path_or_bytes: The path or byte array to the artifact. :param str artifact_id: The id of the artifact. :param str|None filename: The run-relative filename of the artifact. :param ArtifactType artifact_type: The type of the artifact.

log_dict

log_dict(
    dict_name: str,
    data: Dict[str, Union[str, int, float, Any]],
) -> None

Logs a dictionary of data with the given dict_name.

log_figure

log_figure(
    fig: Union[
        plotly.graph_objs.Figure,
        matplotlib.figure.Figure,
        matplotlib.axes.Axes,
    ],
    artifact_id: str,
) -> None

Logs a figure to the experiment run.

log_image

log_image(
    src_path_or_bytes: Union[str, bytes],
    artifact_id: str,
    filename: Optional[str] = None,
) -> None

Logs an image with the given artifact_id and either the path to the image file (src_path_or_bytes is a string) or the contents of the image file (src_path_or_bytes is bytes). If a filename is not provided, the artifact_id is used as the filename. The image is stored as a PNG artifact in the artifacts dictionary of the ExperimentRun object.

log_metric

log_metric(key: str, value: Union[str, int, float]) -> None

Logs a metric to the experiment run.

log_metrics

log_metrics(
    data: Dict[str, Union[str, int, float]]
) -> None

Appends data to the metrics dict.

log_param

log_param(key: str, value: Union[str, int, float]) -> None

Logs a parameter to the experiment run.

log_params

log_params(data: Dict[str, Union[str, int, float]]) -> None

Appends data to the params dict.

log_text

log_text(text: Union[str, bytes], artifact_id: str) -> None

Logs text with the given artifact_id and text (str or bytes that represent utf-8). The text is stored as a binary artifact in the ExperimentRun.

list_experiments

list_experiments(
    registry_uri: str,
    fs: Optional[AbstractFileSystem] = None,
) -> List[str]

List experiments in the given registry.

Parameters:

Name	Type	Description	Default
`registry_uri`	`str`	The URI of the registry to list experiments for.	required
`fs`	`Optional[fsspec.AbstractFileSystem]`	The filesystem to use to access the registry, by default None.	`None`

Returns:

Type	Description
`List[str]`	List[str]: A list of experiment names.

list_runs

list_runs(
    registry_uri: str,
    experiment_id: str,
    variant_id: str,
    fs: Optional[AbstractFileSystem] = None,
) -> List[str]

List the runs for the given experiment and variant.

Parameters:

Name	Type	Description	Default
`registry_uri`	`str`	The URI of the registry where the experiment is stored.	required
`experiment_id`	`str`	The ID of the experiment.	required
`variant_id`	`str`	The ID of the variant.	required
`fs`	`Optional[fsspec.AbstractFileSystem]`	The filesystem to use. If None, a new filesystem will be created using fsspec. Defaults to None.	`None`

Returns:

Type	Description
`List[str]`	List[str]: A list of strings representing the runs for the given experiment and variant.

list_variants

list_variants(
    registry_uri: str,
    experiment_id: str,
    fs: Optional[AbstractFileSystem] = None,
) -> List[str]

List the variants available for an experiment in the given registry URI and experiment ID.

Parameters:

Name	Type	Description	Default
`registry_uri`	`str`	The URI of the registry to list the variants from.	required
`experiment_id`	`str`	The ID of the experiment to list the variants from.	required
`fs`	`Optional[fsspec.AbstractFileSystem]`	The filesystem to use to access the registry. If None, a new one will be created for the given experiment ID. Defaults to None.	`None`

Returns:

Type	Description
`List[str]`	List[str]: A list of variant names stored for the experiment.

load_run_from_path

load_run_from_path(run_path: str) -> ExperimentRun

Load an ExperimentRun object from a given path.

Parameters:

Name	Type	Description	Default
`run_path`	`str`	The path to the ExperimentRun directory.	required

Returns:

Name	Type	Description
`ExperimentRun`	`ExperimentRun`	An ExperimentRun object populated with data from the given path.

load_run_from_registry

load_run_from_registry(
    experiment_registry_path: str,
    experiment_id: str,
    variant_id: str = "main",
    run_id: Optional[str] = None,
) -> ExperimentRun

Load an experiment run from its registry path.

Parameters:

Name	Type	Description	Default
`experiment_registry_path`	`str`	The root path of the experiment registry.	required
`experiment_id`	`str`	The ID of the experiment.	required
`variant_id`	`str`	The ID of the variant. Defaults to "main".	`'main'`
`run_id`	`str`	The ID of the run. If not provided, the latest run for the variant is used.	`None`

Returns:

Name	Type	Description
`ExperimentRun`	`ExperimentRun`	The loaded experiment run.

Raises:

Type	Description
`ValueError`	If the experiment or variant ID is invalid.
`FileNotFoundError`	If the run path does not exist.

save_run_to_path

save_run_to_path(
    er: ExperimentRun,
    path: str,
    fs: Optional[fsspec.AbstractFileSystem] = None,
    overwrite: bool = False,
) -> None

Save an experiment run to a file system path.

Parameters:

Name	Type	Description	Default
`er`	`ExperimentRun`	The experiment run to save.	required
`path`	`str`	The path to save the experiment run to.	required
`fs`	`Optional[fsspec.AbstractFileSystem]`	The file system to use. Defaults to None.	`None`
`overwrite`	`bool`	Whether to overwrite an existing experiment run at the same path. Defaults to False.	`False`

Raises:

Type	Description
`FileExistsError`	If a run already exists at the given path and `overwrite` is set to False.

Returns:

Type	Description
`None`	None

save_run_to_registry

save_run_to_registry(
    er: ExperimentRun,
    experiment_registry_path: str,
    fs: Optional[fsspec.AbstractFileSystem] = None,
    overwrite: bool = False,
) -> str

Saves an ExperimentRun object to a registry on a file system.

Parameters:

Name	Type	Description	Default
`er`	`ExperimentRun`	The ExperimentRun object to save.	required
`experiment_registry_path`	`str`	The path to the experiment registry directory.	required
`fs`	`Optional[fsspec.AbstractFileSystem]`	The file system to use. If None, get_fs_from_uri(experiment_registry_path) is used.	`None`
`overwrite`	`bool`	Whether to overwrite an existing run with the same ID. If False and a run with the same ID exists, a ValueError is raised.	`False`

Returns:

Name	Type	Description
`str`	`str`	The path to the saved run.