numpydantic/docs/contributing/testing.md

file_format	mystnb	myst
mystnb	output_stderr render_text_lexer render_markdown_format remove python myst	enable_extensions colon_fence

Testing

---
tags: [hide-cell]
---

from pathlib import Path
from rich.console import Console
from rich.theme import Theme
from rich.style import Style
from rich.color import Color

theme = Theme({
    "repr.call": Style(color=Color.from_rgb(110,191,38), bold=True),
    "repr.attrib_name": Style(color="slate_blue1"),
    "repr.number": Style(color="deep_sky_blue1"),
    "repr.none": Style(color="bright_magenta", italic=True),
    "repr.attrib_name": Style(color="white"),
    "repr.tag_contents": Style(color="light_steel_blue"),
    "repr.str": Style(color="violet") 
})
console = Console(theme=theme)

Also see the [`numpydantic.testing` API docs](../api/testing/index.md)
and the [Writing an Interface](../interfaces.md) guide

Numpydantic exposes a system for combinatoric testing across dtypes, shapes, and interfaces in the {mod}numpydantic.testing module.

These helper classes and functions are included in the distributed package so they can be used for downstream development of independent interfaces (though we always welcome contributions!)

Validation Cases

Each test case is parameterized by a {class}.ValidationCase.

The case is intended to be able to be partially filled in so that multiple validation cases can be merged together, but also used independently by falling back on default values.

There are three major parts to a validation case:

• Annotation specification: {attr}~.ValidationCase.annotation_dtype and {attr}~.ValidationCase.annotation_shape specifies how the {class}.NDArray {attr}.ValidationCase.annotation that is used to test against is generated
• Array specification: {attr}~.ValidationCase.dtype and {attr}~.ValidationCase.shape specify that array that will be generated to test against the annotation
• Interface specification: An {class}.InterfaceCase that refers to an {class}.Interface, and provides array generation and other auxilary logic.

Typically, one specifies a dtype along with an annotation dtype or a shape along with an annotation shape (or implicitly against the defaults for either), along with a value for passes that indicates if that combination is valid.

from numpydantic.testing import ValidationCase

dtype_case = ValidationCase(
    id="int_int", 
    dtype=int, 
    annotation_dtype=int, 
    passes=True
)
shape_case = ValidationCase(
    id="cool_shape", 
    shape=(1,2,3), 
    annotation_shape=(1,"*","2-4"), 
    passes=True
)

merged = dtype_case.merge(shape_case)
console.print(merged.model_dump(exclude={'annotation', 'model'}, exclude_unset=True))

When merging validation cases, the merged case only passes if all the original cases do.

from numpydantic.testing import ValidationCase

dtype_case = ValidationCase(
    id="int_int", 
    dtype=int, 
    annotation_dtype=int, 
    passes=True
)
shape_case = ValidationCase(
    id="uncool_shape", 
    shape=(1,2,3), 
    annotation_shape=(9,8,7), 
    passes=False
)

merged = dtype_case.merge(shape_case)
console.print(merged.model_dump(exclude={'annotation', 'model'}, exclude_unset=True))

We provide a convenience function {func}.merged_product for creating a merged product of multiple sets of test cases.

For example, you may want to create a set of dtype and shape cases and validate against all combinations

from numpydantic.testing.helpers import merged_product

dtype_cases = [
    ValidationCase(dtype=int, annotation_dtype=int, passes=True),
    ValidationCase(dtype=int, annotation_dtype=float, passes=False)
]
shape_cases = [
    ValidationCase(shape=(1,2,3), annotation_shape=(1,2,3), passes=True),
    ValidationCase(shape=(4,5,6), annotation_shape=(1,2,3), passes=False)
]

iterator = merged_product(dtype_cases, shape_cases)

console.print([i.model_dump(exclude_unset=True, exclude={'model', 'annotation'}) for i in iterator])

You can pass constraints to the {func}.merged_product iterator to filter cases that match some value, for example to get only the cases that pass:

iterator = merged_product(dtype_cases, shape_cases, conditions={"passes": True})
console.print([i.model_dump(exclude_unset=True, exclude={'model', 'annotation'}) for i in iterator])

Interface Cases

Validation cases can be paired with interface cases that handle generating arrays for the given interface from the specification in the validation case.

Since some array interfaces like Zarr have multiple possible forms of an array (in memory, on disk, in a zip file, etc.) an interface may have multiple cases that are important to test against.

The {meth}.InterfaceCase.make_array method does what you'd expect it to, creating an array, and returning the appropriate input type for the interface:

from numpydantic.testing.interfaces import NumpyCase, ZarrNestedCase

NumpyCase.make_array(shape=(1,2,3), dtype=float)

ZarrNestedCase.make_array(shape=(1,2,3), dtype=float, path=Path("__tmp__/zarr_dir"))

Interface cases also define when an interface should skip a given test parameterization. For example, some array formats can't support arbitrary object serialization, and the video class can only support 8-bit arrays of a specific shape

from numpydantic.testing.interfaces import VideoCase

VideoCase.skip(shape=(1,1), dtype=float)

This, and the array generation methods are propagated up into a ValidationCase that contains them

case = ValidationCase(shape=(1,2,3), dtype=float, interface=VideoCase)
case.skip()

The {func}.merged_product iterator automatically excludes any combinations of interfaces and test parameterizations that should be skipped.

Making Fixtures

Pytest fixtures are a useful way to re-use validation case products. To keep things tidy, you may want to use marks and ids when creating them so that you can run tests against specific interfaces or conditions with the pytest -m mark system.

import pytest

@pytest.fixture(
    params=(
        pytest.param(
            p, 
            id=p.id, 
            marks=getattr(pytest.mark, p.interface.interface.name)
        )
        for p in iterator
    )
)
def my_cases(request):
    return request.param