declearn.utils.TomlConfig

Base class to define TOML-parsable configuration containers.

This class aims at wrapping multiple, possibly optional, sets of hyper-parameters, each of which is specified through a dedicated class, dataclass, or base python type. The use of some type-hint annotations is supported: List, Tuple, Optional and Union may be used as long as they are annotated with concrete types.

It also enables parsing these configuration from a TOML file, as well as instantiating from Python objects, possibly using field- wise parsers to convert inputs into the desired type.

Instantiation classmethods

from_toml: Instantiate by parsing a TOML configuration file. from_params: Instantiate by parsing inputs dicts (or objects).

Source code in declearn/utils/_toml_config.py

@dataclasses.dataclass
class TomlConfig:
    """Base class to define TOML-parsable configuration containers.

    This class aims at wrapping multiple, possibly optional, sets of
    hyper-parameters, each of which is specified through a dedicated
    class, dataclass, or base python type. The use of some type-hint
    annotations is supported: List, Tuple, Optional and Union may be
    used as long as they are annotated with concrete types.

    It also enables parsing these configuration from a TOML file, as
    well as instantiating from Python objects, possibly using field-
    wise parsers to convert inputs into the desired type.

    Instantiation classmethods
    --------------------------
    from_toml:
        Instantiate by parsing a TOML configuration file.
    from_params:
        Instantiate by parsing inputs dicts (or objects).
    """

    @classmethod
    def from_params(
        cls,
        **kwargs: Any,
    ) -> Self:
        """Instantiate a structured configuration from input keyword arguments.

        The input keyword arguments should match this class's fields' names.
        For each and every dataclass field of this class:

        - If unprovided, set the argument to None.
        - If a `parse_{field.name}` method exists, use that method.
        - Else, use the `default_parser` method.

        Notes
        -----
        - If a field supports being None, not passing a kwarg for it will
          by default result in setting it to None.
        - If a field has a default value and does not support being None,
          not passing a kwarg for it will by default result in using its
          default value.
        - If a field both has a default value and supports being None, it
          may be preferrable to pass an empty dict kwarg so as to use the
          field's default value rather than a non-default None value.
        - The former remarks hold up when the field does not benefit from
          a specific parser (that may implement arbitrary processing).

        Raises
        ------
        RuntimeError
            In case a field failed to be instantiated using the input key-
            word argument (or None value resulting from the lack thereof).

        Warns
        -----
        UserWarning
            In case some keyword arguments are unused due to the lack of a
            corresponding dataclass field.
        """
        fields = {}  # type: Dict[str, Any]
        # Look up expected kwargs and parse them.
        for field in dataclasses.fields(cls):
            parser = getattr(cls, f"parse_{field.name}", cls.default_parser)
            inputs = kwargs.pop(field.name, None)
            try:
                fields[field.name] = parser(field, inputs)
            except Exception as exc:  # pylint: disable=broad-except
                raise RuntimeError(
                    f"Failed to parse '{field.name}' field: {exc}"
                ) from exc
        # Warn about unused keyword arguments.
        for key in kwargs:
            warnings.warn(
                f"Unsupported keyword argument in {cls.__name__}.from_params: "
                f"'{key}'. This argument was ignored.",
                category=RuntimeWarning,
            )
        return cls(**fields)

    @classmethod
    def default_parser(
        cls,
        field: dataclasses.Field,  # future: dataclasses.Field[T] (Py >=3.9)
        inputs: Union[str, Dict[str, Any], T, None],
    ) -> Any:
        """Default method to instantiate a field from python inputs.

        Parameters
        ----------
        field: dataclasses.Field[<T>]
            Field that is being instantiated.
        inputs: str or dict or <T> or None
            Provided inputs to instantiate the field.
            - If valid as per `field.type`, return inputs.
            - If None (and None is invalid), return the field's default value.
            - If dict, treat them as kwargs to the field's type constructor.
            - If str, treat as the path to a TOML file specifying the object.

        Notes
        -----
        If `inputs` is str and treated as the path to a TOML file,
        it will be parsed in one of the following ways:

        - Call `field.type.from_toml` if `field.type` is a TomlConfig.
        - Use the file's `field.name` section as kwargs, if it exists.
        - Use the entire file's contents as kwargs otherwise.

        Raises
        ------
        TypeError
            If instantiation failed, for any reason.

        Returns
        -------
        object: <T>
            Instantiated object that matches the field's specifications.
        """
        # Case of valid inputs: return them as-is (including valid None).
        if _isinstance_generic(inputs, field.type):  # see function's notes
            return inputs
        # Case of None inputs: return default value if any, else raise.
        if inputs is None:
            if field.default is not dataclasses.MISSING:
                return field.default
            if field.default_factory is not dataclasses.MISSING:
                return field.default_factory()
            raise TypeError(
                f"Field '{field.name}' does not provide a default value."
            )
        # Case of str inputs poiting to a file: try parsing it.
        if isinstance(inputs, str) and os.path.isfile(inputs):
            try:
                with open(inputs, "rb") as file:
                    config = tomllib.load(file, parse_float=_parse_float)
            except tomllib.TOMLDecodeError as exc:
                raise TypeError(
                    f"Field {field.name}: could not parse secondary TOML file."
                ) from exc
            # Look for a subsection, otherwise keep the entire file.
            section = config.get(field.name, config)
            # Recursively call this parser.
            return cls.default_parser(field, section)
        # Case of dict inputs: try instantiating the target type.
        if isinstance(inputs, dict):
            return _instantiate_field(field, **inputs)
        # Otherwise, raise a TypeError.
        raise TypeError(f"Failed to parse inputs for field {field.name}.")

    @classmethod
    def from_toml(
        cls,
        path: str,
        warn_user: bool = True,
        use_section: Optional[str] = None,
        section_fail_ok: bool = False,
    ) -> Self:
        """Parse a structured configuration from a TOML file.

        The parsed TOML configuration file should be organized into sections
        that are named after this class's fields, and provide parameters to
        be parsed by the field's associated dataclass.

        Notes
        -----
        - Sections for fields that have default values may be missing.
        - Parameters with default values may also be missing.
        - All None values should be written as nan ones, as TOML does
          not have a null data type.

        Parameters
        ----------
        path: str
            Path to a TOML configuration file, that provides with the
            hyper-parameters making up for the FL "run" configuration.
        warn_user: bool, default=True
            Boolean indicating whether to raise a warning when some
            fields are unused. Useful for cases where unused fields are
            expected, e.g. in declearn-quickrun mode.
        use_section: optional(str), default=None
            If not None, points to a specific section of the TOML that
            should be used, rather than the whole file. Useful to parse
            orchestrating TOML files, e.g. in declearn-quickrun mode.
        section_fail_ok: bool, default=False
            If True, allow the section specified in use_section to be
            missing from the TOML file without raising an Error.

        Raises
        ------
        RuntimeError
            If parsing fails, whether due to misformatting of the TOML
            file, presence of undue parameters, or absence of required
            ones.

        Warns
        -----
        UserWarning
            In case some sections of the TOML file are unused due to the
            lack of a corresponding dataclass field.
        """
        # Parse the TOML configuration file.
        try:
            with open(path, "rb") as file:
                config = tomllib.load(file, parse_float=_parse_float)
        except tomllib.TOMLDecodeError as exc:
            raise RuntimeError(
                "Failed to parse the TOML configuration file."
            ) from exc
        # Look for expected config sections in the parsed TOML file.
        if isinstance(use_section, str):
            try:
                config = config[use_section]
            except KeyError as exc:
                if not section_fail_ok:
                    raise KeyError("Specified section not found") from exc
        params = {}  # type: Dict[str, Any]
        for field in dataclasses.fields(cls):
            # Case when the section is provided: set it up for parsing.
            if field.name in config:
                params[field.name] = config.pop(field.name)
            # Case when the section is missing: raise if it is required.
            elif (
                field.default is dataclasses.MISSING
                and field.default_factory is dataclasses.MISSING
            ):
                raise RuntimeError(
                    "Missing required section in the TOML configuration "
                    f"file: '{field.name}'."
                )
        # Warn about remaining (unused) config sections.
        if warn_user:
            for name in config:
                warnings.warn(
                    f"Unsupported section encountered in {path} TOML file: "
                    f"'{name}'. This section will be ignored.",
                    category=RuntimeWarning,
                )
        # Finally, instantiate the FLConfig container.
        return cls.from_params(**params)

default_parser(field, inputs) classmethod

Default method to instantiate a field from python inputs.

Parameters:

Name	Type	Description	Default
field	dataclasses.Field	Field that is being instantiated.	required
inputs	Union[str, Dict[str, Any], T, None]	Provided inputs to instantiate the field. - If valid as per field.type, return inputs. - If None (and None is invalid), return the field's default value. - If dict, treat them as kwargs to the field's type constructor. - If str, treat as the path to a TOML file specifying the object.	required

Notes

If inputs is str and treated as the path to a TOML file, it will be parsed in one of the following ways:

• Call field.type.from_toml if field.type is a TomlConfig.
• Use the file's field.name section as kwargs, if it exists.
• Use the entire file's contents as kwargs otherwise.

Raises:

Type	Description
TypeError	If instantiation failed, for any reason.

Returns:

Name	Type	Description
object	Any	Instantiated object that matches the field's specifications.

Source code in declearn/utils/_toml_config.py

@classmethod
def default_parser(
    cls,
    field: dataclasses.Field,  # future: dataclasses.Field[T] (Py >=3.9)
    inputs: Union[str, Dict[str, Any], T, None],
) -> Any:
    """Default method to instantiate a field from python inputs.

    Parameters
    ----------
    field: dataclasses.Field[<T>]
        Field that is being instantiated.
    inputs: str or dict or <T> or None
        Provided inputs to instantiate the field.
        - If valid as per `field.type`, return inputs.
        - If None (and None is invalid), return the field's default value.
        - If dict, treat them as kwargs to the field's type constructor.
        - If str, treat as the path to a TOML file specifying the object.

    Notes
    -----
    If `inputs` is str and treated as the path to a TOML file,
    it will be parsed in one of the following ways:

    - Call `field.type.from_toml` if `field.type` is a TomlConfig.
    - Use the file's `field.name` section as kwargs, if it exists.
    - Use the entire file's contents as kwargs otherwise.

    Raises
    ------
    TypeError
        If instantiation failed, for any reason.

    Returns
    -------
    object: <T>
        Instantiated object that matches the field's specifications.
    """
    # Case of valid inputs: return them as-is (including valid None).
    if _isinstance_generic(inputs, field.type):  # see function's notes
        return inputs
    # Case of None inputs: return default value if any, else raise.
    if inputs is None:
        if field.default is not dataclasses.MISSING:
            return field.default
        if field.default_factory is not dataclasses.MISSING:
            return field.default_factory()
        raise TypeError(
            f"Field '{field.name}' does not provide a default value."
        )
    # Case of str inputs poiting to a file: try parsing it.
    if isinstance(inputs, str) and os.path.isfile(inputs):
        try:
            with open(inputs, "rb") as file:
                config = tomllib.load(file, parse_float=_parse_float)
        except tomllib.TOMLDecodeError as exc:
            raise TypeError(
                f"Field {field.name}: could not parse secondary TOML file."
            ) from exc
        # Look for a subsection, otherwise keep the entire file.
        section = config.get(field.name, config)
        # Recursively call this parser.
        return cls.default_parser(field, section)
    # Case of dict inputs: try instantiating the target type.
    if isinstance(inputs, dict):
        return _instantiate_field(field, **inputs)
    # Otherwise, raise a TypeError.
    raise TypeError(f"Failed to parse inputs for field {field.name}.")

from_params(**kwargs) classmethod

Instantiate a structured configuration from input keyword arguments.

The input keyword arguments should match this class's fields' names. For each and every dataclass field of this class:

• If unprovided, set the argument to None.
• If a parse_{field.name} method exists, use that method.
• Else, use the default_parser method.

Notes

• If a field supports being None, not passing a kwarg for it will by default result in setting it to None.
• If a field has a default value and does not support being None, not passing a kwarg for it will by default result in using its default value.
• If a field both has a default value and supports being None, it may be preferrable to pass an empty dict kwarg so as to use the field's default value rather than a non-default None value.
• The former remarks hold up when the field does not benefit from a specific parser (that may implement arbitrary processing).

Raises:

Type	Description
RuntimeError	In case a field failed to be instantiated using the input key- word argument (or None value resulting from the lack thereof).

Warns:

Type	Description
UserWarning	In case some keyword arguments are unused due to the lack of a corresponding dataclass field.

Source code in declearn/utils/_toml_config.py

@classmethod
def from_params(
    cls,
    **kwargs: Any,
) -> Self:
    """Instantiate a structured configuration from input keyword arguments.

    The input keyword arguments should match this class's fields' names.
    For each and every dataclass field of this class:

    - If unprovided, set the argument to None.
    - If a `parse_{field.name}` method exists, use that method.
    - Else, use the `default_parser` method.

    Notes
    -----
    - If a field supports being None, not passing a kwarg for it will
      by default result in setting it to None.
    - If a field has a default value and does not support being None,
      not passing a kwarg for it will by default result in using its
      default value.
    - If a field both has a default value and supports being None, it
      may be preferrable to pass an empty dict kwarg so as to use the
      field's default value rather than a non-default None value.
    - The former remarks hold up when the field does not benefit from
      a specific parser (that may implement arbitrary processing).

    Raises
    ------
    RuntimeError
        In case a field failed to be instantiated using the input key-
        word argument (or None value resulting from the lack thereof).

    Warns
    -----
    UserWarning
        In case some keyword arguments are unused due to the lack of a
        corresponding dataclass field.
    """
    fields = {}  # type: Dict[str, Any]
    # Look up expected kwargs and parse them.
    for field in dataclasses.fields(cls):
        parser = getattr(cls, f"parse_{field.name}", cls.default_parser)
        inputs = kwargs.pop(field.name, None)
        try:
            fields[field.name] = parser(field, inputs)
        except Exception as exc:  # pylint: disable=broad-except
            raise RuntimeError(
                f"Failed to parse '{field.name}' field: {exc}"
            ) from exc
    # Warn about unused keyword arguments.
    for key in kwargs:
        warnings.warn(
            f"Unsupported keyword argument in {cls.__name__}.from_params: "
            f"'{key}'. This argument was ignored.",
            category=RuntimeWarning,
        )
    return cls(**fields)

from_toml(path, warn_user=True, use_section=None, section_fail_ok=False) classmethod

Parse a structured configuration from a TOML file.

The parsed TOML configuration file should be organized into sections that are named after this class's fields, and provide parameters to be parsed by the field's associated dataclass.

Notes

• Sections for fields that have default values may be missing.
• Parameters with default values may also be missing.
• All None values should be written as nan ones, as TOML does not have a null data type.

Parameters:

Name	Type	Description	Default
path	str	Path to a TOML configuration file, that provides with the hyper-parameters making up for the FL "run" configuration.	required
warn_user	bool	Boolean indicating whether to raise a warning when some fields are unused. Useful for cases where unused fields are expected, e.g. in declearn-quickrun mode.	True
use_section	Optional[str]	If not None, points to a specific section of the TOML that should be used, rather than the whole file. Useful to parse orchestrating TOML files, e.g. in declearn-quickrun mode.	None
section_fail_ok	bool	If True, allow the section specified in use_section to be missing from the TOML file without raising an Error.	False

Raises:

Type	Description
RuntimeError	If parsing fails, whether due to misformatting of the TOML file, presence of undue parameters, or absence of required ones.

Warns:

Type	Description
UserWarning	In case some sections of the TOML file are unused due to the lack of a corresponding dataclass field.

Source code in declearn/utils/_toml_config.py

@classmethod
def from_toml(
    cls,
    path: str,
    warn_user: bool = True,
    use_section: Optional[str] = None,
    section_fail_ok: bool = False,
) -> Self:
    """Parse a structured configuration from a TOML file.

    The parsed TOML configuration file should be organized into sections
    that are named after this class's fields, and provide parameters to
    be parsed by the field's associated dataclass.

    Notes
    -----
    - Sections for fields that have default values may be missing.
    - Parameters with default values may also be missing.
    - All None values should be written as nan ones, as TOML does
      not have a null data type.

    Parameters
    ----------
    path: str
        Path to a TOML configuration file, that provides with the
        hyper-parameters making up for the FL "run" configuration.
    warn_user: bool, default=True
        Boolean indicating whether to raise a warning when some
        fields are unused. Useful for cases where unused fields are
        expected, e.g. in declearn-quickrun mode.
    use_section: optional(str), default=None
        If not None, points to a specific section of the TOML that
        should be used, rather than the whole file. Useful to parse
        orchestrating TOML files, e.g. in declearn-quickrun mode.
    section_fail_ok: bool, default=False
        If True, allow the section specified in use_section to be
        missing from the TOML file without raising an Error.

    Raises
    ------
    RuntimeError
        If parsing fails, whether due to misformatting of the TOML
        file, presence of undue parameters, or absence of required
        ones.

    Warns
    -----
    UserWarning
        In case some sections of the TOML file are unused due to the
        lack of a corresponding dataclass field.
    """
    # Parse the TOML configuration file.
    try:
        with open(path, "rb") as file:
            config = tomllib.load(file, parse_float=_parse_float)
    except tomllib.TOMLDecodeError as exc:
        raise RuntimeError(
            "Failed to parse the TOML configuration file."
        ) from exc
    # Look for expected config sections in the parsed TOML file.
    if isinstance(use_section, str):
        try:
            config = config[use_section]
        except KeyError as exc:
            if not section_fail_ok:
                raise KeyError("Specified section not found") from exc
    params = {}  # type: Dict[str, Any]
    for field in dataclasses.fields(cls):
        # Case when the section is provided: set it up for parsing.
        if field.name in config:
            params[field.name] = config.pop(field.name)
        # Case when the section is missing: raise if it is required.
        elif (
            field.default is dataclasses.MISSING
            and field.default_factory is dataclasses.MISSING
        ):
            raise RuntimeError(
                "Missing required section in the TOML configuration "
                f"file: '{field.name}'."
            )
    # Warn about remaining (unused) config sections.
    if warn_user:
        for name in config:
            warnings.warn(
                f"Unsupported section encountered in {path} TOML file: "
                f"'{name}'. This section will be ignored.",
                category=RuntimeWarning,
            )
    # Finally, instantiate the FLConfig container.
    return cls.from_params(**params)