fovi.training.loading

Restore training state from an existing experiment.

class fovi.training.loading.Sequence

Bases: Reversible, Collection

All the operations on a read-only sequence.

Concrete subclasses must override __new__ or __init__, __getitem__, and __len__.

count(value) → integer -- return number of occurrences of value

index(value[, start[, stop]]) → integer -- return first index of value.

Raises ValueError if the value is not present.

Supporting start and stop arguments is optional, but recommended.

class fovi.training.loading.OmegaConf

Bases: object

OmegaConf primary class

__init__() → None

static structured(obj: Any, parent: BaseContainer | None = None, flags: Dict[str, bool] | None = None) → Any

static create(obj: str, parent: BaseContainer | None = None, flags: Dict[str, bool] | None = None) → DictConfig | ListConfig

static create(obj: List[Any] | Tuple[Any, ...], parent: BaseContainer | None = None, flags: Dict[str, bool] | None = None) → ListConfig

static create(obj: DictConfig, parent: BaseContainer | None = None, flags: Dict[str, bool] | None = None) → DictConfig

static create(obj: ListConfig, parent: BaseContainer | None = None, flags: Dict[str, bool] | None = None) → ListConfig

static create(obj: Dict[Any, Any] | None = None, parent: BaseContainer | None = None, flags: Dict[str, bool] | None = None) → DictConfig

static load(file_: str | Path | IO[Any]) → DictConfig | ListConfig

static save(config: Any, f: str | Path | IO[Any], resolve: bool = False) → None

Save as configuration object to a file

Parameters:

• config – omegaconf.Config object (DictConfig or ListConfig).

• f – filename or file object

• resolve – True to save a resolved config (defaults to False)

static from_cli(args_list: List[str] | None = None) → DictConfig

static from_dotlist(dotlist: List[str]) → DictConfig

Creates config from the content sys.argv or from the specified args list of not None

Parameters:

dotlist – A list of dotlist-style strings, e.g. ["foo.bar=1", "baz=qux"].

Returns:

A DictConfig object created from the dotlist.

static merge(*configs: DictConfig | ListConfig | Dict[str | bytes | int | Enum | float | bool, Any] | List[Any] | Tuple[Any, ...] | Any) → ListConfig | DictConfig

Merge a list of previously created configs into a single one

Parameters:

configs – Input configs

Returns:

the merged config object.

static unsafe_merge(*configs: DictConfig | ListConfig | Dict[str | bytes | int | Enum | float | bool, Any] | List[Any] | Tuple[Any, ...] | Any) → ListConfig | DictConfig

Merge a list of previously created configs into a single one This is much faster than OmegaConf.merge() as the input configs are not copied. However, the input configs must not be used after this operation as will become inconsistent.

Parameters:

configs – Input configs

Returns:

the merged config object.

static register_resolver(name: str, resolver: Callable[[...], Any]) → None

static legacy_register_resolver(name: str, resolver: Callable[[...], Any]) → None

static register_new_resolver(name: str, resolver: Callable[[...], Any], *, replace: bool = False, use_cache: bool = False) → None

Register a resolver.

Parameters:

• name – Name of the resolver.

• resolver – Callable whose arguments are provided in the interpolation, e.g., with ${foo:x,0,${y.z}} these arguments are respectively “x” (str), 0 (int) and the value of y.z.

• replace – If set to False (default), then a ValueError is raised if an existing resolver has already been registered with the same name. If set to True, then the new resolver replaces the previous one. NOTE: The cache on existing config objects is not affected, use OmegaConf.clear_cache(cfg) to clear it.

• use_cache – Whether the resolver’s outputs should be cached. The cache is based only on the string literals representing the resolver arguments, e.g., ${foo:${bar}} will always return the same value regardless of the value of bar if the cache is enabled for foo.

classmethod has_resolver(name: str) → bool

static clear_resolvers() → None

Clear(remove) all OmegaConf resolvers, then re-register OmegaConf’s default resolvers.

classmethod clear_resolver(name: str) → bool

Clear(remove) any resolver only if it exists.

Returns a bool: True if resolver is removed and False if not removed.

Parameters:

name – Name of the resolver.

Returns:

A bool (True if resolver is removed, False if not found before removing).

static get_cache(conf: BaseContainer) → Dict[str, Any]

static set_cache(conf: BaseContainer, cache: Dict[str, Any]) → None

static clear_cache(conf: BaseContainer) → None

static copy_cache(from_config: BaseContainer, to_config: BaseContainer) → None

static set_readonly(conf: Node, value: bool | None) → None

static is_readonly(conf: Node) → bool | None

static set_struct(conf: Container, value: bool | None) → None

static is_struct(conf: Container) → bool | None

static masked_copy(conf: DictConfig, keys: str | List[str]) → DictConfig

Create a masked copy of of this config that contains a subset of the keys

Parameters:

• conf – DictConfig object

• keys – keys to preserve in the copy

Returns:

The masked DictConfig object.

static to_container(cfg: Any, *, resolve: bool = False, throw_on_missing: bool = False, enum_to_str: bool = False, structured_config_mode: SCMode = SCMode.DICT) → Dict[str | bytes | int | Enum | float | bool, Any] | List[Any] | None | str | Any

Resursively converts an OmegaConf config to a primitive container (dict or list).

Parameters:

• cfg – the config to convert

• resolve – True to resolve all values

• throw_on_missing – When True, raise MissingMandatoryValue if any missing values are present. When False (the default), replace missing values with the string “???” in the output container.

• enum_to_str – True to convert Enum keys and values to strings

• structured_config_mode –

  Specify how Structured Configs (DictConfigs backed by a dataclass) are handled.

  • By default (structured_config_mode=SCMode.DICT) structured configs are converted to plain dicts.

  • If structured_config_mode=SCMode.DICT_CONFIG, structured config nodes will remain as DictConfig.

  • If structured_config_mode=SCMode.INSTANTIATE, this function will instantiate structured configs (DictConfigs backed by a dataclass), by creating an instance of the underlying dataclass.

  See also OmegaConf.to_object.

Returns:

A dict or a list representing this config as a primitive container.

static to_object(cfg: Any) → Dict[str | bytes | int | Enum | float | bool, Any] | List[Any] | None | str | Any

Resursively converts an OmegaConf config to a primitive container (dict or list). Any DictConfig objects backed by dataclasses or attrs classes are instantiated as instances of those backing classes.

This is an alias for OmegaConf.to_container(…, resolve=True, throw_on_missing=True,

structured_config_mode=SCMode.INSTANTIATE)

Parameters:

cfg – the config to convert

Returns:

A dict or a list or dataclass representing this config.

static is_missing(cfg: Any, key: str | bytes | int | Enum | float | bool) → bool

static is_interpolation(node: Any, key: int | str | None = None) → bool

static is_list(obj: Any) → bool

static is_dict(obj: Any) → bool

static is_config(obj: Any) → bool

static get_type(obj: Any, key: str | None = None) → Type[Any] | None

static select(cfg: Container, key: str, *, default: Any = _DEFAULT_MARKER_, throw_on_resolution_failure: bool = True, throw_on_missing: bool = False) → Any

Parameters:

• cfg – Config node to select from

• key – Key to select

• default – Default value to return if key is not found

• throw_on_resolution_failure – Raise an exception if an interpolation resolution error occurs, otherwise return None

• throw_on_missing – Raise an exception if an attempt to select a missing key (with the value ‘???’) is made, otherwise return None

Returns:

selected value or None if not found.

static update(cfg: Container, key: str, value: Any = None, *, merge: bool = True, force_add: bool = False) → None

Updates a dot separated key sequence to a value

Parameters:

• cfg – input config to update

• key – key to update (can be a dot separated path)

• value – value to set, if value if a list or a dict it will be merged or set depending on merge_config_values

• merge – If value is a dict or a list, True (default) to merge into the destination, False to replace the destination.

• force_add – insert the entire path regardless of Struct flag or Structured Config nodes.

static to_yaml(cfg: Any, *, resolve: bool = False, sort_keys: bool = False) → str

returns a yaml dump of this config object.

Parameters:

• cfg – Config object, Structured Config type or instance

• resolve – if True, will return a string with the interpolations resolved, otherwise interpolations are preserved

• sort_keys – If True, will print dict keys in sorted order. default False.

Returns:

A string containing the yaml representation.

static resolve(cfg: Container) → None

Resolves all interpolations in the given config object in-place.

Parameters:

cfg – An OmegaConf container (DictConfig, ListConfig) Raises a ValueError if the input object is not an OmegaConf container.

static missing_keys(cfg: Any) → Set[str]

Returns a set of missing keys in a dotlist style.

Parameters:

cfg – An OmegaConf.Container, or a convertible object via OmegaConf.create (dict, list, …).

Returns:

set of strings of the missing keys.

Raises:

ValueError – On input not representing a config.

fovi.training.loading.open_dict(config: Container) → Generator[Container, None, None]

fovi.training.loading.find_config(base_fn: str, load: bool, model_dirs: Sequence[str | Path] | None = None, device: str | device = 'cuda') → tuple[DictConfig, dict[str, dict[str, Tensor]] | None, str | None]

Search explicit local locations, then download a model from the Hub.

An existing local model with malformed configuration or missing weights raises immediately. It must not silently select a different checkpoint from the Hub.

class fovi.training.loading.HiddenPrints(enabled=True)

Bases: object

Context manager to suppress stdout output.

Temporarily redirects stdout to devnull to hide print statements from called functions.

Parameters:

enabled (bool, optional) – Whether to suppress prints. If False, acts as a no-op. Defaults to True.

Example

>>> with HiddenPrints():
...     print("This won't be shown")
>>> print("This will be shown")
This will be shown

__init__(enabled=True)

class fovi.training.loading.Trainer(gpu, cfg: DictConfig, load_checkpoint=True)

Bases: object

__init__(gpu, cfg: DictConfig, load_checkpoint=True)

Initialize trainer with hydra configuration

Parameters:

• gpu – which gpu to run on (or None to use cpu)

• cfg – Hydra configuration object

• load_checkpoint – Whether to load checkpoint

setup_distributed()

Initialize distributed training process group.

cleanup_distributed()

Clean up distributed training process group.

create_optimizer()

Create and configure optimizers for model and probes.

Sets up separate optimizers for the main model and linear probes, with appropriate weight decay settings and learning rate scaling.

create_train_loader(train_dataset, subset=None, batches_ahead=3, phase='train')

Create training data loader with appropriate transforms and augmentation.

Parameters:

• train_dataset (str) – Path to training dataset file

• subset (float, optional) – Fraction of dataset to use for faster prototyping

• batches_ahead (int) – Number of batches to prefetch

• phase (str) – Training phase (‘train’ or other)

Returns:

Configured data loader for training

Return type:

FlashLoader

create_val_loader(val_dataset, subset=None, ratio=1.)

Create validation data loader with center crop transforms.

Parameters:

• val_dataset (str) – Path to validation dataset file

• subset (float, optional) – Fraction of dataset to use

• ratio (float, optional) – crop linear ratio

Returns:

Configured data loader for validation

Return type:

FlashLoader

create_standard_loader(dataset, batch_size, num_workers, resolution)

Create standard data loader with basic transforms.

Parameters:

• dataset – Dataset to create loader for

• batch_size (int) – Batch size for the data loader

• num_workers (int) – Number of worker processes for data loading

• resolution (int) – Target resolution to resize images to

Returns:

Standard PyTorch data loader with basic image transforms

(ToTensor, Resize, Normalize) applied

Return type:

DataLoader

create_model_and_scaler()

Create and configure the neural network model and gradient scaler. :returns:

(model, scaler) where model is the configured neural network

and scaler is the gradient scaler for mixed precision training

Return type:

tuple

reset_model()

Reset the model by recreating it from scratch.

train()

Execute the main training loop.

Runs training for the specified number of epochs, performing validation at regular intervals and saving checkpoints. Handles learning rate scheduling and early stopping.

Returns:

Training statistics for all epochs

Return type:

dict

eval_and_log(extra_dict={})

Run validation and log results.

Parameters:

extra_dict (dict) – Additional data to include in logging

Returns:

Validation statistics

Return type:

dict

load_checkpoint(ckpt=None)

Load model and optimizer state from checkpoint.

Parameters:

ckpt (dict, optional) – Checkpoint dictionary. If None, loads from default checkpoint file in log folder.

checkpoint(epoch)

Save checkpoint at regular intervals based on checkpoint frequency.

save_checkpoint(epoch)

Save model and optimizer state to checkpoint file.

Parameters:

epoch (int) – Current training epoch

train_loop(epoch, max_batches=None)

Execute one epoch of training.

Parameters:

• epoch (int) – Current epoch number

• max_batches (int, optional) – Maximum number of batches to process

Returns:

(average_loss, training_stats)

Return type:

tuple

val_loop(return_preds=False, repeats=None)

Execute validation loop.

Computes validation metrics for all n_fixations values in self.n_fixations_val. Runs a single forward pass with max(n_fixations_val) and slices outputs to evaluate at each fixation count.

Parameters:

• return_preds (bool) – Whether to return predictions

• repeats (int, optional) – Number of times to repeat validation

Returns:

Validation statistics, optionally with predictions and labels

Return type:

dict or tuple

compute_activations(loader, layer_names=['projector'], fixation_size=None, area_range=None, training=False, n_fixations=None, max_batches=None, setting='supervised', do_postproc=False, **kwargs)

Extract activations from specified layers for a given data loader.

Runs the model on data from the loader and captures intermediate activations from the specified layers using forward hooks.

Parameters:

• loader – Data loader to iterate over.

• layer_names (list, optional) – List of layer names to capture activations from. Defaults to [‘projector’].

• fixation_size (int or tuple, optional) – Size of fixation patches. Defaults to None.

• area_range (list, optional) – [min, max] range of crop areas. Defaults to None.

• training (bool, optional) – Whether to use training mode. Defaults to False.

• n_fixations (int, optional) – Number of fixations per image. Defaults to None.

• max_batches (int, optional) – Maximum number of batches to process. Defaults to None.

• setting (str, optional) – Forward pass setting (‘supervised’ or ‘ssl’). Defaults to ‘supervised’.

• do_postproc (bool, optional) – Whether to apply post-processing. Defaults to False.

• **kwargs – Additional arguments passed to get_activations.

Returns:

(outputs, activations, targets) where:

• outputs (np.ndarray): Model outputs of shape (N, …).

• activations (dict): Dict mapping layer names to activation arrays.

• targets (np.ndarray): Target labels of shape (N,).

Return type:

tuple

initialize_logger()

Initialize logging system and create log directory.

copy_hydra_outputs()

Copy Hydra output files to our log directory.

initialize_remote_logger()

Initialize remote logging (e.g., wandb) for experiment tracking.

log(content, phase)

Log training/validation statistics.

Parameters:

• content (dict) – Statistics to log

• phase (str) – Phase name (‘train’ or ‘val’)

classmethod exec(gpu, cfg)

Execute training with the given configuration.

Parameters:

• gpu (int) – GPU device ID

• cfg (DictConfig) – Training configuration

classmethod launch_from_args(cfg)

Launch training with the given configuration.

Parameters:

cfg (DictConfig) – Training configuration

add_supervised_meters()

Add supervised training metrics for logging.

final_accuracy(iterations=10)

Compute the final accuracy of the model by averaging over a number of iterations.

Parameters:

iterations (int) – Number of validation runs to average over

Returns:

DataFrame containing averaged validation statistics

Return type:

pd.DataFrame

fovi.training.loading.get_trainer_from_base_fn(base_fn: str, load: bool = True, load_strict: bool = True, quiet: bool = False, allow_distributed: bool = False, gpu: int | None = 0, model_dirs: Sequence[str] = ('../models', SAVE_DIR + '/logs', SLOW_DIR + '/logs'), **kwargs: str | float | bool | None) → Trainer

Get a Trainer instance based on a base filename and optional parameters.

This function loads a model configuration and optionally its weights from a specified directory, creates a Trainer instance with the loaded configuration, and returns it.

Parameters:

• base_fn (str) – The base filename to look for in the logs directory.

• load (bool, optional) – Whether to load the model weights. Defaults to True.

• load_strict (bool, optional) – Whether to strictly enforce matching keys when loading weights. Defaults to True.

• quiet (bool, optional) – Whether to suppress print statements. Defaults to False.

• allow_distributed (bool, optional) – Whether to allow distributed training configuration. Defaults to False.

• **kwargs – Additional keyword arguments to override or add to the configuration.

Returns:

An instance of Trainer with the specified configuration and optionally loaded weights.

Return type:

Trainer

Note

The function searches for the model in both SLOW_DIR and SAVE_DIR. It prioritizes loading final weights over non-final weights if available.