fovi.sensing.samplers

class fovi.sensing.samplers.BaseGridSampler(*args: Any, **kwargs: Any)[source]

Bases: Module

Base class for grid samplers.

Note: objects of the BaseGridSampler family should not be used directly; it is much more convenient to use RetinalTransform, which stores a BaseGridSampler child, due to its handling of fixation parameters.

property output_dtype: dtype | None
_transform_fix_grid(img_shape, fix_loc, fixation_size, cmf_indices=None)[source]

Transform fixation grid to image coordinates.

Parameters:

  • img_shape (tuple) – Image shape (height, width).

  • fix_loc (torch.Tensor) – Fixation location.

  • fixation_size (torch.Tensor) – Fixation size.

Returns:

Transformed grid coordinates.

Return type:

torch.Tensor

selected_grid(cmf_indices: Tensor | None = None) → Tensor[source]

Return a shared grid or gather per-image grids from a CMF bank.

selected_mask(cmf_indices: Tensor | None = None) → Tensor[source]

Return the scalar or per-image FoV mask.

selected_radius(cmf_indices: Tensor | None = None) → Tensor[source]

Return normalized eccentricity for color decay.

_prep_grid_for_grid_sample(cartesian_grid)[source]

Prepare grid for torch.nn.functional.grid_sample.

Parameters:

cartesian_grid (torch.Tensor) – (n,2) coordinates, where each 2-vector coordinate is specified in (x,y) in the typical math sense (normalized to [-1,1])

Returns:

(1,1,n,2) coordinates, where each 2-vector coordinate is specified in (-y, x) or (row, col) format for grid_sample (normalized to [-1,1])

Return type:

torch.Tensor

_prepare_sampling_args(img, fix_loc, fixation_size, dtype=torch.float32)[source]

Return batched fixation tensors on img.device.

Keeping conversion in one place makes standalone samplers accept the numpy/list fixation forms that RetinalTransform historically normalizes for them.

_direct_pixel_coords(img_shape, fix_loc, fixation_size, cmf_indices=None)[source]

Compute canonical pixel-boundary coordinates for direct sampling.

_direct_grid(img_shape, fix_loc, fixation_size, cmf_indices=None)[source]

Return a grid_sample-format grid describing the direct pixel coordinates.

static _mask_invalid_samples(samples, valid_mask, all_valid)[source]

Apply deterministic FoV padding without changing forward signatures.

class fovi.sensing.samplers.GridSampler(fov, cmf_a, resolution, device='cuda', dtype=torch.float, mode='nearest', style='isotropic', coords=None, isotropic_plotting_type='v1like', backend='auto', output_dtype=None, fov_type='circular', field_geometry='planar', camera_model=None, gaze_convention='camera_xyz', radius_norm=2.0)[source]

Bases: BaseGridSampler

Grid sampler for foveated vision using regular grid sampling.

This sampler uses standard grid sampling with nearest neighbor or bilinear interpolation to sample from the foveated sampling grid.

__init__(fov, cmf_a, resolution, device='cuda', dtype=torch.float, mode='nearest', style='isotropic', coords=None, isotropic_plotting_type='v1like', backend='auto', output_dtype=None, fov_type='circular', field_geometry='planar', camera_model=None, gaze_convention='camera_xyz', radius_norm=2.0)[source]

Initialize the GridSampler.

Parameters:

  • fov (float) – Field of view diameter in degrees.

  • cmf_a (float or sequence of float) – CMF parameter in degrees. A sequence builds a fixed-layout bank for warped Cartesian grids.

  • resolution (int) – Resolution parameter.

  • device (str, optional) – Device to run on. Defaults to ‘cuda’.

  • dtype (torch.dtype, optional) – Data type. Defaults to torch.float.

  • mode (str, optional) – Sampling mode (‘nearest’ or ‘bilinear’). Defaults to ‘nearest’.

  • style (str, optional) – Sampling style. Defaults to ‘isotropic’.

  • coords (SamplingCoords, optional) – Pre-computed sampling coordinates. Defaults to None.

  • backend (str, optional) – Sampling backend: auto, torch, cuda, or compiled (spherical geometry only). For floating inputs, torch selects torch.grid_sample; for uint8 it selects direct indexing. auto prefers eligible native CUDA kernels. Spherical torch uses eager calibrated projection/gather; compiled selects the same operation through torch.compile. Inputs requiring gradients retain the Torch implementation.

  • output_dtype (torch.dtype, optional) – Optional dtype for the compact output. This never rescales values. Nearest sampling otherwise preserves dtype; bilinear integer sampling naturally produces float32.

property camera_model: CameraModel | None
property mode: str
property output_dtype: dtype | None
property gaze_convention: str
_torch_direct_sample(img, fix_loc, fixation_size, cmf_indices=None)[source]

Direct moving-grid gather used as the canonical-coordinate oracle.

_torch_grid_sample(img, fix_loc, fixation_size, cmf_indices=None)[source]

Sample through grid_sample using the floating opmath coordinate dtype.

property last_backend: str | None

Backend used by the most recent forward call, or None before sampling.

calibrated_pixels(fix_loc: Tensor, rotation: Tensor | None = None, cmf_indices: Tensor | None = None) → tuple[Tensor, Tensor][source]

Return gaze-dependent (B, N, 2) pixels and (B, N) validity.

Fixations are normalized (row, column) in the source image. An explicit (B, 3, 3) rotation maps retinal camera coordinates into source-camera coordinates and bypasses fixation-to-rotation conversion.

_sample_calibrated(image: Tensor, pixels: Tensor, valid: Tensor) → Tensor[source]

Gather compact samples without converting the full uint8 image.

calibrated_forward(image: Tensor, fixation: Tensor, rotation: Tensor | None, cmf_indices: Tensor | None = None) → tuple[Tensor, Tensor][source]

Eager calibrated sampling returning values and source pixels; supports autograd.

forward(img, fix_loc=None, fixation_size=None, return_coords=False, direct=False, rotation=None, cmf_indices=None)[source]

Forward pass for grid sampling.

Parameters:

  • img (torch.Tensor) – Input image tensor.

  • fix_loc (torch.Tensor, optional) – Fixation location. Defaults to None.

  • fixation_size (torch.Tensor, optional) – Fixation size. Defaults to None.

  • return_coords (bool, optional) – Whether to return sampling coordinates. Defaults to False.

  • direct (bool, optional) – Force the explicit Torch direct-index oracle. Defaults to False.

  • cmf_indices (torch.Tensor, optional) – Int64 level indices, shape (B,), on the image device. Required when sampling a CMF bank.

Returns:

Sampled image tensor.

Return type:

torch.Tensor

all_coords(device=None)[source]

Get all sampling coordinates.

Parameters:

device (str, optional) – Device to place coordinates on. Defaults to None.

Returns:

All sampling coordinates.

Return type:

torch.Tensor

class fovi.sensing.samplers.PoolingGridSampler(fov: float, cmf_a: float, resolution: int, res_mult: float = 3, cmf_a_mult: float = 1, fixation_size: int = 3000, k: int | None = None, style: str = 'isotropic', sample_cortex: bool | str = True, dtype: dtype = torch.float, device: str | device = 'cuda', isotropic_plotting_type: str = 'v1like', backend: str = 'auto', output_dtype: dtype | None = None, fov_type: str = 'circular', field_geometry: str = 'planar', radius_norm: float = 2.0, camera_model: CameraModel | CameraCalibration | None = None, gaze_convention: str = 'camera_xyz', pool_mode: str = 'avg', gauss_sigma: float = 10.0)[source]

Bases: BaseGridSampler

High-resolution image sampling followed by retinal pooling.

This sampler pools a high-resolution sensor array into a lower-resolution sensor array: _as_grid styles use 2D windows (circularly padded in angle for log-polar grids) and sample-set styles use KNN neighborhoods. - highres_coords: akin to photoreceptors: there are more of them - coords: akin to retinal ganglion cells: there are less of them, and they integrate over a local pool of photoreceptors (highres_coords)

__init__(fov: float, cmf_a: float, resolution: int, res_mult: float = 3, cmf_a_mult: float = 1, fixation_size: int = 3000, k: int | None = None, style: str = 'isotropic', sample_cortex: bool | str = True, dtype: dtype = torch.float, device: str | device = 'cuda', isotropic_plotting_type: str = 'v1like', backend: str = 'auto', output_dtype: dtype | None = None, fov_type: str = 'circular', field_geometry: str = 'planar', radius_norm: float = 2.0, camera_model: CameraModel | CameraCalibration | None = None, gaze_convention: str = 'camera_xyz', pool_mode: str = 'avg', gauss_sigma: float = 10.0) → None[source]

Initialize the pooling sampler.

Parameters:

  • fov (float) – Field of view diameter in degrees.

  • cmf_a (float) – A parameter from the CMF: M(r)=1/(r+a). Smaller = stronger foveation.

  • resolution (int) – Resolution parameter.

  • res_mult (float, optional) – Resolution multiplier for photoreceptor layer vs. rgc layer. Defaults to 3.

  • cmf_a_mult (float, optional) – CMF_a multiplier for photoreceptor layer vs. rgc layer. Defaults to 1.

  • fixation_size (int, optional) – Fixation size in pixels. Defaults to 3000.

  • k (int, optional) – Neighborhood size; for grid2d, the squared kernel side. None uses the sample-count ratio or squared resolution ratio respectively.

  • style (str, optional) – Sampling style. Defaults to ‘isotropic’.

  • sample_cortex (bool, optional) – Whether to sample from cortex. Defaults to True.

  • dtype (torch.dtype, optional) – Data type. Defaults to torch.float.

  • device (str, optional) – Device to run on. Defaults to ‘cuda’.

  • camera_model – Source calibration required for spherical image sampling.

  • gaze_convention – Gaze rotation convention for calibrated sampling.

  • pool_mode – Average or Gaussian weighted pooling.

  • gauss_sigma – Gaussian sigma in normalized neighborhood-distance units.

property output_dtype: dtype | None
property camera_model: CameraModel | None

Calibration owned by the high-resolution image sampler.

forward(img, fix_loc=None, fixation_size=None, direct=False, rotation=None)[source]

Sample the high-resolution retina and pool to the output grid.

Parameters:

  • img (torch.Tensor) – Input image tensor.

  • fix_loc (torch.Tensor, optional) – Fixation location. Defaults to None.

  • fixation_size (torch.Tensor, optional) – Fixation size. Defaults to None.

Returns:

Pooled samples from KNN grid sampling.

Return type:

torch.Tensor

all_coords(device=None)[source]

Get all sampling coordinates.

Parameters:

device (str, optional) – Device to place coordinates on. Defaults to None.

Returns:

Cartesian, polar, and plotting coordinates.

Return type:

tuple