<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>Fit Pixels, Get Labels: Meta-learned Implicit Networks forImage Segmentation</title></titleStmt>
			<publicationStmt>
				<publisher>Springer Nature Switzerland</publisher>
				<date>09/21/2025</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10668824</idno>
					<idno type="doi">10.1007/978-3-032-04947-6_19</idno>
					
					<author>Kushal Vyas</author><author>Ashok Veeraraghavan</author><author>Guha Balakrishnan</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[Not Available]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1">Introduction</head><p>Implicit neural representations (INRs) have demonstrated impressive performance in encoding complex natural signals. A typical INR f &#952; (&#8226;) is a multilayer perceptron (MLP) with parameters &#952; that maps a coordinate x &#8712; R d to signal value I(x) &#8712; R D and is iteratively fit to a specific signal. Because they offer continuous signal representations, excellent reconstruction performance, and implicit signal priors, INRs have gained significant attention in the computer vision community for compactly modeling large signals <ref type="bibr">[13,</ref><ref type="bibr">17,</ref><ref type="bibr">16]</ref>, and in the medical imaging community for inverse imaging tasks such as accelerated MRI <ref type="bibr">[1]</ref>, and sparse-view CT reconstruction <ref type="bibr">[18,</ref><ref type="bibr">23]</ref>. Despite these successes, learned INR representations are highly specific to a given signal and to the way its parameters are initialized. As a result, unlike architectures such as U-Nets <ref type="bibr">[15]</ref> and vision transformers <ref type="bibr">[9]</ref>, INR-produced features lack structural or semantic coherence, making them less used for tasks requiring learning over an image distribution. </p><p>9 H E X 2 p p y H I P 0 8 8 k h I / t I K 6 E d J U I / r u y J + r s j x 7 G U w z j Q l X q / n p z 1 x u J / X j t T 0 a W f U 5 5 m C j i Z D o o y Z q v E H q d i h 1 Q A U W y o C S a C 6 l 1 t 0 s M C E 6 W z M 3 U I 7 u z J 8 6 R x U n H P K 2 e 3 p + X q V R F H C e 2 j Q 3 S M X H S B q u g G 1 V A d E f S A n t A L e j U e j W f j z X i f l i 4 Y R c 8 e + g P j 4 x t 6 U J p J &lt; / l a t e x i t &gt; Results from recent studies suggest that this shortcoming of INRs may be fixable. In particular, by fitting an INR across multiple images starting from the same parameter intialization, the final parameters exhibit clear semantic, structural properties for datasets such as faces or MRI scans <ref type="bibr">[6,</ref><ref type="bibr">21,</ref><ref type="bibr">22]</ref>. And as a result, these learned initializations allow the INR to fit unseen (test) images rapidly with far fewer gradient updates while learning more generalizable features <ref type="bibr">[2,</ref><ref type="bibr">21,</ref><ref type="bibr">22]</ref>. In this study, we leverage this insight to develop an INR fitting strategy for medical image segmentation. In particular, we present the surprising discovery that an INR carefully optimized to fit many training pairs of images and associated segmentation maps can predict a segmentation map for an unseen test image when simply fine-tuned to reconstruct the image's pixels.</p><p>We propose MetaSeg, a novel meta-learning <ref type="bibr">[7]</ref> INR framework that performs this strategy. MetaSeg uses an INR that simultaneously predicts both image pixels and segmentation labels given a coordinate (Fig. <ref type="figure">1b</ref>), and aims to find an optimal setting of the INR parameters (&#952; and &#981;), such that the network can be quickly fine-tuned to fit any image-mask pair starting from these parameters. MetaSeg uses a nested, meta-learning procedure (Fig. <ref type="figure">1a</ref>). In the inner optimization, the INR is fit to reconstruct a randomly selected batch of training image and segmentation mask pairs. The outer (meta) optimization applies a gradient descent step to the current values of &#952; and &#981; based on the converged inner parameters. These steps iterate to converge to an initialization that works well for both image reconstruction and segmentation objectives from the given image distribution. At test time, we simply fit the INR initialized with the learned optimal parameters to the test image (without needing any segmentation information), generating viable segmentation masks in just 2 gradient descent updates and near state-of-the-art quality masks after 100 updates.</p><p>MetaSeg draws inspiration from Neural Implicit Segmentation Functions (NISFs) <ref type="bibr">[20]</ref>, which also use an INR to predict pixel and segmentation values. However, instead of forcing the INR to learn image-label correlations, NISF takes an additional latent vector as input to represent an image's semantic content, which must then be optimized at test time to produce a mask. Due to this strategy, NISF must apply several regularization losses on INR parameters and the latent space to succeed. In contrast, MetaSeg offers a much simpler training methodology based on meta-learning, only requiring reconstruction and segmentation losses during training and only a reconstruction loss during testing.</p><p>We evaluated MetaSeg on 2D and 3D magnetic resonance imaging (MRI) segmentation tasks using the OASIS-MRI <ref type="bibr">[8,</ref><ref type="bibr">12]</ref> dataset. We first find that MetaSeg achieves similar performance for 2D (Dice score: 0.93) and 3D (Dice score: 0.91) MRI segmentation tasks to widely used U-Net baselines <ref type="bibr">[4,</ref><ref type="bibr">14]</ref> with 90% fewer parameters, and significantly outperforms NISF <ref type="bibr">[20]</ref>. Second, we show that MetaSeg performs well for fine-grained segmentation tasks, accurately segmenting image regions which may be sparse and span only a few pixels. Third, we show that when trained on low-resolution scans, MetaSeg fares better on highresolution scans than U-Nets, likely due to its continuous representation. Finally, we qualitatively demonstrate through principal component analysis (PCA) visualizations that MetaSeg's learned initial parameters encode semantic information correlated with ground truth segmentations.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2">Methods</head><p>We assume a given dataset D = {I j , S j } N j=1 of N subjects, where I j and S j are a d-dimensional image scan and segmentation map, respectively (d = 2 and 3 in our experiments). At location x &#8712; R d , we have I j (x) &#8712; R D and S j (x) &#8712; {0, 1} C , where D is the number of image channels, and C is the number of classes.</p><p>We propose the MetaSeg framework to estimate &#348; for an unseen scan I at test time. MetaSeg consists of an INR (Fig. <ref type="figure">1b</ref>) with an L-layer MLP f &#952; : R d &#8594; R D that predicts &#206;(x) given coordinate x, and a segmentation head g &#981; : R h &#8594; R C that predicts C class probabilities given the penultimate features, of dimensionality h, computed by f &#952; : f L-1 &#952; (x). At test time, we aim to fit f &#952; (&#8226;) on I for T f steps, and then simply compute</p><p>In Sec. 2.1, we describe MetaSeg's strategy for learning optimal values &#952; * , &#981; * over D, such that f &#952; * (&#8226;) and g &#981; * (&#8226;) (networks initialized with &#952; * and &#981; * ) may be easily fine-tuned to reconstruct and segment a scan from the distribution. In Sec. 2.2, we describe how the INR initialized with these parameters may be used at inference time to estimate a segmentation map for a scan.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.1">Training: Meta-Learning and Segmentation Head Optimization</head><p>MetaSeg uses a MAML <ref type="bibr">[7]</ref> meta-learning strategy to learn optimal parameters &#952; * , &#981; * , consisting of a nested optimization with inner and outer (meta) routines (Fig. <ref type="figure">1a</ref>). Starting with initial random values &#952; 0 and &#981; 0 at iteration t = 0, the outer routine takes T o gradient descent steps to move the parameters to generalizable values. The gradient at each outer step is based on the results of the inner routine, which fits f &#952; t (&#8226;) and g &#981; t (&#8226;) for T i steps on one data pair (I j , S j ). The outer routine uses the final inner parameter values as a gradient signal to update &#952; t , &#981; t . To avoid a noisy gradient due to overfitting to example j, we set T i to a small value in practice (e.g., T i = 2).</p><p>In the inner routine, at time step t, we optimize both f &#952; t (&#8226;) and g &#981; t (&#8226;) for a single subject j with a loss function:</p><p>where</p><p>2 is a per-pixel reconstruction loss, and</p><p>is a per-pixel multiclass focal classification loss <ref type="bibr">[11]</ref>, where &#947; is a hyperparameter. We use a focal loss to account for heavy class imbalance across pixels. In the outer step, we make a gradient descent update:</p><p>where gradient &#8711; [&#952; t ,&#981; t ] (&#8226;) computes the difference between the converged parameters (&#952; t j , &#981; t j ) of the inner optimization and current parameters &#952; t , &#981; t . After T o outer gradient descent steps, we obtain parameters &#952; To and &#981; To .</p><p>We freeze &#952; * = &#952; To , but further optimize &#981; To such that when, at test time, f &#952; * (&#8226;) is fit for T f iterations on any random scan, we can obtain accurate segmentation predictions, in one step. To do so, we first fit f &#952; * (&#8226;) separately on each training scan I j for T f iterations and populate a dataset of learned scanspecific features and segmentation maps, {f L-1 &#952; * j (x), S j (x)} N j=1 . We then globally optimize g &#981; (&#8226;) by minimizing the loss function L seg :</p><p>We freeze the converged parameters as &#981; * .</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.2">Inference: Fit Pixels, Get Labels</head><p>At inference time, we assume a given (unseen) scan I. We fit f &#952; * (&#8226;) to I for T f iterations, solely optimizing the pixelwise reconstruction loss L recon . Upon completion, we compute &#348;(x) = g &#981; * (f L-1 &#952; * (x)), and apply softmax and argmax operations to &#348;(x) to obtain the predicted class per pixel. </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3">Experiments and Results</head><p>We evaluated MetaSeg on the tasks of 2D and 3D MRI image segmentation. We used 414 T1 brain MRI scans from the OASIS-MRI <ref type="bibr">[8,</ref><ref type="bibr">12]</ref> dataset, randomly split into 214 training, 100 validation, and 100 test scans. OASIS-MRI provides 5 coarse (background, white matter, gray matter, cortex, and cerebrospinal fluid), 24 fine-grained segmentation labels for 2D slices, and 35 segmentation labels for full 3D MRI volumes. We used aligned coronal sections provided in OASIS-MRI for 2D segmentation and aligned 3D volumes for 3D segmentation. We normalized all images and volumes to have intensity in 0, 1 . We padded and resized images to resolution 192&#215;192, and cropped 3D volumes to 160&#215;160&#215;200 followed by 2&#215; downsampling to yield 80 &#215; 80 &#215; 100 resolution scans.</p><p>Implementation. For all experiments, we used a SIREN <ref type="bibr">[19]</ref> INR with L = [6, 5, 5] layers, width h = [128, 512, 256], and w 0 = 30. We set the segmentation head g &#981; (&#8226;) as one fully connected layer with Leaky-ReLU activation, followed by a linear layer with C output values. We used the Adam <ref type="bibr">[10]</ref> optimizer with a learning rate of 10 -4 for inner and outer meta-learning optimizations. When tuning the segmentation head for 5 classes, we reduced the learning rate to 5e -5. We set T i = 2 and T f = 100, and ran the outer loop for T o = 10 epochs over the training data while validating the performance of optimized parameters &#952; t , &#981; t every 50 steps. For 2D segmentation with 5 classes, we set the focal loss hyperparameter &#947; = 1.0, and for 24 output classes we set &#947; = 2.0. For 3D MRI volumes which exhibit high background class imbalance, we set &#947; = 3.0 and further scaled the reconstruction loss by 0.1 for background pixels.  Baselines: We adopt a widely used U-Net <ref type="bibr">[15]</ref> baseline for 2D MRIs <ref type="bibr">[4]</ref> and SegResNet <ref type="bibr">[14]</ref> from the Monai <ref type="bibr">[5]</ref> package for 3D MRIs. We trained all models using a learning rate of 1e-3 until the validation loss saturated. We also compare against NISF <ref type="bibr">[20]</ref>, the most closely related INR method in the literature.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.1">Results</head><p>Table <ref type="table">1</ref> presents average Dice <ref type="bibr">[3]</ref> scores with standard deviations for MetaSeg and baseline methods for both 2D and 3D cases. For 2D, MetaSeg achieves an excellent Dice score of 0.93 &#177; 0.012 for 5 classes, comparable to that achieved by the U-Net, but with 90% fewer parameters. Interestingly, as Fig. <ref type="figure">2</ref> illustrates, MetaSeg fits a signal remarkably fast, yielding a viable segmentation in just 2 update steps to f &#952; (&#8226;) (Dice: 0.85), and reaching peak performance at T f steps. If fit further, MetaSeg's segmentation accuracy eventually falls off as parameters overfit to the particular scan. MetaSeg also performs well on the finer-grained segmentation task with 24 classes, achieving a Dice score of 0.86 &#177; 0.032 and outperforming the baseline U-Net. As shown in Fig. <ref type="figure">3</ref>, MetaSeg provides consistently accurate segmentation results for the ventricles and brain stem, which have drastic variations in size and area across examples. MetaSeg also performs well for 3D segmentation, achieving an average Dice score of 0.91 0.011 with 90% fewer parameters than SegResNet, and significantly outperforms NISF. Fig. <ref type="figure">4</ref> depicts rendered coronal, sagittal, and axial planes from the segmented volume showing high agreement with ground truth segmentation planes, also demonstrating that MetaSeg-encoded volumes can be readily queried at any viewing plane. We also explored whether MetaSeg can generate high resolution (2&#215;) 3D segmentation maps when only trained on low resolution scans, due to its underlying continuous representation. MetaSeg achieved a Dice score of 0.78 &#177; 0.011 on this task, while SegResNet <ref type="bibr">[14]</ref> achieved a Dice score of 0.73 &#177; 0.019. Hence, while there is a significant drop in performance in this scenario, MetaSeg still outperforms U-Nets.</p><p>Finally, we conducted an ablation study on parameter initialization strategy for 2D segmentation, with results reported in Table <ref type="table">2</ref>. MetaSeg performs significantly better than INRs initialized with random, fixed, and image-only meta-learning strategies. The latter result, in particular, demonstrates the benefit of learning structural priors with joint segmentation supervision.  baselines <ref type="bibr">[4,</ref><ref type="bibr">14]</ref> but with 90% fewer parameters. In addition, MetaSeg significantly outperforms NISF <ref type="bibr">[20]</ref>, a related INR approach for segmentation. Ablation results demonstrate that our meta-learning strategy to initialize parameters with joint reconstruction and segmentation supervision is crucial for success.</p><p>MetaSeg offer a new perspective on the capabilities of INRs for imaging tasks, particularly beyond signal representation, which has been their predominant application in the literature. In particular, the results demonstrate that the metalearning strategy can be useful for image translation tasks such as segmentation, where there is strong cross-correlation across signals. Indeed, as demonstrated by PCA decomposition in Fig. <ref type="figure">5</ref>, MetaSeg learns low-dimensional embeddings that seem to jointly model both pixel intensity and semantic structure.</p><p>Because an INR explicitly conditions its features on input coordinates, it is likely that an INR will learn highly signal-specific features localized in space. While this is a good property for signal representation, it may lead to poor performance on tasks requiring generalization, such as segmentation. We conducted a pilot test to measure the sensitivity of MetaSeg to the spatial alignment of test scans to the training set distribution by applying random orientation and translation augmentations. We found that Dice score drops by 2%-6% for test images rotated randomly in [5 &#8226; , 15 &#8226; ], and drops by 3%-9% for test images with a random translation in <ref type="bibr">[5 -10]</ref> pixels. Hence, MetaSeg can be somewhat sensitive to spatial alignment, though further experimentation is needed to understand whether this can be mitigated itself with proper training augmentation.</p><p>MetaSeg learns fast, is resource-friendly, and scales well with more dimensions, unlike typical vision and transformer models. It also is easy to train, using only standard reconstruction and classification losses. In an age where AI re-quires increasing resource and financial investments, MetaSeg provides a fresh new perspective to solving image segmentation tasks.</p></div></body>
		</text>
</TEI>
