Self-Supervised Temporal Pattern Mining for planetary geology survey missions with ethical auditability baked in

It started with a dataset I had no business touching. I was three weeks into a side project, fiddling with a public archive of Mars Reconnaissance Orbiter imagery, when I realized that the bottleneck wasn't the images—it was the time component. We had thousands of snapshots of the same crater, the same canyon, the same polar ice cap, taken days, weeks, and months apart. But no one was modeling the evolution of these features in a way that an autonomous agent could act on.

I remember sitting in my home office, staring at a heatmap of dust devil tracks that had shifted 40 meters between two orbital passes. A traditional convolutional neural network would classify both images as "dust devil tracks" and call it a day. But the temporal signature—the fact that these tracks appeared, migrated, and faded—was invisible to the model. That was the moment I realized that planetary geology surveys, especially those run by autonomous rovers or orbital agents, needed a different kind of learning paradigm.

This article is the result of my exploration into that problem. It's a deep dive into how I built a self-supervised temporal pattern mining system for planetary geology, and why I believe ethical auditability isn't just a compliance checkbox—it's an architectural requirement for any AI system that will make decisions in a remote, high-stakes environment like Mars.

The Problem: Static Models on a Dynamic Planet

When we think about planetary geology, we often think of static features: a crater formed by an impact, a canyon carved by ancient water, a volcano's layered slopes. But the reality is that planetary surfaces are dynamic. On Mars, we see:

• Seasonal frost appearing and sublimating
• Dust devil tracks migrating across plains
• Slope lineae (recurring streaks) that darken and fade
• Avalanches and landslides that alter cliff faces
• Polar ice cap growth and retreat

Each of these processes has a temporal signature. The challenge is that we don't always have labeled data for these events. We can't tell a model "this is a slope lineae formation event" because we barely understand the mechanics ourselves. This is where self-supervised learning becomes not just useful, but necessary.

In my exploration of this space, I discovered that the key insight is to treat time as the supervisory signal. Instead of asking "what is this feature?", we ask "how does this region change over time?" The answer to that question becomes the training signal for a model that can then be fine-tuned for specific geological tasks.

Technical Background: Temporal Pattern Mining as a Self-Supervised Task

The core idea is to build a model that learns a representation of a planetary surface patch that is predictive of its future state. This is a form of predictive coding, but applied to spatiotemporal data.

Let me break down the architecture I settled on after several failed experiments:

1. The Input Representation

Instead of feeding raw images, I create temporal stacks. Given a location on the planet, I sample a series of images at times t_0, t_1, ..., t_n. Each image is a multi-spectral patch (visible light, thermal IR, and sometimes radar reflectivity). The temporal stack is a tensor of shape (n_time_steps, n_channels, height, width).

2. The Encoder

A 3D convolutional neural network (ConvNet) or a Vision Transformer (ViT) with temporal attention processes this stack. The goal of the encoder is to compress the spatiotemporal information into a latent vector z.

3. The Self-Supervised Task

This is where it gets interesting. I found that a simple reconstruction loss isn't enough—it leads to the model just learning to copy the input. Instead, I used a contrastive predictive coding (CPC) approach.

The idea: Given the latent vector z_t at time step t, the model should be able to predict the latent vector z_{t+k} at a future time step t+k. We train the model to distinguish between real future latent vectors and negative samples (latent vectors from different locations or different time periods).

Here's the core code snippet that demonstrates this concept:

import torch
import torch.nn as nn
import torch.nn.functional as F

class TemporalCPC(nn.Module):
    def __init__(self, encoder, latent_dim=256, prediction_steps=4):
        super().__init__()
        self.encoder = encoder  # 3D ConvNet or ViT
        self.prediction_steps = prediction_steps

        # A simple predictor network
        self.predictor = nn.Sequential(
            nn.Linear(latent_dim, latent_dim * 2),
            nn.ReLU(),
            nn.Linear(latent_dim * 2, latent_dim * prediction_steps)
        )

        self.latent_dim = latent_dim

    def forward(self, temporal_stack):
        # temporal_stack shape: (batch, time_steps, channels, H, W)
        batch_size, time_steps, channels, H, W = temporal_stack.shape

        # Encode each time step independently (shared encoder)
        latents = []
        for t in range(time_steps):
            # Reshape to (batch, channels, H, W) for the encoder
            x_t = temporal_stack[:, t, :, :, :]
            z_t = self.encoder(x_t)  # shape: (batch, latent_dim)
            latents.append(z_t)

        # Stack latents: (batch, time_steps, latent_dim)
        latents = torch.stack(latents, dim=1)

        # Use the last observed latent to predict future latents
        z_current = latents[:, -1, :]  # (batch, latent_dim)
        predictions = self.predictor(z_current)  # (batch, latent_dim * prediction_steps)
        predictions = predictions.view(batch_size, self.prediction_steps, self.latent_dim)

        return predictions, latents

    def contrastive_loss(self, predictions, latents, negative_samples):
        # predictions: (batch, pred_steps, latent_dim)
        # latents: (batch, time_steps, latent_dim) - includes future steps
        # negative_samples: (batch, num_negatives, latent_dim)

        loss = 0.0
        for k in range(self.prediction_steps):
            # Get the true future latent at step k+1
            positive = latents[:, -self.prediction_steps + k, :]  # (batch, latent_dim)

            # Compute similarity between predictions and positives
            pos_sim = F.cosine_similarity(predictions[:, k, :], positive, dim=-1)

            # Compute similarity between predictions and negatives
            neg_sim = F.cosine_similarity(
                predictions[:, k, :].unsqueeze(1),
                negative_samples,
                dim=-1
            )

            # InfoNCE loss
            logits = torch.cat([pos_sim.unsqueeze(1), neg_sim], dim=1)
            labels = torch.zeros(logits.shape[0], dtype=torch.long, device=logits.device)
            loss += F.cross_entropy(logits / 0.1, labels)

        return loss / self.prediction_steps

4. The "Ethical Auditability" Layer

Now, here's where my exploration took an unexpected turn. As I was experimenting with this model, I realized that the latent space it learned was a black box. If an autonomous rover used this model to decide that a region was "changing" and therefore warranted a closer look, we couldn't explain why the model made that decision.

For a mission to Mars, this is unacceptable. If the rover decides to burn precious battery power to investigate a slope, the science team on Earth needs to know what temporal pattern triggered that decision. This is where the "ethical auditability baked in" part comes into play.

I built a post-hoc interpretability module that works alongside the CPC model. The idea is that for every prediction the model makes, we can generate a temporal attribution map that highlights which pixels and which time steps were most influential in the model's decision.

Here's how I implemented it using a gradient-based attribution method:

def temporal_attribution(model, temporal_stack, target_time_step=3):
    """
    Generate a temporal-spatial attribution map for the model's prediction.

    Returns a tensor of shape (time_steps, H, W) showing which pixels
    at which times were most influential.
    """
    model.eval()
    temporal_stack.requires_grad_(True)

    # Forward pass
    predictions, _ = model(temporal_stack)

    # Get the prediction for the target time step
    target_prediction = predictions[:, target_time_step, :]

    # Compute the gradient of the prediction with respect to the input
    # We use the norm of the prediction as a proxy for "change importance"
    prediction_norm = torch.norm(target_prediction, dim=-1)
    prediction_norm.backward()

    # Get the gradient
    gradients = temporal_stack.grad  # (batch, time_steps, channels, H, W)

    # Average over channels to get a spatial-temporal importance map
    attribution = torch.mean(torch.abs(gradients), dim=2)  # (batch, time_steps, H, W)

    # Normalize
    attribution = attribution / (attribution.max(dim=2, keepdim=True)[0].max(dim=3, keepdim=True)[0] + 1e-8)

    return attribution[0]  # (time_steps, H, W)

The beauty of this approach is that it generates an audit trail. Every decision the model makes comes with a human-interpretable explanation: "The model flagged this region because it detected a significant change at time step 3, specifically in the top-left quadrant of the thermal IR channel."

Implementation Details: Putting It All Together

Let me walk you through a complete workflow that I tested on a small subset of Mars Reconnaissance Orbiter data.

Step 1: Data Preparation

The first challenge was aligning images from different orbits. The MRO has a repeating orbit, but the exact ground track varies. I had to georeference each image and then resample them to a common grid.

import rasterio
import numpy as np
from scipy.ndimage import affine_transform

def align_temporal_stack(image_paths, reference_bounds, target_shape):
    """
    Align a series of satellite images to a common grid.
    """
    aligned_images = []

    for path in image_paths:
        with rasterio.open(path) as src:
            # Compute the affine transform to align to reference bounds
            transform = src.transform
            # ... (simplified: use rasterio.warp.reproject for real implementation)
            resampled = src.read(
                out_shape=(src.count, target_shape[0], target_shape[1]),
                resampling=rasterio.enums.Resampling.bilinear
            )
            aligned_images.append(resampled)

    # Stack into temporal tensor
    temporal_stack = np.stack(aligned_images, axis=0)  # (time, channels, H, W)
    return temporal_stack

Step 2: Negative Sampling Strategy

One of the trickiest parts of contrastive learning is choosing good negative samples. I found that random crops from different locations worked okay, but the model learned better when I used hard negatives—locations that looked similar in a single frame but had different temporal dynamics.

def sample_hard_negatives(temporal_stacks, encoder, num_negatives=8):
    """
    Sample negatives that are spatially similar but temporally different.
    """
    model.eval()
    all_latents = []

    with torch.no_grad():
        for stack in temporal_stacks:
            latents = []
            for t in range(stack.shape[0]):
                z = encoder(stack[t].unsqueeze(0))
                latents.append(z)
            all_latents.append(torch.stack(latents))

    # Find pairs of locations that have similar static appearance
    # but different temporal signatures
    # (Simplified: use k-NN on the mean latent, then filter by temporal variance)
    mean_latents = torch.stack([l.mean(dim=0) for l in all_latents])
    # ... (k-NN implementation)

    return hard_negatives

Step 3: Training Loop

The training loop is fairly standard, but I want to highlight one insight I gained: the learning rate schedule matters enormously. I found that a cosine annealing schedule with a warmup period was critical for avoiding collapse in the contrastive learning.

optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
scheduler = torch.optim.lr_scheduler.OneCycleLR(
    optimizer,
    max_lr=3e-4,
    total_steps=num_epochs * num_batches,
    pct_start=0.1
)

for epoch in range(num_epochs):
    for batch in dataloader:
        optimizer.zero_grad()

        temporal_stack = batch["temporal_stack"].to(device)
        negatives = batch["negatives"].to(device)

        predictions, latents = model(temporal_stack)
        loss = model.contrastive_loss(predictions, latents, negatives)

        loss.backward()
        optimizer.step()
        scheduler.step()

Real-World Applications: Beyond Mars

While my primary focus was on Mars survey missions, my exploration revealed that this approach has broader applications:

1. Autonomous Rover Navigation

A rover equipped with this model can detect active geological processes (like sand dune migration) and automatically adjust its traverse plan to investigate them, all while maintaining a full audit trail of its decisions for the science team.

2. Orbital Change Detection

Orbital assets can use this to prioritize which regions to image more frequently. The model flags regions with high temporal "surprise" (low prediction confidence), and those get added to the next imaging queue.

3. Climate Model Validation

The same temporal pattern mining can be applied to Earth observation data to validate climate models. The self-supervised nature means we don't need labeled "events"—we just need the raw time series.

4. Quantum Computing Integration

This is where my research took a fascinating turn. I started exploring whether quantum computing could accelerate the contrastive learning process. The idea is to use a quantum annealer to solve the hard negative mining problem—finding the most informative negative samples in high-dimensional space.

I built a proof-of-concept using D-Wave's quantum annealer to select hard negatives. The problem was formulated as a Quadratic Unconstrained Binary Optimization (QUBO):

# Pseudo-code for quantum-enhanced hard negative mining
def quantum_hard_negative_selection(latent_vectors, num_negatives):
    """
    Use a quantum annealer to find the most diverse set of negatives.
    """
    # Formulate as QUBO:
    # Objective: maximize diversity (minimize similarity) among selected negatives
    # while ensuring they are not too similar to the positive samples

    Q = build_qubo_matrix(latent_vectors)

    # Sample from D-Wave
    response = dwave_sampler.sample_qubo(Q, num_reads=100)

    # Extract the best solution
    best_solution = response.first.sample
    selected_indices = [i for i, bit in best_solution.items() if bit == 1]

    return latent_vectors[selected_indices]

While the quantum approach didn't dramatically improve accuracy in my small-scale tests, it showed promise for scaling to larger datasets where classical hard negative mining becomes computationally prohibitive. The key insight was that quantum annealers can efficiently explore the combinatorial space of negative sample combinations.

Challenges and Solutions

Challenge 1: Temporal Misalignment

The biggest problem I encountered was that images from different orbits aren't perfectly aligned. Atmospheric effects, slight camera angle changes, and thermal drift all introduce noise.

Solution: I implemented a temporal smoothing approach that uses a Kalman filter to estimate the "true" surface state at each time step, treating each observation as a noisy measurement.

Challenge 2: Model Collapse

In my early experiments, the contrastive model collapsed—all latent vectors became identical. This is a well-known problem in contrastive learning.

Solution: I added a variance-invariance-covariance (VICReg) regularization term. This encourages the latent vectors to have high variance across the batch, be invariant to augmentation, and have low covariance between dimensions.

Challenge 3: Interpretability vs. Performance

There's often a trade-off between model performance and interpretability. My gradient-based attribution worked, but it was noisy.

Solution: I switched to Integrated Gradients, which provides more stable attributions by integrating gradients along a path from a baseline to the input.

def integrated_gradients(model, input_stack, baseline, steps=50):
    """
    Compute integrated gradients for temporal attribution.
    """
    scaled_inputs = [baseline + (float(i) / steps) * (input_stack - baseline)
                     for i in range(steps + 1)]

    gradients = []
    for scaled_input in scaled_inputs:
        scaled_input.requires_grad_(True)
        output = model(scaled_input)
        output_norm = torch.norm(output[0][0])
        output_norm.backward()
        gradients.append(scaled_input.grad)

    # Average gradients and multiply by input difference
    avg_gradients = torch.stack(gradients).mean(dim=0)
    integrated_gradients = (input_stack - baseline) * avg_gradients

    return integrated_gradients.squeeze(0)

Future Directions

My exploration of this field has opened up several