Leveraging Local Data Sampling Strategies to Improve Federated Learning

In Leveraging Local Data Sampling Strategies to Improve Federated Learning, the authors investigate another important problem that arises from the decentralized nature of Federated Learning (FL): the data available to different clients are often imbalanced and distributed differently. Rather than proposing another modification to the federated aggregation algorithm, the authors ask a simpler but important question: can the performance of Federated Learning be improved simply by changing how clients select and sample the data they use for local training?

The motivation comes from the fact that clients in a federated system rarely possess identical datasets. One client may have a large number of samples from one class and very few from another, while a different client may have almost the opposite distribution. For example, one client might have 900 Cat samples and 100 Dog samples, while another has 100 Cat samples and 900 Dog samples. Although both clients are participating in the same learning task, their local models are exposed to very different distributions. Such imbalance can influence local training and, consequently, the quality of the global model produced through federated aggregation.

The authors therefore focus on local data sampling as a way of mitigating these differences. Their work is particularly interesting because it does not require the fundamental Federated Learning procedure to be replaced. Instead, sampling is introduced as a local data-processing step that takes place before the client performs its normal training.

The paper investigates several possible approaches to handling local data imbalance. These include undersampling, oversampling, dynamic sampling, and hybrid sampling strategies. With undersampling, a client reduces the number of samples belonging to an overrepresented class. For instance, a dataset containing 900 samples from Class A and 100 samples from Class B could be reduced to approximately 100 samples from each class. Oversampling takes the opposite approach by increasing the representation of the minority class, potentially producing a dataset containing 900 samples from each class. Dynamic and hybrid strategies provide more flexibility by adapting the sampling process or combining different approaches.

An important design decision in the study is that this sampling process takes place at the client level rather than at the server. Although the server could theoretically perform some form of centralized sampling if it had access to client data, this would conflict with the privacy-preserving motivation of Federated Learning. By keeping the sampling process local, each client can examine and process its own dataset without exposing its raw data to the central server.

The authors then go beyond simply comparing fixed sampling techniques by introducing an Optimized sampling strategy. The underlying idea is that there should not necessarily be one sampling method that is considered optimal for every client or every situation. Instead, a client can locally determine which sampling strategy is most appropriate for its own data and training conditions.

In other words, rather than making a universal assumption such as “oversampling is always better” or “undersampling is always better,” the proposed approach allows the client to evaluate the available alternatives and select the one that provides the most favourable outcome.

The optimization considers factors such as model performance, convergence, and training time. This is significant because improving model accuracy alone is not necessarily sufficient in a federated environment. A strategy that provides slightly better performance but requires substantially more computation may be unsuitable for a resource-constrained client. The authors' experiments indicate that both oversampling and their Optimized strategy generally perform well across the conditions they investigate.

However, this adaptive approach introduces an important trade-off. To determine the best sampling strategy, the client may need to experiment with several alternatives before beginning its actual local training. This makes the Optimized approach more computationally expensive. In a federated environment where clients may have limited processing power, memory, energy, or time, this additional computation can become a practical limitation.

To address this issue, the authors also investigate a correlation-based alternative. Rather than repeatedly testing all possible sampling strategies, the client can use correlations between characteristics of its local data and the observed behaviour of different sampling strategies to identify an appropriate strategy more efficiently. The goal is to reduce the computational cost of the optimization process while maintaining comparable performance and convergence characteristics.

Overall, the architecture proposed by the paper remains relatively close to conventional Federated Learning. A client first receives the global model from the server. It then examines its private local dataset and applies an appropriate sampling strategy. The resulting sampled or rebalanced dataset is used for local model training, after which the client sends its model update back to the server for aggregation.

Conceptually, the process can be viewed as:

             Central Server
                   │
             Global Model
                   │
        ┌──────────┼──────────┐
        ▼          ▼          ▼
     Client 1   Client 2   Client 3
        │          │          │
   Local Data  Local Data  Local Data
        │          │          │
        ▼          ▼          ▼
   Local Sampling / Optimization
        │          │          │
        ▼          ▼          ▼
   Sampled/     Sampled/    Sampled/
   Rebalanced   Rebalanced  Rebalanced
      Data         Data        Data
        │          │          │
        ▼          ▼          ▼
   Local Training
        │          │          │
        └──────────┼──────────┘
                   ▼
             Model Updates
                   │
                   ▼
             Server Aggregation
                   │
                   ▼
              Global Model

The authors evaluate their sampling strategies under different conditions, particularly focusing on data imbalance, cohort size, and the number of samples available to each client. These experiments are useful because they demonstrate that the effectiveness of local data processing can depend on the characteristics of the federated environment.

Nevertheless, the study also leaves several areas open for further investigation. The experiments concentrate on a limited set of data-specific properties, meaning that more complicated forms of heterogeneity are not fully explored. For example, real-world federated systems can involve feature distribution shifts, label noise, temporal changes, and more complex forms of class imbalance. Model heterogeneity and client dropout are also not central components of the evaluation. Furthermore, the computational cost associated with the full Optimized strategy remains a concern for clients with limited resources.

One of the most important observations from this work is that the data-processing stage before local training can itself be an important design dimension in Federated Learning. The paper demonstrates that clients do not necessarily have to use their local data exactly as it is stored. Instead, they can make local decisions about how that data should be sampled or rebalanced before contributing to the global model.

However, the work primarily operates at the sampling-strategy level. The client decides how its dataset should be sampled rather than determining the usefulness or relevance of every individual sample. This creates an important distinction between this work and more granular forms of data selection.

For example, the approach in this paper can be summarized as:

Local Dataset
      │
      ▼
Choose Sampling Strategy
      │
      ▼
Rebalance/Sample Dataset
      │
      ▼
Local Training

Whereas a more fine-grained data-selection approach could instead ask:

Local Dataset
      │
      ▼
Evaluate Individual Samples
      │
      ▼
Which samples are useful for
the current learning task?
      │
      ▼
Selected Dataset
      │
      ▼
Local Training

This distinction is particularly relevant to the broader direction of Task-Aware Local Data Selection. While Leveraging Local Data Sampling Strategies to Improve Federated Learning shows that adaptively deciding how local data should be sampled can improve FL, it does not fully address the question of whether individual samples are relevant to the current task, model, or learning objective.

The paper therefore strengthens the broader argument that data selection should not necessarily be treated as a fixed preprocessing decision in Federated Learning. It shows that giving clients greater control over which portions of their local data are used for training can improve the learning process, while also revealing the computational and generalization challenges that arise when such decisions become adaptive.