Raspberry Pi Cluster mit KI-basierter Objekterkennung, MPI und Monitoring
This documentation describes the MPI configuration, benchmark applications, measurement methodology, and scalability analysis of the Raspberry Pi compute cluster.
The underlying cluster infrastructure — including DHCP, TFTP, NFS, network boot, storage, Internet routing, and remote access through Tailscale — is documented separately in the Cluster Infrastructure and Remote Access documentation.
This document focuses specifically on:
The MPI and HPL experiments were executed on the existing Raspberry Pi compute cluster.
For the performance experiments, the relevant hardware configuration is:
| Property | Configuration |
|---|---|
| Head Node | Raspberry Pi 5 |
| Compute Workers | 8 × Raspberry Pi 3 Model B v1.2 |
| Operating System | Debian 13 (Trixie) |
| Architecture | ARM64 / aarch64 |
| Worker Nodes | rpi1 to rpi8 |
| Internal Network | 192.168.50.0/24 |
| MPI Interface | eth0 |
The Raspberry Pi 5 is primarily used for:
The final scalability measurements use the Raspberry Pi 3 Worker Nodes as the compute resources. MPI communication between the Worker Nodes uses the physical Ethernet interface:
eth0
The configuration of the underlying private network is part of the separate infrastructure documentation and is therefore not repeated here.
OpenMPI starts processes on remote Worker Nodes through SSH. Passwordless SSH authentication was therefore configured between the Head Node and all Worker Nodes.
Generate an SSH key on the Head Node:
ssh-keygen -t ed25519
Copy the public key to all workers:
ssh-copy-id pi@rpi1
ssh-copy-id pi@rpi2
ssh-copy-id pi@rpi3
ssh-copy-id pi@rpi4
ssh-copy-id pi@rpi5
ssh-copy-id pi@rpi6
ssh-copy-id pi@rpi7
ssh-copy-id pi@rpi8
The SSH configuration is stored in:
~/.ssh/config
Configuration:
Host rpi1 rpi2 rpi3 rpi4 rpi5 rpi6 rpi7 rpi8
User pi
IdentityFile ~/.ssh/id_ed25519
Set the required permissions:
chmod 600 ~/.ssh/config
Verify connectivity:
for NODE in rpi1 rpi2 rpi3 rpi4 rpi5 rpi6 rpi7 rpi8
do
ssh pi@$NODE hostname
done
Every Worker Node should return its hostname without requesting a password. This is required so that MPI processes can be started automatically without interactive authentication.
OpenMPI was installed on the Head Node:
sudo apt update
sudo apt install -y openmpi-bin libopenmpi-dev
The same packages were installed on all Worker Nodes through a loop:
for NODE in rpi1 rpi2 rpi3 rpi4 rpi5 rpi6 rpi7 rpi8
do
ssh -t pi@$NODE \
"sudo apt install -y openmpi-bin libopenmpi-dev"
done
Verify the local installation:
mpirun --version
mpicc --version
Verify OpenMPI on all workers:
for NODE in rpi1 rpi2 rpi3 rpi4 rpi5 rpi6 rpi7 rpi8
do
ssh pi@$NODE "mpirun --version | head -n 1"
done
The documented cluster installation used:
Open MPI 5.0.7
The MPI hostfile specifies which Worker Nodes may participate in distributed jobs.
The file is located at:
/home/cloud-computing/hosts
Content:
rpi1 slots=1
rpi2 slots=1
rpi3 slots=1
rpi4 slots=1
rpi5 slots=1
rpi6 slots=1
rpi7 slots=1
rpi8 slots=1
Each Worker Node provides one MPI slot.
Verify that all workers can be reached:
mpirun \
--hostfile /home/cloud-computing/hosts \
-np 8 \
hostname
A successful execution returns the hostnames of the participating workers.
Before running the benchmark applications, a small MPI test program was used to verify distributed execution.
File:
hello_mpi.c
Source:
#include <mpi.h>
#include <stdio.h>
int main(int argc, char** argv)
{
MPI_Init(&argc, &argv);
int rank;
int size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
printf(
"Hello from rank %d out of %d running on %s\n",
rank,
size,
processor_name
);
MPI_Finalize();
return 0;
}
Compile:
mpicc hello_mpi.c -o hello_mpi
Copy the executable to all Worker Nodes:
for NODE in rpi1 rpi2 rpi3 rpi4 rpi5 rpi6 rpi7 rpi8
do
scp hello_mpi $NODE:/home/pi/
done
Execute:
mpirun \
--hostfile /home/cloud-computing/hosts \
-np 8 \
/home/pi/hello_mpi
Each MPI rank reports the physical Raspberry Pi on which it is running.
This verifies:
Remote MPI processes initially produced locale warnings such as:
bash: warning: setlocale:
LC_ALL: cannot change locale (en_US.UTF-8)
The existing locale configuration was checked using:
locale
locale -a
If required, en_US.UTF-8 was enabled:
sudo sed -i \
's/^# *en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' \
/etc/locale.gen
Generate the locale:
sudo locale-gen
Set the default language:
sudo env -u LC_ALL update-locale LANG=en_US.UTF-8
LC_ALL was intentionally not configured permanently because it overrides the remaining locale configuration.
The benchmark scripts therefore use:
export LANG=en_US.UTF-8
unset LC_ALL
During MPI startup, the following warning occurred:
plm:ssh: Warning:
setpgid(...) failed in parent
with errno=Permission denied(13)
The MPI applications still executed, but the warning generated unnecessary output in the benchmark logs. The following OpenMPI MCA option was used:
--mca plm_rsh_no_tree_spawn 1
This setting was later included in the permanent OpenMPI configuration.
The Head Node contains several network interfaces. During debugging, the system included interfaces such as:
lo
eth0
wlan0
tailscale0
docker_gwbridge
docker0
veth...
After a restart, basic MPI commands such as:
mpirun \
--hostfile /home/cloud-computing/hosts \
-np 8 \
hostname
still succeeded.
However, actual MPI applications failed with errors such as:
WARNING: Open MPI failed to TCP connect to a peer MPI process.
connect() to 172.17.0.1:1025 failed
Error: Connection refused (111)
The available interfaces were inspected using:
ip -br addr
The address:
172.17.0.1
belonged to a Docker network. The intended MPI communication interface is:
eth0
OpenMPI was therefore explicitly restricted to this interface:
--mca btl_tcp_if_include eth0
--mca oob_tcp_if_include eth0
A test execution used:
mpirun \
--mca plm_rsh_no_tree_spawn 1 \
--mca btl_tcp_if_include eth0 \
--mca oob_tcp_if_include eth0 \
--hostfile /home/cloud-computing/hosts \
-np 8 \
/home/pi/montecarlo_pi \
10000 \
"test"
After successful verification, the settings were stored permanently. Create the configuration directory:
mkdir -p ~/.openmpi
Configuration file:
/home/cloud-computing/.openmpi/mca-params.conf
Content:
plm_rsh_no_tree_spawn = 1
btl_tcp_if_include = eth0
oob_tcp_if_include = eth0
This prevents OpenMPI from selecting unrelated interfaces such as Docker, WLAN, or Tailscale for communication between the compute nodes.
Two MPI applications with different parallel characteristics were selected.
| Property | Monte Carlo π | Matrix Multiplication |
|---|---|---|
| Independent calculations | Very high | Lower |
| Communication overhead | Very low | Higher |
| Network dependency | Low | Higher |
| Memory dependency | Low | High |
| Expected scalability | Close to linear | More workload dependent |
| Purpose | Highly parallel reference workload | Communication- and memory-intensive workload |
The combination of both applications allows the scalability of the cluster to be evaluated under different computational conditions.
The first MPI application estimates π using the Monte Carlo method. Random points are generated inside a square.
A point is inside the unit circle if:
x² + y² <= 1
The approximation is calculated using:
π ≈ 4 × points_inside / total_points
Individual random points can be evaluated independently.
The total number of points can therefore be divided across MPI processes with very little communication.
Only the final partial results must be combined using:
MPI_Reduce
This makes Monte Carlo an almost completely parallel workload. The program was adapted for automated benchmarking.
Source:
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
int main(int argc, char** argv)
{
MPI_Init(&argc, &argv);
int rank;
int size;
long long total_points = 100000000;
long long local_points;
long long remainder;
long long local_inside = 0;
long long global_inside = 0;
const char* timestamp = "unknown";
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
if (argc > 1)
{
total_points = atoll(argv[1]);
}
if (argc > 2)
{
timestamp = argv[2];
}
local_points = total_points / size;
remainder = total_points % size;
if (rank < remainder)
{
local_points++;
}
unsigned int seed =
time(NULL) + rank * 1337;
double start = MPI_Wtime();
for (long long i = 0; i < local_points; i++)
{
double x =
(double)rand_r(&seed) / RAND_MAX;
double y =
(double)rand_r(&seed) / RAND_MAX;
if (x * x + y * y <= 1.0)
{
local_inside++;
}
}
MPI_Reduce(
&local_inside,
&global_inside,
1,
MPI_LONG_LONG,
MPI_SUM,
0,
MPI_COMM_WORLD
);
double end = MPI_Wtime();
if (rank == 0)
{
double pi =
4.0 * global_inside / total_points;
double runtime = end - start;
printf(
"%d,%lld,%.10f,%.6f,%s\n",
size,
total_points,
pi,
runtime,
timestamp
);
}
MPI_Finalize();
return 0;
}
Compile:
mpicc montecarlo.c \
-o /home/pi/montecarlo_pi
Example execution:
mpirun \
--mca plm_rsh_no_tree_spawn 1 \
--mca btl_tcp_if_include eth0 \
--mca oob_tcp_if_include eth0 \
--hostfile /home/cloud-computing/hosts \
-np 8 \
/home/pi/montecarlo_pi \
10000000 \
"test"
Only rank 0 outputs the final result, producing one CSV-compatible record per run.
The second MPI application performs distributed matrix multiplication.
Source file:
mpi_matrix_mul.c
Compile:
mpicc \
mpi_matrix_mul.c \
-o /home/pi/mpi_matrix_mul
Example execution:
mpirun \
--mca plm_rsh_no_tree_spawn 1 \
--mca btl_tcp_if_include eth0 \
--mca oob_tcp_if_include eth0 \
--hostfile /home/cloud-computing/hosts \
-np 4 \
/home/pi/mpi_matrix_mul \
800
Unlike Monte Carlo, distributed matrix multiplication requires significantly more communication and memory access.
Collective MPI operations used by the application include:
MPI_Bcast
MPI_Scatter
MPI_Gather
The application is therefore influenced by:
This makes matrix multiplication a useful counterpart to the highly parallel Monte Carlo workload.
The expected behavior of the two applications is different. Monte Carlo performs almost all calculations locally and requires only a small final reduction step.
Matrix multiplication moves significantly more data between processes and is additionally influenced by synchronization, memory bandwidth, cache behavior, and data distribution.
The initial expectation was, therefore:
Monte Carlo:
high parallel efficiency, especially for sufficiently large workloads
Matrix Multiplication:
stronger influence of communication, synchronization, and memory-related overhead
The final measurements confirm this general difference, but they also reveal an additional effect: the observed scalability of both applications strongly depends on problem size.
For Monte Carlo, increasing the fixed workload from 10 million to one billion samples improves the eight-worker efficiency from:
75.0 % -> 99.4 %
A similar effect is visible for matrix multiplication. Increasing the fixed matrix dimension from:
N = 800
to:
N = 1600
improves the eight-worker efficiency from:
64.2 % -> 88.2 %
The results therefore show that scalability cannot be characterized only by the algorithm itself. The ratio between useful computation and parallel overhead must also be considered.
The Monte Carlo application was initially used for automated benchmark collection.
The automated series tested:
MPI processes:
1, 2, 4, 8
and:
Point counts:
10,000
100,000
1,000,000
10,000,000
Each series therefore contained:
4 process counts × 4 point counts = 16 executions
The benchmark script is located at:
/home/cloud-computing/run_montecarlo_benchmark.sh
Script:
#!/bin/bash
export LANG=en_US.UTF-8
unset LC_ALL
export PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
HOSTFILE="/home/cloud-computing/hosts"
PROGRAM="/home/pi/montecarlo_pi"
LOGDIR="/home/cloud-computing/benchmarks"
CSVFILE="$LOGDIR/montecarlo_benchmark.csv"
ERRORFILE="$LOGDIR/montecarlo_errors.log"
mkdir -p "$LOGDIR"
TIMESTAMP=$(date +"%Y-%m-%d_%H-%M-%S")
if [ ! -s "$CSVFILE" ]; then
echo \
"Anzahl Nodes,Anzahl Punkte,Pi,Laufzeit,Durchlauf Zeitstempel" \
>> "$CSVFILE"
fi
for NODES in 1 2 4 8
do
for POINTS in \
10000 \
100000 \
1000000 \
10000000
do
mpirun \
--mca plm_rsh_no_tree_spawn 1 \
--mca btl_tcp_if_include eth0 \
--mca oob_tcp_if_include eth0 \
--hostfile "$HOSTFILE" \
-np "$NODES" \
"$PROGRAM" \
"$POINTS" \
"$TIMESTAMP" \
>> "$CSVFILE" \
2>> "$ERRORFILE"
done
done
Make the script executable:
chmod +x /home/cloud-computing/run_montecarlo_benchmark.sh
Execute manually:
/home/cloud-computing/run_montecarlo_benchmark.sh
The automated Monte Carlo results are stored in:
/home/cloud-computing/benchmarks/montecarlo_benchmark.csv
The columns are:
| Column | Purpose |
|---|---|
| Anzahl Nodes | Number of MPI processes |
| Anzahl Punkte | Number of Monte Carlo samples |
| Pi | Calculated approximation of π |
| Laufzeit | Runtime in seconds |
| Durchlauf Zeitstempel | Identifier of the benchmark series |
Check the latest results:
tail -n 20 /home/cloud-computing/benchmarks/montecarlo_benchmark.csv
MPI errors are written to:
/home/cloud-computing/benchmarks/montecarlo_errors.log
Check the error log:
cat /home/cloud-computing/benchmarks/montecarlo_errors.log
The Monte Carlo benchmark was initially configured to execute automatically every six hours.
The original cron job was:
0 */6 * * * /home/cloud-computing/run_montecarlo_benchmark.sh >> /home/cloud-computing/benchmarks/cron.log 2>&1
This generated a benchmark series at:
00:00
06:00
12:00
18:00
The purpose of the automation was to demonstrate an unattended benchmark collection and to create measurements over time. For the final scalability evaluation, the measurement strategy was changed.
Instead of drawing general conclusions from only one problem size, multiple fixed-size strong-scaling series were executed where practical.
Three independent fixed problem sizes were evaluated:
| Problem Size | Workers | Repetitions | Measurements |
|---|---|---|---|
| 10,000,000 samples | 1, 2, 4, 8 | 5 per configuration | 20 |
| 100,000,000 samples | 1, 2, 4, 8 | 5 per configuration | 20 |
| 1,000,000,000 samples | 1, 2, 4, 8 | 5 per configuration | 20 |
The final Monte Carlo strong-scaling data set therefore contains:
3 problem sizes × 4 worker counts × 5 runs = 60 measurements
The matrix multiplication experiment was also extended to a second fixed problem size:
| Matrix Size | Workers | Repetitions | Measurements |
|---|---|---|---|
| N = 800 | 1, 2, 4, 8 | 5 per configuration | 20 |
| N = 1600 | 1, 2, 4, 8 | 5 per configuration | 20 |
The matrix strong-scaling data set therefore contains:
2 problem sizes × 4 worker counts × 5 runs = 40 measurements
Within every strong-scaling series, the problem size remains fixed and only the number of workers changes. Comparing independent fixed-size series adds a second experimental dimension and makes it possible to determine how workload size affects parallel efficiency.
This distinction is important:
Amdahl’s Law is evaluated separately for each fixed-size workload. Changing the workload between independent series does not violate the definition of strong scaling.
The final scalability analysis therefore relies primarily on deliberately executed benchmark series under comparable conditions. The six-hour cron job remains part of the implemented automation, but it is not the basis of the final performance conclusions.
High Performance LINPACK was installed to measure floating-point performance in GFLOPS.
Install the required dependencies:
sudo apt install -y \
build-essential \
gfortran \
libblas-dev \
liblapack-dev \
wget
The HPL 2.3 sources were downloaded from Netlib:
mkdir -p ~/hpl
cd ~/hpl
wget https://www.netlib.org/benchmark/hpl/hpl-2.3.tar.gz
tar -xzf hpl-2.3.tar.gz
cd hpl-2.3
An ARM64-specific configuration was created:
Make.Linux_ARM64
It defines the compiler, MPI environment, BLAS/LAPACK libraries, and build parameters.
Compile HPL:
make arch=Linux_ARM64
The resulting binary is:
bin/Linux_ARM64/xhpl
A separate runtime directory was created:
/home/pi/hpl-run
It contains:
xhpl
HPL.dat
The relevant HPL.dat parameters are:
| Parameter | Description |
|---|---|
N |
Matrix dimension |
NB |
Block size |
P |
Process-grid rows |
Q |
Process-grid columns |
The process grid satisfies:
P × Q = number of MPI processes
The configurations used were:
| Workers | P | Q |
|---|---|---|
| 1 | 1 | 1 |
| 2 | 1 | 2 |
| 4 | 2 | 2 |
| 8 | 2 | 4 |
The block size was:
NB = 192
For all HPL measurements, a constant block size of NB = 192 was used. According to the official HPL documentation, suitable block sizes are typically between 32 and 256.
Before the final worker measurements, HPL was tested locally on the Raspberry Pi 5 Head Node.
Configuration:
N = 8000
NB = 192
P = 1
Q = 1
Execution:
cd /home/pi/hpl-run
mpirun -np 1 ./xhpl
The initial validation produced:
Runtime = 206.15 seconds
Performance = 1.6562 GFLOPS
Status = PASSED
This result was used only to verify that HPL was functioning. It is not used as the baseline of the final scaling analysis because the final measurements use Raspberry Pi 3 Worker Nodes.
At the End HPL was evaluated using three dimensions (N = 5000, 8000, 18000) to investigate the scaling behavior for different problem sizes. According to the official HPL documentation, the problem size is chosen with regard to the available memory. The three values are therefore selected to represent different workload sizes.
Example distributed execution with eight workers:
mpirun \
--mca plm_rsh_no_tree_spawn 1 \
--mca btl_tcp_if_include eth0 \
--mca oob_tcp_if_include eth0 \
--hostfile /home/cloud-computing/hosts \
--wdir /home/pi/hpl-run \
-np 8 \
/home/pi/hpl-run/xhpl
During the first distributed attempts, HPL failed because the remote processes could not find:
HPL.dat
The option:
--wdir /home/pi/hpl-run
was added so that every MPI process starts in the directory containing both:
xhpl
HPL.dat
The final Monte Carlo and matrix strong-scaling configurations were executed five times. For the remaining benchmarks, five repetitions were used where technically possible.
Example:
for RUN in {1..5}
do
echo "========== RUN $RUN of 5 =========="
date
mpirun \
--mca plm_rsh_no_tree_spawn 1 \
--mca btl_tcp_if_include eth0 \
--mca oob_tcp_if_include eth0 \
--hostfile /home/cloud-computing/hosts \
-np <PROCESSES> \
<PROGRAM>
echo "========== RUN $RUN FINISHED =========="
date
done
Repeated runs allow the calculation of:
After consultation with the professor, the final benchmark data was collected during a focused measurement period instead of relying primarily on measurements distributed over a much longer period. The purpose of this approach was to create comparable test conditions while still collecting enough repeated measurements for statistical evaluation.
The previously configured six-hour cron job remains part of the benchmark automation. However, the final scalability analysis is based primarily on deliberately executed benchmark series.
The experiments use:
1, 2, 4, and 8 MPI workers
Where technically possible, every configuration was executed five times.
Repeated measurements reduce the influence of temporary variations caused by:
The mean is used as the representative runtime. The sample standard deviation describes the variation between the individual runs.
For Monte Carlo strong scaling, three fixed problem sizes were measured independently:
10,000,000 samples
100,000,000 samples
1,000,000,000 samples
For matrix multiplication, two fixed matrix dimensions were measured independently:
N = 800
N = 1600
For each fixed problem size, the worker count was varied from 1,2,4,8 while the workload itself remained constant.
This experimental design separates two questions:
The second question is important because parallel overhead represents a larger fraction of the runtime when the computational workload is small.
The Monte Carlo program measures time using MPI_Wtime() after MPI initialization and directly before the computational loop.
The measured interval therefore includes the local Monte Carlo computation and the final MPI_Reduce, but it does not include remote process startup or MPI_Init.
The same matrix multiplication executable and MPI configuration were used for both matrix sizes so that the influence of problem size could be compared under equivalent conditions.
Strong-scaling speedup is calculated as:
S(p) = T(1) / T(p)
where:
T(1) = mean runtime with one worker
T(p) = mean runtime with p workers
Parallel efficiency is:
E(p) = S(p) / p
or as a percentage:
E(p) = S(p) / p × 100 %
Ideal scaling would produce:
| Workers | Ideal Speedup | Ideal Efficiency |
|---|---|---|
| 1 | 1 | 100 % |
| 2 | 2 | 100 % |
| 4 | 4 | 100 % |
| 8 | 8 | 100 % |
Amdahl’s Law describes the acceleration of a fixed-size problem when additional computing resources are added.
The defining condition is:
Problem size remains constant.
Worker count increases.
The theoretical speedup is:
S(p) = 1 / (α + (1 - α) / p)
where:
p = number of workers
α = serial or non-scaling fraction
The effective non-scaling fraction can be estimated from measured speedup using:
α = (1 / S(p) - 1 / p) / (1 - 1 / p)
In measured systems, this value should not be interpreted only as serial source code.
It can also summarize effects that do not scale ideally, including:
Three independent fixed-size series were measured:
| Series | Total Samples | Workers | Runs per Configuration |
|---|---|---|---|
| 1 | 10,000,000 | 1, 2, 4, 8 | 5 |
| 2 | 100,000,000 | 1, 2, 4, 8 | 5 |
| 3 | 1,000,000,000 | 1, 2, 4, 8 | 5 |
Each row represents a separate strong-scaling experiment because the problem size remains constant within that series. Comparing the three series reveals whether a larger workload reduces the relative impact of non-scaling overhead.
Two independent fixed-size matrix series were measured:
| Series | Matrix Dimension N | Workers | Runs per Configuration |
|---|---|---|---|
| 1 | 800 | 1, 2, 4, 8 | 5 |
| 2 | 1600 | 1, 2, 4, 8 | 5 |
For each series, the matrix dimension remains fixed while only the number of MPI workers changes.
An individual configuration was executed repeatedly using the same MPI parameters.
Example for N = 1600 with one worker:
for RUN in {1..5}
do
echo "========== RUN $RUN von 5 =========="
date
mpirun \
--mca plm_rsh_no_tree_spawn 1 \
--mca btl_tcp_if_include eth0 \
--mca oob_tcp_if_include eth0 \
--host rpi1:1 \
-np 1 \
/home/pi/mpi_matrix_mul 1600
echo "========== RUN $RUN FERTIG =========="
date
done
Equivalent runs were executed for:
1, 2, 4, and 8 workers
and for both:
N = 800
N = 1600
The two matrix problem sizes were selected to provide clearly different computational workloads while remaining practical for repeated measurements. Classical matrix multiplication has approximately:
O(N³)
computational complexity.
Increasing the matrix dimension from 800 to 1600 therefore theoretically increases the arithmetic workload by approximately:
(1600 / 800)³ = 8
The second strong-scaling series therefore allows the experiment to determine whether the larger computation-to-overhead ratio improves parallel efficiency.
Gustafson’s Law considers a different scenario. Instead of keeping the problem size constant, additional processors are used to process a larger workload.
The scaled speedup is:
S_G(p) = p - α(p - 1)
The sample count was increased proportionally to the number of workers:
| Workers | Total Samples |
|---|---|
| 1 | 100,000,000 |
| 2 | 200,000,000 |
| 4 | 400,000,000 |
| 8 | 800,000,000 |
Each worker therefore receives approximately:
100,000,000 samples
The ideal result is an approximately constant runtime.
Classical matrix multiplication has approximately:
O(N³)
computational complexity.
The matrix dimension was therefore increased approximately according to:
N(p) ≈ N(1) × p^(1/3)
The tested dimensions were:
| Workers | Matrix N |
|---|---|
| 1 | 800 |
| 2 | 1008 |
| 4 | 1272 |
| 8 | 1600 |
This produces approximately:
1×
2×
4×
8×
the computational workload.
For the scaled workload experiments, weak-scaling efficiency is calculated as:
E_weak(p) = T(1) / T(p)
An ideal value is:
100 %
HPL was tested with:
N = 5000
N = 8000
N = 18000
N = 5000 and N = 8000 were executed with:
1, 2, 4, and 8 workers
N = 18000 exceeded the available memory with one and two workers.
This experiment therefore also demonstrates the effect of distributed memory capacity.
| Workers | Mean GFLOPS | Std. Dev. GFLOPS | Mean Time [s] | Speedup | Efficiency |
|---|---|---|---|---|---|
| 1 | 0.16837 | 0.00005 | 495.22 | 1.00 | 100.0 % |
| 2 | 0.29600 | 0.00003 | 281.66 | 1.76 | 87.9 % |
| 4 | 0.53494 | 0.00064 | 155.87 | 3.18 | 79.4 % |
| 8 | 0.93857 | 0.00128 | 88.80 | 5.58 | 69.7 % |
Three valid runs were available for the one-worker configuration.
The other configurations contain five runs.
With eight workers, the runtime decreases from approximately 495.2 seconds to 88.8 seconds.
The measured speedup is:
5.58
corresponding to an efficiency of:
69.7 %
| Workers | Mean GFLOPS | Std. Dev. GFLOPS | Mean Time [s] | Speedup | Efficiency |
|---|---|---|---|---|---|
| 1 | 0.16825 | 0.00007 | 2029.51 | 1.00 | 100.0 % |
| 2 | 0.30932 | 0.00002 | 1103.84 | 1.84 | 91.9 % |
| 4 | 0.57496 | 0.00148 | 593.25 | 3.42 | 85.4 % |
| 8 | 1.01614 | 0.00483 | 335.07 | 6.06 | 75.7 % |
The larger workload produces better parallel efficiency than N = 5000.
With eight workers:
Performance = 1.016 GFLOPS
Speedup = 6.06
Efficiency = 75.7 %
The larger problem provides a better computation-to-communication ratio.
| Workers | Mean GFLOPS | Std. Dev. GFLOPS | Mean Time [s] | Result |
|---|---|---|---|---|
| 1 | – | – | – | HPL memory allocation failed |
| 2 | – | – | – | Linux OOM killer terminated xhpl |
| 4 | 0.61665 | 0.01749 | 6356.20 | Successful |
| 8 | 1.15298 | 0.01264 | 3374.15 | Successful |
A conventional speedup relative to one worker cannot be calculated because no valid single-worker baseline exists. Increasing from four to eight workers reduces the runtime from:
6356.20 s
to:
3374.15 s
corresponding to an improvement of approximately:
1.87×
The highest measured mean HPL performance was:
1.153 GFLOPS
with eight workers.
The experiment also demonstrates the benefit of distributed memory: a problem that could not be executed with one or two Raspberry Pis became executable with four and eight workers.
The Monte Carlo strong-scaling experiment was repeated for three fixed problem sizes. For every size, the total number of samples remained constant while the number of workers increased from 1,2,4,8.
Each configuration contains five runs.
| Workers | Mean Runtime [s] | Std. Dev. [s] | Speedup | Efficiency |
|---|---|---|---|---|
| 1 | 1.0287 | 0.0180 | 1.000 | 100.0 % |
| 2 | 0.5259 | 0.0177 | 1.956 | 97.8 % |
| 4 | 0.2997 | 0.0206 | 3.432 | 85.8 % |
| 8 | 0.1715 | 0.0153 | 5.997 | 75.0 % |
With eight workers:
Speedup = 6.00
Efficiency = 75.0 %
The mean estimated effective non-scaling fraction across the 2-, 4-, and 8-worker measurements is approximately:
α ≈ 0.0418
or:
4.18 %
The small workload still benefits from additional workers, but the deviation from ideal scaling becomes clearly visible at four and eight workers.
At this size, the parallel computation is so short that communication, synchronization, and other fixed runtime costs represent a significant fraction of the total measured time.
| Workers | Mean Runtime [s] | Std. Dev. [s] | Speedup | Efficiency |
|---|---|---|---|---|
| 1 | 9.9973 | 0.0275 | 1.000 | 100.0 % |
| 2 | 5.0261 | 0.0150 | 1.989 | 99.5 % |
| 4 | 2.5353 | 0.0135 | 3.943 | 98.6 % |
| 8 | 1.2977 | 0.0065 | 7.704 | 96.3 % |
With eight workers:
Speedup = 7.70
Efficiency = 96.3 %
The mean estimated effective non-scaling fraction is approximately:
α ≈ 0.00526
or:
0.53 %
At 100 million samples, the computation-to-overhead ratio is substantially better.
The eight-worker result is already close to ideal linear scaling.
| Workers | Mean Runtime [s] | Std. Dev. [s] | Speedup | Efficiency |
|---|---|---|---|---|
| 1 | 99.7706 | 0.0160 | 1.000 | 100.0 % |
| 2 | 49.9419 | 0.0070 | 1.998 | 99.9 % |
| 4 | 25.0000 | 0.0089 | 3.991 | 99.8 % |
| 8 | 12.5415 | 0.0115 | 7.955 | 99.4 % |
With eight workers:
Speedup = 7.96
Efficiency = 99.4 %
The mean estimated effective non-scaling fraction is approximately:
α ≈ 0.00090
or:
0.09 %
This workload exhibits almost ideal strong scaling on the available eight-node cluster.
The most important result is not only that Monte Carlo scales well, but that the measured scalability improves systematically as the fixed problem size increases.
| Problem Size | T(1) [s] | T(8) [s] | Speedup at 8 | Efficiency at 8 | Mean effective α |
|---|---|---|---|---|---|
| 10,000,000 | 1.0287 | 0.1715 | 5.997 | 75.0 % | 4.18 % |
| 100,000,000 | 9.9973 | 1.2977 | 7.704 | 96.3 % | 0.53 % |
| 1,000,000,000 | 99.7706 | 12.5415 | 7.955 | 99.4 % | 0.09 % |
At eight workers, efficiency rises from approximately:
75 %
for 10 million samples to more than:
99 %
for one billion samples. The effective non-scaling fraction decreases at the same time. This provides experimental evidence that the relative cost of MPI communication and synchronization becomes less important when more useful computation is performed between communication events.
The matrix strong-scaling experiment was performed for two fixed matrix dimensions:
N = 800
N = 1600
Within each series, the matrix size remained constant while the number of workers increased from 1 to 8. Each configuration contains five runs.
| Workers | Mean Runtime [s] | Std. Dev. [s] | Speedup | Efficiency |
|---|---|---|---|---|
| 1 | 56.151 | 2.539 | 1.000 | 100.0 % |
| 2 | 28.925 | 0.833 | 1.941 | 97.1 % |
| 4 | 16.123 | 0.448 | 3.483 | 87.1 % |
| 8 | 10.929 | 0.054 | 5.138 | 64.2 % |
With eight workers:
Speedup = 5.138
Efficiency = 64.2 %
The mean estimated effective non-scaling fraction across the 2-, 4-, and 8-worker measurements is approximately:
α ≈ 0.0531
or:
5.31 %
The scaling is initially good but increasingly deviates from the ideal as more workers are added.
| Workers | Mean Runtime [s] | Std. Dev. [s] | Speedup | Efficiency |
|---|---|---|---|---|
| 1 | 914.455 | 31.547 | 1.000 | 100.0 % |
| 2 | 454.969 | 13.357 | 2.010 | 100.5 % |
| 4 | 235.438 | 8.919 | 3.884 | 97.1 % |
| 8 | 129.614 | 3.790 | 7.055 | 88.2 % |
With eight workers:
Speedup = 7.055
Efficiency = 88.2 %
The mean estimated effective non-scaling fraction is approximately:
α ≈ 0.008
or approximately:
0.8 %
The deviation is small and may result from:
It is therefore treated as effectively ideal scaling within the measurement uncertainty.
The comparison between both fixed matrix sizes shows a substantial improvement in strong-scaling efficiency as the workload increases.
| Matrix Size | T(1) [s] | T(8) [s] | Speedup at 8 | Efficiency at 8 | Mean effective α |
|---|---|---|---|---|---|
| N = 800 | 56.151 | 10.929 | 5.138 | 64.2 % | 5.31 % |
| N = 1600 | 914.455 | 129.614 | 7.055 | 88.2 % | ~0.8 % |
At eight workers, parallel efficiency improves from:
64.2 %
to:
88.2 %
when the matrix dimension increases from 800 to 1600. This is an important result.
The poorer scaling observed for N = 800 is therefore not a fixed property of the matrix multiplication implementation.
When N = 1600 is used, substantially more useful computation is performed per worker and the relative influence of non-scaling overhead decreases.
The result supports the same general principle already observed in the Monte Carlo experiments:
larger workload
-> higher computation-to-overhead ratio
-> better parallel efficiency
However, the matrix results do not identify one specific bottleneck. Communication, synchronization, memory bandwidth, cache behavior, and data locality are plausible contributing factors, but these components were not measured independently.
The extended measurements show that neither Monte Carlo nor matrix multiplication has one universal parallel efficiency.
The measured scalability depends on both:
At eight workers:
| Workload | Speedup at 8 | Efficiency at 8 | Mean effective non-scaling fraction |
|---|---|---|---|
| Monte Carlo – 10 million samples | 5.997 | 75.0 % | 4.18 % |
| Monte Carlo – 100 million samples | 7.704 | 96.3 % | 0.53 % |
| Monte Carlo – 1 billion samples | 7.955 | 99.4 % | 0.09 % |
| Matrix Multiplication – N = 800 | 5.138 | 64.2 % | 5.31 % |
| Matrix Multiplication – N = 1600 | 7.055 | 88.2 % | ~0.8 % |
The comparison reveals two important effects.
First, Monte Carlo reaches the highest measured strong-scaling efficiency. Its independent calculations and small communication requirement allow the large workloads to approach ideal linear scaling. Second, matrix multiplication also improves strongly when its fixed problem size is increased.
Increasing the matrix dimension from:
N = 800
to:
N = 1600
raises eight-worker efficiency from:
64.2 %
to:
88.2 %
This means that the difference between the applications cannot be explained only by saying that Monte Carlo scales well and matrix multiplication scales poorly. Instead, the observed result depends on the relationship between:
useful computation
and
parallel overhead
The large Monte Carlo workloads still scale more efficiently than matrix multiplication, which is consistent with their different communication and memory characteristics. However, the additional matrix experiment demonstrates that workload size must be considered before drawing conclusions about application scalability.
| Workers | Total Samples | Mean Runtime [s] | Std. Dev. [s] | Weak Efficiency | Gustafson S_G |
|---|---|---|---|---|---|
| 1 | 100,000,000 | 9.9882 | 0.0042 | 100.0 % | 1.000 |
| 2 | 200,000,000 | 10.0024 | 0.0085 | 99.9 % | 1.995 |
| 4 | 400,000,000 | 10.0168 | 0.0073 | 99.7 % | 3.986 |
| 8 | 800,000,000 | 10.3206 | 0.1643 | 96.8 % | 7.968 |
The workload increases by a factor of eight, while the runtime increases only from approximately:
9.99 s
to:
10.32 s
With eight workers:
Weak efficiency = 96.8 %
Gustafson S_G = 7.97
This demonstrates very good scaled workload behavior.
| Workers | Matrix N | Mean Runtime [s] | Std. Dev. [s] | Weak Efficiency | Gustafson S_G |
|---|---|---|---|---|---|
| 1 | 800 | 57.7126 | 1.0771 | 100.0 % | 1.000 |
| 2 | 1008 | 116.8567 | 2.4913 | 49.4 % | 1.947 |
| 4 | 1272 | 121.4670 | 1.4845 | 47.5 % | 3.841 |
| 8 | 1600 | 131.3311 | 3.3267 | 43.9 % | 7.628 |
The computational workload increases approximately with the worker count, but the runtime does not remain constant.
At eight workers:
Weak efficiency = 43.9 %
Gustafson S_G = 7.63
The measured runtime therefore shows that the matrix workload does not maintain ideal weak-scaling behavior.
Potential contributing factors include:
These effects were not individually profiled and should therefore not be interpreted as separately proven bottlenecks.
| Metric at 8 Workers | Monte Carlo | Matrix Multiplication |
|---|---|---|
| Workload increase | 8× | approximately 8× |
| Mean runtime | 10.321 s | 131.331 s |
| Weak efficiency | 96.8 % | 43.9 % |
| Gustafson S_G | 7.968 | 7.628 |
Monte Carlo processes approximately eight times more work with almost the same runtime.
Matrix multiplication cannot maintain a constant runtime under the scaled workload.
The difference shows that increasing both workload and worker count is much more effective for the highly independent Monte Carlo workload than for the matrix implementation.
Monte Carlo requires very little MPI communication. Each worker performs almost all calculations independently, and only the final partial results are combined.
Matrix multiplication and HPL require more communication and synchronization. Their performance can therefore be influenced more strongly by network traffic and collective MPI operations.
The benchmark results are consistent with such overhead, but communication time was not measured independently. It should therefore be treated as a potential contributing factor rather than as an isolated, proven bottleneck.
Collective MPI operations require participating processes to synchronize. If one process reaches a synchronization point later than the others, the remaining processes must wait.
Synchronization can therefore reduce parallel efficiency, particularly when the useful computation between synchronization points is relatively small. Synchronization time was not profiled separately and is therefore considered a plausible contributing factor.
Communication between physical workers takes place through the Ethernet cluster network. Distributed applications can therefore be affected by:
This is especially relevant for matrix multiplication and HPL.
Matrix multiplication and HPL process large matrix datasets. Performance can therefore depend not only on CPU execution time but also on:
These effects were not measured independently, but they represent plausible explanations for deviations from ideal scaling.
The HPL N = 18000 experiment demonstrated a directly observed memory-capacity limitation.
The workload failed with one and two workers but executed successfully with four and eight workers.
Unlike the potential communication and memory-bandwidth effects, this bottleneck was directly observed during execution.
Distributed execution therefore provides not only additional processing resources but also additional aggregate memory capacity.
The extended measurements provide strong evidence that workload size affects observed parallel efficiency.
For Monte Carlo at eight workers:
10 million samples -> 75.0 % efficiency
100 million samples -> 96.3 % efficiency
1 billion samples -> 99.4 % efficiency
For matrix multiplication at eight workers:
N = 800 -> 64.2 % efficiency
N = 1600 -> 88.2 % efficiency
For HPL:
N = 5000 -> 69.7 % efficiency at 8 workers
N = 8000 -> 75.5 % efficiency at 8 workers
Across all three benchmark types, larger workloads generally make additional workers more effective. The reason is that a larger amount of useful computation can reduce the relative importance of fixed or slowly growing parallel overhead.
However, this effect is application-dependent. Increasing the workload can also introduce new limitations related to:
The HPL N = 18000 experiment demonstrates this trade-off because its larger workload exceeds the memory capacity of one and two workers.
The Monte Carlo timer begins after MPI_Init and immediately before the computational loop.
The measured runtime therefore includes the local calculation and the final reduction but excludes SSH process launch and MPI initialization.
This is relevant when interpreting the 10-million-sample series: its lower efficiency cannot be attributed solely to process startup.
| Purpose | Path |
|---|---|
| MPI Hostfile | /home/cloud-computing/hosts |
| OpenMPI Configuration | /home/cloud-computing/.openmpi/mca-params.conf |
| Monte Carlo Executable | /home/pi/montecarlo_pi |
| Monte Carlo Benchmark Script | /home/cloud-computing/run_montecarlo_benchmark.sh |
| Monte Carlo Results | /home/cloud-computing/benchmarks/montecarlo_benchmark.csv |
| Monte Carlo Error Log | /home/cloud-computing/benchmarks/montecarlo_errors.log |
| Cron Log | /home/cloud-computing/benchmarks/cron.log |
| Matrix Multiplication Executable | /home/pi/mpi_matrix_mul |
| HPL Runtime Directory | /home/pi/hpl-run |
| HPL Executable | /home/pi/hpl-run/xhpl |
| HPL Configuration | /home/pi/hpl-run/HPL.dat |
The Raspberry Pi cluster was successfully configured for distributed MPI execution. The central configuration challenge was explicitly binding OpenMPI to eth0, after the system had initially attempted to use a Docker network interface for internode communication.
The benchmark results show that scalability is not a fixed property of an application but depends on the ratio between useful computation and parallel overhead. For Monte Carlo, eight-worker Amdahl efficiency rises from 75.0 % (10 million samples) to 99.4 % (1 billion samples); for matrix multiplication, from 64.2 % (N=800) to 88.2 % (N=1600). Larger fixed problem sizes substantially improve efficiency in both cases.
Under Gustafson’s scaled-workload scenario, a clear difference emerges: Monte Carlo remains close to ideal with 96.8 % weak efficiency, while matrix multiplication reaches only 43.9 %. This suggests that communication volume (O(N²)) grows faster than the per-worker computation, which is kept constant by design.
HPL confirms this pattern from a different angle: the highest measured performance was 1.153 GFLOPS (N=18000, 8 workers). At N=18000, execution failed with one and two workers due to insufficient memory — a directly observed demonstration that distributed systems provide not only additional compute power but also additional aggregate memory capacity.
Overall, the experiments provide a practical demonstration of the core concepts of distributed-memory scalability: Amdahl’s and Gustafson’s Law can be observed in real measurements, scalability conclusions require testing multiple problem sizes, and communication and memory effects limit achievable efficiency even on a fully functional cluster.