A monoid is a concept borrowed from Category theory. It describes a set with an associative binary operation and an identity. While this seems abstract, it is a powerful way of thinking about data.
Take for example a regular retrieval of data. Each snapshot represents a single timestep.
To add this data together we want a binary operator that takes two stores and returns a store.
What happens if we only have one timestep, how do we combine it using our binary operator?
Well, we need a store that doesn't affect addition. We also need to treat our data as immutable. Otherwise, performing multiple operations may affect the result of future or previous operations.
Let's consider S3, Amazon's simple storage service. A bucket contains objects. Each object requires some latency to read or write. A single text file containing the latest meta-data describing our dataset would be useful.
{
"timesteps": [
"20260201T080000Z",
"20260201T081500Z",
"20260201T083000Z",
"20260201T084500Z",
],
"physical_resolution": 512,
"physical_tiles": [
[0, 0, 0],
[1, 0, 0],
],
}
This data structure implies every timestep has every physical tile. It makes more sense to omit the per timestep settings.
{
"timesteps": [
"20260201T080000Z",
"20260201T081500Z",
"20260201T083000Z",
"20260201T084500Z",
],
}
Then the bucket contents would be.
/prefix/
store.json
20260201T080000Z/
store.json
0.0.0.png
...
But this violates our algebra.
We want to be able to add infinitely.
We shouldn't need to be aware of the
past of a value.
store.json should always appear
at the same directory level.
/prefix/
latest/
store.json
20260201T080000Z/
store.json
0.0.0.png
...
This separates the arithmetic from the values.
A peculiarity of how we work with data is
the usage of a latest.txt
to indicate which dataset is freshest.
/prefix/
latest.txt
latest/
store.json
20260201T080000Z/
store.json
0.0.0.png
...
The contents of latest.txt is the word latest.
By thinking about associative operations with an identity value we were able to design a data structure. The individual values became immutable. The running accumulation can be written independently.
Is our operation associative?
If we think carefully,
we can convince ourselves that
(a + b) + c = a + (b + c).
Firstly, adding sorted lists results in a sorted list containing the elements of a, b and c.
Secondly, applying a max length constraint will discard elements too early in the time window.