The Set of Integers With a Unique Maximum
How much useful information can a softmax layer hold?
This is Part 2 in our What Every Programmer Should Know About Enumerative Combinatorics series. Find Part 1 here.
1.0 Motivation
Neural networks are fancy probability calculators. After performing a bunch of matrix multiplications, the final layer tends to find the most probable outcome.
The most probable outcome tends to be the one with the highest softmax value. We claim that we can understand the softmax layer by enumerating the set of integers with a unique maximum.
2.0 Primer on Softmax
In a neural network, a softmax layer outputs a probability distribution over a vector (Nielsen, 2015)1 and is defined as:
The softmax function is used in machine learning classification tasks since it has this special property: the largest element in the input vector remains the largest element after softmax is applied while the information about the other, non-maximal elements is retained in a reversible way (Taxel, 2016)2.
The example below illustrates the difference between the softmax function and a regular maximum function (Meerkat, 2023)3:
After performing the softmax the class with the highest probability is usually* taken as the network’s output.
*Temperature settings and random sampling in LLMs best determine the best probability in classification.
3.0 Enumerating the Set
If you know little about enumerative combinatorics then check this out:
This section demonstrates our approach to enumerating the set of integers with a unique maximum. We make this assumptions:
All possible inputs are positive integers.
The maximal element is known.
This means we know the number base we are working in, ie base-2 binary or base-10.
The maximum element is unique.
This means {0,0,0} and {1,6,6} are invalid vectors while {1,6,2} is valid.
3.1 The Set of Integers With a Right-most Maximum Element
3.1.1 Definitions
These are some important definitions in Set Theory:
Maximal element is an element for which no other element is larger while
Maximum element is an element that is larger when compared to every other element in a set (Sylvestre, 2025)4.
The distinction between maximum and maximal is subtle.
For the rest of the paper we use base (b) in place of maximal element.
The set of integers with a right-most maximum element is the set of numbers where the rightmost digit is always bigger than the rest.
The cardinality of a set is the number of elements within the set.
3.1.2 The Cardinality of our Set
From our previous article, we saw that one of the best ways to approach set enumeration is by finding the set’s cardinality. We also saw that making observations eventually leads to a formula.
Thus, we observe that a n-digit number of base b within our right-maximum set satisfies these properties:
The right-most digit must be strictly greater than all digits.
This means {0,0,0} and {1,6,2} are invalid vectors while {1,2,6} is valid.
Let the right-most digit be k, where 1 ≤ k ≤ b-1. Then
The remaining n-1 digits must all be less than k.
For each k choices, there are k choices for each of the n-1 digits.
This means there are k^(n-1) possible combinations.
Summing over all possible k values gives the total size of the set.
From these observations, we establish that the cardinality of our set is given by the formula:
In C, we can write our formula like this:
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
//clear && gcc maximum.c -lm -o m.o && ./m.o
int FindCardinality(int base, int n)
{
int cardinality = 0;
for(int k = 1; k <= base - 1; k++)
{
cardinality += pow(k, n-1);
}
return cardinality;
}
int main()
{
int base = 4;
int numberOfDigits = 2;
int cardinality = FindCardinality(base,numberOfDigits);
printf("Cardinality : %d\n", cardinality);
return 0;
}If we run the code above with a base of 4 and 2 digits, we observe that our set has a cardinality of 6 elements. This can be easily verified by hand-enumeration (or bruteforce):
3.1.3 Generating All Elements Within the Set
We can generate the entire set of right-most maximal elements given our knowledge of the set’s cardinality. This observations must be made however:
The individual summands of k^(n-1) count the number of digits with a specific maximum value.
For the example in Section 3.1.2, the summands are 1, 2 and 3. This tells us that in the right-most column, 1 occurs once, 2 occurs twice and 3 occurs thrice.
The right-most column becomes a local maximal ie, the current number base.
This means for instance, if the right-most column is 2 then the other columns must be binary (0, 1) digits.
Therefore, we can generate the entire set by :
Listing all the numbers from 0 to k^(n-1) in base. We use this helper function to convert from base-10 to an arbitrary base X:
void PrintIntArray(int length, int*array) { for(int i = 0; i < length; i++){printf("%2d,", array[i]);} } double LogN(double number, int base) { return log((double)number) / log((double)base); } void Base10ToBaseX(int number, int base, int baseLength, int *baseHolder) { int index = baseLength - 1; //zero the array memset(baseHolder, 0, baseLength*sizeof(baseHolder[0])); while(number > 0) { baseHolder[index] = number % base; number /= base; index -= 1; if(index < 0){break;} } }Appending our current right-most digit. This variable starts at 1 and increase by 1 after every iteration. This resembles:
void PrintEntireSet(int base, int numberOfDigits) { int rightMostDigit = 1; int baseLength = numberOfDigits-1; int *baseHolder = calloc(baseLength, sizeof(int)); int cardinality = FindCardinality(base,numberOfDigits); printf("Cardinality : %d\n", cardinality); int index = 0; for(int k = 1; k <= base - 1; k++) { int count = pow(k, numberOfDigits-1); for(int i = 0; i < count; i++) { printf("%3d:", index); //Base conversion Base10ToBaseX(i, rightMostDigit, baseLength, baseHolder); PrintIntArray(baseLength, baseHolder); //Append rightMostDigit printf("%d\n", rightMostDigit); index += 1; } rightMostDigit += 1; } free(baseHolder); assert(index == cardinality); } void TestSetGenerator() { int base = 4; int numberOfDigits = 2; PrintEntireSet(base, numberOfDigits); }
Running the TestSetGenerator function yields this permutation of the example in Section 3.1.2:
Feel free to compare your code with mine here.
3.1.4 Indexing Elements Within the Set
The previous sections covered cardinality search and set generation. This information permits us answer the following questions:
What is the element at index x of our set?
To achieve this, we need to find out the location of x in the summation:
x exists somewhere in the summation Then we subtract the lower summand from x and convert to a different number base like we did in Section 3.1.3:
void TestIndexToSetElement() { int base = 4; int numberOfDigits = 2; PrintEntireSet(base, numberOfDigits); int index = 3; //Index we are searching for int baseLength = numberOfDigits-1; int *baseHolder = calloc(baseLength, sizeof(int)); int cardinalityLow = 0; int cardinalityHigh = 0; int rightMostDigit = 2; for(int k = 2; k <= base - 1; k++) { //Find location of index in the summation cardinalityLow += pow(k-1, numberOfDigits-1); cardinalityHigh += pow(k, numberOfDigits-1); if(index >= cardinalityLow && index <= cardinalityHigh) { printf("Found at index %d:\n", index); //Subtract lower cardinality from index index -= cardinalityLow; //Convert base Base10ToBaseX(index, rightMostDigit, baseLength, baseHolder); PrintIntArray(baseLength, baseHolder); printf("%d\n", rightMostDigit); break; } rightMostDigit += 1; } free(baseHolder); }
What index does a given set element correspond to?
First, we find the location of the right-most digit then we convert the remaining digits to base 10. We add this converted value to our count:
void TestSetElementToIndex() { int base = 4; int numberOfDigits = 2; PrintEntireSet(base, numberOfDigits); int index = -1; int baseHolder[] = {1,3}; int baseLength = sizeof(baseHolder) / sizeof(int); int cardinalityLow = 0; int cardinalityHigh = 0; int rightMostDigit = 1; for(int k = 1; k <= base - 1; k++) { //Find location of index in the summation cardinalityLow += pow(k-1, numberOfDigits-1); if(rightMostDigit == baseHolder[baseLength-1]) { //Convert from right to left in base 10 int localIndex = 0; int currentPower = 0; for(int j = baseLength-2; j >= 0; j--) { localIndex += baseHolder[j] * pow(rightMostDigit, currentPower); currentPower += 1; } cardinalityLow += localIndex; printf("%d\n", cardinalityLow); break; } rightMostDigit += 1; } }
3.1.5 Matrices Yielding the Set of Integers With a Right-most Maximum Element
An interesting question arises: Can we enumerate the set of matrices whose product yields a vector with a right-most maximum in its integer elements?
This section attempts to answer this question in the 1 Dimensional case.
3.1.5.1 Establishing Matrix Dimensions and Product Values
We are working with one-dimensional vectors of length n. Therefore, we establish that a 1*m vector multiplied by a m*n matrix yields a 1*n vector.
We also know that the elements of the 1*n vector are in fact sums of products of the matrix elements on the left. This is illustrated below:
Thus, we conclude that finding matrix weights is an exercise in additive number theory. Backpropagation is not needed.
3.2 Enumerating the Entire Set of Integers With a Unique Maximum
The previous section tackled the problem of enumerating the right-most maximal set. This section generalizes the problem to enumerating the entire set of integers with a unique maximum.
Commonsense suggests that for a n-digit number, there are n positions where a unique maximum can be observed. If we know the set of right-most maximum elements, it follows that multiplying this set by n yields the entire set of integers with a unique maximum.
This concludes Section 3. We succeeded in enumerating all elements in the set of integers with a unique maximum.
Footnotes
Taxel, P,. (2016). Why is the softmax function called that way?. MathStackExchange.
Meerkat, M,. (2023). Why is the softmax function called that way?. MathStackExchange.
Sylvestre. J,. (2025). Elementary Foundations: An Introduction to Topics in Discrete Mathematics. GNU FDL. Chapter 19.5:Maximal/Minimal Elements.