Solving Pell Equations with Index Calculus

Coding Implementation of Lenstra's Paper : Solving The Pell Equation

Aug 03, 2025

LeetArxiv is Leetcode for implementing Arxiv and other research papers.

**Abstract for Lenstra’s paper,** **Solving the Pell Equation**

1.0 Paper Introduction

In 2008, Hendrik Lenstra wrote Solving the Pell Equation (Lenstra, 2008)1 to demonstrate an index calculus approach to solving the pell equation using smooth numbers.

Lenstra’s thesis states that using index calculus to solving Pell-type equations is much faster than finding the period of a continued fraction.

The paper is summarized below:

Lenstra Paper Summary
The paper, Solving the Pell Equation introduces index calculus-based solutions for Pell’s equation.
Lenstra demonstrates how to use small prime numbers (smooth numbers) to derive fundamental solutions for Pell’s equation.
He asserts that the index calculus approach is much faster than the continued fraction approach to solving Pell’s equation.
Lenstra’s primary claims are:
All solutions to Pell’s equation can be ordered by magnitude.
Solving Pell’s equation is equivalent to knowing the fundamental solution, (x₁ , y₁).
One can solve Pell’s equation involving large D integers by solving a Pell equation involving smaller integers, D'.
“All math papers should be a blog post with relevant code!” ~ Jesse, W (paid subscriber)

2.0 Introduction to Pell’s Equation

For the integers x, y and a positive integer D, that is not a square, an equation of the form

is called Pell’s equation (Conrad, 2020)2.

Pell’s equation is visually a hyperbola:

**Pell's equation for** D = 2 and six of its integer solutions. Taken from Wikipedia

Pell’s equation is parameterized by the formula (Joyal, 2013)3:

where t is the slope of the line joining two rational points on the Pell equation hyperbola. We can use this parameterization to generate a third rational point (much like elliptic curve addition).

Pell’s equation can be written in the field K = ℚ(√D) in its algebraic form (Lenstra, 2008):

*High school math called these surds.

For all solutions to Pell’s equation, this identity holds (Lenstra, 2008):

The solution (x, y) of Pell’s equation correspond to units in the ring ℤ[√D], where a unit is an element of norm 1.

If D is constant, then we can order our points as powers of the fundamental solution:

3.0 Lenstra’s Thesis

**Fundamental solution of Pell’s equation. Taken from page 1 of (Lenstra, 2008)**

Lenstra’s thesis is : one can solve Pell’s equation involving large D integers by solving a Pell equation involving smaller integers, D'.

Lenstra claims that all solutions to Pell’s equation can be ordered by magnitude:

Lenstra defines the fundamental solution as the smallest possible values, (x₁ , y₁₎, that solve Pell’s equation.

He asserts that solving Pell’s equation is equivalent to knowing the fundamental solution.

3.1 Index Calculus Approach to Solving the Cattle Problem

In this section, Lenstra introduces a famous problem involving Pell’s equation and demonstrates how to solve a large D integer using a smaller d' integer.

**The** **cattle problem** **can be solved by Pell’s equation.** Taken from page 3 of (Lenstra, 2008)

Archimedes’ cattle problem asks one to find the composition of male and female cattle that are white, black, spotted or brown in a herd with the following conditions (Weisstein, 2025)4:

**Archimedes cattle problem conditions.** Taken from Wolfram Alpha

Lenstra introduces us to Amthor’s solution (Amthor, 1880)5 :

**Amthor’s solution to the cattle problem.** Taken from page 8 of (Lenstra, 2008)

Lenstra defines these variables:

d = 410 286 423 278 424 .
- Amthor required one to solve the Pell equation involving d:
- Whose solution would resemble:
d^'= 4 729 494 .
- This is the factorization of d into a square and squarefree form:
  - (2 • 4657)²* 4 729 494 = 410 286 423 278 424
  - Amthor’s requirement is equivalent to solving the Pell equation:
  - The solution resembles:
  - This simplifies to:
u. This is the fundamental solution to the simplified form:
u is the fundamental solution. Taken from page 8 of (Lenstra, 2008)

Lenstra states that the simplified form involving d^' is much easier to solve than the original form involving d.

He relates the simplified form to the fundamental form:

**Relationship between simplified form and fundamental form**. Taken from page 8 of (Lenstra, 2008)

3.2 Finding the fundamental solution, u, using Index Calculus

**Section 7: Smooth numbers and index calculus.** Taken from page 15 of (Lenstra, 2008)

A b-smooth number is a number whose prime factorization is composed of prime numbers less than or equal to b.

In section 7, Lenstra wants to find the smallest (x, y) pair to solve the cattle problem. That is, he wants to find the fundamental solution:

He wants to use Amthor’s approach, finding a smaller square, but 4729494 cannot factor into smaller squares. That is, 4729494 = 2¹ x 3¹ x 7¹ x 11¹ x 29¹ x 353¹

Instead, he suggests an index calculus approach, where one finds smooth numbers close to 4729494 and solves these.

He shows that smooth numbers close to our desired D value always yield the fundamental solution, u.

u in its power representation form. Taken from page 17 of (Lenstra, 2008)

4.0 Coding Lenstra’s Method

This section demonstrates Lenstra’s approach to solving Pell’s equation using index calculus and smooth numbers, in the C programming language. We use the stb_ds.h library for variable length arrays and libgmp for big integers.

4.1 Finding Smooth Numbers in a Quadratic Extension Field

**Step 1 according to Lenstra.** Taken from page 14 of (Lenstra, 2008)

This section demonstrates how to find smooth numbers in a quadratic extension field, as described in (Lenstra, 2008).

Step 1 (Starter code): write a struct to hold smooth numbers in an algebraic extension field x+y√D where the norm is:

#include <stdio.h>
#include <stdint.h>
#include <string.h>
#include <stdlib.h>
#include <assert.h>
#include <stdbool.h>
#include <math.h>
#include <gmp.h>
#define STB_DS_IMPLEMENTATION
#include "stb_ds.h"
#include "ff_asm_primes.h"
#define PRIMES_COUNT 5000

typedef struct smooth_number_struct *SmoothNumber;
struct smooth_number_struct
{
	mpz_t x;
	mpz_t y;
	mpz_t D;
	mpz_t norm;
        int normSign;
	int *primes;
	int *powers;
};

SmoothNumber CreateSmoothNumber()
{
	SmoothNumber smoothNumber = malloc(sizeof(struct smooth_number_struct));
	mpz_init(smoothNumber->x);
	mpz_init(smoothNumber->y);
	mpz_init(smoothNumber->D);
	mpz_init(smoothNumber->norm);
	smoothNumber->normSign = 0;
	smoothNumber->primes = NULL;
	smoothNumber->powers = NULL;
	return smoothNumber;
}

void DestroySmoothNumber(SmoothNumber smoothNumber)
{
	if(smoothNumber)
	{
		mpz_clear(smoothNumber->x);
		mpz_clear(smoothNumber->y);
		mpz_clear(smoothNumber->D);
		mpz_clear(smoothNumber->norm);
		arrfree(smoothNumber->primes);
		arrfree(smoothNumber->powers);
		free(smoothNumber);
	}
}

void SmoothNumber_FindNorm(SmoothNumber smoothNumber)
{
	mpz_t temp;
	mpz_inits(temp, NULL);
	//Set norm to x^2
	mpz_mul(smoothNumber->norm, smoothNumber->x, smoothNumber->x);
	//Set temp to y^2
	mpz_mul(temp, smoothNumber->y, smoothNumber->y);
	//multiply temp by D
	mpz_mul(temp, smoothNumber->D, temp);
	
	//Set norm to x^2 - Dy^2
	mpz_sub(smoothNumber->norm, smoothNumber->norm, temp);
        //Find sign of norm
	smoothNumber->normSign = mpz_sgn(smoothNumber->norm);
	//Set norm to it's absolute value
	mpz_abs(smoothNumber->norm, smoothNumber->norm);
	
	mpz_clears(temp, NULL);
}

void PrintSmoothNumber(SmoothNumber smoothNumber)
{
	gmp_printf("(%Zd + %Zd√%Zd): Norm: %d * %Zd\n",smoothNumber->x,smoothNumber->y, smoothNumber->D,smoothNumber->normSign, smoothNumber->norm);
}

void TestSmoothNumber()
{
	SmoothNumber smoothNumber = CreateSmoothNumber();
	mpz_set_ui(smoothNumber->x, 1);
	mpz_set_ui(smoothNumber->y, 1);
	mpz_set_ui(smoothNumber->D, 2);
	SmoothNumber_FindNorm(smoothNumber);
	PrintSmoothNumber(smoothNumber);
	DestroySmoothNumber(smoothNumber);
}

int main()
{

	TestSmoothNumber();
	return 0;
}

Step 2: Write a prime factorization algorithm to find algebraic integers with a smooth norm.

Note that:

The Legendre symbols of our norm and chosen primes must equal 1.

The legendre symbol is undefined when P is even. We write this function to define the legendre symbol for even numbers.

int SmoothNumber_LegendreSymbol(mpz_t a, mpz_t p)
{
	if(mpz_cmp_ui(p,2) == 0)
	{
	        if(mpz_even_p(a)){return 0;}	
		mpz_t mod8; mpz_init(mod8);mpz_mod_ui(mod8, a, 8);
		int res;
		switch(mpz_get_ui(mod8))
		{
			case 1: case 7: res =  1; break;
			case 3: case 5: res = -1; break;
			default: res = 0; // Shouldn't happen since a is odd
		}
		mpz_clear(mod8);
		return res;
	}
	else
	{
		return mpz_legendre(a, p);
	}
}
void TestLegendreSymbol()
{
	mpz_t D, p;
	mpz_inits(D,p, NULL);
	//Set D
	mpz_set_ui(D, 4729494);
	for(int i = 0; i < 12; i++)
	{
		mpz_set_ui(p, first5239[i]);
		if(SmoothNumber_LegendreSymbol(D, p) == 1)
		{
			printf("%d\n", first5239[i]);
		}
	}
	mpz_clears(D,p,NULL);
}

Running our test function yields similar primes to (Lenstra, 2008):

**5,13,17,23,31,41,43,47 are the primes suitable for D=4729494**

We use a list of primes to factorize our norms while checking to ensure smoothness and that we are factoring into primes that satisfy the Legendre property:

bool SmoothNumber_Factorize(mpz_t integer, mpz_t D, int bSmooth, int **primes, int **powers)
{
	//Factorize while ensuring legendre check
	mpz_t remainder, primeHolder;mpz_inits(remainder,primeHolder,NULL);
	mpz_set(remainder, integer);
	bool isSmooth = true;

	for(int i = 0; i < PRIMES_COUNT; i++)
	{
		int primeNumber = first5239[i];
		//Stop if primeNumber is smooth
		if(primeNumber > bSmooth){break;}
		int count = 0;
		while(mpz_divisible_ui_p(remainder, primeNumber))
		{
			mpz_divexact_ui(remainder, remainder, primeNumber);
			count++;
		}
		if(count > 0)
		{
			//Check legendre symbol == 1
			mpz_set_ui(primeHolder,primeNumber);
			if(SmoothNumber_LegendreSymbol(D, primeHolder) == 1)
			{
				arrput(*primes, primeNumber);
				arrput(*powers, count);
			}
			else
			{
				isSmooth = false;
				break;
			}
			//gmp_printf("%lu^%d ", primeNumber, count);
		}

		// Early exit if fully factored
		if(mpz_cmp_ui(remainder, 1) == 0){break;}
	}
	//Check if remaining factor is 1 or smooth
	if(isSmooth && mpz_cmp_ui(remainder, 1) != 0)
	{
		if(mpz_cmp_ui(remainder, bSmooth) <= 0)
		{
			//gmp_printf("%Zd^1 ", remainder);
			arrput(*primes,  mpz_get_ui(remainder));
			arrput(*powers, 1);
		}
		else
		{
			//gmp_printf("[non-smooth factor: %Zd] ", remainder);
			isSmooth = false;
		}
	}
	mpz_clears(remainder,primeHolder,NULL);
	return isSmooth;
}

void TestSmoothNumber()
{
	SmoothNumber smoothNumber = CreateSmoothNumber();
	mpz_set_ui(smoothNumber->x, 4351);
	mpz_set_ui(smoothNumber->y, 2);
	mpz_set_ui(smoothNumber->D, 4729494);
	SmoothNumber_FindNorm(smoothNumber);
	
	int smoothnessFactor = 50;
	bool isSmooth = SmoothNumber_Factorize(smoothNumber->norm, smoothNumber->D, smoothnessFactor, &(smoothNumber->primes), &(smoothNumber->powers));
	if(isSmooth)
	{
		PrintSmoothNumber(smoothNumber);
	}

	DestroySmoothNumber(smoothNumber);
}

Running our factorization algorithm on the algebraic number yields the correct prime factorization:

**Prime factorization of 4351 + 2** in our field

Step 3: Searching for smooth numbers using Lenstra’s bounds.

Lenstra suggests success in finding smooth numbers when the value of y is small and x is close to y√D


SmoothNumber* SmoothNumber_FindSmoothNumbers(mpz_t D, int bSmooth, int maxFoundCount, int maxY, int maxX)
{
	SmoothNumber *acceptedNumbers = NULL;
	mpz_t sqrtD, ySquareD, xSearch,xSearchBound,gcd;
	mpz_inits(sqrtD, ySquareD, xSearch, xSearchBound, gcd, NULL);
	
	//Set squareRoot of D and variables
	int foundCount = 0;
	mpz_sqrt(sqrtD, D);
	for(int y = 1; y < maxY; y++)
	{
		//Set xSearchBound to ySquareD
		mpz_mul_ui(xSearchBound, sqrtD, y);
		//Set ySquareD
		mpz_mul_ui(ySquareD, D, y);
		mpz_mul_ui(ySquareD, ySquareD, y);
		for(int x = -maxX; x < maxX; x++)
		{
			//Search x^2 values close to ySquareD
			mpz_set(xSearch, xSearchBound);
			if(x < 0)
			{
				mpz_sub_ui(xSearch, xSearch, abs(x));
			}
			else
			{
				mpz_add_ui(xSearch, xSearch, x);	
			}

			//Ensure x is coprime to y
			mpz_gcd_ui(gcd, xSearch, y);
			if(mpz_cmp_ui(gcd, 1) == 0)
			{
				//gmp_printf("\t%Zd ^2 - %d^2 * %Zd = ",xSearch, y, D);
				mpz_mul(xSearch, xSearch, xSearch);
				mpz_sub(xSearch, xSearch, ySquareD);
				
				//Find absolute value
				int sign = mpz_sgn(xSearch);
				//Set norm to it's absolute value
				mpz_abs(xSearch, xSearch);
				//gmp_printf("%Zd\n",xSearch);
		
				//Test for smoothness
				int *primes = NULL;
				int *powers = NULL;
				bool isSmooth = SmoothNumber_Factorize(xSearch, D, bSmooth, &primes, &powers);
				if(isSmooth)
				{
					SmoothNumber smoothNumber = CreateSmoothNumber();
					mpz_set(smoothNumber->x, xSearch);
					mpz_set_ui(smoothNumber->y, y);
					mpz_set(smoothNumber->D, D);
					SmoothNumber_FindNorm(smoothNumber);
					smoothNumber->primes = primes;
					smoothNumber->powers = powers;
					arrput(acceptedNumbers, smoothNumber);
				}
				else
				{
					arrfree(primes);
					arrfree(powers);
				}
			}
		}
	}
	
	mpz_clears(sqrtD, ySquareD, xSearch,xSearchBound,gcd,NULL);
	return acceptedNumbers;
}
void TestGenerateSmoothNumbers()
{
	mpz_t D;mpz_inits(D, NULL);
	mpz_set_ui(D, 4729494);
		
	int smoothness = 50;
	int maxFoundCount = 20;
	int maxY = 30;
	int maxX = 200;
	SmoothNumber *smoothNumbers = SmoothNumber_FindSmoothNumbers(D, smoothness,maxFoundCount,maxY,maxX);
	for(size_t i = 0; i < arrlen(smoothNumbers); i++)
	{
		PrintSmoothNumber(smoothNumbers[i]);
		printf("\n");	
	}
	for(size_t i = 0; i < arrlen(smoothNumbers); i++){DestroySmoothNumber(smoothNumbers[i]);}
	arrfree(smoothNumbers);
	mpz_clears(D,NULL);
}

Running this code gives us these smooth numbers:

**Smooth numbers and their prime factorizations**

Step 2: Generating a Matrix of Prime Ideal Factorizations

The second step is generating a matrix similar to the one above. This can be done by finding the null space of the reduced row echelon form.

5.0 Implementation Fine Details

This section answers FAQs concerning the implementation.

Where did Lenstra get the smoothness bounds 31 and 43?

This is the Canfield-Erdős-Pomerance bound6. It describes the probability of a random integer being smooth.

A more practical bound is (Odlyzko, 1985)7:

double Find_Log_MPZ_Double(mpz_t x, double base){if(mpz_cmp_ui(x, 0) == 0){return 0;}signed long int ex;const double di = mpz_get_d_2exp(&ex, x);return ( (log(di) + log(base) * (double) ex) /log(base));}

double PomeranceSmoothnessBound(mpz_t N)
{
	//Find log2 of N
	double logBase2 = Find_Log_MPZ_Double(N, 2);	
	
	//Find natural log
	double ln_N = logBase2 * log(2.0);
	double ln_ln_N = log(ln_N);
	
	//Find bound
	double B = exp(sqrt(ln_N * ln_ln_N));
	return B;
}

double OdlyzkoBound(mpz_t N)
{
	double logBase2 = Find_Log_MPZ_Double(N, 2);	
	double c = 1 / (4 * log(2));
	c = sqrt(c);
	double B = c * sqrt(logBase2 * log(logBase2));
	B = pow(2, B);
	return B;
}

void TestSmoothnessBound()
{
	mpz_t D;mpz_inits(D, NULL);
	mpz_set_ui(D, 4729494);
		
	double smoothnessBound = PomeranceSmoothnessBound(D);
	double smoothnessBoun = SMm(D);
	printf("Rough B:%.3f\n Better B:%.3f \n",smoothnessBound,smoothnessBoun);
	mpz_clears(D,NULL);
}

Why do we use primes with Legendre symbol = 1?
- Only primes with Legendre symbol 1 split Q√D (Lynn, 2025)8

References

Lenstra, H,. (2008). Solving the Pell Equation. Algorithmic Number Theory Journal. Link.

Conrad, K,. (2020). Pell’s Equation. Link.

Joyal, B,. (2013). Answer To Rational Solutions of Pell’s Equation. MathStackExchange. Link.

Weisstein, E. (2025). Archimedes' Cattle Problem. MathWorld--A Wolfram Resource. https://mathworld.wolfram.com/ArchimedesCattleProblem.html

Amthor, A,. & Krumbiegel B,. (1880). Das Problema bovinum des Archimedes. Z. Math. Phys. 25, 121-171.

Thome, E,. (2002). The Number Field Sieve. PDF.

Odlyzko, M,. (1985). Discrete logarithms in finite fields and their cryptographic significance. Springer-Verlag, Lecture Notes in Computer Science #209.

Lynn, B,.(2025). Splitting of Primes in Extensions. Link.

LeetArxiv