Every semester I teach Karatsuba algorithm by Anatoly Karatsuba in the first or second lecture. Well, to impress my students how a normal task like integer multiplication can be improved by clever algorithms. I still remember the “Aha” moment of reading the algorithm. By using a small trick to save one multiplication in each stage, we can achieve a faster algorithm!

The Daughter

But that is not the only Karatsuba having impact on me. When I was a PhD student, one thing I worked on is to use a computational model called Chemical Reaction Network (CRN), to compute real number. The Number that I extremely interested in computing was Euler’s $\gamma$, which can be defined as $$ \gamma = \lim_{n\to \infty} \bigg(-\log n + \sum_{k=1}^{n}\frac{1}{k}\bigg). $$ The first few digits of the number are 0.5772156649 …

The above formula can not be used to actually compute $\gamma$. Firstly, it is very slow! Secondly, the CRN model I used is a continuous model. I need something continuous, such as an integral. I tried many many formula but failed.

Then one day I ran into this thread on StackOverflow and people talked about a method that can compute $\gamma$ fast. The author? Ekatherina Karatsuba, the daughter of Anatoly Karatsuba. Her research interest are around fast algorithms.

The field of computational mathematics, which is called Fast Algorithms, was born in 1960. By an algorithm, not formalizing this concept, we mean a rule or a way of computation.

I found the formula I need in her paper to do the work for me. Of course, it is not that straightforward. The formula looks like.

\[ \gamma =\Gamma^{\prime}(2) -1, \] where $\Gamma$ is the Gamma function and $\Gamma^{\prime}(2)=\int_0^{\infty} e^{-t}t\log(t)d t$.

The list of her publications contains many other work on computing mathematical constants and special function. One thing I found specifically very interested is this one. Titled: Fast Computation Of ζ(3) and of Some Special Integrals Using The Ramanujan Formula and Polylogarithms.

$$\zeta(3) = -\frac{1}{2}\bigg(\Gamma^{(3)}(1) -3\Gamma^{(2)}(1)\Gamma^{(1)}(1) + 2\Big(\Gamma^{(1)}(1)\Big)^{3}\bigg),$$

which is something I identified that can be computed by CRN using the tricks I developed previous to compute $\gamma$. This time we can compute Apéry’s constant.

Coincidentally, I just figured out a way to do it by another formula in the Wiki page Apéry’s constant.

\[ \zeta(3) = -\frac{1}{2}\bigg( \Gamma^{(3)}(1) + \gamma^{3} + \frac{1}{2}\pi^{2}\gamma\bigg) = -\frac{1}{2}\varphi^{(2)}(1). \] But I only use the first half of the formula. I have no idea how to deal with the polygamma function in the second equation, where $$ \varphi^{(m)}(z)= (-1)^{m+1}\int_0^{\infty}\frac{t^m e^{-zt}}{1-e^{-t}} dt, $$ although it looks manageable.

“举一反三” (jǔ yī fǎn sān) means “from one example, deduce solutions for many other problems.” This emphasizes the concept of applying a solution or understanding from one specific problem to a variety of other, related problems, which is a fundamental skill in algorithmic thinking and problem-solving.

“融会贯通” (róng huì guàn tōng) means “integrate knowledge and achieve a thorough understanding.” This suggests not only learning various algorithms and concepts but also synthesizing them into a comprehensive, interconnected understanding, crucial in algorithm design and application.

You can see these ideas reflected in my lab and homework assignments:

• Understanding an algorithm means you can generalize it.
• Understanding an algorithm means you can adapt it to different, perhaps non-classical, scenarios.
• Understanding an algorithm means you can integrate it with other methods to solve bigger problems.
• Understanding an algorithm means you can reinvent it.

Basic Ideas

The following simple properties can guide us on developing the algorithm.

Property 1: $(p \cdot q)\% m = [(p \% m) \cdot(q \% m)] \% m. $

That is , if we need to do the product of $p\cdot q$ modulo $m$, we can instead do the $p$ modulo $m$ and $q$ modulo $m$ separately, and multiply the results together. But then the result might exceed $m$, so we need to modulo $m$ for another time.

Now let’s consider computing $a^b \,\%m$, we need the following property.

Property 2:

• if b is even:

$$ a^b \ \% m = (a ^ {\frac{b}{2}} \cdot a ^ {\frac{b}{2}}) \% m, \quad \text{by rewriting } a ^ b $$ $$ = ((a ^ {\frac{b}{2}} \% m) \cdot(a ^ {\frac{b}{2}} \% m)) \% m, \quad \text{by property 1.} $$

This suggests a recursive algorithm.

By now, we should be able to handle even exponents. What if $b$ is odd? Well, we just need to take one out of the $b$. That is , if $b-1$ is an even number, we can just rely on the recursive call on computing that.

• if b is odd:

$$ a ^ b \% m = (a \cdot(a ^ {b - 1})) \% m = (a \% m) \cdot(a ^ {b - 1}) \% m $$

Special Case:

We will eventually reduce $b$ to 0, in which case $a^0 \equiv 1 \pmod{m}$.

Pseudocode

By the above analysis, we can implement the function as what follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Computing a^b modulo m, where b is large. 
def largePower(a, b, m):
    # Base Cases
    if (b == 0):
        return 1
    # If b is Even
    if (b % 2 == 0):
        y = largePower(a, b / 2, m)
        return (y * y) % m
    # If b is Odd
    else:
        y = a % m
        return (y * largePower(a, b - 1, m)) % m

Iterative Version

We can also implement the method by loops. The iterative process has a lot to do with the binary representation of $b$. Consider $b=20=2^4+2^2$ or $b=10100_2$ in binary. Note that we have two $1$’s in “10100”. Or we can view “$10100_2 = 10000_2 +100_2$”. From right to left, we will run into the first “1” after two digits; and the second “1” after four digits. The program keeps squaring until it runs into a “1”. So by the time it runs into the first “1”, we have $(a^2)^2 = a^{2\times 2}= a^{2^2}$ already. By the time it runs into the second “1”, we have see four digits and have been squaring four times, so we have $((((a^2))^2)^2)^2 = a^{2\times 2\times 2 \times 2}= a^{2^4}$.

Another observation is that, each time when we see a “1”, we multiply the squaring result to the output result. We did that twice for $b=10100$ for the two 1’s. So we have result= $a^{2^4 + 2^2}=a^{20} = a^{10100_2}$.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

def largePowerIterative(a, b, m):
    result = 1
    a = a % m

    while b > 0:
        # If b is odd, multiply a with result
        if b % 2 != 0:
            result = (result * a) % m

        # b must be even now
        b = b // 2  # Use floor division to avoid float
        a = (a * a) % m

    return result

Generalization

We know binary number are not so special. Now if your treat $b$ as a ternary number. How will you implement the algorithm?

Test

Try a = 7, b = 327, m = 853 to see what you get. (I made up these numbers just now. You can surely make up more tests. Use Wolfram Alpha to get quick results.)

The other day, I sent an announcement to my students about the concept of an open-ended lab, stating they could pick any problem as long as “it wasn’t as trivial as computing the $n$-th Fibonacci number”. But is it really trivial? In the world of algorithms, even straightforward tasks like integer multiplication have spawned a range of fast algorithms, such as Karatsuba, Toom-Cook, and Schönhage-Strassen (See Wiki: Multiplication algorithm), not to mention the recent $O(n\log n)$ method for $n$- bit two numbers. As as the numbers grow, what seems fast can become slow—just think about the grade school multiplication algorithm.

The $O(n)$ Time Complexity Is Not Good For A Number Theoretical Algorithm

Every semester, in my algorithm course I review the Big-O notations, as many people do. One thing that I always emphasize is that the naive primary test is not a polynomial algorithm, and it is not deemed fast. The reason is that we measure complexity of an algorithms by its input length. A number $n$ is presented in $\log(n)$ bits. A polynomial algorithms needs to be in polynomial of $\log(n)$. So, a $O(n)$ algorithm is an exponential algorithm in this sense.

One can read more on this in CLRS Chapter 31: Number - theoretic Algorithms.

In this chapter, a “large input” typically means an input containing “large integers” rather than an input containing “many integers” (as for sorting). Thus, we shall measure the size of an input in terms of the number of bits required to represent that input, not just the number of integers in the input. An algorithm with integer inputs $a_1, a_2, \cdots, a_k$ is a $\textbf{polynomial - time algorithm}$ if it runs in time polynomial in $\log a_1, \log a_2, \cdots, \log a_k$, that is, polynomial in the lengths of its binary-encoded inputs.

Now let’s come back to Fibonacci number. Every semester, I use it to introduce recursion and some basic dynamic programming (memoization trick) to my students. Everyone is glad that we can avoid the naive recursive function implementation to achieve $O(n)$ time and $O(1)$ space!

But wait! From the naive primary test we know the $O(n)$ algorithm for Fibonacci number is not a polynomial algorithm with respect to the input length.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

def fib_linear(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    a, b = 0, 1
    for _ in range(2, n + 1):
        a, b = b, (a + b)
    return b

Can we find a polynomial algorithm for computing the $n$-th Fibonacci number? Ideally in $O(\log n)$? Before we continued, we need to solve a problem: $n$-th Fibonacci number is about $O(n)$ long (Why? See this Quora thread for more info. But it is not our focus here.) It is exponentially long with respect to the input length! How can our algorithm output so long a thing in $O(\log n)$ time? Well, we can avoid this by taking a modulo just as in Leetcode 790. In that way, every number we compute in with in fixed length, and the arithmetic operations between two numbers will not grow as $n$ grows.

Matrix Method

To calculate the n-th Fibonacci number efficiently, we can use a technique known as matrix exponentiation, which allows us to achieve a time complexity of O(log n). The Fibonacci sequence can be represented by a matrix. Although the matrix representation looks very scary, it is not. It is just a fancy way to pack the following two formula altogether. $$ \begin{align} &F_{n+1}= F_n + F_{n-1} \\ &F_n = F_n \end{align} $$

We can rewrite it into the following. \[ \begin{bmatrix} F(n+1) \\ F(n) \end{bmatrix} =\begin{bmatrix} 1 & 1 \\ 1 & 0 \end{bmatrix}\begin{bmatrix} F(n) \\ F(n-1) \end{bmatrix} \]

If we denote the matrix

\[ \begin{bmatrix} 1 & 1 \\ 1 & 0 \end{bmatrix} \]

as $M$, we can express $F(n)$ as:

\[ \begin{bmatrix} F(n) \\ F(n-1) \end{bmatrix} =M^{(n-1)} \begin{bmatrix} F(1) \\ F(0) \end{bmatrix} \]

where F(1) = 1 and F(0) = 0.

The key here is to compute $M^{(n-1)}$ efficiently. We can do this using the concept of exponentiation by squaring, which is a method where we divide the power in half at each step, squaring the result as we go along. This method reduces the number of multiplications needed to calculate $M^n$, resulting in a logarithmic time complexity. The method is not all that mysterious. I talked about the Exponential by Squaring algorithm in my last post. That one is for computing modulo of large number’s large exponent, which makes it a very useful algorithm in Cryptography! Here the matrix exponent computation is more of less the same.

Since the Fibonacci numbers can grow quite large, we will use modulo $10^9 + 7$ to keep the numbers within an integer range.

Here’s the Python code implementing this approach:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

MOD = 10**9 + 7

def matrix_multiply(A, B):
    result = [[0, 0], [0, 0]]
    for i in range(2):
        for j in range(2):
            for k in range(2):
                result[i][j] = (result[i][j] + A[i][k] * B[k][j]) % MOD
    return result

def matrix_power(matrix, n):
    result = [[1, 0], [0, 1]]  # Identity matrix
    while n > 0:
        if n % 2 == 1:
            result = matrix_multiply(result, matrix)
        matrix = matrix_multiply(matrix, matrix)
        n //= 2
    return result

def fib(n):
    if n == 0:
        return 0
    M = [[1, 1], [1, 0]]
    M_power = matrix_power(M, n - 1)
    return M_power[0][0]  # F(n) is located at the top left of the matrix

# Test the function with a sample value
print(fib(10))

Explanation of the code:

1. matrix_multiply(A, B): Multiplies two 2x2 matrices.
2. matrix_power(matrix, n): Efficiently calculates the power of a matrix to the nth power using exponentiation by squaring.
3. fib(n): Calculates the n-th Fibonacci number using matrix exponentiation by calling matrix_power and then returning the appropriate element of the resulting matrix.

Since the exponent $n$ shrinks by half every time. This algorithm has a time complexity of O(log n), making it very efficient for large values of n, and it also handles large Fibonacci numbers by using modulo (10^9 + 7) to keep the numbers within the range of a 32-bit integer.

If you like recursive function better, here is the recursive version of the matrix_power() function.

1
2
3
4
5
6
7
8
9

def matrix_power_recursive(matrix, n):
    if n == 1:
        return matrix
    if n % 2 == 0:
        half_power = matrix_power_recursive(matrix, n // 2)
        return matrix_multiply(half_power, half_power)
    else:
        half_power = matrix_power_recursive(matrix, n // 2)
        return matrix_multiply(matrix_multiply(half_power, half_power), matrix)

No Matrix, Please.

If you hate matrix and don’t want to see it for like at all. You can unpack the calculation in the following way.

The key relationship we’ll use is based on the identities:

1. $F_{2k} = F_k \cdot (2F_{k+1} − F_k)$
2. $F_{2k+1} = F_{k+1}^2 + F_k^2$

where $F_k$ is the $k$-th Fibonacci number. One can prove this by induction. But you must not be satisfied with that. Check out this thread to see how to derive this relation. You will understand why I called this an unpack of the the matrix. (It uses the formula $M^{2n}=M^n\cdot M^n$.) The Wiki page for Fibonacci number a discussion on the matrix forms. Donald Knuth also includes a section in of the chapter of his famous book series, TAOCP, in volume 1: Fundamental Algorithms.

These identities allow us to compute Fibonacci numbers at even and odd indices efficiently, enabling a divide-and-conquer approach similar to the matrix exponentiation method but without explicitly using matrices. But I think the matrix method is more elegant, and easier to generalize. You will see why I say so.

Here’s the Python code implementing this approach:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

MOD = 10**9 + 7

def fib_logn(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    
    # Compute F(n/2) and F(n/2 + 1)
    if n % 2 == 0:
        a = fib_logn(n // 2)
        b = fib_logn(n // 2 + 1)
        return (a * ((2 * b) - a)) % MOD  # F(2k) = F(k) * (2F(k+1) − F(k))
    else:
        a = fib_logn((n - 1) // 2)
        b = fib_logn((n - 1) // 2 + 1)
        return (a * a + b * b) % MOD  # F(2k+1) = F(k+1)^2 + F(k)^2

# Test the function with a sample value
print(fib_logn(10))

Explanation:

• The function fib_logn(n) computes the n-th Fibonacci number using the properties of Fibonacci numbers.
• If n is even, we compute $F_{\frac{n}{2}}$ and $F_{\frac{n}{2} + 1}$, then use the formula for $F_{2k}$.
• If n is odd, we compute $F_{\frac{n-1}{2}}$ and $F_{\frac{n-1}{2} + 1}$, then use the formula for $F_{2k+1}$.

The time complexity remains O(log n) due to the divide-and-conquer nature of the algorithm.

Performance

We want to compare the three version of the implementation: recursive, iterative, and O(n)/linear version. One can find the full test code here. When the test case is n=1000 and we sample every hundred. We get the following result. Several observation:

1. The linear algorithm leads until about $n=400$. Then its time consumption exceeds the other two algorithms.
2. There are some fluctuation between $n=800$ to $n=1000$ in the recursive algorithm. You won’t believe it takes less time to do $n=900$ than $n=800$, right? What happen in the test is that, the jobs of calculating $n=800$ or $n=900$ do not take too much of time, and it might be comparable to the background operating system scheduling. It could be the time we have for $n=800$ is the actually calculation time plus the scheduling time. Actually, the testing result differs time to time. I attached another testing result showing the fluctuation happening somewhere else. The way to mitigate the phenomenon? Well, just use bigger number so that the actual jobs take more time to finish, so the scheduling time become negligible.

Picture below: Random fluctuations causes by scheduling.

Use bigger number: n=10,000,000. The data show that for $n$ equals to ten million, the recursive version takes 0.107 ms, while the iterative version takes 0.0997 ms, but the $O(n)$ version takes 822 ms. So the $\log(n)$ algorithms are now 8000 times more efficient than the $O(n)$ version!

Generalization

Now we have some new fancy ways to computer Fibonacci numbers. We want to ask further questions.

1. Can we compute the $n$-th Domino-Tromino Tiling number in $O(\log(n))$ time? As we remember the recurrence is $f(n)= 2*f(n-1) + f(n-3)$. What is the matrix you need to use for this recurrence?
2. For a general Linear Recurrence Relations, how do we compute the $n$-th number in $\log(n)$? What is the form of the matrix?

The answer is not so terrible to figure out. To help you see that. I attached the solution for the Domino-Tromino Tiling problem here. I was so exciting after I asked ChatGPT wrote this up for me. (No, I did not write the code I attached.) I thought I was going to beat everyone on Leetcode. The result? Well, it only ranks me at 93.06% in time complexity! Surely I know my program will take more memory, but time?

And what is the fastest algorithm on Leetcode then? Well, it is only the $O(n)$ algorithm. I attached in the above source code and ran a local test. Here is the result.

The result shows that when $n=1,000,000$, the $O(\log(n))$ algorithm takes 0.36 ms, while the $O(n)$ algorithm takes 193.8 ms. A more than 500 times different. While the Leetcode tests only tests the maximum number of $n=1000$. The number is not big enough to show the advantage of the $O(\log n)$ algorithm yet.

Other Methods I Tried but Failed

I know there are closed-form formula to compute Fibonacci number directly.

\[ F(n) = \frac{\phi^n - (1 - \phi)^n}{\sqrt{5}} \]

where:

• $F(n)$is the nth Fibonacci number.
• $\phi$ is the golden ratio, approximately equal to $1.61803398875$.
• $n$is the position in the Fibonacci sequence.

The generating function methodology that lead to this kind of formula is a topic that I like to talk about.

However, though attempting, we cannot use the above method to get a fast computation of Fibonacci number. The reasons:

1. Floating number has errors. When $n$ becomes very large, the result you get is not correct any more.
2. How to compute a floating number to a large exponent fast? Well, I believe one can still use the successive squaring method. But the precision of of the floating number itself determines how far you can go.

I guess if we restrict to a finite modular, like $10^9+7$, we can pick a precision for $\phi$. So that our calculation always return correct result when we round it to the nearby integer.

One thing I try for the domino-tromino tiling number is I found its solutions to the characterization function of the recurrence through Wolfram Alpha. And the $n$-th tiling number $f(n)$ can be written as

\[ f(n) = \left(\frac{1}{3} - \frac{1}{354} (1 + i \sqrt{3}) \left(\frac{3481}{2} - \frac{177 \sqrt{177}}{2}\right)^{\frac{1}{3}} - \frac{(1 - i \sqrt{3}) \left(\frac{1}{2} (59 + 3 \sqrt{177})\right)^{\frac{1}{3}}}{6 \cdot 59^{\frac{2}{3}}}\right) \left(\frac{2}{3} - \frac{1}{6} (1 - i \sqrt{3}) \left(\frac{43}{2} - \frac{3 \sqrt{177}}{2}\right)^{\frac{1}{3}} - \frac{1}{6} (1 + i \sqrt{3}) \left(\frac{1}{2} (43 + 3 \sqrt{177})\right)^{\frac{1}{3}}\right)^n \]

\[+ \left(\frac{1}{3} - \frac{1}{354} (1 - i \sqrt{3}) \left(\frac{3481}{2} - \frac{177 \sqrt{177}}{2}\right)^{\frac{1}{3}} - \frac{(1 + i \sqrt{3}) \left(\frac{1}{2} (59 + 3 \sqrt{177})\right)^{\frac{1}{3}}}{6 \cdot 59^{\frac{2}{3}}}\right) \left(\frac{2}{3} - \frac{1}{6} (1 + i \sqrt{3}) \left(\frac{43}{2} - \frac{3 \sqrt{177}}{2}\right)^{\frac{1}{3}} - \frac{1}{6} (1 - i \sqrt{3}) \left(\frac{1}{2} (43 + 3 \sqrt{177})\right)^{\frac{1}{3}}\right)^n \]

\[+ \frac{1}{59} \left(59 + \left(\frac{3481}{2} - \frac{177 \sqrt{177}}{2}\right)^{\frac{1}{3}} + \left(\frac{59}{2} (59 + 3 \sqrt{177})\right)^{\frac{1}{3}}\right) 3^{-n - 1} \left(2 + \left(\frac{43}{2} - \frac{3 \sqrt{177}}{2}\right)^{\frac{1}{3}} + \left(\frac{1}{2} (43 + 3 \sqrt{177})\right)^{\frac{1}{3}}\right)^n \]

Quite a monster with a mixture of algebraic numbers and complex numbers. I asked ChatGPT to generate the formula for me, too. But ChatGPT got it wrong. It is not so good at Math!

I asked ChatGPT to code the above formula into Python (Yes, it takes LaTex input pretty well). Here is the source code. The program fails at $n=50$: it outputs 451995436 while the expected is 451995198.

Final Remarks

1. It is quite a learning experience for me to putting this up.
2. Teaching forces me to reflect. After so many semesters talking about the naive primary test not being polynomial, I finally realized that I did not treat the calculation of Fibonacci number in the same caliber.
3. ChatGPT does a fantastic job. Not a single line of the above code is written by me; The codes are all by ChatGPT. But it takes some extra effort to make sure it does things correctly. See the conversation I had with ChatGPT. (Please ignore that part where I asked it to help my English writing.) Of course, I read quite some other things before I talked to ChatGPT: I knew all the algorithms already from other materials and I knew all the math I need. The more you know, the more helpful ChatGPT is.
4. The future best programming language are Math and English.

1. Euclidean algorithm and Extended Euclidean algorithm. One might say its faster to just do loops. But it is indeed a good problem on recursion anyway.
2. Exponentiation by squaring (The Successive Squaring Algorithm) for modular arithmetic.
3. Catalan number Recurrence. $C_0=0$ and $C_{n+1}=\displaystyle\sum_{i=0}^{n}C_i C_{n-i}$ for $n\geq 0$. And many other recurrence about Catalan number.
4. Derangement. My personal favorite recursion and counting problem. $D(n)=(n-1)\cdot(D(n-1)+ D(n-2))$.
5. Variation of Hanoi Tower Problems. Bi-color[pdf], 3-color, and more.
6. Colin Mallows’s Recurrence on Golomb sequence. $a(1) = 1$; $a(n+1) = 1 + a(n + 1 - a(a(n)))$. I had had a lot of fun deriving this recurrence. It lied somewhere in my old notes. I should share it some day. I don’t have the trick to do the asymptotic analysis as Colin Mallows did. He associated that with golden ratio. This will be a very interesting topic to read more about.

Tiling a $2\times n$ Board with Dominoes and Trominoes

I assigned Leetcode 790 as a Lab to my students. In this problem, we are tasked with finding the number of ways to tile a $2\times n$ board using dominoes and trominoes. Many online resources do not explain the deduction of the recurrence very well. So I decide to write one here.

Mutual Recursion

It will be convenient to introduce another problem $g(x)$ defined as below to help us attack the problem.

We define two functions to represent the number of tiling configurations:

• $f(n)$: Number of ways to tile a $2\times n$ board.
• $g(n)$: Number of ways to tile a $2\times n$ board with one of the grids in the first column missing.

The phenomenon of two functions recursively call each other is called mutual recursion. In a way, the two functions help implementing each other. We will see that in a minute.

Recurrence Relations

The recurrence relations for $f(n)$ and $g(n)$ are derived based on the ways to cover the first column of the board:

For $f(n)$:

1. Cover the first column with a vertically placed domino. This leaves a $2 \times(n - 1)$ board.
2. Cover the first column with two horizontally placed dominoes. This leaves a $2\times(n - 2)$ board.
3. Cover the first column with a tromino in two possible orientations, each leaving a configuration that fits $g(n - 1)$.

This gives us the relation:

$$ f(n) = f(n - 1) + f(n - 2) + 2g(n - 1). $$

Or one can see the following picture for a better visualization.

For $g(n-1)$:

1. Place a domino to cover the remaining grid in the first column of a $2 \times(n - 1)$ board with one grid missing. This leaves a configuration that fits $g(n - 2)$.
2. Place a tromino to cover the remaining grid in the first column. This leaves a standard $2 \times(n - 3)$ board.

This gives us the relation:

$$ g(n - 1) = g(n - 2) + f(n - 3). $$

One can consult the following picture for better understanding.

Eliminate the $g()$ function:

From the expression for $f(n)$, we can express $g(n - 1)$ in terms of $f(n).$ $$ g(n - 1) = \frac{f(n) - f(n - 1) - f(n - 2)}{2} $$ and similarly for $g(n - 2)$.

Substituting these into the recurrence for $g(n - 1)$, we get: $$ \frac{f(n) - f(n - 1) - f(n - 2)}{2} =\frac{f(n - 1) - f(n - 2) - f(n - 3)}{2} + f(n - 3). $$

Rearranging and solving for $f(n)$, we finally get: \[f(n) = 2f(n - 1) + f(n - 3). \]

Python Implementation

Dynamic Programming with Memoization:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

MOD = 10**9 + 7

def f_dp(n):
    if n == 0:
        return 0
    if n == 1 or n== 2:
        return n
    if n == 3:
        return 5
    f = [0] * (n + 1)
    f[0], f[1], f[2], f[3] = 1, 1, 2, 5
    for i in range(4, n + 1):
        f[i] = (2 * f[i - 1] + f[i - 3]) % MOD
    return f[n]

# Example usage:
n = 5
print(f_dp(n))

Conclusion Remarks

Don’t overthink the notion of mutual recursion. It is just you find a helper to help you implement your recursive function. Or it is just a name you give to some special intermediate calculation.

I generate some of the math in markdown by ChatGPT. I would hope it can help me draw the pictures but it did not do so good a job in writing Tikz. I ended up using Mathcha to draw the picture I need. It is a very good online what-you-see-is-what-you-get solution for LaTex/Tikz. I highly recommend it when it comes to draw math pictures.

Understanding Bayes’s Rule Through a Concrete Example

Bayes’s rule is a fundamental concept in probability theory, offering a robust framework for understanding conditional probabilities. This article delves into Bayes’s rule, illustrating its principles through a practical example and visualizations. The example is taken from Example 1.10.1 (Page 59 - 62) of the book An Introduction to Kolmogorov Complexity and Its Application, Fourth Edition, authored by Ming Li and Paul Vitányi.

Formal Statement of Bayes’s Rule

Bayes’s rule can be formally stated as follows:

$$ P(A | B) = \frac{P(B | A)P(A)}{P(B)} $$

where:

• $P(A | B)$ is the posterior probability of event $A$ given event $B$.
• $P(B | A)$ is the likelihood of event $B$ given event $A$.
• $P(A)$ is the prior probability of event $A$.
• $P(B)$ is the marginal probability of event $B$.

Illustrative Example: The Dice Urn

Consider an urn filled with dice, each having a unique probability $p$ of showing the number 6. The probability $p$ may differ from the 1 / 6 expected from a fair die. If we draw a die from the urn and roll it $n$ times, observing the number 6 $m$ times, we can use Bayes’s rule to understand the probabilities involved.

Prior Distribution

Let $P(X=p)$ represent the prior probability of drawing a die with probability $p$ of showing a 6. According to von Mises’s interpretation, the relative frequency of drawing a die with a specific $p$ converges to $P(X=p)$ over many draws.

Likelihood Function

The likelihood of observing $m$ outcomes of 6 in $n$ throws for a die with probability $p$ is given by the binomial distribution:

$$ P(Y=m | n, p) = \binom{n}{m} p ^ m(1 - p) ^ {n - m} $$

This represents the number of ways to choose $m$ successful outcomes from $n$ trials, multiplied by the likelihood of those successes and failures.

Posterior Distribution

Therefore, the probability of drawing a die with probability $p$ and subsequently observing $m$ outcomes of 6 in $n$ throws is the product $P(X=p)P(Y=m | n, p)$.

In this context, Bayes’s problem involves determining the probability of observing $m$ outcomes of 6 in $n$ throws due to a die with a specific probability $p$. The solution is given by the posterior or inferred probability distribution:

$$ P(X=p | n, m) = \frac{P(X=p)P(Y=m | n, p)}{\sum_{p} P(X=p)P(Y=m | n, p)} $$

Here, the denominator is the sum over all possible values of $p$, ensuring the probabilities sum to 1.

Experiments and Visualization

To understand these concepts better, let’s conduct a series of experiments and visualize the results.

Experiment 1: Uniform Prior

Let $p$ take values 0.1, 0.2, …, 0.9, each with equal probability $P(X=p) = 1 / 9$. We consider two cases: $n = 5, m = 3$ and $n = 500, m = 300$.

Case 1: $n=5, m=3$

Case 2: $n=500, m=300$

Experiment 2: Weighted Prior

Let the new prior distribution be $P(X=p) = i / 45, i = 1, …, 9$. We analyze the inferred probabilities for the same two cases.

Case 1: $n=5, m=3$

Case 2: $n=500, m=300$

These experiments demonstrate that as the number of trials increases, the limiting value of the observed relative frequency of success approaches the true probability of success, regardless of the initial prior distribution. Bayes’s rule proves to be a powerful tool for making inferences based on a large number of observations. However, for small sequences of observations, knowledge of the initial probability is crucial for making justified inferences.

The codes for the experiments are generated by ChatGPT.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

import numpy as np
import matplotlib.pyplot as plt
from scipy.special import comb

# Define the binomial likelihood function


def binomial_likelihood(p, n, m):
    return comb(n, m) * (p**m) * ((1 - p)**(n - m))

# Calculate the posterior distribution


def posterior(p_values, prior, n, m):
    likelihoods = np.array([binomial_likelihood(p, n, m) for p in p_values])
    unnormalized_posterior = prior * likelihoods
    posterior = unnormalized_posterior / np.sum(unnormalized_posterior)
    return posterior


# Define p values and prior distributions
p_values = np.arange(0.1, 1, 0.1)
uniform_prior = np.array([1 / 9] * 9)
weighted_prior = np.array([i / 45 for i in range(1, 10)])

# Experiment 1: Uniform prior
posterior_uniform_5_3 = posterior(p_values, uniform_prior, 5, 3)
posterior_uniform_500_300 = posterior(p_values, uniform_prior, 500, 300)

# Experiment 2: Weighted prior
posterior_weighted_5_3 = posterior(p_values, weighted_prior, 5, 3)
posterior_weighted_500_300 = posterior(p_values, weighted_prior, 500, 300)


# Function to plot with more accurate numbers on top of the bars and save the image
def plot_with_accurate_labels_and_save(p_values, posteriors, titles, xlabel, ylabel, filename, figsize=(18, 14)):
    plt.figure(figsize=figsize)

    for i, (posterior, title) in enumerate(zip(posteriors, titles), 1):
        plt.subplot(2, 2, i)
        bars = plt.bar(p_values, posterior, width=0.08)
        plt.title(title)
        plt.xlabel(xlabel)
        plt.ylabel(ylabel)
        # Adjust y-limit to max value + 0.1 for better visualization
        plt.ylim(0, max(posterior) + 0.1)
        for bar, value in zip(bars, posterior):
            # More accurate label with 6 decimal places
            label = f"{value:.6f}" if value > 0.0001 else "< 0.0001"
            plt.text(bar.get_x() + bar.get_width() / 2, bar.get_height(), label,
                     ha='center', va='bottom', fontsize=8)

    plt.tight_layout()
    plt.savefig(f"/mnt/data/{filename}.png")
    plt.show()


# Call the function with all four experiments
plot_with_accurate_labels_and_save(
    p_values,
    [posterior_uniform_5_3, posterior_uniform_500_300,
        posterior_weighted_5_3, posterior_weighted_500_300],
    titles,
    'p',
    'P(X=p | n, m)',
    'bayes_experiments_accurate_labeled',
    figsize=(18, 14)
)

Conclusion

This approach quantifies the intuition that when the number of trials $n$ is small, the inferred distribution $P(X=p | n, m)$ heavily depends on the prior distribution $P(X=p)$. However, as $n$ increases, the inferred probability $P(X=p | n, m)$ increasingly concentrates around the empirical frequency $m / n = p$, regardless of the prior.

• Show your understanding of a topic related to the course.
• Presentation skills.

Format

• A talk of length 10-15 minutes.
• Slides. (Required for everyone. You need to submit those slides. )
• You can use the whiteboard during your talk.

Structure

1. Introduce the topic. Given the basic definition. Or state the problem you try to solve. (Be friendly to you audience.) You can lightly use some Youtube video to help. But please note that it is your talk, you don’t want to use the video for too long.
2. Solve the problem, typically you present a proof or some calculation (typically for counting and probablity problems). Or if you are introducing some mathematical software, demo what kind of problem it can solve and how to solve it.
3. Summerize the skills that you used form this course.
4. Q and A.

Topics.

I don’t sugguest topics. You need to pick you own topics. However, the following topics are examples to help you understand what is expected. Once you have selected a topic, you need to send me an email to seek my approval on the topic.

Topics needs to be related to contents in Week 1 through Week 6, i.e., Logic, Set Theory, Functions, Proofs, and Theory of Computation.

Set Theory and Mathematical Logic:

1. More application of Cantor’s Diagonalization trick in proof.
2. Cantor-Schroeder-Bernstein theorem.
3. Lean theorem prover.
4. SAT solver tutorial.
5. How to test a well formed formula of propositional logic?

Functions

1. The Cantor-Schroeder-Bernstein theorem.

Theory of Computation

1. Finite State Machine and Regular Expression.
2. Stack Machine.
3. Product of two finite state machine
4. Pigeonhole Principle and Pumping Lemma The above list is not intended to be complete. Again, I expect that you will do you own research to pick your own topic.

Topics from previous semesters

• Fall 2023. For some reason the class likes gambling games and probability a lot.

• Show your understanding of a topic related to the course.
• Presentation skills.

Format

• A talk of length 10-15 minutes.
• Slides. (Required for everyone. You need to submit those slides. )
• You can use the whiteboard during your talk.

Structure

1. Introduce the topic. Given the basic definition. Or state the problem you try to solve. (Be friendly to you audience.) You can lightly use some Youtube video to help. But please note that it is your talk, you don’t want to use the video for too long.
2. Solve the problem, typically you present a proof or some calculation (typically for counting and probablity problems). Or if you are introducing some mathematical software, demo what kind of problem it can solve and how to solve it.
3. Summerize the skills that you used form this course.
4. Q and A.

Topics.

I don’t sugguest topics. You need to pick you own topics. However, the following topics are examples to help you understand what is expected. Once you have selected a topic, you need to send me an email to seek my approval on the topic.

Set Theory and Mathematical Logic:

1. More application of Cantor’s Diagonalization trick in proof.
2. Cantor-Schroeder-Bernstein theorem.
3. Lean theorem prover.
4. SAT solver tutorial.

Recursive function, inductive proof, and counting

1. Problems that has beautiful recursive solutions that we haven’t talked about yet in the class. For example,
   • Derangement
   • Domino and Tromino Tiling
2. Inductive proofs and possibly the recursive algorithms inspired by them.
3. Counting with recursion. For instance, the Catalan Numbers.
4. Generating functions of counting problems. See example in Generatingfunctionology.
5. Correctness of an algorithm by induction.
6. Recursive definitions and structural inductive proof. Example: propositional logic string parsing.

Graphs and trees.

1. Recursive algorithms on Graphs and trees.
2. Proofs on graphs and trees.

Probability.

1. Understand Bayes' Rule via simulation. Example 1.10.1 (Page 59-62), An Introduction to Kolmogorov Complexity and Its Application, Fourth Edition. The above list is not intended to be complete. Again, I expect that you will do you own research to pick your own topic.

Topics from previous semesters

• Fall 2023. For some reason the class likes gambling games and probability a lot.

• Dynamic Programming

Problem Statement

You can find the statement through Beebrowd 1798 (You might need to login and search with the problem number). Note that you need to submit you code and report on Canvas for grading. I won’t maintain the class page for the problem. You can just use the public page to test your code. No ChatGPT generated codes are allowed for this task. You can always play it for fun after submitting your work.

Some Ideas

A full solution down to the pseudo-code level will not be provided. You are taking an algorithm course, not a programming course. To some extend, pseudo-code itself is the solution already.

Some hints:

• It is a Knapsack Problem variant. Think about how to interpret the problem with Knapsack Problem’s terminology.
• But this time for each element, you can take it multiple times. It not either take it or leave it; you can “take” something and take it as many times as you can while complying the constraints. For example, in the sample input, you can take 10= 6+ 2+2, with value 3+1+1 =5, where you take element with length 2 for twice. (Note that the total length is ten, and 6+2+2 already use all the length, so you can not “take” anything anymore.) What do you have to change to the recurrence under this new condition?
• Modify the code from your previous lab on Knapsack Problem to obtain a solution for this one.
• Think about how to make your memoization more efficient. Can you just use 1-D array(s) instead of 2-D array?
• Please find test cases through Beecrowd/UDebug. You can do text comparison if you don’t like Beecrowd auto test.

Submission

Beecrowd

1. Submit it to Beecrowd public page and take a screenshot. Or do output comparison yourself and should me you have the right answer.

Canvas

1. Beecrowd passing screenshot. If you can not pass the tests, tell me that you cannot pass it in the submission comment.
2. You source file. I assume it just take one file.
3. A report on the things you try and your understanding on the problem.
   • It should be something you can read and understand after one year. Write some short introduction to help achieving that goal.
   • What is new in your solution comparing to the lab.
   • What is your understanding about the more space-efficient 1-D array implementation.

Last Note: You are already allowed to use a lot of other resources from previous Lab. No ChatGPT allowed for this task.

Purpose

• Dynamic Programming.
• Application of Sorting in Algorithm Design.

Problem Statement

Let’s schedule some jobs again! We have had a lab on job scheduling problem before and this time we consider jobs with weights.

The Lab is excerpted from Section 12.3 of Algorithm Design and Applications by Michael T. Goodrich and Roberto Tamassia. We also talked about weighted job scheduling in Kleinberg and Tardos. I will not repeat most of the contents in the books. I will just emphasis what I believe are the important aspects of the problem.

• Input: A list of jobs $L$. A job $i$ is specified by a triple: ($s_i$, $f_i$, $b_i$) meaning the starting time, finishing time, and the benefit of performing the job $i$.
• Output: A selected subset of the jobs, so that
  • No jobs in the subset are conflict with each other.
  • The sum of the benefits are maximized.

In the first book above, the problem is introduced via a vivid language and is called “telescope scheduling problem”. Below is an example taking out of the book.

Figure 1: The telescope scheduling problem. The left and right boundary of each rectangle represent the start and finish times for an observation request. The height of each rectangle represents its benefit. We list each request’s benefit on the left and its predecessor on the right. The requests are listed by increasing finish times. The optimal solution has total benefit 17 = 5+ 5+ 2+ 5.

Solution of The Problem

Dynamic Programming and Memoization

As we discussed in the lecture, we need the following notion.

We first assume $L$ is ordered by nondecreasing finish times.

\[ B_i = \text{the maximum benefit that can be achieved with the first $i$ request in $L$.} \] So, as a boundary condition, we get $B_0=0$.

We also define the predecessor, $pred(i)$, for each job $i$, to be the largest index $j$, such that $j$<$i$ and job $i$ and $j$ don’t conflict. If there is no such index, then define the predecessor of to be 0 (use -1 if index starting from 0).

The definition of the predecessor of each job lets us easily reason about the effect that including or not including a job, $i$, in a schedule that includes the first $i$ jobs in $L$. That is, in a schedule that achieves the optimal value, $B_i$, for $i>1$, either it includes the job $i$ or it doesn’t; hence, we can reason as follows:

• If the optimal schedule achieving the benefit $B_i$ includes job $i$, then $B_i= B_{pred(i)} + b_i$. If this were not the case, then we could get a better benefit by substituting the schedule achieving $B_{pred(i)}$ for the one we used from among those with indices at most $pred(i)$.
• On the other hand, if the optimal schedule achieving the benefit does not include job $i$ , then $B_{i-1}$. If this were not the case, then we could get a better benefit by using the schedule that achieves $B_{i-1}$.

Therefore, we can make the following recursive definition:

\[ B_i = \max\{B_{i-1}, B_{pred(i)}+ b_i\} \]

It is most efficient for us to use memoization when computing $B_i$. We use the arrays $B$ and $P$, and $B[i]$ and $P[i]$ for $B_i$ and $pred[i]$ respectively in the following code.

1
2
3

B[0]=0
for i=1  to n do:
    B[i]=max{B[i-1], B[P[i]]+ b_i}

We can see this algorithm runs in $O(n)$, where $n$ is the length of $L$. The bottleneck of the algorithm lies in the sorting of $L$, which costs $O(n\log(n))$. However, we still haven’t discuss how to compute $P[i]$ or $pred(i)$.

How to Compute $pred(i)$

Naive Linear Search

If you do a Google search, you will easily find solutions using naive way to solve the problem.

For example, this one’s Python implementation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Find the latest job (in sorted array) that
# doesn't conflict with the job[i]. If there
# is no compatible job, then it returns -1
def latestNonConflict(arr, i):

    for j in range(i - 1, -1, -1):
        if arr[j].finish <= arr[i - 1].start:
            return j

    return -1

does a linear search for each job $i$, and you need to do it for every $i$. So the complexity of calculating $P[]$ in this way is $O(n^2)$.

Binary Search

You can improve the above process by using binary search. See this link for the full solution. Basically, since the jobs already sorted by finish time, you just need to binary search for the first job so that its finish time is earlier than the target job’s start time.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# A Binary Search based function to find the latest job
# (before current job) that doesn't conflict with current
# job.  "index" is index of the current job.  This function
# returns -1 if all jobs before index conflict with it.
# The array jobs[] is sorted in increasing order of finish
# time.
def binarySearch(job, start_index):

    # Initialize 'lo' and 'hi' for Binary Search
    lo = 0
    hi = start_index - 1

    # Perform binary Search iteratively
    while lo <= hi:
        mid = (lo + hi) // 2
        if job[mid].finish <= job[start_index].start:
            if job[mid + 1].finish <= job[start_index].start:
                lo = mid + 1
            else:
                return mid
        else:
            hi = mid - 1
    return -1

This method takes $O(\log(n))$ time to compute $pred(i)$ for each $i$. So in total it costs $O(n\log(n))$ time.

Some implementations use in Python use bisect module for binary search.

An $O(n)$ Algorithm Once and For All

The previous solutions compute $pred(i)$ for each $i$ separately. We now introduce a method to compute $pred(i)$ for all $i$ all at once.

In the Algorithms and Their Applications, a proposed way to do it

In particular, given a listing of $L$ ordered by finish times and another listing, $L'$, ordered by start times, then a merging of these two lists, as in the merge-sort algorithm, gives us what we want.

The predecessor of request $i$ is literally the index of the predecessor in $L$ of the value, $s_i$, in $L'$.

1. We have job $i$’s start time, $s_i$, in our hand.
2. Look at the merged list of $L$ and $L'$, find out $s_i$’s predecessor (but it need to be a finish time $e$). (If its predecessor is a start time, keep looking until you find an end time).
3. Report the index, $j$, of the finish time. (Which job it is coming form.) That is, set pred(i)=j.

In implementation, I modified the above process slightly.

1. I don’t really store the merged list to save space. But I do write out the merging process.
2. The two cursors in the merging algorithm will keep a record on the index of the end time naturally. So I don’t need to “keep looking until I find the end time”.
3. But I need to maintain a map between the start time and its original index before sorting. Note that I originally indexed the jobs by the end time.

Index the job by end time.

1
2
3
4

# zip the three lists
jobs = sorted([(e, s, p) 
       for e, s, p 
       in zip(endTime, startTime, profit)])

Bind the start time and the job index.

1
2
3
4
5
6

# map between the start time and its original job index.
jobs_by_start = sorted([
         (s, index)
        for index, (e, s, p) 
        in enumerate(jobs)])

The merging process. Computing pred() for all jobs once and for all.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# ends: "jobs" as in the above
# starts: "jobs_by_start" as in the above
# P: the pred() function, stored in a list. 
def merge(self, ends, starts, P):
        j = 0 # cursor for end time 
        i = 0 # cursor for start time
        while j < len(ends) and i < len(starts):
            if ends[j][0] <= starts[i][0]:
                j += 1
            else:
                # we might want to do i-1, if we want things starting from 0
                P[starts[i][1]] = j - 1
                i += 1

Note that the algorithm does not work theoretically better than the previous binary searching algorithm, since it require the input to be sorted.

The full implementation can be found here. I changed some variables' name for consistency with the current blog piece.

• Read in a graph.
• Run Dijkstra’s Algorithm to get a subgraph consist of all the shortest paths.
• Remove the subgraph via BFS algorithm.
• Dijkstra’s Algorithm again.

It is a neat problem that can touch many topics we cover in the course.

• Heap.
• Dijkstra;s Algorithm. (And they need to hack it.)
• BFS algorithm.

In the beginning of this semester, I thought some of the concepts in the ASP problem is not so easy to understand by ChatGPT. So it will still take quite some efforts for a student to solve it, even with the help of ChatGPT, since after all, the first time I ran ChatGPT and asked it to implement Dijkstra’s algorithm a few month ago, it did not even compile. But as we know, ChapGPT evolved quickly in the past few months. (And I have not even try GPT-4 yet.) Now it can easily generate all the useful pieces to solve my homework assignment.

Today I will demonstrate how to solve the problem via ChatGPT.

Input Processing

Beecrowd, as many ICPC programming contest sites, feeds formatted input through standard input. Each semester I have students complain they spend too much time on processing the input. Generally speaking, input processing is easy, and ChapGPT can make it easier: You can just speak English and it understands you.

Below is my conversation with ChatGPT. My occasional grammar error does not affect its “understanding”.

Person:

• create python code to read the following format data into a graph: First line: two numbers, the first number “n” represent the number of nodes of the graph, the second number “m” represent the number of edges . Second line: two numbers. Read the first one into a variable name “start”, the second “dest” Follow by m lines. Each line is tuple (a, b, w), which means an edge from a to b with weight w.

• We have multiple such graph as input, until we run into a line “0 0”. Then the input ends.

• Create a loop and read each session of the data into a graph

ChatGPT:

• Sure, here’s the code to read the input data and create a graph:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

def read_graph():
    n, m = map(int, input().split())  # read n and m from first line
    if n == 0 and m == 0:  # end of input
        return None
    start, dest = map(int, input().split())  # read start and dest from second line
    graph = {i: {} for i in range(n)}  # initialize graph with n nodes
    for _ in range(m):
        a, b, w = map(int, input().split())  # read edge from input
        graph[a][b] = w  # add edge to graph
    return start, dest, graph

while True:
    data = read_graph()
    if data is None:
        break
    start, dest, graph = data
    # do something with start, dest, and graph

• The read_graph function reads the input data for a single graph session and returns a tuple of start, dest, and graph. If the first line of input is “0 0”, it returns None to indicate the end of input.

• The while loop repeatedly calls read_graph until it returns None, indicating the end of input. For each graph session, it unpacks the tuple into start, dest, and graph variables, and you can do whatever you want with them inside the loop.

I found that it did not use adjacency list but matrix. So I request it to do so.

Person:

• use adj list for the graph in the above code

ChatGPT:

• Sure, here’s the modified code that uses an adjacency list to represent the graph:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

def read_graph():
    n, m = map(int, input().split())  # read n and m from first line
    if n == 0 and m == 0:  # end of input
        return None
    start, dest = map(int, input().split())  # read start and dest from second line
    graph = [[] for _ in range(n)]  # initialize graph with n nodes
    for _ in range(m):
        a, b, w = map(int, input().split())  # read edge from input
        graph[a].append((b, w))  # add edge to graph
    return start, dest, graph

while True:
    data = read_graph()
    if data is None:
        break
    start, dest, graph = data
    # do something with start, dest, and graph

The graph variable is initialized as a list of n empty lists. Each element of the list represents a node in the graph, and the list contains tuples representing edges from that node. Each tuple contains two values: the destination node and the weight of the edge.

When reading an edge from the input, we append a tuple of (b, w) to the graph[a] list to represent an edge from node a to node b with weight w.

Min Heap Implementation

I asked ChatGPT to implement a min heap with extra requirements. It understood my request about the swap function.

Person:

• implement a min heap in python; add a dict name “position” to map the data items to their indices; add a changePriority() function to allow a user change an item’s priority; add a swap function, in side the swap function, maintenance the position mapping.

ChatGPT:

• Sure, here’s an implementation of a min heap in Python that includes the position dictionary and a changePriority() function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

class MinHeap:
    def __init__(self):
        self.heap = []
        self.position = {}
        
    def parent(self, i):
        return (i-1)//2
    
    def left_child(self, i):
        return 2*i+1
    
    def right_child(self, i):
        return 2*i+2
    
    def swap(self, i, j):
        self.heap[i], self.heap[j] = self.heap[j], self.heap[i]
        self.position[self.heap[i][1]] = i
        self.position[self.heap[j][1]] = j
    
    def insert(self, item):
        self.heap.append(item)
        n = len(self.heap)
        self.position[item[1]] = n-1
        self.sift_up(n-1)
        
    def sift_up(self, i):
        while i > 0 and self.heap[i][0] < self.heap[self.parent(i)][0]:
            self.swap(i, self.parent(i))
            i = self.parent(i)
        
    def extract_min(self):
        min_item = self.heap[0]
        del self.position[min_item[1]]
        last_item = self.heap.pop()
        if self.heap:
            self.heap[0] = last_item
            self.position[last_item[1]] = 0
            self.sift_down(0)
        return min_item
    
    def sift_down(self, i):
        n = len(self.heap)
        min_idx = i
        l = self.left_child(i)
        if l < n and self.heap[l][0] < self.heap[min_idx][0]:
            min_idx = l
        r = self.right_child(i)
        if r < n and self.heap[r][0] < self.heap[min_idx][0]:
            min_idx = r
        if i != min_idx:
            self.swap(i, min_idx)
            self.sift_down(min_idx)
    
    def changePriority(self, item, new_priority):
        if item in self.position:
            i = self.position[item]
            old_priority = self.heap[i][0]
            self.heap[i] = (new_priority, item)
            if new_priority < old_priority:
                self.sift_up(i)
            else:
                self.sift_down(i)

In this implementation, the heap is represented as a list of tuples, where the first element of each tuple is the priority value and the second element is the data item. The position dictionary maps each data item to its index in the heap.

The swap() function swaps two elements in the heap and updates their positions in the position dictionary accordingly.

The changePriority() function takes a data item and a new priority value and updates the heap and position dictionary accordingly. If the new priority is lower than the old priority, the function calls sift_up() to move the item up the heap, otherwise it calls sift_down() to move the item down the heap.

Modified Dijkstra’s Algorithm

I asked ChatGPT to implement Dijkstra’s algorithm for me. I asked it to preserve all the alternative paths with the same distance. And use a list of lists to maintain that information. It correctly implement the functionality I requested. I then tweaked it a little bit, asking it use list, instead of dict for distances.

Person:

• Dijkstra algorithm implementation via heap

ChatGPT:

• I skip the content here. ChapGPT gave me an standard implementation, not satisfying my needs.

Person:

• in the above implementation: the alternative path where new_distance == distances[neighbor] is discard; use a data structure having the type list of lists with name prev, to maintain that information

ChatGPT:

• Sure, we can modify the previous implementation to keep track of the predecessor of each node on the shortest path. Here’s the updated Python code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

import heapq

def dijkstra_heap(graph, start):
    distances = {node: float('inf') for node in graph}  # set all distances to infinity
    distances[start] = 0  # set the distance to the start node to 0
    heap = [(0, start)]  # initialize heap with starting node and its distance
    prev = {node: [] for node in graph}  # initialize predecessor list for all nodes
    while heap:
        (distance, node) = heapq.heappop(heap)  # get node with smallest distance from heap
        if distance > distances[node]:  # if node is already processed, skip it
            continue
        for neighbor, weight in graph[node].items():  # visit neighbors of the node
            new_distance = distances[node] + weight
            if new_distance < distances[neighbor]:  # if new distance to neighbor is shorter, update it
                distances[neighbor] = new_distance
                prev[neighbor] = [node]  # set the node as the only predecessor of the neighbor
                heapq.heappush(heap, (new_distance, neighbor))  # add neighbor and its distance to heap
            elif new_distance == distances[neighbor]:  # if there's a tie in distance, add the node as a predecessor
                prev[neighbor].append(node)
    return distances, prev

The prev dictionary is initialized with empty lists as the predecessor of each node. In the main loop, if a shorter distance to a neighbor is found, its predecessor list is updated with the current node. If there’s a tie in distance, the current node is added to the predecessor list.

The modified function returns two values: distances and prev. distances is the same as before, containing the shortest distances to all nodes from the starting node. prev is a dictionary where each key is a node, and its value is a list of nodes that are its predecessors on the shortest path.

Person:

• use list for distances, and name it dist instead

ChatGPT:

• Sure, here’s the modified implementation that uses a list for distances instead of a dictionary, and names it dist:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

import heapq

def dijkstra_heap(graph, start):
    n = len(graph)
    dist = [float('inf')] * n  # set all distances to infinity
    dist[start] = 0  # set the distance to the start node to 0
    heap = [(0, start)]  # initialize heap with starting node and its distance
    prev = [[] for _ in range(n)]  # initialize predecessor list for all nodes
    while heap:
        (distance, node) = heapq.heappop(heap)  # get node with smallest distance from heap
        if distance > dist[node]:  # if node is already processed, skip it
            continue
        for neighbor, weight in graph[node].items():  # visit neighbors of the node
            new_distance = dist[node] + weight
            if new_distance < dist[neighbor]:  # if new distance to neighbor is shorter, update it
                dist[neighbor] = new_distance
                prev[neighbor] = [node]  # set the node as the only predecessor of the neighbor
                heapq.heappush(heap, (new_distance, neighbor))  # add neighbor and its distance to heap
            elif new_distance == dist[neighbor]:  # if there's a tie in distance, add the node as a predecessor
                prev[neighbor].append(node)
    return dist, prev

The dist list is initialized with float('inf') for all nodes except the starting node, which is set to 0. The prev list is initialized with empty lists as the predecessor of each node.

In the main loop, the distance variable is renamed to dist[node] for readability, and the distances dictionary is replaced with dist. The n variable is added to store the number of nodes in the graph.

The shortest_path function from the previous answer should still work without any modification.

BFS Backtrack Deletion

I ask ChatGPT to delete the edges stored in prev.

Person:

• Given dist, prev as the returning result of the dijkstra_heap function. We consider deleting all the edge recorded in prev subgraph using a BFS algorithm. The function’s signature looks like delete(graph, prev, t), where t is the starting vertex of hte algorithm. For any (u,v) in prev, delete it from graph.

ChatGPT:

• Sure, here’s an implementation of the delete function that takes in the graph graph, the previous node list prev, and the starting vertex t. It uses a breadth-first search (BFS) to traverse the previous node list and remove all the edges in the subgraph represented by prev.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

from collections import deque

def delete(graph, prev, t):
    # Initialize visited and queue for BFS
    visited = set()
    queue = deque([t])
    visited.add(t)

    # Traverse the previous node list with BFS
    while queue:
        u = queue.popleft()
        for v in prev[u]:
            if v not in visited:
                visited.add(v)
                queue.append(v)
            # Remove the edge (u, v) from the graph
            graph[u] = [(w, weight) for w, weight in graph[u] if w != v]

    return graph

The function starts by initializing a set of visited nodes and a queue for the BFS. We add the starting vertex t to the visited set and enqueue it to the queue.

We then traverse the previous node list with BFS. For each node u in the queue, we loop over all its previous nodes v and add them to the visited set if they haven’t been visited before. We also remove the edge (u, v) from the graph by creating a new list of edges for node u that excludes the edge (u, v).

Finally, we return the modified graph.

Person:

• add another parameter “s” to the above BFS algorithm. After the line “u = queue.popleft()”, check if u is s, if true, exit the loop.

ChatGPT:

• Sure, here’s the modified implementation of the delete function that adds a parameter s and checks if the current node u is equal to s. If u is equal to s, we exit the loop early.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

from collections import deque

def delete(graph, prev, t, s):
    # Initialize visited and queue for BFS
    visited = set()
    queue = deque([t])
    visited.add(t)

    # Traverse the previous node list with BFS
    while queue:
        u = queue.popleft()
        if u == s:
            break
        for v in prev[u]:
            if v not in visited:
                visited.add(v)
                queue.append(v)
            # Remove the edge (u, v) from the graph
            graph[u] = [(w, weight) for w, weight in graph[u] if w != v]

    return graph

We add the parameter s to the function’s signature, which represents the node at which we want to stop the BFS. Inside the while loop, we check if u is equal to s. If it is, we exit the loop early by using the break statement.

Note that this modification assumes that s is reachable from t in the graph. If s is not reachable from t, the function will continue to remove edges until the BFS has traversed the entire graph.

Person:

• Instead of remoing (u,v) from the graph, we should remove (v, u) from the graph, since prev records predesessors…….

ChatGPT:

• You’re correct! Since the prev list records the predecessors, we should remove the edge (v, u) instead of (u, v) from the graph.

Here’s the modified implementation of the delete function that removes the edge (v, u) instead of (u, v):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

from collections import deque

def delete(graph, prev, t, s):
    # Initialize visited and queue for BFS
    visited = set()
    queue = deque([t])
    visited.add(t)

    # Traverse the previous node list with BFS
    while queue:
        u = queue.popleft()
        if u == s:
            break
        for v in prev[u]:
            if v not in visited:
                visited.add(v)
                queue.append(v)
            # Remove the edge (v, u) from the graph
            graph[v] = [(w, weight) for w, weight in graph[v] if w != u]

    return graph

Inside the for loop, we remove the edge (v, u) from the graph by creating a new list of edges for node v that excludes the edge (u, v).

Thanks for bringing this to my attention!

Putting Everything Together

By now I have everything we need. Let’s put them all together. There are very few lines of codes that I really need to write, in the case I need to do so, my GitHub Copilot plugin give me very good suggestion: I just need to type a few initial letters and then hit Tab.

Besides changing a few variables/functions name. Here are some modification I did.

Break Early In Dijkstra’s Algorithm

We don’t care about the shortest distance of other nodes more than necessary. When dest is extracted from the heap, its shortest distance is not going to change anymore. Hence we break the loop. See line 21-23. (GitHub Copilot promote the right code and write the comments for me, although its not always accurate.)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

def dijkstra(graph, start):
    n = len(graph)
    dist = [float('inf')] * n  # set all distances to infinity
    dist[start] = 0  # set the distance to the start node to 0
    heap = MinHeap()  # initialize heap
    heap.insert((0, start))  # initialize heap with starting node and its distance
    prev = [[] for _ in range(n)]  # initialize predecessor list for all nodes
    while heap.isempty() == False:
        (distance, node) = heap.extract_min()  # get node with smallest distance from heap
        if distance > dist[node]: # if node is already processed, skip it
            continue
        for neighbor, weight in graph[node]:  # visit neighbors of the node
            new_distance = dist[node] + weight
            if new_distance < dist[neighbor]:  # if new distance to neighbor is shorter, update it
                dist[neighbor] = new_distance
                prev[neighbor] = [node]  # set the node as the only predecessor of the neighbor
                heap.insert((new_distance, neighbor))  # add neighbor and its distance to heap
            elif new_distance == dist[neighbor]:  # if there's a tie in distance, add the node as a predecessor
                prev[neighbor].append(node)

    # break early if we reached the destination
        if node == dest:  # if we reached the destination, stop
            break
    return dist, prev

No changeKey Needed

Note that in this implementation, no changeKey() function is needed. The trick happens in line 10-11 of the above code block.

1
2

        if distance > dist[node]: # if node is already processed, skip it
            continue

The distance is saved in two places: the heap and the array dist. The distances dist[node] get updated more often than the copy in the heap: whenever some neighbor of the node get popped from the heap, it might receive an alternative value. If we want to make all the copy of the dist[node] up-to-date, we need to do a changeKey. However, the changeKey operation is expensive: we need to look though the underlying array of the heap, or use a fast look-up table to find where the (dist[node], node) is stored in the heap and changing it will cause a heapify process.

The above code use a lazy strategy. When a alternative (smaller) distance is found, we just insert into the heap the node and its new distance. The heap-copy of the distance is now outdated. But we still leave it in the heap, until at some point we pop it and do the above comparison to find that it is outdated, then discard it. This strategy is more efficient, but it can incur higher space cost since we will have many copy of the (dist, node) pair in the heap. The correctness is always guarantee, for the smallest copy always get processed first since we are using a min heap!

BFS Deletion

Similarly to the early exit of Dijkstra’s algorithm, I add a line of code to exit the BFS deletion function early. See line 10-11.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

def delete(graph, prev, s, t):
    # Initialize visited and queue for BFS
    visited = set()
    queue = deque([t])
    visited.add(t)

    # Traverse the previous node list with BFS
    while queue:
        u = queue.popleft()
        if u == s:
            break
        for v in prev[u]:
            if v not in visited:
                visited.add(v)
                queue.append(v)
            # Remove the edge (u, v) from the graph
            graph[v] = [(w, weight) for w, weight in graph[v] if w != u]


    return graph

Note that we just want to delete shortest paths starting from s and ending at t, and we begin this BFS in a backward way. When s is pop from the queue, we need to end immediately, otherwise the algorithm might delete other useful edges.

The following test case can make the idea clearer.

1
2
3
4
5
6
7
8
9

5 6
0 1
0 1 1
0 2 1
2 3 1
2 4 1
3 1 1
4 1 1
0 0

You can easily draw the graph out. After the first round, if you break early from Dijkstra’s algorithm, your prev will basically contain two edges {(0,1),(0,2)}. The BFS algorithm will only need to delete (0,1), since s=0, t=1. If you delete also (0,2), you will get a wrong answer! I had to run a debugger to figure this out.

The above few lines of code are the real codes that I add to this project! Easy task? No! If you don’t know the algorithm well, there is no way you know what to do! That is why you still need to learn.

The Full Solution

This blog has already becoming too long. You can download the file here.

Conclusion

1. ChatGPT and GitHub Copilot dramatically changes the way we write codes. In the above project, I only “wrote” four lines of codes, which are generated by Copilot after I typed the first few letters. With ChatGPT and Copilot, one can have great confidence even when writing in an unfamiliar language.
2. A lot of laborious intermediate-level software development jobs will be accelerated by them, if not replace by them. ChatGPT will become a basic tool, just like the calculator in your pocket. You can not get a job by claiming you know how to press the buttons on a calculator. You need to go beyond that.
3. Learning is still important. As you can see from the above, insights into the algorithms are critical. Without a good understanding of the algorithms, you can not go very far.

1. The programs ChatGPT gave are correct. But it did make a mistake: it’s example test cases output does not correctly correspond to its program. The order of reflected codes and non-reflected codes were switched. I pointed that out and ChatGPT fixed it.
2. The inductive proof it wrote misses did not discuss the adjacent codes (last one in the non-reflected codes and the first one in the reflected codes). I pointed out and ChatGPT seemed to try to fixed it but mainly just copy my words.
3. The space complexity $O(1)$ ChatGPT claimed is not true.

Below is the conversation I have with ChatGPT on Gray’s code.

Person:

• Can you recursively generate all Gray' binary code of length n and explain the code to me?

ChatGPT:

• Sure! The Gray code is a binary numeral system where two consecutive values differ in only one bit position. To recursively generate all Gray codes of length n, we can use the following algorithm:

1. If n = 1, the Gray code is ['0', '1'].
2. Otherwise, generate the Gray code of length n-1 recursively.
3. Reflect the Gray code of length n-1 around its middle (i.e., reverse the order of the elements).
4. Add a 0 prefix to each element of the reflected Gray code.
5. Add a 1 prefix to each element of the original Gray code of length n-1.
6. Concatenate the two lists obtained in steps 4 and 5 to obtain the Gray code of length n.

Here is the implementation of the above algorithm in Python:

1
2
3
4
5
6
7
8
9

def gray_code(n):
    if n == 1:
        return ['0', '1']
    else:
        prev_code = gray_code(n-1)
        reflected_code = prev_code[::-1]
        zero_prefixed = [f'0{c}' for c in reflected_code]
        one_prefixed = [f'1{c}' for c in prev_code]
        return zero_prefixed + one_prefixed

To use this function, you can simply call gray_code(n) where n is the length of the Gray code you want to generate. For example:

1
2

result = gray_code(3)
print(result)

This will output:

1

['000', '001', '011', '010', '110', '111', '101', '100']

which are all the possible Gray codes of length 3.

Note that this algorithm has a time complexity of O(2^n) since it generates all possible Gray codes of length n.

Person:

• Do you have a way to generate the codes with less space?

ChatGPT:

• Yes, there is a more memory-efficient way to generate the Gray code sequence of length n using an iterative approach, which uses only O(1) extra space.

The basic idea is to start with the Gray code of length 1, and then generate the Gray code of length n by iteratively inserting a new bit to the left of the previous sequence, while reflecting the previous sequence and adding a new bit to the left of the reflected sequence. This process generates all possible Gray codes of length n without storing all of them explicitly in memory.

Here is the implementation of the iterative algorithm in Python:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

def gray_code(n):
    if n == 0:
        return ['']
    
    prev_code = ['0', '1']
    for i in range(2, n+1):
        reflected_code = prev_code[::-1]
        zero_prefixed = [f'0{c}' for c in reflected_code]
        one_prefixed = [f'1{c}' for c in prev_code]
        prev_code = zero_prefixed + one_prefixed
    
    return prev_code

This algorithm starts by initializing the prev_code variable with the Gray code of length 1 (['0', '1']), and then generates the Gray code of length n by iteratively applying the steps described above.

To use this function, you can simply call gray_code(n) where n is the length of the Gray code you want to generate. For example:

1
2

result = gray_code(3)
print(result)

This will output:

1

['000', '001', '011', '010', '110', '111', '101', '100']

which are all the possible Gray codes of length 3.

Note that this algorithm has a time complexity of O(2^n) since it generates all possible Gray codes of length n. However, it only uses O(1) extra space, which makes it more memory-efficient than the recursive approach.

Person:

• I felt like you need to do prev_code in zero_prefixed = [f'0{c}' for c in reflected_code] and reflected_code in the one_prefixed to get the order you output up there

ChatGPT:

• You are correct! Thank you for pointing that out. The order of the lists in the concatenation step should be reversed to get the correct order of the Gray code sequence. Here is the corrected implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

def gray_code(n):
    if n == 0:
        return ['']
    
    prev_code = ['0', '1']
    for i in range(2, n+1):
        reflected_code = prev_code[::-1]
        zero_prefixed = [f'0{c}' for c in prev_code]
        one_prefixed = [f'1{c}' for c in reflected_code]
        prev_code = zero_prefixed + one_prefixed
    
    return prev_code

Now, calling gray_code(3) will produce the expected output:

1

['000', '001', '011', '010', '110', '111', '101', '100']

Person:

• Can you write a inductive proof that the python code above indeed produces correct Gray code, i.e., the consecutive codes differ by exactly one bit.

ChatGPT:

• Sure! Here’s an inductive proof that the Python code provided earlier in our conversation indeed generates a correct Gray code sequence, where each consecutive code differs by exactly one bit:

• Claim: The Python code generates a correct Gray code sequence of length n, where each consecutive code differs by exactly one bit.

• Base Case: When n = 1, the code generates the list [‘0’, ‘1’], which is a correct Gray code sequence of length 1, where each consecutive code differs by exactly one bit.

• Inductive Hypothesis: Assume that the Python code generates a correct Gray code sequence of length k, where k > 1, and each consecutive code differs by exactly one bit.

• Inductive Step: We will show that the Python code generates a correct Gray code sequence of length k+1, where each consecutive code differs by exactly one bit.

  • First, we obtain the Gray code sequence of length k using the Python code, which we assume to be correct by the inductive hypothesis. Then, we reverse this sequence to obtain a reflected Gray code sequence of length k, which also has the property that each consecutive code differs by exactly one bit. We then add a ‘0’ prefix to each code in the original Gray code sequence, and a ‘1’ prefix to each code in the reflected Gray code sequence. We then concatenate these two lists to obtain a new list of codes of length 2k. This new list of codes has the property that each consecutive code differs by exactly one bit, because adding a ‘0’ prefix to each code in the original Gray code sequence ensures that the first bit in each code changes from 0 to 1 as we go from one code to the next, while adding a ‘1’ prefix to each code in the reflected Gray code sequence ensures that the first bit in each code changes from 1 to 0 as we go from one code to the next. Since the remaining k-1 bits in each code remain the same as we move from one code to the next, we have that each consecutive code in the new list differs by exactly one bit. Therefore, the Python code generates a correct Gray code sequence of length k+1, where each consecutive code differs by exactly one bit.

  • By the principle of mathematical induction, the claim holds for all n > 0. Therefore, the Python code provided earlier in our conversation indeed produces a correct Gray code sequence, where the consecutive codes differ by exactly one bit.

Person:

• You should emphasize that the last code in the prev block and the first code in the reflected prev block are the same, but there are added a prefix of “0” and “1” respectively, so now in the new code they differ by one bit.

ChatGPT:

• You’re absolutely right, thank you for pointing that out! Let me clarify that part of the proof.

  • When we concatenate the list of zero-prefixed codes from the original Gray code sequence with the list of one-prefixed codes from the reflected Gray code sequence, the last code in the zero-prefixed block and the first code in the one-prefixed block are the same. However, since we add a prefix of “0” to the last code in the zero-prefixed block and a prefix of “1” to the first code in the one-prefixed block, these codes become different in the new Gray code sequence. Specifically, the last code in the zero-prefixed block becomes the first code in the new Gray code sequence, and the first code in the one-prefixed block becomes the second code in the new Gray code sequence. Since the last code in the zero-prefixed block and the first code in the one-prefixed block differ by exactly one bit (the first bit), this ensures that all consecutive codes in the new Gray code sequence differ by exactly one bit, as required.

• Thank you for bringing this to my attention, and I apologize for any confusion my previous explanation may have caused.

Person:

• You are a little wordy. Can you give a more conciser summary?

ChatGPT:

• Certainly! Here is a more concise summary of the inductive proof that the Python code generates a correct Gray code sequence, where each consecutive code differs by exactly one bit:
• The base case of the proof is when n = 1, and the code generates the list [‘0’, ‘1’], which is a correct Gray code sequence of length 1, where each consecutive code differs by exactly one bit.
• The inductive hypothesis is that the Python code generates a correct Gray code sequence of length k, where k > 1, and each consecutive code differs by exactly one bit.
• The inductive step involves using the previous Gray code sequence of length k to generate a new Gray code sequence of length k+1, where each consecutive code differs by exactly one bit. This is achieved by taking the previous Gray code sequence, reversing it to obtain a reflected Gray code sequence, and then adding a prefix of “0” to each code in the previous sequence, and a prefix of “1” to each code in the reflected sequence. Concatenating these two lists yields a new Gray code sequence of length 2k, where each consecutive code differs by exactly one bit due to the added prefix, as well as the fact that the previous Gray code sequence and its reflection have the property that each consecutive code differs by exactly one bit.
• By the principle of mathematical induction, the claim holds for all n > 0.

I hope this summary is helpful! Let me know if you have any further questions.

One of my student, Eli Exner, said he would do things differently, but need some time to work out the detail. It turned out his implement the way we would have used to enumerate the powerset. Say, let $n=4$. He would enumerate the sets as follows:

That is a very natural way to enumerate the sets. Basically that is to generate all $\binom{n}{k}$ size $k$ subsets. But how to program it? Eli observed the following features of the enumeration.

1. We enumerate sets in batches of the same size. First batch, size zero (empty set); then size one, size two, etc.
2. The initial set of each batch is a set that packing all number together, starting from 1. For example, for the size two batch, it will start with {1,2}.
3. After listing the initial set, we will increase the last number in the set, until it heats $n$. For example, after {1,2}, we list {1,3}, and then {1,4}.
4. Once we heat $n$ in the previous step. We increase the second to the last number, and pack numbers starting from the increased number, until we have enough numbers (hit the size of the current batch). For example, after {1,2,4}, we would increase the second to the last number, which is 2. Now we have 3. Starting from 3, we need to pack numbers, so we have a 4. Therefore, we get {1,3,4}. Note that 1 remains the same, we don’t touch it. After that we repeat step 3.

There is something special about the above process: why are we interested in the last number, and second to the last number etc.? What does the locations of these numbers that we are increasing mean? Eli develop this notion that he called “depth”.

Problem: Leetcode 416 - Partition Equal Subset Sum.

Problem Statement: Given an integer array nums, return true if you can partition the array into two subsets such that the sum of the elements in both subsets is equal or false otherwise.

You can find the full statement here on Leetcode.com.

Reduction

You might want to say this is not the version of Subset Sum we discussed in our class. True. The version we discussed has a target number $t$. Then what is the target number in this problem? Let’s denote the set of number as $S$, then we let

$$ t =\frac{\sum_{n\in S} n}{2}. $$

If you can find a subset of number sum up to $t$, then the rest of the number must also sum up to $t$. Hence you partition the set into two partitions with the same sum! This idea, turning a problem to another solved problem, is called reduction. We will talk about it in the coming weeks!

Coding!

Now it is time to do some coding! This is a Lab, not an assignment. So feel free to try whatever you want to try! One very helpful link is this, where people discuss the implementation of classical Knapsack Problem. Here are a few things you want to do:

1. If you haven’t familiarize yourself with the tabular method, you can try the example there. (I was a little struggled doing the table, too.)
2. Try an implementation with 2-D array (the table!)
3. Try an implementation with 1-D array. (Scroll down and you will find people talk about it.)

In the lecture I talk about the wavefront of memory when you did memoization. The wavefront of the 2-D table is actually just a single line! So that is why you can improve it by using only one line (the 1-D array)!

Submission via Canvas

1. A short report on the lab, in PDF format.
2. The codes you try.
3. A few Leetcode screenshots (Don’t do too many).

Recursion is a commonly used, yet often difficult to grasp problem solving technique. Recursion itself is not an algorithm, but a process that can be implemented as part of an algorithm. The notion of recursion sometimes conjures up mental images like the one shown below, which is reminiscent of what one might see when two mirrors are placed so they are facing each other, allowing an image to be reflected back and forth indefinitely:

{width=“563” height=“450”}

We may think about solving a problem recursively during the process of problem decomposition, where we are trying to break a complex problem into smaller, less complex sub-problems that are easier to solve. For certain types of problems we may find that we can repetitively decompose a problem such that each sub-problem is really just a smaller version of the larger problem. In those cases, recursion may be an excellent way to solve those problems. The notion of a larger problem being repetitively broken down into similar but smaller problems is encapsulated in the diagram above.

In order for recursion to work, however, there must be a base case, which can be computed without using recursion. The base case serves as a termination point for recursion. Without a base case, infinite recursion results, which is similar to an infinite loop. A base case is conspicuously absent in the above diagram.

Some Classical Examples of Recursive Problems

Examples of problems that lend themselves to recursive solutions abound. We’ll look briefly at a few of the more classical examples of recursive problems here, but this list is by no means exhaustive.

Calculating Fibonacci Numbers

The Fibonacci sequence is a famous mathematical sequence:

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, …

If squares whose sides measure individual Fibonacci numbers (except 0) are created and arranged as shown in the following diagram, a spiral that is found throughout nature can be drawn:

{width=“320” height=“202”}

Furthermore, for any consecutive pair of Fibonacci numbers, if you divide the greater number by the lesser number; e.g., $\frac{13}{8}$, you will get a decimal value that is very close to the Golden Ratio, an irrational number approximately equal to $1.61803…$

The rule for computing the $n$th Fibonacci number is very simple: $$ F(n)=\begin{cases} 0,&\text{for } n=0.\ 1,&\text{for } n=1.\ F(n-1)+ F(n-2),&\text{otherwise.} \end{cases} $$ In plain English, any given Fibonacci number is equal to the sum of the previous two Fibonacci numbers in the sequence. The exceptions, of course, are the first two Fibonacci numbers, which are given as 0 and 1, respectively. Once you have the first two numbers in the sequence you can use the above rule to compute any other Fibonacci number.

To see how recursion might be used to find the $n$th Fibonacci number, you would begin at $n$, then you need to compute $F(n-1)$ and $F(n-2)$. To compute $F(n-1)$ you would compute $F(n-2)$ and $F(n-3)$ , and to compute $F(n-2)$ you would compute $F(n-3)$ and $F(n-4)$, and so on, until $n = 1$ and $n=0$, whose Fibonacci number are given by definition. So the first and second Fibonacci number serves as a base case. You can see the computation of $F(6)$ in the following picture.

{width=“500” }

It is important to note that the Fibonacci sequence is a 0-based sequence. That is, the first number of the sequence, $F(0)$, is 0, the second number of the sequence, $F(1)$, is 1, the third number of the sequence, $F(2)$, is 1, and so on.

Euclid’s Algorithm

To find the GCD of two integers A and B:

1. If A = 0 then GCD(A,B)=B, since the GCD(0,B)=B, and we can stop.
2. If B = 0 then GCD(A,B)=A, since the GCD(A,0)=A, and we can stop.
3. Write A in quotient remainder form (A = B·Q + R)
4. Find GCD(B,R) using the Euclidean Algorithm since GCD(A,B) = GCD(B,R)

1
2
3
4
5

def gcd(a, b):
    if a == 0:
        return b
 
    return gcd(b % a, a)

Euclid’s Algorithm is a classical algorithm for finding the greatest common divisor (GCD) of two integers. The algorithm can be formally stated as follows:

For example, suppose you want to find the GCD of 72 and 54. The following illustrates how the algorithms works:

A = 72, B = 58

Find GCD(A, B), or GCD(72, 58):

A = B * Q + R: 72 = 58 * 1 + 14

Next we must find GCD(B, R). We use the same procedure as in the previous step. When we write the quotient form, we are essentially setting A = B and B = R. Each time we do this we will get a new value for B and a new value for R.

Find GCD(B, R), or GCD(54, 14):

54 = 14 * 3 + 12

Find GCD(B, R), or GCD(14, 12):

14 = 12 * 1 + 2

Find GCD(B, R), or GCD(12, 2):

12 = 2 * 6 + 0

Find GCD(B, R), or GCD(2, 0):

Since B = 0, we can stop, and say that GCD(2, 0), and therefore GCD(72, 58) = 2.

Euclid’s Algorithm lends itself to a recursive solution since at each step we are solving a smaller version of the same problem. We are still solving GCD(A, B), we just keep reducing the values of A and B until one of them reaches 0, at which point we have a base case and can conclude that the GCD is whichever number, A or B, is not 0.

In this example, finding that GCD(72, 58) = 2 might appear to be in error since 2 is such a small value. However, 58 is the product of 2 and 29, both which are prime numbers and cannot be factored any further.

The Towers of Hanoi

The Towers of Hanoi is a classic problem or puzzle that appears in many CS textbooks. Given 3 pegs labeled A, B, and C, and a tower of N disks arranged on peg A such that the disks are stacked in descending order of diameter, as shown in the following diagram:

{width=“500”}

The problem is to move all N disks from peg A to another peg (peg C, for example) such that the following rules are always obeyed:

1.Only one disk may be moved at a time. 2.Only the topmost disk of a stack may be moved; i.e., a disk may not be moved until all the disks on top of that disk have been moved. 3.No disk ever has a larger disk stacked on top of it, on any peg. 4.Peg B may be used as a designated spare to temporarily hold disks.

You can find here a live demo.

The ideal goal is to move all the disks from peg A to peg C using the fewest number of moves. The problem sounds simple, but developing an algorithm to solve it turns out to be somewhat complex. Using recursion, however, a relatively elegant solution can be designed to solve this problem. Keeping in mind the notion of repetitively going from a larger problem to a similar but smaller problem, a recursive solution by necessity needs to begin at the largest problem, which is moving disk 4 along with all the disks on top of it (i.e., the entire stack of disks) from peg A to peg C. We can think of the recursive solution as something like the following:

| Move 4 disks starting at disk 4 from peg A to peg C (can’t do this directly since disk 4 has disks on top of it).

| Move 3 disks starting at disk 3 from peg A to peg C (can’t do this directly since disk 3 has disks on top of it).

| Move 2 disks starting at disk 2 from peg A to peg C (can’t do this directly since disk 2 has disks on top of it).

| Move 1 disk starting at disk 1 from peg A to peg C (can do this — base case).

Once disk 1 has been moved to peg C (the spare peg), we have solved the simplest case of the problem, which is moving a single disk with no disks on top of it. We can return to the previous step, to move disk 2, which we can now move since it no longer has disk 1 on top of it. We move disk 2 to peg B, then move disk 1 from peg C to peg B. We have then solved the next highest case of the problem, which is moving 2 disks.

Generalizing the algorithm above is necessary to prevent problems that will arise if we specify pegs by name. We don’t really care so much where we move the disks so long as we obey the rules of the problem. Either peg B or peg C can be used as the spare peg, as needed, and at times we may need to use peg A as a spare as well. So for each move we need to specify the source peg, the destination peg, and the spare peg, along with the number of disks to move. A generalized solution in Python might look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

## move n disk from "source (s)" peg to "destinate (t)" peg 
## with the help of "helper (h)" peg
def Hanoi(n, s, t, h):
    if n==1:
        move(1,s,t)
    else:
        Hanoi(n-1,s, h, t)
        move(n,s,t)
        Hanoi(n-1,h,t,s)
        
# move disk n form s to t
def move(n,s,t):
    print("Move disk {} from {} to {}".format(n, s, t))

Hanoi(4,"A", "B","C")

Golomb’s tromino Problem

A unit square has been removed from a $2^n \times 2^n$ board. Gave an algorithm to tile the rest of the board with L-shaped trominos.

You can find the solution (and proof) in this document.

The 8 Queens Problem

The 8 Queens problem can be stated as follows:

Given a 8x8 chessboard, place 8 queens on the chessboard such that no queen can capture another queen.

Put another way, no queen may share a row, column, or diagonal with another queen. The diagram below shows one possible solution to this problem:

{width=“282” height=“282”}

Other solutions are possible, but the problem only requires that we find a solution, it doesn’t matter which one.

The problem is commonly formulated by representing the chessboard as a 8x8 array of integers:

0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0

where a 1 indicates the presence of a queen on a square, and a 0 indicates no queen is present on that square. A solution to the problem would then look something like this:

1 0 0 0 0 0 0 0
0 0 0 0 0 0 1 0
0 0 0 0 1 0 0 0
0 0 0 0 0 0 0 1
0 1 0 0 0 0 0 0
0 0 0 1 0 0 0 0
0 0 0 0 0 1 0 0
0 0 1 0 0 0 0 0

The most straightforward approach to solving the problem is a brute-force search. We begin by placing a queen at (row = 0, col = 0), where (0,0) refers to the upper left corner of the 2D array. Then, for each row, including the row we are currently on, we shift one column to the right and see if we can place a queen there safely. In this case we would try to place a queen at (0, 1), which will fail since the queen can be captured by the queen we previously placed at (0,0). We continue incrementing the column and attempting placements until we reach the end of the row, at which point we must reset col to 0 and increment the row. This process continues until we either successfully place all 8 queens, or we encounter a situation where there is no square left to place the current queen. When this happens we must backtrack to the last queen that was placed, and remove it from that square, since having a queen in that particular square does not allow for a solution. We then move to the next possible location for the queen we just removed and see if we can place it at the new location. Eventually, either a solution will be found, or we will get the result that there is no solution given the parameters of the problem.

Conceptually, the process for placing the queens is easy to understand. What may not be immediately intuitive, though, is that backtracking is a naturally recursive process, and as such any problem that requires backtracking lends itself to a recursive solution. We need to think about each queen as being able to remember its own placement history independently of the placement of the other queens. If a queen is placed in a location that ultimately produces a board state from which a solution is not possible, that queen needs to be able to know where it left off in its placement history so that it can undo its current placement and move naturally to its next logical placement. Trying to implement this iteratively can be a complex and confusing task. However, the nature of recursion and how recursion uses a stack to keep track of its history makes recursion a good strategy for solving a problem like the 8 8Queens problem. We don’t need to take special pains to keep track of each queen’s placement history because the recursion does that for us automatically.

The recursive portion of a recursive solution the the 8 Queens problem can be described in pseudocode as shown below. The framework of representing the chessboard and the queens, as well as a function for determining whether a placement is safe or not, have been omitted for brevity.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

Begin at row = 0, col = 0, N = 8 (where N = one dimension of the chessboard)

function placeQueen(row, col)

  // base case: all queens placed
  if col >= 8 return True

  // recursive case
  for row = 0 to N - 1
    if safePlacement
    // currently safe to place queen at (row, col)
      place queen at (row, col)

      // recurse
      if placeQueen(row, col + 1) == True return True
      // turns out placing queen at (row, col) did not lead to a solution,
      // so remove queen from (row, col)
      remove queen from (row, col)

  // queen cannot be placed safely in any row in current column
  return False

One thing to note about a recursive backtracking solution such as this one is the efficiency of the algorithm. Since N = 8 in this case, we would expect the most efficient solution will require only 8 queen placements, one placement per queen. However, with backtracking there will be some trial and error involved before the solution is found, and this will result in a number of attempted queen placements that is greater than 8. The exact number of attempted placements will depend on several factors, including:

1. Which square is tried first (in our example it is the square at location (0, 0), but it could be any one of the 64 squares on the board)

2. The nature of the successor function that determines the order in which the squares are tried (we’ve opted for the straightforward left to right, top down order, but any ordering, even random ordering, would work)

3. Whether the safety check occurs after a queen is added (post-check) or before the queen is added (pre-check); pre-checking will require fewer actual queen placements than will post-checking

Thus, computing the efficiency of a backtracking algorithm is not a trivial process.

Solving a Maze

Mazes are another type of problem in which backtracking plays a key role in finding a solution. The basic principles are very similar to those of the 8 Queens problem, so I will just give a general description here.

When attempting to solve a maze, the key is determining the correct sequence of branchpoints to take. When the only move is to move forward, there are no decisions to be made. When there is a branchpoint, however, in the absence of any information leading us to favor one branch over the others, finding the correct branch will be trial and error. We choose one branch path and proceed until we encounter another branchpoint, a dead end, or the maze exit. Both the maze exit and a dead end are considered base cases. Finding the exit means we have solved the maze. Finding a dead end forces us to backtrack to the most recent branchpoint, where we then proceed down a different branch until we find the exit, a dead end, or another branchpoint, and so on.

Comparing the maze to the 8 Queens problem, each branchpoint in the maze is analogous to a queen. Selecting the correct branch in a maze equates to placing a queen safely. One way in which the two problems differ is that there are multiple solutions to the 8 Queens problem, all of which have the same intrinsic value. A maze, however, often has only one path from the entrance to the exit, so finding the correct path through the maze is equivalent to finding one specific solution to the 8 Queens problem.

As with the recursive solution to the 8 Queens problem, the efficiency of a recursive backtracking solution to solving a maze will depend on a number of factors, including:

1. The number of branchpoints in the maze. More branchpoints increase the complexity of the problem, and will generally increase the time needed to find a solution.

2. The order in which the branchpoints are taken. For example, if left branchpoints are always attempted first, but only the right branchpoints are the correct ones to take, a left-centric algorithm will be less efficient than a right-centric algorithm.

Depth-first Searching

Depth-first searching is a general search algorithm that applies to tree and graph structures, and any structure that resembles a tree or graph. The problem with searching branched structures like trees and graphs is that it usually isn’t known beforehand what the depth of the structure is, or whether the branching is consistent. When the depth is arbitrary, trying to use an iterative solution can be problematical since you don’t know when you have hit the end of a branch. Thus, depth-first searches, like maze-solving algorithms, are naturally suited to being implemented recursively.

Relationship Between Recursion, Recurrence Relations, and Induction

Recursion has its basis in mathematics in the form of recurrence relations. A recurrence relation is a relation that is defined in terms of itself (much like a recursive method calls itself). In general, a recurrence relation requires 2 elements:

1. One or more predefined terms that serve as the initial case, or boundary condition.
2. A rule for defining the next term in the relation based on the value(s) of previously defined terms.

Recurrence relations typically define sequences of numbers. The Fibonacci sequence can be defined using a recurrence relation, as can the sequence of factorial numbers, and the sequence of counting numbers, to name just a few. As an example, let’s look at the recurrence relation that defines the factorial function. Loosely defined, we can say that n! is the product of an integer n and all the integer values less than n, but greater than 0. So, if n = 5, n! = 5 * 4 * 3 * 2 * 1 = 120. This is easy to understand in plain English, but we need to be a bit more formal if we want to define the factorial operation as a recurrence relation.

To define n! as a recurrence relation we first need to define the boundary condition. We know the recurrence relation has to be able to define the next term in the sequence in terms of the relation itself, but in order to do that we must have at least one predefined value. For n! we can define the boundary condition as n = 1. The sequence used in calculating n! never goes less than 1, so using 1! as the boundary condition is appropriate. We can now write a more formal definition of n! as a recurrence relation:

• Boundary Case: $a_1 = 1$
• General Case: $a_n = n * a_{n-1}$, for all n > 1

So we have a rule for defining $n!$, and if we look at a few values we can see that the relation holds for $n = 2$, $n = 3$, and $n = 4$. It seems like it should hold for all integers $n > 1$, but we need to prove the general case before we can be assured our rule is correct. To prove a recurrence relation we need to use a technique known as mathematical induction.

The basic principle behind using mathematical induction to prove a recurrence relation is that, given both the boundary and the general case, we assume the relation holds for some arbitrary value, $k$. We then show that the relation also holds for the value $k + 1$. The idea is that if we can show that the recurrence relation holds for two arbitrary consecutive values, $k$ and $k + 1$, inductive logic states that the relation will hold for all values within the specified range, in this case all positive integer values.

As an example, let’s look at how to use an inductive proof to show that $1 + 2 + 3 + … + n = \frac{n(n + 1)}{2}$, a counting formula commonly used to express the efficiency of a nested loop.

Proof by Induction that $1 + 2 + 3 + … + n =\frac{n(n + 1)}{2}$

Let $P(n) \equiv 1 + 2 + 3 + … + n = \frac{n(n + 1)}{2}$

Boundary Case:

When n = 1, the left hand side of P(n) equals 1 and the right-hand side of P(n) equals 1(1 + 1)/2 = 1. Therefore, P(1) is true.

Inductive Hypothesis:

Assume that $P(k) = \frac{k(k + 1)}{2}$ for some positive integer, k. That is, show that $1 + 2 + 3 + \cdots + k =\frac{k(k + 1)}{2}$

Induction Step:

Show that $P(k + 1) = \frac{(k + 1)(k + 2)}{2}$. That is, show that $1 + 2 + 3 + … + k + k + 1 = \frac{(k + 1)(k + 2)}{2}$

Proof:

We begin by writing out $P(k + 1)$:

$$ P(k + 1) = 1 + 2 + 3 + … + k + (k + 1) $$

Our goal is to make this sequence equal to the right-hand side of the inductive step, $\frac{(k + 1)(k + 2)}{2}$.

Step 1: Group terms

Using parentheses we can rewrite the left-hand side as:

$$P(k + 1) = (1 + 2 + 3 + … + k) + (k + 1)$$

This does not change the sequence, it merely separates the sequence into the sum of two groups of terms.

Step 2: Use the inductive hypothesis to substitute for the leftmost group of terms

Our inductive hypothesis states that $1 + 2 + 3 + … + k = \frac{k(k + 1)}{2}$, so we can substitute for the left-hand group of terms to give:

P(k + 1) = k(k + 1)/2 + (k + 1)

Step 3: Place the right hand side of P(k + 1) under a common denominator

The next step is to simplify the right-hand side of P(k + 1) by putting both terms over a common denominator. Doing so gives:

P(k + 1) = [k(k + 1) + 2(k + 1)] / 2

Step 4: Use factoring to further simplify the right hand side

In the numerator of the right-hand side we see a common term, (k + 1), so we can further simplify by factoring out this term to give:

$$P(k + 1) = \frac{(k + 1)(k + 2) }{2}$$

The right-hand side of P(k + 1) now matches the right-hand side from the inductive step.

Conclusion:

$P(k + 1)$ is true. Therefore, $P(n) = \frac{n(n + 1)}{2}$for all n > 0.

Structure of a Recursive Algorithm

Imperative programming languages use methods or functions that may contain a number of variables whose scope is local to a given method, and the values of those local variables can change depending on the logic contained in the method. Execution of a program involves a sequence of method invocations. The state of each method invocation must be stored in memory to ensure execution within the method continues correctly when the overall flow of execution comes back to that method. Every time a method is invoked, a record of the method invocation is added to the stack frame for that method. The record contains the values of the input parameters, the values for any local variables declared within the method, and address where execution should continue when the method terminates.

Practically speaking, a recursive algorithm involves a method or function that calls itself with the parameters necessary to specify the next smallest problem size. Strictly speaking, a recursive method does not actually call itself, but rather calls another instance of the same method. Like all other method invocations, when a recursive method calls itself another record is added to the method’s stack frame, but with different values for the parameters and local variables.

A simple example will illustrate how the structure of a recursive method uses the stack frame. The following example uses a recursive algorithm to compute the $3$rd Fibonacci number, and uses a technique known as the box trace technique to show how the stack frame is used. The code for the recursive method is as follows:

1
2
3
4
5
6
7

def fib(n)
{
    if (n <= 1)
       return n;
    else
       return fib(n - 1) + fib(n - 2);
}

There are 2 base cases for this algorithm, which correspond to the first two Fibonacci numbers, 0 and 1. (Recall that the first two Fibonacci numbers must be pre-defined, since you need to already have at least two numbers in order to compute the next number in the sequence.) Every other case for n > 1 is thus a recursive case. The diagram below shows how the box trace technique might be used to represent the stack frame as the algorithm’s execution is manually traced. Keep in mind that the stack frame actually would just look like a single column, as records are pushed onto and popped off the stack. I have expanded the stack frame sideways to better illustrate both the past and current changes to the local variables and return values.

In reality, for this example the stack frame would have at most 3 records, since each recursive call will eventually be replaced by a return value when one recursive call hits a base case. Following each row in the diagram from left to right shows chronologically how each of the 3 records changes as the algorithm progresses. When both recursive calls in the first record have been replaced by literal values, the recursion is over, and the computed value can be returned.

Fundamental Rules for Using Recursion

There are 4 fundamental rules, or guidelines that must be observed when designing any recursive algorithm:

1. There must be a base case that can be solved without using recursion.

The recursive process is used to simplify a complex problem by breaking it into a similar but smaller problem. At some point, however, this decomposition process must result in a smaller problem that can be solved directly, without using recursion. This smaller problem, or base case as it is called, essentially serves to put the brakes on the recursion and allow the solution to the smaller problem to be used to solve the next largest problem. Without a base case the recursion will proceed indefinitely, or practically speaking, until the computer’s storage capacity is exhausted. There is no hard and fast rule to how many base cases a recursive solution must have. Most contain only a single base case, but some contain multiple base cases.

2. Every recursive call must progress towards the base case.

The basic tenet of recursion is that a complex problem is simplified by breaking it into a smaller, but similar problem. Therefore, any recursive solution must progress from a larger or more complex problem to a smaller, simpler problem. If this is not the case, infinite recursion will result.

3. Always assume the recursive calls will work.

This rule seems counterintuitive since we are often taught the dangers of making assumptions. To clarify, this rule does not mean to simply assume your recursive algorithm is correct. It simply means that if you know the base case is handled correctly, and you have verified your algorithm works for several larger problem sizes, you can assume the logic of the algorithm will continue to work correctly down to the base case. This assumption is a natural extension of inductive proofs, a strategy that has already been proven to be sound mathematically. Thus, you do not need to prove a recursive algorithm will work for every possible problem size. This assumption does not account for limitations with respect to computer hardware or programming languages, however. A perfectly sound recursive algorithm may be rendered useless if it exhausts the storage capacity of a computer, or if it must use data values that are beyond the capacity of a particular programming language to represent.

4. If you have multiple recursive calls in an algorithm, make sure they don’t solve the same instance of a problem more than once.

In some cases, when a recursive solution contains more than one recursive call, it is possible that the inputs to one recursive call will overlap the inputs used in a previous recursive call. This results in the same instance of a problem being solved more than once by the algorithm, which can significantly reduce the algorithm’s efficiency.

Direct Recursion vs. Indirect Recursion

Most of the time when we write a recursive algorithm we have a method that directly calls itself. This is an example of direct recursion. It is also possible to have one method call another method, which in turn calls the first method. This is an example of indirect recursion, since in this case no method directly calls itself. Any number of methods could lie between the initial method call and the method that eventually calls the initial method. A simple example of indirect recursion is an algorithm that traverses a directory structure in a file system, as shown in the following pseudocode:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

displayDirectoryContents(Directory d)
{
    if d is a valid Directory
        processDirectory(d)
    else
        print error message
}

processDirectory(d)
{
    for each item in d
    {
        print filename of item

        if item is a directory
            displayDirectoryContents(item)
    }
}

In this case it really isn’t necessary to use indirect recursion, as this algorithm can easily be written using direct recursion, but this example is enough to illustrate what indirect recursion looks like.

Tail Recursion

Tail recursion, sometimes called “tail call”, is a form of recursion where the last statement of a recursive method is the recursive call. Consider the following recursive algorithms for computing factorial values:

Standard Recursion

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

int factorial(int n)
{
    int r;

    if (n <= 1)
        r = 1;
    else
        r = n * factorial(n - 1);

    return r;
}

Tail Recursion

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

int factorial(int n)
{
    return tail_factorial(1, n);
}

int tail_factorial(int m, int n)
{
    if (n <= 1)
        return m;

    return tail_factorial(m * n, n - 1);
} 

In the standard recursive algorithm, the stack frame will fill with records since the recursive case of the problem cannot be evaluated until the base case is reached. Each record will contain a pending computation of n * factorial(n - 1). Once the base case is reached, and execution proceeds back up through the stack, all the pending computations can be quickly resolved, but there is inherent inefficiency in having all the unresolved recursive calls on the stack.

The tail recursive algorithm avoids the problem of storing unresolved recursive calls at the expense of some extra code. The original factorial(int n) method now uses a helper method that takes an additional argument to store the current product in the calculation. Using the tail recursive algorithm there will be no unresolved recursive calls waiting on the stack. This avoids the problem of exhausting the available stack space. Also, many compilers are capable of optimizing tail recursion by converting the recursion into a loop, so although the source code may be recursive, the compiled code is not.

While tail recursion would seem to be the de facto standard when implementing a recursive algorithm based on these benefits, it may not always be easy to implement tail recursion for every recursive problem. Additionally, using tail recursion may result in code that is somewhat more difficult to understand.

Efficiency of Recursive Algorithms

While recursive algorithms are elegant in terms of the simplicity of their logic, they are not particularly efficient. The available stack space memory of a computer determines how deep a recursive algorithm can go. The amount of data that must be stored for each record in the stack frame also plays a role. The less space required for local variables, the more records there can be, and thus the deeper the recursion can go. Depending on the limitations of the available stack space, it may be that a recursive algorithm will not work, simply because there aren’t enough resources to accommodate the storage of all the recursive calls.

The running time of a recursive algorithm is often difficult to determine, as it may or may not match what one would expect from an inspection of the problem. Using the Fibonacci example, an iterative solution to find the nth Fibonacci number should require no more than n - 1 computations (since both $F_0$ and $F_1$ are given):

$F_2 = F_1 + F_0 = 1 + 0 = 1$ (n = 2; 1 computation)

$F_3 = F_2 + F_1 = 1 + 1 = 2$ (n = 3; 2 computations)

$F_4 = F_3 + F_2 = 2 + 1 = 3$ (n = 4; 3 computations)

If we look at the recursive solution shown in the box trace diagram, we see that the value for the first Fibonacci number is returned twice when computing the 3rd Fibonacci number. Although the box trace example doesn’t show it, this redundancy in computation increases exponentially as n increases.

Recursion vs. Iteration

Generally speaking, an iterative solution is preferable to a recursive one. Recursion is typically used to develop relatively simple, elegant solutions to complex problems where the formulation of an iterative solution proves to be very difficult. There are a number of pitfalls associated with using recursion. As previously mentioned, an improperly designed recursive algorithm may result in infinite recursion, or redundantly solve the same instance of a problem repeatedly. Even a well formed recursive solution may prove to be unworkable. There is a limit to the number of recursive calls that can practically be made, depending on the storage capacity of the computer running the algorithm and the amount of information that must be stored in each activation record on the call stack.

Iterative solutions are generally more efficient than recursive solutions, in terms of both storage usage and running time. One issue common to recursive algorithms is that once a base case has been reached, execution must proceed back up the call stack. This tends to cause a performance hit, although with modern computer hardware the hit is not as noticeable as it was in timed past.

As a rule, any problem that can be solved recursively can also be solved iteratively. An iterative solution will usually be faster and require less storage than a recursive algorithm. However, an iterative solution to a problem that can be solved easily using recursion often requires a great deal more logic, and this logic may be significantly more convoluted and subsequently error-prone.

It is challenging to teach students recursion, especially in an introductory course where students are not yet familiar with complex data structures. Therefore, teaching concepts like searching on a tree might be premature. I realized why we often start with the Hanoi Tower problem when introducing recursive algorithms.

It’s an excellent problem for teaching because:

• It doesn’t involve complex data structures.
• The problem itself is an interesting game.
• It illustrates beautiful recursive ideas.

In the past, I have used this problem to introduce recursion. I tried to add a twist by assigning them a variation of the problem as an assignment.

The Variation

If all moves must be between adjacent pegs (i.e., given pegs A, B, C, one cannot move directly between pegs A and C, and vice versa), how should we write the recursive function?

Consider we want to move a tower of ‘n’ from a peg named “from” to another peg “to” by using the peg “aux”, that is:

1

move(n, from, to, aux)

We need to ensure the condition:

1

not (from == "A" and to == "C" or from == "C" and to == "A")

Or equivalently:

1

not (aux == "B").

If the auxiliary peg is not B, the task is straightforward. We proceed as in the classic Hanoi Tower solution:

1
2
3

move(n-1, from, aux, to)
moveDisk(n, from, to)
move(n-1, aux, to, from)

Here, moveDisk() function moves disk #n from “from” to “to”.

If the auxiliary peg is B, the process becomes more complex, as demonstrated below:

Question arises: How can we move from A to C directly? This is where our recursive function comes into play. We assume it is correct. It is not the moveDisk() function where we move one disk directly and are bound by the moving rule immediately.

Pseudo-code for the core function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

move(n, from, to, aux):
  if n==0: return
  if aux!="B":
    move(n-1, from, aux, to)
    moveDisk(n, from, to)
    move(n-1, aux, to, from)
  else:
    move(n-1,from, to, aux)
    moveDisk(n, from, aux)
    move(n-1, to, from, aux)
    moveDisk(n, aux, to)
    move(n-1, from, to, aux)

Java Implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

import java.util.Scanner;
public class Hanoi_2 {
  // Main method
  public static void main(String[] args) {
    // Create a Scanner
    Scanner input = new Scanner(System.in);
    System.out.print("Enter number of disks: ");
    int n = input.nextInt();
    // Find the solution recursively
    System.out.println("The moves are:");
    move(n, 'A', 'C', 'B');
    input.close();
  }
  
  // The method for finding the solution to move n disks
  // from fromTower to toTower with auxTower
  public static void move(int n, char from, char to, char aux) {
    // Simplified stop condition. Goes through one more recursive step
    // than the previous solution.
    if (n == 0) // Stopping condition
      return;
    if(aux!='B') {
        move(n - 1, from, aux, to);
        moveDisk(n,from, to);
        move(n - 1, aux, to, from);
    }
    else {   
        move(n-1,from, to, aux);
        moveDisk(n,from, aux);
        move(n-1,to, from, aux);
        moveDisk(n,aux,to);
        move(n-1,from, to, aux);
    }
  }
  
  public static void moveDisk(int n, char from, char to) {
    System.out.println("Move disk " + n + " from " + from + " to " + to);
  }
}

Algorithm Analysis:

How many steps does it take to move a tower of n from one place to another? Before we answer this, we must consider the starting and ending points, as there are obviously two different scenarios:

1. The steps needed when the distance from source and destination is 1.
2. The steps needed when the distance is 2.

Let:

• $f(n)$ be the number of steps needed to move a tower of n when the moving distance is 1.
• $g(n)$ be the number of steps needed to move a tower of n when the moving distance is 2.

Observing the pseudo-code of the core function, we have:

\[ f(n) = g(n-1) + f(n-1) + 1 \] \[ g(n) = 3g(n-1) + 2 \]

Detailed Solution for $ g(n) $:

We define a new function $ h(n) = g(n) + 1 $. The recurrence relation for $ g(n)$ becomes:

\[ h(n) - 1 = 3(h(n-1) - 1) + 2 \]

Simplifying this gives:

\[ h(n) = 3h(n-1) \]

Using the base case $ h(0) = g(0) + 1 = 1 $, we find:

\[ h(n) = 3^n \]

Substituting back for $ g(n) $:

\[ g(n) = h(n) - 1 \] \[ g(n) = 3^n - 1 \]

Detailed Solution for $ f(n) $:

Using the solution for $ g(n)$, we can solve $f(n)$:

\[ f(n) = (3^{n-1} - 1) + f(n-1) + 1 \]

Recognizing this as a geometric series, we can express it as:

\[ f(n) = \frac{3^n - 1}{2} \]

Output for $n=3$:

When $n=3$, the sequence of moves for the Hanoi Tower variation is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

1. Move disk 1 from A to B
2. Move disk 1 from B to C
3. Move disk 2 from A to B
4. Move disk 1 from C to B
5. Move disk 1 from B to A
6. Move disk 2 from B to C
7. Move disk 1 from A to B
8. Move disk 1 from B to C
9. Move disk 3 from A to B
10. Move disk 1 from C to B
11. Move disk 1 from B to A
12. Move disk 2 from C to B
13. Move disk 1 from A to B
14. Move disk 1 from B to C
15. Move disk 2 from B to A
16. Move disk 1 from C to B
17. Move disk 1 from B to A
18. Move disk 3 from B to C
19. Move disk 1 from A to B
20. Move disk 1 from B to C
21. Move disk 2 from A to B
22. Move disk 1 from C to B
23. Move disk 1 from B to A
24. Move disk 2 from B to C
25. Move disk 1 from A to B
26. Move disk 1 from B to C

This sequence shows all the steps needed to move a tower of 3 disks from peg A to peg C using the variation of the rules where each move must be between adjacent pegs.