Posts Tagged ‘prob8’

Problem 8 – Some Improvement

June 5, 2009

I was able to make an improvement in the performance of my script for problem 8. I’m still not particularly sure how to extend my method to taking products of other length substrings of the string of digits (5, in the statement of the problem).

The idea is that we’d like to trim down the number of multiplications we do, remembering some values along the way for future use, instead of doing them over again. The trick seems to be storing the right combinations of products. Here’s the function, which I’ll try to explain below.

def bigprodfive(s):
    """ Largest product of n consectuve digits of s

    We assume that len(s) >= 5, and that it contains only digits

    t = time.time()

    # split the string into a bunch of substrings that have no 0s
    # throw out any substrings that have fewer than 5 characters
    blocks = filter(lambda l: len(l) >= 5, s.split("0"))

    # if there are no substrings, any product would be 0
    if not len(blocks):
        t = time.time() - t
        return (0,t)

    ret = 0
    for b in blocks:
        d = map(int, b) # convert to list of integers

        # store some products
        pair = d[2]*d[3]
        quad = d[0]*d[1]*pair

        # some of the indices we might use are out of bounds
        # and will throw an exception. but that's a good sign
        # we can just stop with this substring (b), and move on
            idx = 0
            while(idx <= len(b)-5):
                # see if the string starting at idx gives a bigger product
                ret = max(ret, quad * d&#91;idx+4&#93;)

                # update our stored products
                quad,pair = pair, d&#91;idx+4&#93;*d&#91;idx+5&#93; # may throw up
                quad *= pair

                # see if the string starting at idx+1 gives a bigger product
                ret = max(ret,d&#91;idx+1&#93;*quad)

                idx += 2
            # don't do anything, "catch" the exception so it
            # doesn't interrupt our calculations

    t = time.time() - t
    return (ret,t)

Hopefully, with the somewhat excessive comments, the main thing I should describe is the stored products. Here's the picture that gave me the idea, d_i is d&#91;i&#93;:
<pre>0: d_0*d_1  d_2*d_3  d_4
1:     d_1  d_2*d_3  d_4*d_5
2:          d_2*d_3  d_4*d_5  d_6
3:              d_3  d_4*d_5  d_6*d_7
4:                   d_4*d_5  d_6*d_7  d_8</pre>
On the left we have the variable idx, increasing from 0, and we show the terms that get multiplied together to determine the product for the substring starting at idx. Notice that, e.g., from idx=1 to idx=2, four of the terms are unchanged. And then from idx=3 to idx=4, same thing. My variable "quad" in the code above stores these products. Notice, furthermore, that to get from the quad for idx=2 to the quad for idx=3, we keep a pair (d_4*d_5, for this particular idx change). Instead of re-calculating that product yet again, we store that product (as "pair"), and then to get the next quad, we can multiply this pair by the next one (which would be d_6*d_7, continuing on).

If we keep track of quads and pairs this way, they only change when we get to an odd value for "idx". My original code didn't wrap things in the try block, and within the while loop just tested the parity of "idx" to decide about updating values or not. But that's a lot of testing, so I thought I'd just take care of idx and idx+1 at the same time, starting with an even idx. But then you might run into trouble at the very end, because you try indexing the idx+5 position in the string, which might not be there (it might be the length of the string, one too far). Well, I could put test code back in my while loop to avoid this issue, but it'll pass every time except possibly once. So I realized I could just wrap the entire while loop in the try block (obtaining the code above), and if we index out of range, we knew we were done anyway, so there's no real harm.

I tried keeping track of how many multiplications this does, based on the input string, and comparing to some of my other solutions. The original solution I posted, which doesn't remember any saved values, does approximately 4 times the string length multiplications. This newer code does approximately 2 times the string length multiplications.

I don't have any sweet performance comparison graphs for you this time, but my little bit of testing indicates that this new solution is approximately 5-6 times faster than my first solution.

I'd still love to get a good generalization of this, taking products of substrings of length other than 5...

In the mean time, I'll post yet another solution. I've been thinking very hard about what products to store, because I want to avoid doing too many multiplications. An unstated goal was that I also wanted to do no divisions, for some reason deciding that this is a slow operation. Of course, if you allow divisions, the process is much simpler, and easier to generalize. To get from the product starting at idx, to the product starting at idx+1, you just divide by the digit at idx, and multiply by an end digit (idx+4 in the problem, but idx+len-1 for length len substrings). I realized, while I was writing this post for the solution above, that I'd never bothered coding this solution up, and that I really should, to see how it compared. In the code above, we just replace the for loop, over blocks, with the following code:

for b in blocks:
    d = map(int, b)

    p = d[0]*d[1]*d[2]*d[3]
    f = 1 # the "first digit" of the previous product

    for idx in xrange(0,len(b)-5+1):
        p = p // f # pull off old first digit
        p *= d[idx+4] # tack on new last digit
        f = d[idx] # update first digit
        ret = max(ret, p)

That’s not only quicker to write, easier to understand, and easier to generalize, it’s faster as well! Not a lot faster, but seems to be perhaps about 10% faster than the first solution above. Silly me.

Problem 8 – Files and Random Numbers

June 2, 2009

I decided that problem 8 made a good problem to learn about reading from files. Then I can store the large number that they suggest in a file, and open the file each time I run my script.

A quick search turned up the section on File Objects in the Python Library Reference. Looks easy enough, especially using the “from __future__ import with_statement” line. I also decided to set up my program to just generate a random string of digits if no file was specified. Unsurprisingly enough, there is a module called random, with a function random() that returns a floating point value in the interval [0,1). My guess it’s a good idea to do random.seed() after importing random. Perhaps somebody who knows more about that could chime in.

So anyway, here’s a quick function to generate a random string of digits:

import random

def randstr(n):
    """ a string, of length n, of random digits """

    return "".join(str(int(10*random.random())) for i in xrange(0,n))

I also guess that a function that takes a string of digits and multiplies them all together would be helpful. Python’s map/reduce routines are pretty fun.

def prod(s):
    """ Take a string of digits, s, and return the product of all those digits """

    return reduce(lambda a,b:int(a)*int(b),s)

I think it’s pretty awesome that you can put a string in a for loop, in place of a list or tuple, and Python will just iterate over the characters in the string. I mean, it makes sense, they’re all “iterable” sorts of objects. But still, it’s pretty sweet.

I thought, for a bit of robustness, I might set up my program to take any file, even if it has non-digits in it. Notice, for example, that the prod function above will throw an exception (a ValueError), if the string contains a non-digit character. So let’s write a little function that’ll take a filename and build a string of all of the digits from that file.

from __future__ import with_statement

def digits(f):
    """ Open the file named 'f', and return a string of all the digits in 'f' """

    ret = ""
    with(open(f)) as file:
        for line in file: # iterate over lines
            ret += filter(lambda c: c.isdigit(),line)
    return ret

The function isdigit above, that gets called on each character in the “line” string is one of the many helpful String Methods.

Now, we’ve got all of our helper bits in play, we just need a function that’ll solve this problem – which is to find the largest product of 5 consecutive digits from a string. I’d love to see (/come up with) a clever solution to this, that does as few multiplications as possible. In the mean time, here’s a one-liner (if you can’t come up with quickly executing code, at least don’t spend much time typing it) that uses my “prod” function above:

def maxsubprod(s, n=5):
    """ The largest product of n consecutive digits from s """

    return max([prod(s[i:i+n]) for i in xrange(0,len(s)-n+1)])

I should probably get more careful with exception handling at some point. If the previous function gets passed a string with length less than n, a ValueError is thrown.