Advances in Pencil Science

Environment variables in scripts

2012-01-18T15:58:00.000-08:00

Nice tutorial here

Fixing Excel text exports in OS X

2011-08-12T07:58:00.000-07:00

Excel exports .tsv and .csv with a broken linefeed character.

To fix: copy this file to "fixcel.py"

# fixcel.py

from sys import argv

infnam, outfnam = argv[1:]

inf = file(infnam)

indata = inf.read()

inf.close()

outdata = ""

for ch in indata:

if ord(ch) == 13:

  outdata += "\n"

else:

  outdata += ch

outf = file(outfnam,"w")

outf.write(outdata)

outf.close()

#

Then invoke with:

python fixcel.py oldfilename newfilename

Wordify

2010-12-09T23:43:00.000-08:00

This code can count to a million in words. It could probably be refactored so it was shorter and could count to any number. If you were patient enough.

The "-b" flag puts "and" in the places a Brit would put it. It's optional but preferred.
I have random stopping points to increase the modest entertainment value of it running.

Can you recursively get wordify up to the quintillions? How should the "-b" flag be handled?

Can you do this in French?


from string import ascii_lowercase as alphabet
from sys import argv

units = [""] + "one two three four five six seven eight nine".split()
teens =  "ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen".split()
tenties = 2 * [""] + "twenty thirty forty fifty sixty seventy eighty ninety".split() 


def name(number,terminate = True,continued = False,brit=False):
    assert type(number) == int
    assert number > 0 and number < 1001
    if number == 1000: return "one thousand"
    hundreds,residual = divmod(number,100)
    tens,ones = divmod(residual,10)
    result = ""
    if hundreds:
        result += units[hundreds] + " hundred"
    if residual and hundreds:
        result += " "
    if brit and residual and (hundreds or continued):
        result += "and "
    if (tens == 1):
        result += teens[ones] 
        if terminate:
            result += "."
    else:
        if tens:
            result += tenties[tens]
            if ones: result += "-"
        if ones:
            result += units[ones]
        if terminate:
            result += "."
    return result

def wordify(number,brit=False):
    assert type(number) == int
    assert number > 0 and number < 1000000
    thousands,residual = divmod(number,1000)
    result = ""
    if thousands:
        result = name(thousands,False,brit=brit) + " thousand"
        if residual:
            result += " "
            result += name(residual,continued = True,brit=brit)
        else:
            result += "."
    else:
        result = name(number)
    return result

if __name__ == "__main__":

    from time import sleep
    from random import randint

    brit = "-b" in argv
    
    sleeper = 30
    for i in range(1,1000000):
        if i == sleeper:
            sleep(3)
            sleeper += randint(3000,90000)
        print wordify(i,brit=brit)

output:


one.
two.
three.
four.
five.
six.
seven.
eight.
nine.
ten.
eleven.
twelve.
...
nine hundred and ninety-nine thousand nine hundred and ninety-six.
nine hundred and ninety-nine thousand nine hundred and ninety-seven.
nine hundred and ninety-nine thousand nine hundred and ninety-eight.
nine hundred and ninety-nine thousand nine hundred and ninety-nine.

That's as far as it goes so far. Go fix it.

Up to 11

2010-12-09T17:50:00.000-08:00

I managed to confuse myself for hours on #11 trying to be elegant. But I really like this solution:


grid = [[0]*23]*3 + [[int(x) for x in line.split()]+[0,0,0] for line in
'''08 02 22 97 38 15 00 40 00 75 04 05 07 78 52 12 50 77 91 08
49 49 99 40 17 81 18 57 60 87 17 40 98 43 69 48 04 56 62 00
81 49 31 73 55 79 14 29 93 71 40 67 53 88 30 03 49 13 36 65
52 70 95 23 04 60 11 42 69 24 68 56 01 32 56 71 37 02 36 91
22 31 16 71 51 67 63 89 41 92 36 54 22 40 40 28 66 33 13 80
24 47 32 60 99 03 45 02 44 75 33 53 78 36 84 20 35 17 12 50
32 98 81 28 64 23 67 10 26 38 40 67 59 54 70 66 18 38 64 70
67 26 20 68 02 62 12 20 95 63 94 39 63 08 40 91 66 49 94 21
24 55 58 05 66 73 99 26 97 17 78 78 96 83 14 88 34 89 63 72
21 36 23 09 75 00 76 44 20 45 35 14 00 61 33 97 34 31 33 95
78 17 53 28 22 75 31 67 15 94 03 80 04 62 16 14 09 53 56 92
16 39 05 42 96 35 31 47 55 58 88 24 00 17 54 24 36 29 85 57
86 56 00 48 35 71 89 07 05 44 44 37 44 60 21 58 51 54 17 58
19 80 81 68 05 94 47 69 28 73 92 13 86 52 17 77 04 89 55 40
04 52 08 83 97 35 99 16 07 97 57 32 16 26 26 79 33 27 98 66
88 36 68 87 57 62 20 72 03 46 33 67 46 55 12 32 63 93 53 69
04 42 16 73 38 25 39 11 24 94 72 18 08 46 29 32 40 62 76 36
20 69 36 41 72 30 23 88 34 62 99 69 82 67 59 85 74 04 36 16
20 73 35 29 78 31 90 01 74 31 49 71 48 86 81 16 23 57 05 54
01 70 54 71 83 51 54 69 16 92 33 48 61 43 52 01 89 19 67 48'''.split('\n')
] + [[0]*23]*3

import operator

print max([reduce(operator.mul, [grid[y+n*d[0]][x+n*d[1]] for n in (0,1,2,3)])
for x in xrange(0,20) for y in xrange(3,23)
for d in ((0,1),(1,0),(1,1),(-1,1))])

The halo of zeros is very clever and completely works around what made mine confusing, and despite everything, inelegant:



linelen = len(array[0])
for line in array:
    assert len(line) == linelen

numlines = len(array)
offset = 4

directions = {
              "row":(0,linelen-offset+1,numlines,(0,1)),
              "column":(0,linelen,numlines-offset+1,(1,0)),
              "slash":(offset-1,linelen-offset+1,numlines-offset+1,(-1,1)),
              "backslash":(0,linelen-offset+1,numlines-offset+1,(1,1))
             }

maxprod = None
valmax = 0

for direction in directions.keys():
    x0, xlen, yrg, delta = directions[direction]
    vec0 = [(x0 + i * delta[0] ,i * delta[1]) for i in range(offset)]
    for y in range(yrg):
        vec1 = [(item[0]+y,item[1]) for item in vec0]
        for x in range(xlen):
            vec = [(item[0],item[1]+x) for item in vec1]
            vals = [array[item[0]][item[1]] for item in vec]
            prod = reduce(mul,vals,1)
            if prod > maxprod:
                valmax = vals
                maxprod = prod

print valmax, maxprod

Puzzle 5

2010-12-05T22:22:00.000-08:00

Profiting from my overwrought example 3


from bf7 import *
from sys import argv

if __name__ == "__main__":
    if len(argv) == 1:
        upto = 10
    else:
        upto = int(argv[1]) 
    allprimes = primesupto(upto)

    result = 1
    for prime in allprimes:
        factor = prime
        while factor < upto:
            result *= prime
            factor *= prime
    print result

For me it would have been less work to just write

print 2 * 2 * 2 * 2 * 3 * 3 * 5 * 7 * 11 * 13 * 17 * 19

Best solution, for those who can remember grade school:



def gcd(a, b):
 while(b != 0):
  a, b = b, a%b
 return a
 
def lcm(a,b):
 return a * b / gcd(a, b)
 
print reduce(lcm, range(2, 21))

Puzzle 4

2010-12-05T21:48:00.000-08:00

What's the largest palindrome number that is the product of two three digit numbers.

This time I have one of the better solutions:


def nextpal(low,high,reversed=False):
    assert low < high

    if reversed:
        candidate = high
        final = low
        increment = -1
    else:
        candidate = low
        final = high
        increment = 1
    while candidate != final:
        if str(candidate) == str(candidate)[-1::-1]:
            yield candidate
        candidate += increment

def getfactor(low,high,target):
    for test in range(low,high):
        if not target % test:
            residual = target/test
            if residual in range(low,high):
                return test, residual
    return None

if __name__ == "__main__":
    from time import time
    start = time()
    pal = nextpal(10000,1000000, reversed=True)
    while True:
        target = pal.next()
        result = getfactor(100,1000,target)
        if result:
            print target
            print result
            break
    end = time()
    print end - start

Most started from the factors to see if they built palindromes. Much better to go the other way; far fewer tests (the 94th palindrome succeeded, so less than 9400 calculations. The cast to string for the palindrome test is easy, but is it efficient?

The most interesting items on the discussion were the Pytho one-liner:

max(map(lambda s: bool(s) and max(s) or 0, [(filter(lambda n: n == int(str(n)[::-1]), map(lambda n: i*n, range(999, 99, -1)))) for i in range(999, 99, -1)]))

and the pen-and-paper solution

You can also do this with pen and paper. You have a number:

(100a + 10b + c)(100d + 10e + f)

Which is a palindrone. This factors out to:

100cd + 10ce + cf +
1000bd + 100be + 10bf +
10000ad + 1000ae + 100af

Assuming the first digit is 9, then cf must be equal to nine.
Also, both a and d must then be equal to nine. The only ways
to make the last digit of the product of two integers 9 would
be if those integers were either of:

1 and 9
3 and 3
7 and 7

So, both numbers must start with 9, end with either 1, 3, 7,
or 9, and one of them must be divisible by 11. The only
numbers divisible by 11 from 900 - 1000 are:

902
913
924
935
946
957
968
979
990

You can discard all of those that do not end in either 1, 3,
7, or 9, and you are left with:

913
957
979

So now the presumed answer is either:

(900 + 10 + 3)(900 + 10x + 3)
(900 + 50 + 7)(900 + 10x + 7)
(900 + 70 + 9)(900 + 10x + 1)

Factoring all those out, you get:

810000 + 9000x + 2700 + 9000 + 100x + 30 + 2700 + 30x + 9
824439 + 9130x

Now, for the first digit 824439 + 9130x to be 9, x must be 9
(if x were 8, then 824439 + 9130x = 897479, and the first
digit is 8). And so you have 913 * 993, which is the answer.
You can factor the others out to see if they produce a bigger
answer, which they don't.

Puzzle 3 without cheating

2010-12-04T22:55:00.000-08:00

I am about 2800 times slower than the cheat I downloaded. But I get the right answer, and in a tolerable time (30 seconds). Pure brute forcing this seems like it would take much longer.


from sys import argv
from math import sqrt
from time import time

# PRIME GENERATOR

def nextprime():
    """ generate next prime """

    myprimes = [2]
    candidate = 3
    yield 2
    while True:
        factors = factorize(candidate,myprimes)
        if len(factors.keys()) == 1:
            myprimes.append(candidate)
            yield candidate
        candidate += 2

def primesupto(upto):
    """ collect primes from the generator"""

    result = []
    primes = nextprime()
    p = None
    while not p or p < upto:
        if p:
            result.append(p)
        p = primes.next()
    return result

def factorize(target,allprimes=None,verbose=False):
    """ get factors of an integer

    this works but is very slow if the primes aren't cached """

    factors = {}
    upto = int(sqrt(target))
    residual = target
    prime = nextprime()

    if allprimes:
        prime = allprimes.__iter__()

    while True:  # get factors and multiplicities up to sqrt(N)
        try:
            factor = prime.next()
        except StopIteration:
            break
        if factor > upto:
            break
        while not residual % factor:
            residual /= factor
            factors[factor] = factors.get(factor,0) + 1           

    if residual > 1: # if anything is left it is the sole prime factor > sqrt(N)
        factors[residual] = 1

    return factors
   
if __name__ == "__main__":
    target = int(argv[1]) # get target from command line
    upto = sqrt(target)
    starttime = time()
    allprimes = primesupto(upto)
    print max(factorize(int(argv[1]),allprimes).keys())
    endtime = time()
    print endtime - starttime

Update:

Best answer, wow!


from time import time
from sys import argv

n = int(argv[1])
d = 3

starttime = time()
while (d * d < n):
   if n % d == 0: n /= d
   else: d += 2
endtime = time()
print n
print endtime - startime

Runs in 0.7 msec. I feel like a complete idiot now. Oh well, it was interesting anyway.

Cheating, sorta

2010-12-04T12:10:00.000-08:00

What is the largest prime factor of the number 600851475143 ?


from os import system
from os.path import isfile
from sys import argv

try:
  import numbthy
except:
  if not isfile("numbthy.py"): # don't mess with it if it's already there and broken
     system("wget http://userpages.umbc.edu/~rcampbel/Computers/Python/lib/numb
thy.py")
  import numbthy

f = numbthy.factors(int(argv[1]))
if f: print f[-1]

Level 2

2010-12-04T08:01:00.001-08:00

I'm mostly at level 2 in this chart, except for IDEs. I don't really use them. I like aquamacs for an editor but even there I'm not good at it. I'm also pretty lame at version control, never having had the opportunity to work closely with others; hence never having run into a merge problem in real life.

I read some of the sorts of books he calls level 3.

Number Two

2010-12-03T13:58:00.001-08:00

Python shines on this one. Find the sum of all the even-valued terms in the Fibonacci sequence which do not exceed four million.


from itertools import ifilterfalse

def fib(n1=1,n2=1,fmax=4000000):
    "terminating fibonacci sequence"

    assert 0 < n1 <= n2 < fmax
    l = [n2,n1]
    while l:
        if l[0] < fmax/2:
            l.append(sum(l))
        yield l.pop(0)

if __name__ == "__main__":
    print sum(ifilterfalse(lambda x: x % 2,fib()))

Euler Project

2010-12-03T13:13:00.000-08:00

prob 1

A one-liner.


>>> sum([i for i in range(10) if not (i%3 and i%5)])
23
>>> sum([i for i in range(1000) if not (i%3 and i%5)])
233168

So it's surprisingly interesting:


>>> for n in range(1,9):
...     print sum([i for i in range(10**n) if not (i%3 and i%5)])
... 
23
2318
233168
23331668
2333316668
233333166668
23333331666668
2333333316666668

Can't even come up with a succinct notation for the theorem, never mind any idea how to prove it.

Moving notes to self

2010-11-30T13:08:00.000-08:00

Notes to self or those interested in my daily pain will be moved to http://aardvarknoplay.blogspot.com/ . Old notes being moved.

This blog will actually have content that others may find interesting.

Some Key Papers in Climate Modeling History

2010-10-21T20:43:00.000-07:00

A first pass at a greatest hits list; updates will not necessarily be noted as such. I had a request for key papers in the history of modeling so the emphasis leans that way. Nominations welcome.

Charney, Fjortofft & von Neumann 1950: Numerical integration of the barotropic vorticity equation, Tellus 2:237-254

Lorenz 1963: Deterministic Nonperiodic Flow J. Atmos. Sci 20:130-141

Lorenz 1967: The nature and theory of the general circulation of the atmosphere. WMO. http://eapsweb.mit.edu/research/Lorenz/General_Circ_WMO_1967_Part1.pdf

Manabe & Bryan 1969: Climate calculations with a combined ocean-atmosphere model.

Bryan & Cox 1972: The circulation of the world ocean: a numerical study. Part I, a homogeneous model J Phys Oceanog

Arakawa & Lamb 1977: Computational Design of the basic processes of the UCLA general circulation model. Methods in Computational Physics 17:173-265

Charney et al 1979: Ad Hoc Study Group on Carbon Dioxide and Climate. http://www.atmos.ucla.edu/~brianpm/charneyreport.html

Manabe & Stouffer 1988: Two stable equilibria of a coupled ocean-atmosphere model

Ramanathan et al 1985: Trace gas trends and their potential role in climate change. J Geophys Res 90:5547-5566

Cess et al 1990: Intercomparison and interpretation of climate feedback processes in seventeen atmospheric general circulation models

Bryan 1991: Poleward heat transport in the ocean: a reviwe of a hierarchy of models of incerasing resolution. J Phys Oceanog 18:851-867

Cubasch et al 1992: Time-dependent greenhouse warming computations with a coupled ocean-atmosphere model

Hansen et al 1992: Potential climate impact of Mount Pinatubo eruption GEOPHYSICAL RESEARCH LETTERS, VOL. 19, NO. 2, PP. 215-218, 1992
doi:10.1029/91GL02788

Hasselmann 1997: Multi-patterm fingerprint method for detection and attribution of climate change. Climate Dynamics 13:601-612

Hansen et al 1998: Climate Forings in the Industrial Era. PNAS

Bengtsson 1999: From short-range barotropic modeling to extended-range global weather prediction. Tellus 51 A-B:13-23

Randall 2000: General Circulation Model Development: Past, Present, and Future (International Geophysics) [Hardcover]

Hansen et al 2005: Earth's Energy Imbalance: Confirmation and Implications. Science.

Hack et al 2006: Simulation of the Global Hydrological Cycle in the CCSM Community Atmosphere Model Version 3. J Clim Volume 19, Issue 11 (June 2006)

Schmidt et al 2006: Present-Day Atmospheric Simulations Using GISS ModelE: Comparison to In Situ, Satellite, and Reanalysis Data. J. Climate, 19, 153–192.

Rahmstorf et al 2007: Recent Climate Observations Compared to Predictions. Science 316:709 DOI: 10.1126/science.1136843

More Adventures in Climate Analysis Installs

2010-10-12T13:59:00.000-07:00

So here I document getting Ferret to work on a Mac; getting it to run on the super is another problem. I assumed no privileges and more or less succeeded. However, the install instructions are (as is normal with scientist code) broken.

Here are the absurd installation instructions:

http://ferret.pmel.noaa.gov/static/Downloads/mac_intel_downloads.html

This is basically just three useful links to the actual binaries under "Environment and Executable files"; ignore the rest of it.

Then proceed to

http://ferret.pmel.noaa.gov/static/Downloads/ferret_installation_and_update_guide_v600.html

If you don;t knwo about environment variables in bash, go study up on them.

The first step 1 should be

tar xf $GET_LOCATION/fer_environment.tar

The second step 1 and step 2 [sic] work as noted. Step 3 should be

tar xf $GET_LOCATION/fer_dsets.tar

as the Mac ungzips behind your back. Your next step will have to be backwards, as the Finstall script expects the exectuables to be gzipped. So go back to your download directory and type

gzip fer_executables.tar

Also, you will have to find a place for an extra file in the middle of the next step. Be sensible, create a ~/.Ferret directory before doing the next step. Use that for your $SET_FER .

Now you can run the Finstall per instructions. But the script you ghet out will be a csh script not a bash script. The following edits worked for me:


export FER_DIR=/Users/tobis/Ferret
export FER_DSETS=/Users/tobis/FerSamp

#   if ( $PATH !~ *ferret* ) then
#       export PATH={$PATH}:$FER_DIR/bin
#   else
#        echo replacing ferret path
#      export PATH=`echo $PATH | awk -F: '{for (i=1; i<=NF; i++) {if ($i \!~ /ferret\/bin/) {printf "%s:",$i}}}'`
#      export PATH={$PATH}$FER_DIR/bin
#   endif
 
# =========== Initially make no modifications below this line ===========

#  Default Ferret document browser
#   export FER_WEB_BROWSER="netscape -ncols 60"
   export FER_EXTERNAL_FUNCTIONS="$FER_DIR/ext_func/libs"
   export FER_GO=". $FER_DIR/go $FER_DIR/examples $FER_DIR/contrib"
   export FER_DATA=". $FER_DSETS/data $FER_DIR/go $FER_DIR/examples $FER_DIR/contrib"
   export FER_DESCR=". $FER_DSETS/descr"
   export FER_GRIDS=". $FER_DSETS/grids"
   export TMAP="$FER_DIR/fmt"
   export PLOTFONTS="$FER_DIR/ppl/fonts"
   export SPECTRA="$FER_DIR/ppl"         # for old ferret versions
   export FER_PALETTE=". $FER_DIR/ppl"       # palette search list

So basically, change all the setenvs to exports, put in the equal sign, comment out the if/else block. If you are better with bash scripts, the if/else thing looks harmless.

I did not uncomment the reference to netscape. I don;t know where to get a netscape for os X 10.6 anyway. Sigh.

This actually works!

Meanwhile Chas has bunches of hoops for me to jump through. Will keep you posted, if you exist.

PyNGL Even worse than CDAT?

2010-10-06T12:17:00.000-07:00

Undecided as yet. More to follow. But plenty of the same signs of cluelessness.

I am still hopeful that the interpolation routine I need will work. And the install is much more straightforward, there being suitable binaries for the machines I care about.

But the test routine starts with an absolute path to a python installation at NCAR!

And this is unpythonic, though perhaps this is a Fortran 77 constraint rearing its ugly head


import Ngl
import Nio
import numpy

# create an array and make a netcdf file

ff = Nio.open_file("foo.nc","c")
ff.create_dimension("x",10)
ff.create_dimension("y",10)

ff.create_variable("u","f",("x","y"))

z = numpy.zeros((10,10),"f")

# ff.variables["u"] = z doesn't work

ff.variables["u"].assign_value(z)

# assignment is by value, as the name indicates

z[3,3] = 3.3

ff.variables["u"][4,4] = 4.4

# but after assignment you can pick up a reference

u = ff.variables["u"]

u[2,2] = 2.2

"""
>>> ff.variables["u"][:]
array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  2.2,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  4.4,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]], dtype=float32)
"""

ff.close()

Anthony Doerr

2010-09-25T11:15:00.000-07:00

Read the whole thing. It starts

During my sophomore year, 1992, 1,500 scientists, including more than half the living Nobel laureates, admonished in their Warning to Humanity: “A great change in our stewardship of the earth and the life on it is required if vast human misery is to be avoided and our global home on this planet is not to be irretrievably mutilated.”

So what have we done? Not much. From 1992 to 2007, global CO2 emissions from burning fossil fuels rose 38 percent. Emissions in 2008 rose a full 2 percent despite a global economic slump. Honeybees are dying by the billions1, amphibians by the millions, and shallow Caribbean reefs are mostly dead already.2 Our soil is disappearing faster than ever before, half of all mammals are in decline, and a recent climate change model predicts that the Arctic could have ice-free summers by 2013. Unchecked, carbon emissions from China alone will probably match the current global level by 2030.

“The god thou servest,” Marlowe wrote in Dr. Faustus, almost four hundred years before the invention of internet shopping, “is thine own appetite.” Was he wrong? How significantly have you reduced your own emissions since you first heard the phrase “climate change?” By a tenth? A quarter? A half? That’s better than I’m doing. The shirt I’m wearing was shipped here from Thailand. The Twinkie I just ate had 37 ingredients in it. I biked to work through 91-degree heat this morning but back at my house the air conditioner is grinding away, keeping all three bedrooms a pleasant 74 degrees.

My computer is on; my desk lamp is glowing. The vent on the wall is blowing a steady, soothing stream of cool air onto my shoes.

h/t Andrew Sullivan. Anthony Doerr, whom I had not heard of until today, lives in Idaho, writes "on Science" column for the Boston Globe, and is a 2010 Fellow of the John Simon Guggenheim Memorial Foundation.

Exploring NIO

2010-09-24T12:51:00.000-07:00

The PyNIO install from binary went more or less without incident; I had to dig up a conformant numpy (1.5 is out but the binary required 1.4)

I am deferring a PyNGL install until I figure out what PyNIO can do.

So now I have py 2.5.1, numpy 1.4.1, Nio 1.4.0 and a doc.

The main doc is here

Not hard for me to dig up a netcdf file from the CAM distribution; let's take a 2 D dataset from the land surface model. No idea what is in it.


-rwxr-x---  1 tobis G-25522 7671124 Oct 22  2009 clms_64x128_USGS_c030605.nc

seems to work! The data object has some attributes set by the data file.


>>> import numpy as np
>>> import Nio
>>> data = Nio.open_file("nctest.nc")
>>> data

>>> dir(data)
['Conventions', 'Glacier_raw_data_file_name', 'History_Log', 'Host', 'Inland_water_raw_data_file_name', 'Input_navy_oro_dataset', 'Lai_raw_data_file_name', 'Logname', 'Revision_Id', 'Run_mode', 'Soil_color_raw_data_file_name', 'Soil_texture_raw_data_file_name', 'Source', 'Urban_raw_data_file_name', 'Vegetation_type_raw_data_filename', 'Version', '__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', 'close', 'create_dimension', 'create_variable', 'set_option']

but how to get at the actual contents?

Ah; the directory is incomplete. Apparently the extension module wasn't Pythonic. This can be patched.

So we can get at dv = data.variables() yielding


['PCT_CLAY', 'LATIXY', 'LANDMASK', 'LANDFRAC_PFT', 'PCT_LAKE', 'LANDFRAC', 'NUMLON', 'LONGXY', 'MONTHLY_HEIGHT_BOT', 'PCT_WETLAND', 'MONTHLY_SAI', 'PCT_URBAN', 'PCT_SAND', 'MONTHLY_HEIGHT_TOP', 'PCT_PFT', 'MONTHLY_LAI', 'SOIL_COLOR', 'PCT_GLACIER', 'PFT']

Now, just printing (__str__) data tells us


   dimensions:
      lsmlon = 128
      lsmlat = 64
...
   variables:
...
      float PCT_LAKE [ lsmlat, lsmlon ]

so dv['PCT_LAKE'] ought to be a 128 x 64 array

but dv[2000,0] works. Apparently an overflow is treated as a [-1] index and an underflow as a [0] index. Not very reassuring.

Exasperatingly, though PyNIO is available for py2.5 and numpy 1.4, PyNGL is not. Must get python 2.6... boring...

Install confusion

2010-09-21T13:11:00.000-07:00

Trying to install PyNCL on a Red Hat machine without sudo priv's.

1) Py2.7 from source. Builds fine. Runs fine.

2) numpy from source using aforesaid python. atlas/blas instructions aren't too clear, proceed blindly ahead. build works fine.

3) import numpy or import numarray fails:

ImportError: Error importing numpy: you should not try to import numpy from
its source directory; please exit the numpy source tree, and relaunch
your python intepreter from there.

very little info on Google about this. Try various versions of current working directory. Failure all the same.

4) looks like this code:


if __NUMPY_SETUP__:
    import sys as _sys
    print >> _sys.stderr, 'Running from numpy source directory.'
    del _sys
else:
    try:
        from numpy.__config__ import show as show_config
    except ImportError, e:
        msg = """Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python intepreter from there."""
        raise ImportError(msg)
    from version import version as __version__

is getting invoked. Try setup command, but it tries to install in /usr/local/lib/python2.7

5) How to tell setup where to put the numpy?

setup.py install --help

helps, yielding


Options for 'install' command:
  --prefix            installation prefix
  --exec-prefix       (Unix only) prefix for platform-specific files
  --home              (Unix only) home directory to install under
  --install-base      base installation directory (instead of --prefix or --
                      home)
  --install-platbase  base installation directory for platform-specific files
                      (instead of --exec-prefix or --home)
  --root              install everything relative to this alternate root
                      directory
  --install-purelib   installation directory for pure Python module
                      distributions
  --install-platlib   installation directory for non-pure module distributions
  --install-lib       installation directory for all module distributions
                      (overrides --install-purelib and --install-platlib)
  --install-headers   installation directory for C/C++ headers
  --install-scripts   installation directory for Python scripts
  --install-data      installation directory for data files

It must be one of those!

6) Used "--prefix". setup.py runs without complaint.

Haha! New error. "{PATH}/multiarray.so: cannot open shared object file: No such file"

but the file exists! Now what?

7) Wait - no. I used the system python not the self-installed 2.7. It works! numpy is imported.

8) Now get PyNCL? (ncl is already working on this machine)

NO! Looks like I have to do the numpy thing again, because PyNGL recommends against the g77 compiler and requires gfortran. So my numpy, though working, is wrong.

9) While I'm at it, no BLAS/ATLAS? etc?

The Russia Thing

2010-08-11T20:49:00.000-07:00

I am not sure how to take being definitely regarded as climate scientist by both Roger Pielke Jr (who is in and out of the Tobis-ooh-scary club) and Brad Johnson (a Romm protege), but not quite so definitively by anyone commonly regarded as a climate scientist. I suppose I should accept it as better than nothing. But Johnson's topic is worth following up. And I really liked that Brad got an actual meteorologist to back me up:

Like Dr. Tobis, Carver [Meteorologist Rob Carver, the Research and Development Scientist for Weather Underground] believes that manmade global warming has fundamentally altered weather patterns to produce the killer Russian heat wave. “Without contributions from anthropogenic climate change,” Carver said in an email interview with the Wonk Room, “I don’t think this event would have reached such extremes or even happened at all”:

I agree with Michael Tobis’s take at Only In It For the Gold that something systematic has changed to alter the global circulation and you’ll need a coupled atmosphere/ocean global model to understand what’s going on. My hunch is that a warming Arctic combined with sea-surface-temperature teleconnections altered the global circulation such that a blocking ridge formed over western Russia leading to the unprecedented drought/heat wave conditions. Without contributions from anthropogenic climate change, I don’t think this event would have reached such extremes or even happened at all. (You may quote me on that.)

Cool. Thanks for taking the heat, making a stronger claim than I did, Rob! And thanks for the work on this, Brad!

I want to explain my thinking in some detail on this, but I really don't have the time to do it justice right now. Of course it will be half-wittedly nitpicked and mocked to oblivion if I post it on a blog whether it's brilliant or foolish. I wonder if it rates a real paper somewhere. That would be a novelty...

But a short version occurs to me. Remember when I tried to explain that "global warming" and "anthropogenic climate change" don't mean exactly the same thing? Well, the fact that we now have situations like this one, and last year's in Australia, allows for making the point clear with some examples.

See, what we are worried about is not global warming. Global warming itself causes relatively little damage, at least at first. What we are worried about is climate change.

Anthropogenic global warming does not cause climate change.

Anthropogenic climate change causes global warming,.

Global warming (at the surface, just like global cooling in the stratosphere) is one of the more predictable symptoms of anthropogenic climate change. But if you change the forcing, you change the response. That's not a very deep scientific result, is it?

"Anthropogenic climate change" - well that's a tedious mouthful, isn't it? Not in a sense, no, not really. Not as tedious as what is happening this instant in Pakistan and in Russia, anyway. That's what abrupt climate change looks like.

2010-08-10T12:47:00.000-07:00

Handy FIR filter inherits from list:


   class filter(list):

   """real valued F I R filter with memory"""

   def __init__(self, coeffs, data=None):
            list.__init__([float(item) for item in coeffs])
       if data:
           assert len(data) == len(coeffs)
           self.data =[float(datum) for datum in data]
       else:
           self.data = len(self) * [0.]

   def __call__(self,value=0.):
       return self.filter(value)

   def output(self):
       return sum(map(mul, self.coeffs, self.data))

   def update(self,value):
       self.pop(0)
       self.append(value)

   def filter(self, value=0.):
       value = float(value)
       self.update(value)
       return self.output()

   def innovate(self):
       return self.filter(white())

I'll just use Python

2009-11-03T19:08:00.000-08:00

Captchad

2009-06-26T14:01:00.000-07:00

Pair Programming

2009-06-13T12:34:00.000-07:00

What Pair-Programing is Not

-misko hevery

Let me tell you about Arthur, the last developer which joined us for a month, and how productive he was. Arthur showed up on Monday at 9am. I sat down with him behind his computer and said, “Let’s get you up-to-speed.” I made sure that he was driving, and I was talking and we started installing all of the software which he needed to develop code. The compilers, the database the IDE, checking out the code from the repository and running it. The key here is that Arthur was doing all of the work and I was just telling him what to do. By noon, he was up and running, thanks to a very intensive hand-holding from me. We went to lunch and talked about the application, not in general terms but in specific terms which Arthur could translate to what he did few minutes ego.

After lunch, I said let’s go and fix a bug. I picked a simple bug and for the most part again i was doing most of the talking but made sure that Arthur had the keyboard and understood what, and why he was doing things. By 3 pm we had a fix. We ran all of the tests, and submitted the code. We watched the build to make sure we did not break anything.

This is pairing, and it is exhausting! By 3 pm I was beat and went home, but Arthur stayed to play some more with the code. Turns out he fixed another bug on his own before he went home. Not bad for first day. Now I could have fixed the bugs myself in 1 hour flat both of them, but than Arthur would not learn anything. I sacrificed my productivity to make Arthur productive in a single day. If I did not it would take Arthur weeks before he would figure out how to set everything up how things worked and enough courage to fix a bug. Yet that is exactly what most companies do. Think about the confidence Arthur had on day one working with us. He was up and ready and he fixed two bugs on day one. WOW!

More importantly is what we did not do. We did not get distracted by checking email. How can you, you have a single monitor, and that means that Arthur would read my email. Arthur did not waste time looking for documentation which is poor or does not exist, he was spoon fed the knowledge in the most efficient way, and he did it all by himself, i was just talking, he was typing. It is amazing how much more focused you are when you have two people working on something.

Seeking help from Statistician

2009-04-14T09:40:00.000-07:00

Suppose I have distinct samples of a cumulative density function at 0.01, 0.25, 0.5, 0.75, and 0.99 obtained from an interview. I can provide some constraint that the 0.01 point is not "too close" to the 0.25 point and similarly for the 0.99 and the 0.75 points.

I need to interpolate between the given points.

Now, of course there is not a unique interpolation, but I guess there is a maximum entropy interpolation that is in some sense closest to a Gaussian. I wonder how I would find out what it is.

Secondly, I would like to find distributions of the sum of two such distributions. I am imagining a Monte Carlo method would work, but if there is a closed form, so much the better. It needs to be reasonably computationally efficient and either available as a Python package or relatively easily programmed and tested.

Thirdly, similarly for a product of two such distributions.

If there is not a closed form, I need a way to take samples of the distribution from Python. It's not hard for me to construct a method that is crudely correct in the sense that the biases are unlikely to matter for the purposes at hand, but it seems someone ought to be able to tell me how to do better.

It would be nearly hopeless for a small traffic blog like this to get an answer by itself, but I wonder if Twitter isn't going to prove useful for scientists to answer questions of this sort. So I've tweeted a call for assistance and pointed to this posting. It will be very encouraging if I get a useful response; I have little hangups like this one all the time.

Python float input validation

2009-04-13T12:54:00.000-07:00

If you put 'nan' in as a string, a cast to float will not raise an exception.

I am not sure this is a good design, but the counterintuitive workaround is this.

To test if a Python float is not a number, see if it is equal to itself!
try:
....val = float(raw_input("Air speed of an unladen swallow: "))
except:
....print "foo"
if val != val
....print "foo also!"

Turing Test Passed; Humanity Obsolete

2009-04-02T22:25:00.000-07:00

@LizNeeley tweets:

A reverse image search engine - upload a pic, find where it came from, modified versions. Still in beta, but incredible. http://tineye.com

So I submitted this image, stolen for my main blog from Tom the Dancing Bug here.

I wanted to see if tineye could find the original, but it outdid me, and came up with this from
here: a different clip from the same strip of the same size and shape!

Google chat bug: Majitao

2009-04-02T21:49:00.001-07:00

Who is Majitao?

Buckminster Fuller Tweet

2009-03-05T07:40:00.001-08:00

When I am working on a problem, I never think about beauty but when I have finished, if the solution is not beautiful, I know it is wrong.

Installing MySQL on OS X

2008-12-12T17:08:00.000-08:00

It's actually not required to get started on Django, but I thought it would be a good idea to jump through the MySQL hoops early.

It was a bit of an adventure:

The MySql install instructions are broken for OS X. After you have
downloaded the MySql package and installed it with the usual dance,
you will probably encounter some instructions involving
"groupadd".

Various mortals seem to have gotten confused by this and
run into various smug bastards being unhelpful. Ignore the smug bastards.

===

Instead, follow the instructions at
http://www.macosxhints.com/article.php?story=20080128103022907

However, the lines

===

$ cd /Library/LaunchDaemons
$ sudo chown root com.mysql.mysqld
$ sudo chgrp wheel com.mysql.mysqld
$ sudo chmod 644 com.mysql.mysqld

===

should read:

===

$ cd /Library/LaunchDaemons
$ sudo chown root com.mysql.mysqld.plist
$ sudo chgrp wheel com.mysql.mysqld.plist
$ sudo chmod 644 com.mysql.mysqld.plist

===

Otherwise the instructions are OK.

Now it tells me to check with Console.app ... What is that?

Wahoo! There it is in Utilities. I run it and it claims mysqld is running and
ready for connections.

===

So now to find mysql; open a Terminal window.

OK, now mysql isn't in my path yet, but I can invoke a mysql session
with

/usr/local/mysql/bin/mysql

thus:

/Guido:~ tobis$ /usr/local/mysql/bin/mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.30 MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

MySql is running! Type quit . This is enough of a victory for one day.

===

In future if you want to just type mysql instead of
/usr/local/mysql/bin/mysql, you need to update your path.

Change to your home directory. Issue

touch .profile

Then edit .profile, appending the line

export PATH="$PATH:/usr/local/mysql/bin"

Note this is a hidden file. See

http://www.peachpit.com/articles/article.aspx?p=31442&seqNum=4

http://www.macworld.com/article/51830/2006/07/showallfinder.html

===

In future if you don't want to burden your machine with mysql anymore,
issue

launchctl unload com.mysql.mysqld.plist

Next step Django itself. I am expecting it to be a bit easier.

Proper tarballs from aquamacs projects

2008-10-16T20:38:00.000-07:00

Getting a bunch of dot underscore cruft in your tarballs?

export COPY_EXTENDED_ATTRIBUTES_DISABLE=true

see comment by David Olinsky here

Python First

2008-10-16T12:52:00.000-07:00

"But there is emerging consensus in the scripting community that Python is the right choice for freshman programming. Ruby would also be a defensible choice. Python and Ruby have the enviable properties that almost no one dislikes them, and almost everyone respects them. Both languages support a wide variety of programming styles and paradigms and satisfy practitioners and theoreticians equally. Both languages are carefully enough designed that 'correct' programming practices can be demonstrated and high standards of code quality can be enforced. The fact that Google stands by Python is an added motivation for undergraduate majors." -Ronald Loui

How not to build a bad website

2008-08-16T15:47:00.001-07:00

from xkcd 345

2008-07-28T20:22:00.000-07:00

Reproducibility blog

2008-07-24T09:13:00.000-07:00

John Cook has started a blog on reproducibility in science. H/T Greg Wilson.

Salient comment: "Your result is not reproducible if you can reproduce it. Your result is reproducible if someone else can reproduce it."

I hope this was a joke

2008-07-09T17:10:00.001-07:00

but I think it was serious.

J Wants a Mac (Vista flame)

2008-06-30T15:21:00.000-07:00

MUD transcript:

J wants another Mac.

J bought a cheap laptop to last while he's living in corporate storage, it's VISTA! Holy crap, I have the absolute lowest expectations possible about Windows, well desrved from experience, and somehow Vista blows away every shred of quality or clear thinking. It's truly a self-caricature. Every time you run a program it asks you if you want to run it. Later it'll ask about getting more permissions to contniue to run! Programs will die, and then another program runs to "try to figure out why" the first crashed. It's truly the most pathetic computing experience I've ever had. OH! And the network on the Vista box already hung once. I was appalled. I was restoring a backup from a PII-233MHz to this brand-new box and the best it could do was about 140KB/sec and then it hung.... My G4 Mac had no problem doing 4-6MB/sec to/from the Pentium2.

J says, "And here's another absurdity: Vista is 32bit so it can't see all the memory I bought. M$ spent millions crippling a 64 bit OS."

J says, "However, the laptop is great: 17", dual 2Gz Opteron, 4 USB, 1 FW, crappy graphics card, but here was the deal cincher: Two internal drive bays! This box will become the new R***.com server. I'm sure Linux will run 1000x better on this box than Vista."

mt says, "Yes but Microsoft got the license fee anyway"

client side matplotlib

2008-06-20T23:19:00.000-07:00

way cool

App Engine OS X Confusion

2008-06-07T13:15:00.000-07:00

Google AppEngine docs is inconsistent at this point.

It offers a download of AppEngineLauncher for OS X rather than AppEngine. Installing that does NOT, per docs, put appcfg.py or dev_appserver.py in the path.

It does create instances of those.

Directly invoking dev_appserver.py works; modulo a warning in importing PIL; that seems OK.

I see two instances of appcfg.py: in Applications/GoogleAppEngineLauncher.app//Contents/Resources :

google_appengine/appcfg.py
and
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/appcfg.py

Neither works. They both say ImportError: No module named google. I have deployed test code effectively. I don't actually know what appcfg is supposed to do.

Controversy about Software in Higher Ed

2008-06-04T11:10:00.000-07:00

here and a reply here

The Secret Ingredient

2008-05-13T10:39:00.001-07:00

http://radian.org/notebook/wp-content/uploads/2008/05/guido2.jpg

HPC Considered Harmful

2008-05-12T10:47:00.000-07:00

I'm really looking forward to Greg Wilson's talk called "HPC Considered Harmful" at Scientific Software Days at TACC, for which I am a co-organizer. Also Eric Jones will be talking about "New Directions in Scientific Workflow" which sounds great too.

I'm not sure what these guys will say, but I think this (Carriero et al 2004) is relevant:

The goal of [our] approach is not to reap the maximum efficiency benefits in making changes to a program. Rather the goal is to remove compute time as a rate-limiting step.

Anyway, come on down if you're in the neighborhood!

Life in the Trenches, Generalized

2008-05-05T15:25:00.000-07:00

A really good description of the problem is here.

It's only exacerbated when both the producers and the consumers of the software are scientists who don't think software principles are interesting and/or whose mentors don't believe in time for training on such principles. Invariably we are targetting weird platforms, too.

I recently heard someone say that about 30% of professional programmers' time is spent on build issues. If scientists are as much as half as good as full-timers, that means they spend 60% of their time on build issues, but they also have to spend 50% of their time on reading and writing discipline-related stuff. That means we get negative 10% of our time for actual productive coding. If we're lucky.

Sudden abstractions

2008-04-21T22:26:00.000-07:00

verbatim from "Young Archimedes" by Aldous Huxley, 1928

(an English family spends a year in Italy and their young child befriends an older peasant boy)

"Though fully two and a half years older than little Robin - and at that age thirty months are crammed with half a lifetime's experience - Guido took no undue advantage of his superior intelligence and strength. I have never seen a child more patient, tolerant and non-tyrannical. He never laughed at Robin for his clumsy efforts to imitate his own prodigious feats; he did not tease or bully, but helped his small companion when he was in difficulties and explained when he could not understand. In return, Robin adored him, regarded him as a model and perfect Big Boy, and slavishly imitated him in every way he could.

"These attempts of Robin's to imitate his companion were often exceedingly ludicrous. For by an obscure psychological law, words and actions in themselves quite serious become comic as soon as they are copied; and the more accurately, if the imitation is a deliberate parody, the funnier - for an overloaded imitation of someone we know does not make us laugh so much as one that is almost indistinguishably like the original. The bad imitation is only ludicrous when it is a piece of sincere and earnest flattery that does not quite come off. Robin's imitations were mostly of this kind. His heroic and unsuccessful attempts to perform the feats of strength and skill, which Guido could do with ease, were exquisitely comic. And his careful, long-drawn imitations of Guido's habits and mannerisms were no less amusing. Most ludicrous of all, because most earnestly undertaken and most incongruous in the imitator, were Robin's impersonations of Guido in a pensive mood. Guido was a thoughtful child given to brooding and sudden abstractions. One would find him sitting in a corner by himself, chin in hand, elbow on knee, plunged, to all appearances in the profoundest meditation."

...

(Guido is discovered to have a gift for mathematics.)

"This child, I thought, has had the fortune to be born at a time when he will be able to make good use of his capacities. He will find the most elaborate analytical methods lying readily to his hand; he will have a prodigious experience behind him. Suppose he had been born while Stonehenge was building; he might have spent a lifetime discovering the rudiments, guessing darkly where he might have had a chance of proving. Born at the time of the Norman Conquest, he would have had to wrestle with all the preliminary difficulties created by an inadequate symbolism; it would have taken him long years, for example, to learn the art of dividing MMMCCCCLXXXVIII by MCMXIX. In five years, nowadays, he will learn what it took generations of Men to discover."

(Alas the story ends sadly for the fictitious young Guido, while happily for all of us in real life, the analogy breaks down. Clearly Guido should have contrived to be born not just in the modern age but also in Holland...)

Now let me think a minute, son

2008-04-21T10:53:00.000-07:00

Ah, yes, that can be easily done...

More Python and Climate

2008-03-30T19:27:00.000-07:00

Apparently my rants got linked by GravityLoss, who understands that I am saying something about Python and about modern methodologies, but apparently isn't sure exactly what.

Here's my reply:

Hi and thanks for the links.

Make no mistake, I am a Python fanboy. I really doubt it's possible to do much better than Python at our present level of understanding of how to control computers.

As Paul Graham explains, any assertion that a more advanced technology Y is better than technology X tends to fall on deaf ears on users of technology X, because what they think is doable is defined by technology X. It is perfectly possible to get no advantage from Y because it is always possible to do X-like things with it.

I am proposing to do somewhat different things with Y. I am not the greatest expert on Y, but I do see things to do relevant things with Y that are not entirely X-like.

I would like to have software managers more experienced than myself involved. Google are the folk with the disposable wealth and the farsighted motivation as well as some of the relevant skills. I sort of wish they'd do what I want to do rather than leaving me scrambling to do it. I retain enough hubris to suggest that if they tried they'd do well to get me on the team. Realistically I probably will not have their help nor the help of an experienced software manager, though any pitching in from JM or JM would be most welcome.

There's nothing magic about Google or Python that aren't summed up in Clarke's Law: any sufficiently advanced technology is indistinguishable from magic. The sad thing is that much of the academy, including climate science, is far enough behind the cutting edge of what is possible that comparable productivity would look magical.

The elegance of Fortran

2008-03-21T13:37:00.001-07:00

You know, the point of OOP is to SAVE work.

Yes, the following is the most trivial and trite and pointless possible example of OOP. I am not showing you code that is interesting. I am just comparing it with bolted-on OOP. Have a look at the Fortran equivalent. It's completely equivalent, note.

The "polymorphism" section form the Fortran is expressed in Python as an arbitrarily small amount of whitespace. And you thought whitespace was a drawback of Python!

def vectorsum(vec1,vec2):
...return map(sum,zip(vec1,vec2))

class Circle(object):
...def __init__(self,center,radius):
......self.center = center
......self.radius = radius

...def __str__(self):
......return "circle of radius %s at %s" % (self.radius, self.center)

...def move(self,translation):
......assert len(translation) == len(self.center)
......self.center = vectorsum(self.center, translation)

class Square(Circle):
...""" note: can inherit constructor for circle; treat radius as length"""
...def __str__(self):
......s = self.radius
......shifts = ((0,0),(0,s),(s,0),(s,s))
......corners = tuple([vectorsum(self.center,shift) for shift in shifts])
......return "square at points %s %s %s %s" % corners

PyCon snippets

2008-03-15T07:55:00.000-07:00

Stuff to look into:

iPy1: a spawned interpreter on every CPU

MPI4py: the winner for fine-grained parallelism

distributed NumPy array: Brian Granger

"it's like a beginner trying to make a souffle; you WILL get an egg dish"

stats.bayes.mvs

new buffer type in Py 3, Travis O in charge, imports NumPy arrays; PEP 3118

struct module

Amazon's elastic cloud; Pete Skomorock's ElastiWulf

Trestle

noonhat.com ; saturdayhouse.prg

pyxpcomext.com

freebase

argparse

orbited

math instruction in APL

if you want to start a conversation, ask a question

Insufficient, Inaccurate and Inconsistent

2008-02-15T09:01:00.000-08:00

The data I mean...

The job of the geophysicist, I was told yesterday at a seminar given by Mrinal Sen, is essentially impossible. It is, given data that is insufficient, inaccurate and inconsistent to come up with a useful model.

Interestingly, it is unlikely that the skeptics' squad will make much of this admission, since the utility of the models Sen is specifically interested in is whether they correctly inform placement of oil well drilling efforts.

Nevertheless wells do get drilled. In the past it was a matter of some combination of analysis and intuition. Nowadays, statistics works in there as well.

Does climatology partake of these fundamental problems? Not as much as seismology, really; our observations are relatively accurate and consistent compared to seismic data. They are far from sufficient for long time scale processes, and the formalism of use of paleodata still leaves much to be desired.

Nevertheless, the future will happen, and we will do something about it. The question at hand for climate modelers and their customers is to what extent the models ought to affect what we do.

Computational geosciences and computational biological sciences are very different from computational engineering in flavor. In engineering (aside from civil and architectural, which partakes slightly of our problems) the system is entirely controlled and hence the tradeoffs between model expense and model accuracy are well understood. By contrast, our problem is to establish the right model based on observational data and theory.

This is called "inverse modeling" in some circles. I dislike the name and regret the loss of the appropriate term "cybernetics" to Hollywood and to some rather ill-defined corners of computer science. I propose, therefore, to call what some computational sciences do "meta-modeling", wherein the model itself (and the nature of its relationship to observation and theory) is construed as a legitimate object of study.

It is interesting how well-established this modality of thought is in mineral exploration (where there is, after all, a bottom line to take care of) and how controversial it remains in climate science. I have some thoughts as to why this might be. Some of the problems are directly a consequence of the unfairness of the unfair criticisms to which we have been subjected; this makes a fair assessment of fair crticisms fraught with peril. Others emerge from the history of the field and the social structures in which the work is performed.

It seems obvious to me that if the computational resources applied to climate go up several orders of magnitude, the correct approach to this is not, or at the very least not solely, to build a small number of immensely complex modeling platforms which use all the available computational power. Rather, it is to apply the (expensive but powerful) inverse modeling methodology to explore the model space of simpler models; to improve the expressiveness of model description languages; and to use these tools to build a much larger and more formal ensemble of climate modeling tools.

I also have a question whether the most policy-relevant models and the most science-relevant models are necessarily the same. I suspect otherwise. Climate modelers are so defended against the accusation of heuristics that they fail to apply the heuristics that might apply in an applied science effort.

Sen presented an interesting taxonomy of inverse methods at his talk. Much more to follow on this subject, hopefully.

Computational Silence

2008-02-12T09:40:00.001-08:00

Still not sure whether to take Mr. Hughes seriously, even though he echoes some of my points. His latest looks pretty confused at first reading but it's far from my expertise. Maybe Eli will have a look.

That all said, I am not one who says all is right with computational climatology.

My friend JL sends along an article by (not captain) James Quirk that does a good job outlining the quandary and some of the efforts toward a solution. Climate science is particularly backward in adopting these measures. There's a misperception that all GCMs solve the same problem that is understandable in the public but hard to account for within the field.

The article is called "Computational Science: Same Old Silence, Same Old Mistakes". It appears in a volume on AMR. It is behind a Springer firewall, so if you can't get the PDF note that Google will display a scan of it.

(Remember, rationally, the less we trust the models the more severe our policy constraints on modifying the system should be.)

Why Pencil Science?

2008-02-12T09:19:00.001-08:00

This blog has, at least for now, wandered from its original intent, but its original explanation of purpose is worth saving. I do think computer science is for everybody, not just middle school kids but even climatologists...

This is a blog about the nuts and bolts of computer literacy from a
Pythonic perspective.

I would like not merely to defend the position that computer
programming is "for everybody" (including children, intelligent
adults, and scientists) but also to examine the proposition that
Python is the appropriate vehicle for such a de-professionalization of
programming.

Of course, "computer science is not about computers" but about
formally expressing processes using symbols. The misnomer is
consequential in how most people think about the question of the role
of programming in education.

We don't teach children how to write because we expect them all to be
professional writers. Professional writers are not opposed to literacy
on the grounds that they will lose readership. We do not call the
ability to write "pencil science".

Is writing (pencil science) for everybody? That wasn't clear a
thousand years ago. Can "computer science" be for everybody too?

I'll be reorganizing my online presence soon. Still thinking about what to do...

Bad month index=36; help, you prawns!

2008-02-07T17:00:00.000-08:00

Recall we are trying to get a sensitivity to 2xCO2; the public thinks of this as the primary use case.

Most of what was hanging me up for the past week was building a 6Mb one-time data file (containing climatological mixed layer depth, clamped at 200m; the magic number is somewhat arbitrary and hard-wired, but we suppose others have put thought into it.) It would have been nice if I could just download the file. It turned out I needed a c compiler compatible with the Fortran compiler that built netcdf. The magic word was "icc", specifically

setenv CC=icc

Once that was done, it turned out I had some other data file to dig up. I had trouble tracking down my ncar UID/password (as if this data needed to be kept under wraps for some reason). Then I found that the URL specified in the README was incorrect! Eventually tracked the file down with Google. Built (huzzah) the heat flux boundary condition. Thought I was done.

Today I tried to submit the job. Well, still a use case I;ve never run. I was hung up for some time on the error messsage "BNDDYI: Bad month index=36" which is not mentioned in CAM documentation but is mentioned in the CCM3.6 documentation! (an ancestor)

It seems the month number should be a number between 1 and 12 (reasonable). No idea where the number 36 came from. I look into my data file and see the following for dates:

time = 116.5, 215, 316.5, 416, 516.5, 616, 716.5, 816.5, 916, 1016.5, 1116, 1216.5 ;

There are twelve of them as needed, but...

Those are indeed months in a peculiar sense. Consider it a puzzle, and here is your clue:

double time(time) ;
time:units = "day as %m%d.%f" ;
time:axis = "T" ;

Go figure. No clue where the 36 comes from. Charles advises, though, that I should not specify a start and end date for a slab run. So I make no effort to surgically fix the month numbers and remove the start and end date, specifying a duration instead. (Because I specify in days rather than seconds, I use a negative number...)

The 36 BNDDYI message goes away! Apparently the error "month number is 36" really means "you can't specify dates on a slab run".

The model gets considerably further now. It starts to run the first time step and then proffers this droll message a few times

sublimate away all sea ice
something is probably seriously wrong
ice state at dh stop nstep = 0

amid some other unexpected output, and then aborts.

Three weeks and counting. We wanted a result by next Monday and this will take 48 hours to run. Groan.

Somebody else has come to this pass, describing it as "display the seriously error". Note the useful assistance provided by the community. There's also something in Chinese that Google translates as follows:

help you prawns, the slab ocean model CAM operations serious mistake!

-=-=-=-=-=> Turn CCM and CAM counterparts all know, by adding a simple ocean model (slab ocean model) running CAM, but regardless CAM3.0, or CAM3.1, there are the same mistakes can not continue to run, the Daxia Please help!

Any prawns out there have any advice for me?

Update: Running at last! The last hurdle was knowing that you have to replace the file

cami_0000-09-01_64x128_L26_c030918.nc

with the file

cami_0000-01-01_64x128_T42_L26_SOM_c030918.nc

Unlike the mixed layer depths, this is actually distributed in the initial conditions dataset. I am also using a topography file that differes from the standard one. Not sure why, this is based on lore and not knowledge. It's called

topo-from-cami_0000-01-01_64x128_T42_L26_SOM_c030918.nc

I don't know its provenance, but possibly the name is correct. If that's the case, though, it isn't clear why there are two different files at all. Some totally undocumented use case perhaps?

This is actually consistent with the message from "seriously message" guy. It is not documented as part of the procedure for the use case in the documents. There is a tiny clue in the redme for defineqflux:

Another executable available (but hopefully unnecessary) in the definesomic
module is (also) called definesomic. It adds necessary SOM fields to an
initial dataset if they are not already there.

If there are fields missing from the initial conditions, you'd think something better than "something is seriously wrong" and lots of numerical output would be possible.

Not sure how much further to investigate.

My Deep Naivetee

2008-01-29T09:10:00.000-08:00

I've made a couple of comments on Bryan's blog early this morning. One is a continuation of my usual ranting, and another is an expression of confusion about what web services have to do with anything scientific coders care about.

Meanwhile I am amused to see my position described as "deeply naive" in a blog by Dan Hughes that I haven't spotted before. It seems worth following for the in crowd.

I don't, actually, think a ground-up rewrite would be easy, and I have some ideas for a less ambitious approach to prove the pudding for a higher level abstraction approach. We'll see what NSF says about it.

However, I think the tenacious attachment to the code which Dan himself goes so far as to describe as "very olde, very poorly constructed, undocumented, and yet very actively used" (much further than I would go) is damned peculiar. It seems to imply that what we are doing doesn't matter. Well, if it doesn't matter we should quit, and if it does matter we should bite the bullet and write something clean and maintainable and (dare I say it) even literate.

In the end this matter is too important for anything less than maximal transparency. It completely baffles me that this goal is so thoroughly outside the field of vision of practitioners. I think it's fundamental.

Update: Whoops. I mistook what Dan Hughes was about. He isn't attached to the code. He doesn't work with the codes, he just snipes at them from a distance. He's an "auditor". Likely, he doesn't want to believe that climate modeling is feasible, or at least is playing to an audience that thinks like that. One can hope he does not reach the illogical conclusion that climate protection is not a useful goal of policy pending the substantial improvement of such models.

Let me make it very clear from observing real productive climate scientists that extant models are useful tools for science, and that the scientists are well aware of their imperfections and flaws.

This doesn't mean they/we should be immune from criticism, nor that we have a clear sense of how useful the extant models are for projection. I have a lot doubts on that score, but I take the rational risk-weighted response of being more worried about our collective future, rather than less so, as a consequence.

Further Update: That said, the discussions on Hughes blog are not without value. I found this one especially interesting.

Further Update: While it might appear that there is no intent by Hughes to provide a constructive alternative (see comment #6 to this thread) he does in fact offer us up a strawman proposal for an alternative approach here (linked from #7 in the same thread). His server is slow and unreliable, but it's there on occasion. Reading...

Life in the trenches III

2008-01-28T10:52:00.000-08:00

One of my ensembles has died. The run dropped an error file as follows (excerpt):

print_memusage iam 47 stepon after dynpkg. -1 in the next line means unavailable
print_memusage: size, rss, share, text, datastack= 52301 22403 4114 806 0
print_memusage iam 13 stepon after dynpkg. -1 in the next line means unavailable
print_memusage: size, rss, share, text, datastack= 52441 22616 4121 806 0
print_memusage iam 36 stepon after dynpkg. -1 in the next line means unavailable
print_memusage: size, rss, share, text, datastack= 52135 24788 5987 806 0
[0] [MPI Abort by user] Aborting Program!
done.
/opt/lsf/bin/ibrun: line 6: 545 Killed pam -g 1 mvapich_wrapper $*
[0] [MPI Abort by user] Aborting Program!
done.
/opt/lsf/bin/ibrun: line 6: 13870 Killed pam -g 1 mvapich_wrapper $*
[0] [MPI Abort by user] Aborting Program!
done.
/opt/lsf/bin/ibrun: line 6: 3813 Killed pam -g 1 mvapich_wrapper $*
[0] [MPI Abort by user] Aborting Program!
done.

The error file is 1132 lines long. Python would give me more legible information.

I am simply going to hope it has something to do with Lonestar's I/O problems. I need to rerun the exact case, of course.

The ensemble controller somehow knew something was awry. It wasn't written by a very seasoned programmer and I'm eager to rewrite it but the time never shows up. Fortunately this time it appears to have done the right thing.

Meanwhile

Charles has convinced me that he really has something going with his current proposal so I am trying to help firm up the text (for practical purposes, on my own time)
I still have yet to get sensible output from defineqflux, or understand how to build it for that matter (see Trenches II)
I need to back up my expired files from Chicago
I need to document and version control my December work
I need to finish building the NCAR diagnostic toolkit now that NCL works
round up speakers for scientific software day
somehow we need a local MPI install; yow!
eventually we need a CCSM build on a TACC platform, but don't hold your breath

This is on halftime! I'm not even supposed to be in on Mondays!

Will I ever write a line of my own code again?

Update: The error file says nothing whatsoever. Successful runs have the same errors, up to and not including the Abort messages, where instead they contain 64 lines consisting of "0".

Why Python?

2008-01-25T12:36:00.000-08:00

Other people have other reasons, but mine is succinctly summarized by Paul Prescod in his article On the Relationship between Python and Lisp. Here's the core of the argument:

Paul Graham says that a language designed for "the masses" is actually being designed for "dufuses". My observation is that some of the smartest people on the planet are dufuses when it comes to programming, (or in some cases just dynamic programming languages) and I am pleased to share a language (and code!) with them.

I usually spend a big chunk of my day in Python. But most professional programmers will not be able to do that until Python is a dominant language. In the meantime they must switch back and forth between Python and the language of their day-job. Python is designed to make that easy. During the day they can pound out accounting code and at night switch to hacking distributed object oriented file systems. Every intuititively named function name makes it that much easier to "swap" Python back into your brain. After we have been away from a language for a while, we are all dufuses, and every choice made in favor of the duffers actually benefits even high-end programmers.

I get paid to share my code with "dufuses" known as domain experts. Using Python, I typically do not have to wrap my code up as a COM object for their use in VB. I do not have to code in a crappy language designed only for non-experts. They do not have to learn a hard language designed only for experts. We can talk the same language. They can read and sometimes maintain my code if they need to.

From a purely altruistic point of view, it feels good to me to know that I am writing code (especially open source code) that can become a lesson for a high school student or other programming beginner. I like to give programming classes to the marketing departments at the companies where I work.

Even hard-core geeks get a certain aesthetic pleasure out of using something that feels minimal and simple because most of the complexity has been hidden away or removed.

Code is a communication medium. The machine is not the only reader. Python is the only language explicitly designed with both beginners and experts in mind. This has direct benefits for the transition from beginner to expert, and it also has direct benefits for development collaboration among groups with distinct expertise.

Here is an example. I have never taken a compiler course and I still consider code compilation to be deep magic, though not as much as I did before Python began boosting my sophistication. Nevertheless, I can understand and appreciate the following.

# romanNumerals.py
#
# Copyright (c) 2006, Paul McGuire
#

from pyparsing import *

def romanNumeralLiteral(numeralString, value):
return Literal(numeralString).setParseAction(replaceWith(value))

one         = romanNumeralLiteral("I",1)
four        = romanNumeralLiteral("IV",4)
five        = romanNumeralLiteral("V",5)
nine        = romanNumeralLiteral("IX",9)
ten         = romanNumeralLiteral("X",10)
forty       = romanNumeralLiteral("XL",40)
fifty       = romanNumeralLiteral("L",50)
ninety      = romanNumeralLiteral("XC",90)
onehundred  = romanNumeralLiteral("C",100)
fourhundred = romanNumeralLiteral("CD",400)
fivehundred = romanNumeralLiteral("D",500)
ninehundred = romanNumeralLiteral("CM",900)
onethousand = romanNumeralLiteral("M",1000)

numeral = ( onethousand | ninehundred | fivehundred | fourhundred |
       onehundred | ninety | fifty | forty | ten | nine | five |
       four | one ).leaveWhitespace()

romanNumeral = OneOrMore(numeral).setParseAction( lambda s,l,t : sum(t) )

print romanNumeral.parseString("XLII") # prints "42"

Try doing that in a dozen or so lines of C++ or Java, mate, so that a random reader could get half a clue as to what was happening! Yes of course the "import" statement hides a great deal of magic, but that's the whole point, see?

Life in the Trenches: Part II

2008-01-23T11:34:00.000-08:00

This story is probably more illustrative (than is Part I) of some of our productivity sinks:

We have a modified set of CAM parameters that demonstrably improves climate fidelity, and are looking to establish its CO2 doubling sensitivity. This is an atmosphere-only model with six of its magic numbers tweaked by our (UTIG's, i.e., Sen, Stoffa and Jackson's) smart ensemble search algorithm. Our suspicion is that we will see a sensitivity closer to 3 C (increasingly regarded as the most likely value) rather than NCAR's 2.4 C.

Now, CO2 sensitivity is a very common use case, perhaps the most famous of all. However, rather than a data ocean it requires an interactive ocean in order to make any sense.

This use case is so common it merits a section in the User Guide.

You will note that the description of what is going on is strikingly terse. If you haven't hung around climate modelers you will not be able to parse it at all, I'll wager.

So note if you can or take my word if you must that step 1 seems more or less model independent, though admittedly it is resolution dependent. However, since CAM only supports a few resoilutions, it's unclear why I should have to jump through the hoops.

Nevertheless, deep within the CAM file hierarchy you do find the directory in question with a perfectly fine Makefile to build definemldlxl, definemldbdy and defineqflux, which in my case naturally doesn't work.

You sort of have to know to tell Lonestar module load netcdf and then env to get the netcdf paths from the environment, which you can then use to create the environment variables the NCAR buld file expects. None of this is documented anywhere, and this is enough to get through the compile phase but still no joy at link time.

/opt/apps/netcdf/netcdf-3.6.1/lib/libnetcdf.a(attr.o): In function `dup_NC_attrarrayV':
/home/build/rpms/BUILD/netcdf-3.6.1/src/libsrc/attr.c:199: undefined reference to `_intel_fast_memset'

Scratching my head on this one. Maybe someone at TACC can help; will file a ticket and see what happens.

Meanwhile, we have tracked down an executable for the last one in a directory of a worker who has left and his data file for the first phase; we are hoping that is good enough. Note, though, that the docs do not say what fields need to be in the output of the file you pass to defineqflux. It turns out that the ones we normally save aren't the right ones.

No problem, you say. This is a runtime variable ('namelist' variable in fortranese) and I won't have to rebuild the whole model for that, will I? I'll just add the missing fields (rummage through the source code to find them... rummage, rummage...) and Bob will be my uncle.

Well actually this would have cost me two days, except that Lonestar lost two days to a panic and I was off Monday, so it cost me a whole week. You see, if you do a restart run, (i.e., a continuation of a prior experiment) the fortran will happily ingest your list of output variables and ignore them (without any message to that effect) in favor of the list you used in the spinup run. No, to have your namelist read and actually used, you have to to a "branch" run rather than a "restart" run, which requires significantly different set of namelist parameters that is essentially ill-documented. In fact if you try to do this based on the CAM manual, you will not find sufficient information. The clew, as Sherlock Holmes would say, is here

This appendix describes a small subset of the namelist variables recognized by CLM 3.0. For more information, please see the CLM User's Guide.

Aha; well that does help. After a while you understand that the namelist contains two (!) definitions of NREVSN (the restart file name) in "branch" mode (the namelist apparently can have multiple namespaces, though Fortran jargin calls them some other thing most likely), whereas in "restart" mode it suffices to use the default values for REST_PFILE which is a small text file containing what in the other case is the value of NREVSN.

So by close of business today we should be able to get the flux file we need to actually run the SOM version of CAM, the building of which was another story but it's in the past so I won't bother you with it.

Update: Still haven't pursued the build question.

After rerunning the spinup, the run of defineqflux fails, but later in the process, with the message:

nc_put_vara_double -1 failure: Variable not found.

So another 12 hours spinup run needed, once I figure out what it needs. This is a good example of a bad error message. In a real language it would not be too much trouble to report which variable wasn't found and where. In a reasonable document the requisiste fields would be listed. If I were more competent I'd have done a better job perusing the source. 0 for 3.

Update 1/28: Used default outputs and set it running on Friday. Trioed to run defineqflux today with the same error. I don't think it's a matter of which fields I use. Perhaps the version of defineqflux which I have is defective. Still don't really know how to build it so though it is only modestly complicated C code, I can't debug it.

...
Using SST mid-months 8 and 10 to compute QFLUX mid-month 9
dayfact=62
Using SST mid-months 9 and 11 to compute QFLUX mid-month 10
dayfact=60
Using SST mid-months 10 and 0 to compute QFLUX mid-month 11
dayfact=62
nc_put_vara_double -1 failure: Variable not found
Abort

Update 1/29: I have managed to build against a NetCDF library that I myself built from scratch. Still trying to resolve the error, but at least I can put some tracers in the code now.

Meanwhile, a ticket is pending at TACC for how to build against their library.

Update 1/31: Still no joy. Looks very much as if I need the other pieces of the tool chain. But they require the fortran as well as the C libraries. Linking against TACC's library fails as I reported some time ago. My own build of Fortran NetCDF yields

vcc works; but missing link magic for f77 I/O. - NO fort. None of gcc, cc or vcc generate required names. - f2c : Use #define f2cFortran, or cc -Df2cFortran - NAG f90: Use #define NAGf90Fortran, or cc -DNAGf90Fortran - Absoft UNIX F77: Use #define AbsoftUNIXFortran or cc -DAbsoftUNIXFortran - Absoft Pro Fortran: Use #define AbsoftProFortran - Portland Group Fortran: Use #define pgiFortran - PathScale Fortran: Use #define PATHSCALE_COMPILER"
make[3]: *** [fort-attio.lo] Error 1
make[3]: Leaving directory `/home/utexas/ig/tobis/CAMYA/cam1/models/atm/cam/tools/defineqflux/NCDF/netcdf-3.6.2/fortran'

so here I sit helpless. Two weeks since starting to try to run a standard use case and I am set back to trying to build system-supported libraries. TACC has been episodically helpful but is presently silent.

I am going to try from the beginning on a UTIG machine next.

The Cybernetic Perspective

2008-01-23T09:31:00.000-08:00

In the original sense, "cybernetics" means optimal use of information in decisionmaking.

It emerges that to develop a statistics of information requires a change of perspective from the simple one where developing the model is informed by the system under study to one where the model and the system under study are two components of a larger system. The Wiener filter and its descendant the Kalman filter are the best known examples of this approach but it is more general.

I believe this use of the word "model" is not entirely coincident with the software engineering sense as in "model-driven development" though it does mesh more or less well with "Model-View-Controller". In our world the View is relatively trivial and usually done offline as postprocessing; it's the utter absence of the Controller that troubles me.

(Update: That last is really a stretch on second thought. Probably more sellable is the idea that the M in MVC corresponds to system state (variables) and the C corresponds to executable code (statements). In the end the word "model" is just as much a source of confusion as ever.)

Anyway I think the cybernetic perspective has value in getting climate modeling unstuck.

Quote of the Day

2008-01-22T20:00:00.001-08:00

Chris McAvoy on the ChiPy list:

I don't know for sure, but from a "Zen of Python" perspective, when I
can't do something with Python it usually ends up that what I'm trying
to do shouldn't be done.

Life in the Climate Trenches: Part 1

2008-01-22T09:22:00.000-08:00

I've got not one but two build problems going on. If the rest of my life weren't going well I'd be awfully depressed. I really hate wasting time on this sort of mundane frustration.

Lonestar (the supercomputer) is offline for alternate Tuesday maintenance, so for today I will share build Probleme Numero Un with you. I welcome any skill transfer that can help untangle all this; comments welcome. Rather than polluting the blogspace with innumerable followups I will just provide updates to this entry.

Tomorrow I'll start documenting Probleme Numero Deux: running a standard CAM use case.

It may be that we'll more be documenting my own ignorance than any fundamental problem with the software from some point of view; that will be a learning experience just the same.

You may object that this isn't the sort of thing you want your tax dollars paying PhD's to fumble around with; in that case I would be in agreement with you. In the private sector this would clearly be a systems task and not a scientist task. So I appeal to your mercy as a lousy admin and a potentially productive scientist, albeit of a somewhat eccentirc stripe. The sooner I get these things done the sooner I can be a scientist.

OK, here's the scoop (unix/posix skills required to make any sense out of this):

To participate in NCAR CAM development we need to use their diagnostic package which in turn depends on NCL. Having a legitimate connection to a US research institution, I have long since jumped through the hoops and am registered on the Earth System Grid. So I download the package and all its dependencies.

OK, so the UTIG disk systems are all cross-mounted and so, though I have root on the climate research machine I don't have write privileges on /usr . Our excellent systems folk have set it up so that climate users just define $CLIMATE in their .cshrc which will invoke anything I put (or conceivably somebody else puts) into /usr/local/climate so all I have to do that is nonstandard is insert "climate/" into the path; I make my own ../bin ../lib ../man in there; everyone interested in climate ends up with it in their path, nobody else has their namespace messed up, a very nice approach.

Alas, the lengthy but very clear build instructions do not work (this is standard practice with NCAR products alas; apparently they lack the skills and/or funding to actually make their products portable, though the build instructions are usually very hopeful.) In the present case the snag occurs even before I get to NCAR's stuff.

NetCDF is already available. I have acquired and build jpeglib and zlib as follows:

cd jpeg-6b
make clean
./configure --prefix=/usr/local/climate
make
make install

which puts five executables into /usr/local/climate/bin and

cd zlib-1.2.3
make clean
./configure --prefix=/usr/local/climate
make
make install

which puts a library into /usr/local/climate/lib

My current snag is building HDF-4, which in the configure phase always stops at:

checking zlib.h usability... yes
checking zlib.h presence... yes
checking for zlib.h... yes
checking for compress2 in -lz... yes
checking jpeglib.h usability... no
checking jpeglib.h presence... no
checking for jpeglib.h... no
checking for jpeg_start_decompress in -ljpeg... no
configure: error: couldn't find jpeg library

I have tried various combinations here. What seems to me ought to work is

./configure --prefix=/usr/local/climate --with-zlib=/usr/local/climate \
--with-jpeg=/usr/local/climate --disable-netcdf

but it seems to ignore the prefix altogether. I change --with-zlib to point to a totally bogus address, and yet it finds the zlib stuff anyway. It seems to be ignoring the flags I am setting.

Working hypothesis: I am building jpeg wrong, and some system setting is overriding my value of --with-zlib.

A less satisfying but easier solution could come from downloading binaries. At this point I simply have to swallow my pride and ask which if any of the binaries listed here might work.

Advice would be even more welcome than sympathy.

Update 1/24:

UTIG support advised option 4, which worked more or less smoothly.

Also, from UTIG support:

Oh I didn't see you had your own (up a directory). The command you
were looking for would be

./configure --prefix=/usr/local/climate \
--with-zlib=/usr/local/climate/NCLc3/zlib-1.2.3 \
--with-jpeg=/usr/local/climate/NCLc3/jpeg-6b --disable-netcdf

And I'm idiotically proud of this script, my first conditional in a shell script in a long time:

set hn = `hostname`
if ($hn == 'foo.whatever.utexas.edu') then
echo "climate paths added"
setenv NCARG_ROOT /usr/local/climate/NCLBin/
set path = ( /usr/local/climate/NCLBin/bin/ $path )
endif

So there's finally a happy ending on this one. TACC is still having hardware or configuration problems with Lonestar which is preventing any progress on the other front.

Language of Choice

2008-01-15T07:55:00.000-08:00

Presumably everyone knows that Python is on the rise with a bullet. Of course, now that Python is sexy it will probably attract some weaker programmers, diluting its reputation. Such are the costs of great beauty. Right now, though, the upswing seems to be near its maximum rate.

In the Slashdot article discussing this I saw a ref to a lexicon definition the concept of Language of Choice in the Jargon File. FORTRAN is dismissed as a niche language for scientists, but nothing better is proposed.

So what should the language of choice be for high performance numerical computing?

There are teams working on replacements (Fortress, Chapel, some IBM thing with a forgettable name), but I think much of the high performance work of the future can be done in Python with a code generation strategy. I'm not sure what the target language ought to be for the code generation. Would there be advantages to Fortress or Chapel for this? For contemporary purposes even F77 would work (as I said in the PyNSol papers) but to be honest the F77 target is almost a stunt, just to make a point. I don't expect it would be useful in the long run.

I am currently thinking most of the system should be agnostic as to the target language. If done right, multiple targets can be supported in an architecture-dependent bottom layer. Even Verilog is a candidate. ( F90 and onward are not. I'd sooner use BF or Whitespace. )

Update: Bryan Lawrence has a nice summary of the issues.

Do NCAR or GISS Ask for This?

2008-01-14T20:00:00.000-08:00

This crossed my path on a CS mailing list I follow. The context is about whether students need more software engineering or more computer science. It's a bit harsh on us older types but it certainly is an interesting list. I post it so scientific computing types can consider what they may or may not be missing. I should also note that there are practicing scientific coders for whom I have great respect who would find this list ludicrous.

As a software engineer in the industry and as a person who hires developers I have to say that 2 out of every 3 interviews I take from people with a CS or MS in computer science I do not hire because they are completely inadequate as software engineers. More often than not, they do not know about or have distaste for one or more of the following:
Design Patterns
Agile engineering / programming
Extreme Programming
Test Driven Development
Composition vs. Inheritance
We also give a programming test as a part of the interview process, it is mathematically / algorithmically moderate in challenge and most interviewees (80%) can solve the problem, but only 15-20% solve it with good programming practices and maybe 5-10% write unit tests. Actually, if an interviewee who has good engineering, but does not solve the problem is more appealing than someone who solves the problem.

I do recall from my undergraduate program at Purdue that we did have a "Software Engineering" class that did try to drive home the points I mentioned, but I have to say that it did not go far enough.

I would never diminish CS courses, mathematics, or general studies programs ever. I would say that anyone who only takes their CS and never masters the engineering principles either through other course work (several hours worth) or through an internship does not stand much of a chance getting hired at the company I work for.

In the end, I would say that if I knew 6 years ago what I knew now, I would have gone for the software engineering program and tried for a dual major in CS if I had the bandwidth.

For what it's worth I believe that design patterns in practice are overstressed, and that the list isn't especially elegant or complete. That said I also have the impression that these ideas are only very dimly perceived in the climate modeling centers and a great deal of benefit could be gained from them.

Progress

2008-01-10T23:33:00.000-08:00

OLPC is a great achievement if only for this story:

"Microsoft struggles to port Windows to a device originally conceived to run Linux."

Still waiting on mine...

Why is Climate Modeling Stuck?

2007-12-26T08:23:00.000-08:00

I believe that progress in climate modeling has been relatively limited since the first successes in linking atmosphere and ocean models without flux corrections. (That's about a decade now, long enough to start being cause for concern.) One idea is that tighter codesign of components such as atmosphere and ocean models in the first place would help, and there's something to be said for that, but I don't think that's the core issue.

I suggest that there is a deeper issue based on workflow presumptions. The relationship between the computer science community and the application science community is key. I suggest that the relationship is ill-understood and consequently the field is underproductive.

The relationship between the software development practitioners and the domain scientists is misconstrued by both sides, and both are limited by past experience. Success in such fields as weather modeling and tide prediction provide a context which inappropriately dominates thinking, planning and execution.

Operational codes are the wrong model because scientists do not modify operational codes. Commercial codes are also the wrong model because bankers, CFOs and COOs do not modify operational codes. The primary purpose of scientific codes as opposed to operational codes is to enable science, that is, free experimentation and testing of hypotheses.

Modifiability by non-expert programmers should be and sadly is not treated as a crucial design constraint. The application scientist is an expert on physics, perhaps on certain branches of mathematics such as statistics and dynamics, but is typically a journeyman programmer. In general the scientist does not find the abstractions of computer science intrinsically interesting and considers the program to be an expensive and balky laboratory tool.

Being presented with codes that are not designed for modification greatly limits scientific productivity. Some scientists have enormous energy for the task (or the assistance of relatively unambitious and unmarketable Fortran-ready assistants) and take on the task with energy and panache, but the sad fact is that they have little idea of what to do or how to do it. This is hardly their fault; they are modifying opaque and unwelcoming bodies of code. Under the daunting circumstances these modifications have the flavor of "one-offs", scripts intended to perform a single calculation, and treated as done more or less when the result "looks reasonable". The key abstractions of computer science and even its key goals are ignored, just as if you were writing a five-liner to, say, flatten a directory tree with some systematic renaming. "Hmm, looks right. OK, next issue."

This, while scientific coding has much to learn from the commercial sector, the key use case is rather atypical. The key is in providing an abstraction layer useful to the journeyman programmer, while providing all the verification, validation, replicability, version control and process management the user needs, whether the user knows it or not. As these services become discovered and understood, the value of these abstractions will be revealed, and the power of the entire enterprise will resume its forward progress.

It's my opinion that Python provides not only a platform for this strategy but also an example of it. When a novice Python programmer invokes "a = b + c", a surprisingly large number of things potentially happen. An arithmetic addition is commonly but not inevitably among the consequences and the intentions. The additional machinery is not in the way of the young novice counting apples but is available to the programmer extending the addition operator to support user defined classes.

Consider why Matlab is so widely preferred over the much more elegant and powerful Mathematica platform by application scientists. This is because the scientists are not interested in abstractions in their own right; they are interested in the systems they study. Software is seen as a tool to investigate the systems and not as a topic of intrinsic interest. Matlab is (arguably incorrectly) perceived as better than Mathematica because it exposes only abstractions that map naturally onto the application scientist's worldview.

Alas, the application scientist's worldview is largely (see Marshall McLuhan) formed by the tools with which the scientist is most familiar. The key to progress is the Pythonic way, which is to provide great abstraction power without having it get in the way. Scientists learn mathematical abstractions as necessary to understand the phenomena of interest. Computer science properly construed is a branch of mathematics (and not a branch of trade-school mechanics thankyouverymuch) and scientists will take to the more relevant of its abstractions as they become available and their utility becomes clear.

Maybe starting from a blank slate we can get moving again toward a system that can actually make useful regional climate prognoses. It is time we took on the serious task of determining the extent to which such a thing is possible. I also think the strategies I have hinted at here have broad applicability in other sciences.

I am trying to work through enough details of how to extend this Python mojo to scientific programming to make a credible proposal. I think I have enough to work with, but I'll have to treat the details as a trade secret for now. Meanwhile I would welcome comments.

Fake Steve on OLPC

2007-11-12T14:52:00.000-08:00

Fake Steve Jobs has some brutal (but sort of funny) articles about OLPC. Check this one, or google the site for "Negroponte".

GOGO gone?

2007-11-12T14:44:00.000-08:00

I don't quite get this "limited time only" "give one get one" thing the OLPC people are running.

It's being spun as if I were giving the one I get to a child. But that is silly.

The child's interest will be piqued by the community aspect, and if there will be no XO systems for sale, it will be impossible to build the community. It's almost as if Bell had decided to start the phone system in Ougadougou (a real place with a real OLPC initiative, actually, but I like the name) and had financed it by selling telephones in America.

You know, one telephone isn't worth much. You need at least two.

So in practice it will almost certainly be "give two get two" program, with adult hobbyists buying. Or did I miss something?

Will there be a commercial XO product in the marketplace? If not, why not?

Is the platform stable enough to start thinking about end user apps? I don't realistically have the time to do systems work, but I'm very interested in what runs on it.

Alan Perlis Quote

2007-08-03T14:36:00.000-07:00

From Dethe Elza's sig:

"It goes against the grain of modern education to teach children to program. What fun is there in making plans, acquiring discipline in organizing thoughts, devoting attention to detail and learning to be self-critical?"

--Alan Perlis

Article is Out

2007-05-23T12:14:00.000-07:00

Not really makin a huge splash, but for what it is worth look for the spring 2007 edition of the Python Papers. See pages 8 and following.

In an email Duke Crawford offers the following links to an open source language drill program:

http://www.olpcnews.com/content/ \

localization/learning_language.html

http://sourceforge.net/projects/twexter
http://wiki.laptop.org/go/WiXi
http://wixi.cc/hello_world
http://twext.cc/go/18

Is anybody else out there?

Laptop backlash

2007-05-10T11:54:00.000-07:00

Get it while it's live: the NY Times reports “Seeing No Progress, Some Schools Drop Laptops”.

Responses here.

a snippet:

==>
So the Liverpool Central School District, just outside Syracuse, has
decided to phase out laptops starting this fall, joining a handful of
other schools around the country that adopted one-to-one computing
programs and are now abandoning them as educationally empty — and
worse.

Many of these districts had sought to prepare their students for a
technology-driven world and close the so-called digital divide between
students who had computers at home and those who did not.

"After seven years, there was literally no evidence it had any impact
on student achievement — none," said Mark Lawson, the school board
president here in Liverpool, one of the first districts in New York
State to experiment with putting technology directly into students'
hands. "The teachers were telling us when there's a one-to-one
relationship between the student and the laptop, the box gets in the
way. It's a distraction to the educational process."
<==

It's not surprising, really, but the timing is not good.

FInal Version of Article

2007-05-10T11:49:00.000-07:00

The 3rd instance of the Python Papers will be coming out May 28.

Edu-Python: Python's Killer App?

2007-04-16T17:26:00.000-07:00

DRAFT of an ARTICLE

IS EDUCATION PYTHON'S KILLER APP?

SALIENT QUOTATIONS

"Should the computer program the kid, or should the kid program the computer?"
- Seymour Papert

"We want to explore the notion that virtually everybody can obtain some level of computer programming skills in school, just as they can learn how to read and write."
- Guido van Rossum

"Python makes too much sense as a learning language to not ultimately have an effect in the education arena."
- Arthur Siegel

"If schools won't accept classes that have Python in them, then it doesn't matter how much better Python is than BASIC, the classes cannot be taught."
- Dean Nevins

PYTHON IN EDUCATION

If learning is about information, then surely learning will change somewhat under the influence of pervasive information technology.

This is an exciting time for those of us who believe that software development in general, and Python in particular, has a role in general education. Three of the four keynotes at PyCon '07 revolved around the idea of computing as a medium for education and/or communication. Adele Goldberg talked about the frustrations of getting carefully designed computational curriculum into the public school systems. Robert "rOml" Lefkowitz talked about code as a communication medium. Finally, on a more practical note, Ivan Krstic discussed the very exciting prospects of the One Laptop Per Child project, whose modest goal is to make computing available to all children everywhere.

These discussions gave a boost to the quiet world of educational Python. It seems like a good time to take stock of the entire educational programming landscape and Python's role in it.

COMPUTER LITERACY

In the early days of computing, it became apparent that computers would become ubiquitous and part of daily life. At that time, we had little idea what shape they would take in a generation.

I recall an episode of the futuristic TV news program "The Twentyfirst Century", hosted by the renowned Walter Cronkite, that discussed the implications of home computers. A professor's family had a massive mechanical teletype (a terminal that used a typewriter mechanism rather than a CRT or LCD display) in their home. The demonstartion application was, as I recall, changing the proportions in a recipe so that it served seven rather than four. I am sure it was a great tangle of Lisp under the hood and no great pleasure to use, but it appeared that if you perservered a recasting of your recipe with all the quantities multiplied by 7/4 and converted to standard cooking units loudly clanked out onto the fanfold paper. At the time, the 21st century was promoted in the media to be an almost flawless utopia, but even so one had the sense that both Cronkite and the professor's wife were struggling to express any enthusiasm for the concept of home computing.

If you think about what "computer literacy" means today in high schools, trade schools, and nontechnical programs in university, though, you'll see that it amounts to recipe scaling writ large. Mass market software presents complex and often counterintuitive user interfaces, and the skill of navigating through each of them is valued and promoted.

PROGRAMMING FOR CHILDREN

Of course, we do need to be able to cope with applications. Still, there have always been two visions of computing, one as a world in which there is a very sharp distinction between producers and consumers of software, and another where the continuum is blurred.

The alternative vision of computer literacy was formed by the early users of computers. In those days, there were no "applications". Being a computer user amounted to being a programmer. Many of us felt that the skills we learned by wrestling with the structure and logic of a program would be applicable to many aspects of life. We felt that the advantages we were getting could be democratized. In this view, "computer literacy" was about coding, not about being able to push a mouse around and click on things effectively.

This view is succinctly summarized by Georgia Tech Professor Mark Guzdial's comment on a blog page: "computing FOR learning (as opposed to learning ABOUT computing".[1]

Seymour Papert (with Willy Feurzig) developed Logo[2], the original "learning language in the late 1960's. The ideas of Logo were explicitly based on developmental psychology[3]. Papert not only claimed that remarkably young children (fourth graders, about 9 years old) could be exposed to the ideas of programming, he was successful in demonstrating this. Many subsequent teaching environments have been based on Logo, and especially on the observation that "turtle graphics" were a uniquely compelling development target for novices) the basic idea has not been as successful as Papert envisioned. (It's interesting to contemplate the fact that it was less expensive to construct a mechanical turtle than a graphic display in those days!)

Alan Kay is another key figure in the computing for learning movement. His 1971 "Dynabook" paper[4] is a remarkable exposition of the possibilities of computing from a time long before anyone actually had so muchas a portable display. Kay and Papert are still with us as I write (although, unfortunately, Papert was seriously injured recently), and among their recent efforts, along with Nicholas Negroponte, is the One Laptop Per Child initiative[5], where Python plays a key role, and which largely revives the Dynabook idea.

COMPUTING FOR NOVICES

Despite the strong advocacy of the likes of Papert and Kay, there is no consensus on whether most people can and should learn a modest amount of programming, nor what age is best to start. Still, we can agree that *some* people will become programmers, and they have to start with *some* language.

Among the most famous quotations attribute to the great computer scientist Edsger Dijkstra are critiques of teaching computer science with inappropriate languages, notably that "The use of COBOL cripples the mind; its teaching should, therefore, be regarded as a criminal offense" and "It is practically impossible to teach good programming to students that have had a prior exposure to BASIC: as potential programmers they are mentally mutilated beyond hope of regeneration." Many people have taken DIjkstra's advice to heart, and designed languages intended to expose young adult students to good practice and good theory from the beginning. Pascal and Smalltalk are probably the most successful examples.

Another effort in this direction was the ABC programming language project of Geerts, Meertens and Pemberton at Centrum voor Wiskunde en Informatica in Holland. They list advantages of ABC in comparison to Pascal as 1) small, intuitive syntax 2) maps well onto design principles 3) terseness 4) support for refinement 5) no declarative statements and 6) interactive [5]. Along with these good ideas the CWI group had the inspiration to put the young Guido van Rossum on staff, and it is here that the Python story begins.

While much of the history of the Python community remains an oral history, we do have the following quote from Guido: "Over six years ago, in December 1989, I was looking for a 'hobby' programming project that would keep me occupied during the week around Christmas. My office ... would be closed, but I had a home computer, and not much else on my hands. I decided to write an interpreter for the new scripting language I had been thinking about lately: a descendant of ABC that would appeal to Unix/C hackers." [6] Python, then, is a combination of a well designed learning language and a practical scripting language.

COMPUTER PROGRAMMING FOR EVERBODY

Once Python had attained a certain momentum, Guido van Rossum under the auspices of the Corporation for National Research Initiatives applied for and received a small grant to support a project called Computer Programming for Everybody of CP4E, to investigate democratizing programming. The project was funded for a single year, half the anticipated duration, through DARPA. Guido and much of the core development team moved into the for-profit sector at aboutr that time as well. The proposal [7] lists broad objectives of developing high school and college curriculum, developing tools, and developing a user community, as well as many subcomponents of these objectives.

The motivation section is in the intellectual lienage of Papert and Kay:

===>
If we are successful, non-experts will be able use their computers and other intelligent devices much more effectively, reducing their level of frustration and increasing their productivity and work satisfaction. (New leisure possibilities will undoubtedly ensue as well!) Computer users will be able to solve their own computer problems more often, reducing the need for technical support.

Even if most users do not program regularly, a familiarity with programming and the structure of software will make them more effective users of computers. For example, when something goes wrong, they will be able to make a better mental model of the likely failure, which will allow them to fix or work around the problem. They will also be able to assess better when they can make the changes themselves and when they will need the services of an expert. They will be more able to converse with experts, since they will now share more of a common language. An analogy is obtaining basic literacy in automotive maintenance: you know enough to check your oil and add a few quarts if necessary, but you also know that you shouldn't try to change your own brakes. When the mechanic says "your rotors are warped and you need new pads," you understand what he is talking about.

If this effort is successful, there could be many millions, eventually billions of computer programmers, at various levels of proficiency. The effects this will have on the state of the art of software development is hard to imagine. The nature of software will change to accommodate the needs of these programmers, allowing customization through source code modifications--and personalizations will be plentiful.
<===

The funded CP4E project's principal achievement, was developing the IDLE system. This is not to be minimized, as IDLE remains the development platform of choice for beginners in many contexts.

Still, it must be admitted that many of the goals of CP4E remain unfulfilled to this day. The SIG that was centered on the project, while hosting many fascinating discussions, lost focus. Until recently, a fairly large number of fairly small projects were developed in and around the cluster of ideas that motivated CP4E and the edu-sig. Efforts to create a directory and a repository for this work have so far failed to reach critical mass. There appears to be a cat-herding problem, with each enthusiast having his or her own individual goals. That said, a great deal of fascinating work has been and continues to be happening, and quite a few young people have been exposed to Python as their first programming language and environment. Perhaps even more importantly, the mission of CP4E, and Python's role in it, has been an active goal for quite a few people in the intevening seven years.

The sidebar attemps, and surely fails, to be an exhaustive list of Python related education projects. It includes projects where Python is or has been used to develop a domain specific learning language or learning tools, or where Python is used as the language of discourse in teaching programming, or in using programming to teach other topics. It is limited to resources aimed at novice programmers who are not necessarily going to become technical or computing professionals.

WHY PYTHON?

Pulling together these Python as a First Language/CP4E experiences, formalizing them, and making a case for them in educational settings, remains somewhat
elusive. If we accept Dijkstra's admonitions that a first computing language is important to the inetllectual development of the student, it seems to those of us who appreciate Python for what it is. It's arguably our responsibility to get beyond "we like it because it is good" for the non-Python audience.

The Python community knows what it likes and is amazingly verbose and articulate in discussing this or that programming construct, but we have never been particularly adept at explaining our vision to those who have not experienced it. In researching this article, I made some effort to get Python First enthusiasts to explain the basis of their enthusiasm in the context of the novice programmer.

===
MORE
===

AUDIENCES AND OBSTACLES

In discussing the future of Python as a first language, we need to distinguish between several populations of novices:
1) a child working with a parent or other adult
2) a child working alone
3) a child studying a formal course in school
4) a college student studying a formal course
5) an adult seeking to achieve some personal software goal

These audiences have distinct constraints and distinct objectives. It seems clear that, except for very young children, foe whom a limited environment such as eToys or Alice may be of great value for more than a few weeks, Python is clearly a viable choice.

Python's competitive position is weakest in the second and third categories, for quite distinct reasons.

In the case of a purely self-motivated child, the child's immediate objective is to do something cool on a web page and email the URL to friends and family. Python is, of course, a very powerful weapon in the hands of the experienced programmer, especially in combination with the vast alphabet soup of frameworks and libraries available. Unlike the core language, though, this power does not scale down to simple applications. Most mass market ISPs do not support Python CGIs, and certainly don't support the likes of Twisted or mod-Python. Also, the libraries and frameworks, though powerful, are conceptually inaccessible to the novice. To the extent that frameworks like Django are easy to use, they probably move far away from Dijkstra' concepts of a pedagogically useful language, and at the extreme Zope/Plone is more of a bureaucracy than a development platform. Compare this with the ease of use and remarkably complete and accessible documentation of PHP, or the zero barrier to entry of Javascript, and we lose the budding web program to pedagogically less valuable languages. Perhaps we needn't worry about the self-motivated young programmer, though. Through their enthusiasm they will learn the limitations of their platform, and eventually move to Python or a similar language. As a youth I programmed in uglier things than PHP and yet lived to tell the tale.

More of an issue is the school. Adele Goldberg's PyCon keynote was largely a lament about trying to work within the US public school systems. She is giving up on it as a primary goal of her efforts, for a sad litany of reasons. Although her root causes were familiar to me (too much politics, too much fear, no room for experimentation), despite her sad tale she missed a couple of points that are relevant to adding a computing language to the curriculum that would not agitate the good Dr. Dijkstra. First of all, computing (except for advanced placement) is considered a vocational pursuit, not an intellectual one! This madness is endemic; Jeff Elkner's diligent pursual of all that is Pythonic within a public school system in Virginia all occurs in the context of a vocational track. Python is considered in competition with VBasic in this context. For college-bound students, Java has become the language of choice as a conseqeunce of the structure of the Advanced Placement examinations. (Everyone seems to agree that this is a pedagogical imporvement over C++, which presumably was as baffling to the teachers as the students.) The consequence is that despite the enormous scale of the school system, which would allow for rigorous experimentation and improvement, most schools are stuck teaching Java and VBasic.

In the home or the college, these constraints are not operative. Python First continues to gain momentum in these settings.

ENTER OLPC

Amid all of this mixture of success and frustration, a new five hundred pound gorilla has entered the room. The One Laptop Per Child[] initiative has promise of reviving the original vision of CP4E.

+++>

The project's origins go back more than four decades to the early days of computing, when most machines were still the size of small dinosaurs, and almost no one dreamed they would ever be suitable for children. But pioneering thinkers like Seymour Papert disagreed sharply, and over time led the long march from radical theory to reality proving the immense power of the personal computer as a learning tool for children.
<===

Even better, much of the system, and specifically the operating system GUI (called Sugar)[] will be based in Python. Some clever architecture will even allow the child to play with the code! (Of course Python is available.) The very large scale and funding of this initiative promise to provide new focus to the CP4E community.

OLPC does an end run around the school bureaucracies of developed countries. In less developed countries, bureaucratic and political obstacles may also be less developed. It provides a Python-friendly platform with massive distribution among children who have not seen code. It promises to enhance the prominence of Python and allow much larger scales of experimentation wioth pedagogic ideas, all the while being perceived as great fun.

It might not be in time to save the world, but we can hope for that too.

Python-First Educational Resources

2007-04-11T12:45:00.000-07:00

OLPC and SUGAR

OLPC Interface Guidelines

OLPC/Sugar demo (video)

Sugar

OTHER PYTHON SOFTWARE

Childsplay

Crunchy

Guido van Robot

PataPata

PyLogo

RUR-PLE

PROGRAMMING FOR THE VERY YOUNG

Alice

OLPC-Squeak demo (video)

Squeak

VIDEO

Introducing Python

Various, Especially Python-in-Mathematics Videos - Urner

PRESENTATIONS

The computer revolution hasn't happened yet - Kay 1998

My presentation to the Austin Python User Group last week - Tobis 2007

PAPERS

Models of Growth - Cavallo 2004

Using Python in a High School Computer Science Program - Elkner 2000

A Personal Computer for Children of All Ages - Kay 1972

Promoting Computer Literacy Through Programming Python - Miller 2004 (dissertation)

Python First: A Lab-Based Digital Introduction to Computer Science - Radenski 2006

CP4E - van Rossum 1999

Teaching Scientific Programming Using Python - Williams (master's thesis)

WIKIS, BLOGS, ARTICLES etc.

C2 Wiki: Computer Programming for Everybody

Alan Kay and OLPC - Windley 2006

Interview with Jeff Elkner - Willison 2000

OLPC - IEEE 2007

PORTALS

python.org edu-sig

python.org beginners' resources

Python Bibliotheca

ONLINE BOOKS and COURSES

An Introduction to Programming - Rollins

Software carpentry - Williams

How to Think Like a Computer Scientist: Learning with Python Downey, Elkner and Meyers, 2004; also available in book format

The LiveWires Python Course

PYTHON FIRST TEXTBOOKS

Learn to Program Using Python - Gauld 2001

Object-Oriented Programming in Python - Goldwasser and Letscher 2007 (to appear)

Introduction to Computing and Programming in Python, A Multimedia Approach - Guzdial 2004

Beginning Python: From Novice to Professional - Hetland 2005

Python Programming: An Introduction to Computer Science - Zelle 2003

I: Pedagogical Advantages of Python

2007-03-26T13:56:00.001-07:00

What experience do you have teaching coding to beginners using Python as the target language? Using other languages? Especially welcome are comparison/contrast of teaching with language X vs teaching with Python from a pedagogical point of view.

II: Pythonic Teaching Tools

2007-03-26T13:54:00.000-07:00

What experience do you have using Pythonic tools (environments, libraries) as part of your teaching environment, regardless of the language that the students are learning? What advantages does python present you as an educational tool developer and/or deployer?

III: Attractiveness of Competing First Languages

2007-03-26T13:53:00.000-07:00

What advantages do other languages have over Python in increasing the attractiveness of the language to self-directed learners? e.g., JavaScript's universal accessibility, PHP's ease of use and extensive documentation. Can these disadvantages be overcome, and if so, how so and how quickly?

IV: Institutional Competition

2007-03-26T13:52:00.000-07:00

What advantages do other languages have over Python in their attractiveness to educational institutions? e.g., Java being the language of the Advanced Placement Test in the US, VBasic being perceived as the appropriate medium for vocational preparation. Can these disadvantages be overcome, and if so, how so and how quickly?

V: Pythonic tools

2007-03-26T13:50:00.000-07:00

What existing libraries, tools and resources should the instructor proposing to use Python in education be aware of? What are the most important Pythonic libraries, tools, and resources are under development?

Why Pencil Science?

2007-03-26T12:48:00.000-07:00

This is a blog about the nuts and bolts of computer literacy from a Pythonic perspective.

I would like not merely to defend the position that computer programming is "for everybody" but also to examine the proposition that Python is the appropriate vehicle for it.

Of course, "computer science is not about computers" but about formally expressing processes using symbols. The misnomer is consequential in how most people think about the question of the role of programming in education.

We don't teach children how to write because we expect them all to be professional writers. Professional writers are not opposed to literacy on the grounds that they will lose readership. We do not call the ability to write "pencil science".

Perhaps we should start, though. Maybe it would make the absurdity of how we misrepresent beginning software education as a vocational track more clear.

I am focusing my explorations on an opportunity to write an article for the Python Papers in the very near future. I'd like to use this blog, at least at first, as a gathering place for ideas from others. If you have anything you'd like to add, read on.

Many thanks
Michael Tobis