Installing MySQL on OS X

It's actually not required to get started on Django, but I thought it would be a good idea to jump through the MySQL hoops early.

It was a bit of an adventure:

The MySql install instructions are broken for OS X. After you have
downloaded the MySql package and installed it with the usual dance,
you will probably encounter some instructions involving
"groupadd".

Various mortals seem to have gotten confused by this and
run into various smug bastards being unhelpful. Ignore the smug bastards.

===

Instead, follow the instructions at
http://www.macosxhints.com/article.php?story=20080128103022907

However, the lines

===

$ cd /Library/LaunchDaemons
$ sudo chown root com.mysql.mysqld
$ sudo chgrp wheel com.mysql.mysqld
$ sudo chmod 644 com.mysql.mysqld

===

should read:

===

$ cd /Library/LaunchDaemons
$ sudo chown root com.mysql.mysqld.plist
$ sudo chgrp wheel com.mysql.mysqld.plist
$ sudo chmod 644 com.mysql.mysqld.plist

===

Otherwise the instructions are OK.

Now it tells me to check with Console.app ... What is that?

Wahoo! There it is in Utilities. I run it and it claims mysqld is running and
ready for connections.

===

So now to find mysql; open a Terminal window.

OK, now mysql isn't in my path yet, but I can invoke a mysql session
with

/usr/local/mysql/bin/mysql

thus:

/Guido:~ tobis$ /usr/local/mysql/bin/mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.30 MySQL Community Server (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

MySql is running! Type quit . This is enough of a victory for one day.

===

In future if you want to just type mysql instead of
/usr/local/mysql/bin/mysql, you need to update your path.

Change to your home directory. Issue

touch .profile

Then edit .profile, appending the line

export PATH="$PATH:/usr/local/mysql/bin"

Note this is a hidden file. See

http://www.peachpit.com/articles/article.aspx?p=31442&seqNum=4

http://www.macworld.com/article/51830/2006/07/showallfinder.html

===

In future if you don't want to burden your machine with mysql anymore,
issue

launchctl unload com.mysql.mysqld.plist

Next step Django itself. I am expecting it to be a bit easier.

Proper tarballs from aquamacs projects

Getting a bunch of dot underscore cruft in your tarballs?

export COPY_EXTENDED_ATTRIBUTES_DISABLE=true

see comment by David Olinsky here

Python First

"But there is emerging consensus in the scripting community that Python is the right choice for freshman programming. Ruby would also be a defensible choice. Python and Ruby have the enviable properties that almost no one dislikes them, and almost everyone respects them. Both languages support a wide variety of programming styles and paradigms and satisfy practitioners and theoreticians equally. Both languages are carefully enough designed that 'correct' programming practices can be demonstrated and high standards of code quality can be enforced. The fact that Google stands by Python is an added motivation for undergraduate majors." -Ronald Loui

How not to build a bad website

from xkcd 345

Reproducibility blog

John Cook has started a blog on reproducibility in science. H/T Greg Wilson.

Salient comment: "Your result is not reproducible if you can reproduce it. Your result is reproducible if someone else can reproduce it."

I hope this was a joke

but I think it was serious.

J Wants a Mac (Vista flame)

MUD transcript:

J wants another Mac.

J bought a cheap laptop to last while he's living in corporate storage, it's VISTA! Holy crap, I have the absolute lowest expectations possible about Windows, well desrved from experience, and somehow Vista blows away every shred of quality or clear thinking. It's truly a self-caricature. Every time you run a program it asks you if you want to run it. Later it'll ask about getting more permissions to contniue to run! Programs will die, and then another program runs to "try to figure out why" the first crashed. It's truly the most pathetic computing experience I've ever had. OH! And the network on the Vista box already hung once. I was appalled. I was restoring a backup from a PII-233MHz to this brand-new box and the best it could do was about 140KB/sec and then it hung.... My G4 Mac had no problem doing 4-6MB/sec to/from the Pentium2.

J says, "And here's another absurdity: Vista is 32bit so it can't see all the memory I bought. M$ spent millions crippling a 64 bit OS."

J says, "However, the laptop is great: 17", dual 2Gz Opteron, 4 USB, 1 FW, crappy graphics card, but here was the deal cincher: Two internal drive bays! This box will become the new R***.com server. I'm sure Linux will run 1000x better on this box than Vista."

mt says, "Yes but Microsoft got the license fee anyway"

client side matplotlib

way cool

App Engine OS X Confusion

Google AppEngine docs is inconsistent at this point.

It offers a download of AppEngineLauncher for OS X rather than AppEngine. Installing that does NOT, per docs, put appcfg.py or dev_appserver.py in the path.

It does create instances of those.

Directly invoking dev_appserver.py works; modulo a warning in importing PIL; that seems OK.

I see two instances of appcfg.py: in Applications/GoogleAppEngineLauncher.app//Contents/Resources :

google_appengine/appcfg.py
and
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/appcfg.py

Neither works. They both say ImportError: No module named google. I have deployed test code effectively. I don't actually know what appcfg is supposed to do.

Controversy about Software in Higher Ed

here and a reply here

The Secret Ingredient

http://radian.org/notebook/wp-content/uploads/2008/05/guido2.jpg

HPC Considered Harmful

I'm really looking forward to Greg Wilson's talk called "HPC Considered Harmful" at Scientific Software Days at TACC, for which I am a co-organizer. Also Eric Jones will be talking about "New Directions in Scientific Workflow" which sounds great too.

I'm not sure what these guys will say, but I think this (Carriero et al 2004) is relevant:

The goal of [our] approach is not to reap the maximum efficiency benefits in making changes to a program. Rather the goal is to remove compute time as a rate-limiting step.

Anyway, come on down if you're in the neighborhood!

Life in the Trenches, Generalized

A really good description of the problem is here.

It's only exacerbated when both the producers and the consumers of the software are scientists who don't think software principles are interesting and/or whose mentors don't believe in time for training on such principles. Invariably we are targetting weird platforms, too.

I recently heard someone say that about 30% of professional programmers' time is spent on build issues. If scientists are as much as half as good as full-timers, that means they spend 60% of their time on build issues, but they also have to spend 50% of their time on reading and writing discipline-related stuff. That means we get negative 10% of our time for actual productive coding. If we're lucky.

Sudden abstractions

verbatim from "Young Archimedes" by Aldous Huxley, 1928

(an English family spends a year in Italy and their young child befriends an older peasant boy)

"Though fully two and a half years older than little Robin - and at that age thirty months are crammed with half a lifetime's experience - Guido took no undue advantage of his superior intelligence and strength. I have never seen a child more patient, tolerant and non-tyrannical. He never laughed at Robin for his clumsy efforts to imitate his own prodigious feats; he did not tease or bully, but helped his small companion when he was in difficulties and explained when he could not understand. In return, Robin adored him, regarded him as a model and perfect Big Boy, and slavishly imitated him in every way he could.

"These attempts of Robin's to imitate his companion were often exceedingly ludicrous. For by an obscure psychological law, words and actions in themselves quite serious become comic as soon as they are copied; and the more accurately, if the imitation is a deliberate parody, the funnier - for an overloaded imitation of someone we know does not make us laugh so much as one that is almost indistinguishably like the original. The bad imitation is only ludicrous when it is a piece of sincere and earnest flattery that does not quite come off. Robin's imitations were mostly of this kind. His heroic and unsuccessful attempts to perform the feats of strength and skill, which Guido could do with ease, were exquisitely comic. And his careful, long-drawn imitations of Guido's habits and mannerisms were no less amusing. Most ludicrous of all, because most earnestly undertaken and most incongruous in the imitator, were Robin's impersonations of Guido in a pensive mood. Guido was a thoughtful child given to brooding and sudden abstractions. One would find him sitting in a corner by himself, chin in hand, elbow on knee, plunged, to all appearances in the profoundest meditation."

...

(Guido is discovered to have a gift for mathematics.)

"This child, I thought, has had the fortune to be born at a time when he will be able to make good use of his capacities. He will find the most elaborate analytical methods lying readily to his hand; he will have a prodigious experience behind him. Suppose he had been born while Stonehenge was building; he might have spent a lifetime discovering the rudiments, guessing darkly where he might have had a chance of proving. Born at the time of the Norman Conquest, he would have had to wrestle with all the preliminary difficulties created by an inadequate symbolism; it would have taken him long years, for example, to learn the art of dividing MMMCCCCLXXXVIII by MCMXIX. In five years, nowadays, he will learn what it took generations of Men to discover."

(Alas the story ends sadly for the fictitious young Guido, while happily for all of us in real life, the analogy breaks down. Clearly Guido should have contrived to be born not just in the modern age but also in Holland...)

Now let me think a minute, son

Ah, yes, that can be easily done...

More Python and Climate

Apparently my rants got linked by GravityLoss, who understands that I am saying something about Python and about modern methodologies, but apparently isn't sure exactly what.

Here's my reply:

Hi and thanks for the links.

Make no mistake, I am a Python fanboy. I really doubt it's possible to do much better than Python at our present level of understanding of how to control computers.

As Paul Graham explains, any assertion that a more advanced technology Y is better than technology X tends to fall on deaf ears on users of technology X, because what they think is doable is defined by technology X. It is perfectly possible to get no advantage from Y because it is always possible to do X-like things with it.

I am proposing to do somewhat different things with Y. I am not the greatest expert on Y, but I do see things to do relevant things with Y that are not entirely X-like.

I would like to have software managers more experienced than myself involved. Google are the folk with the disposable wealth and the farsighted motivation as well as some of the relevant skills. I sort of wish they'd do what I want to do rather than leaving me scrambling to do it. I retain enough hubris to suggest that if they tried they'd do well to get me on the team. Realistically I probably will not have their help nor the help of an experienced software manager, though any pitching in from JM or JM would be most welcome.

There's nothing magic about Google or Python that aren't summed up in Clarke's Law: any sufficiently advanced technology is indistinguishable from magic. The sad thing is that much of the academy, including climate science, is far enough behind the cutting edge of what is possible that comparable productivity would look magical.

The elegance of Fortran

You know, the point of OOP is to SAVE work. 

Yes, the following is the most trivial and trite and pointless possible example of OOP.  I am not showing you code that is interesting. I am just comparing it with bolted-on OOP. Have a look at the Fortran equivalent. It's completely equivalent, note.

The "polymorphism" section form the Fortran is expressed in Python as an arbitrarily small amount of whitespace. And you thought whitespace was a drawback of Python!

def vectorsum(vec1,vec2):
...return map(sum,zip(vec1,vec2))

class Circle(object):
...def __init__(self,center,radius):
......self.center = center
......self.radius = radius

...def __str__(self):
......return "circle of radius %s at %s" % (self.radius, self.center)

...def move(self,translation):
......assert len(translation) == len(self.center)
......self.center = vectorsum(self.center, translation)

class Square(Circle):
...""" note: can inherit constructor for circle; treat radius as length"""
...def __str__(self):
......s = self.radius
......shifts = ((0,0),(0,s),(s,0),(s,s))
......corners = tuple([vectorsum(self.center,shift) for shift in shifts])
......return "square at points %s %s %s %s" % corners

PyCon snippets

Stuff to look into:

iPy1: a spawned interpreter on every CPU

MPI4py: the winner for fine-grained parallelism

distributed NumPy array: Brian Granger

"it's like a beginner trying to make a souffle; you WILL get an egg dish"

stats.bayes.mvs

new buffer type in Py 3, Travis O in charge, imports NumPy arrays; PEP 3118

struct module

Amazon's elastic cloud; Pete Skomorock's ElastiWulf

Trestle

noonhat.com ; saturdayhouse.prg

pyxpcomext.com

freebase

argparse

orbited

math instruction in APL

if you want to start a conversation, ask a question

Insufficient, Inaccurate and Inconsistent

The data I mean...

The job of the geophysicist, I was told yesterday at a seminar given by Mrinal Sen, is essentially impossible. It is, given data that is insufficient, inaccurate and inconsistent to come up with a useful model.

Interestingly, it is unlikely that the skeptics' squad will make much of this admission, since the utility of the models Sen is specifically interested in is whether they correctly inform placement of oil well drilling efforts.

Nevertheless wells do get drilled. In the past it was a matter of some combination of analysis and intuition. Nowadays, statistics works in there as well.

Does climatology partake of these fundamental problems? Not as much as seismology, really; our observations are relatively accurate and consistent compared to seismic data. They are far from sufficient for long time scale processes, and the formalism of use of paleodata still leaves much to be desired.

Nevertheless, the future will happen, and we will do something about it. The question at hand for climate modelers and their customers is to what extent the models ought to affect what we do.

Computational geosciences and computational biological sciences are very different from computational engineering in flavor. In engineering (aside from civil and architectural, which partakes slightly of our problems) the system is entirely controlled and hence the tradeoffs between model expense and model accuracy are well understood. By contrast, our problem is to establish the right model based on observational data and theory.

This is called "inverse modeling" in some circles. I dislike the name and regret the loss of the appropriate term "cybernetics" to Hollywood and to some rather ill-defined corners of computer science. I propose, therefore, to call what some computational sciences do "meta-modeling", wherein the model itself (and the nature of its relationship to observation and theory) is construed as a legitimate object of study.

It is interesting how well-established this modality of thought is in mineral exploration (where there is, after all, a bottom line to take care of) and how controversial it remains in climate science. I have some thoughts as to why this might be. Some of the problems are directly a consequence of the unfairness of the unfair criticisms to which we have been subjected; this makes a fair assessment of fair crticisms fraught with peril. Others emerge from the history of the field and the social structures in which the work is performed.

It seems obvious to me that if the computational resources applied to climate go up several orders of magnitude, the correct approach to this is not, or at the very least not solely, to build a small number of immensely complex modeling platforms which use all the available computational power. Rather, it is to apply the (expensive but powerful) inverse modeling methodology to explore the model space of simpler models; to improve the expressiveness of model description languages; and to use these tools to build a much larger and more formal ensemble of climate modeling tools.

I also have a question whether the most policy-relevant models and the most science-relevant models are necessarily the same. I suspect otherwise. Climate modelers are so defended against the accusation of heuristics that they fail to apply the heuristics that might apply in an applied science effort.

Sen presented an interesting taxonomy of inverse methods at his talk. Much more to follow on this subject, hopefully.

Computational Silence

Still not sure whether to take Mr. Hughes seriously, even though he echoes some of my points. His latest looks pretty confused at first reading but it's far from my expertise. Maybe Eli will have a look.

That all said, I am not one who says all is right with computational climatology.

My friend JL sends along an article by (not captain) James Quirk that does a good job outlining the quandary and some of the efforts toward a solution. Climate science is particularly backward in adopting these measures. There's a misperception that all GCMs solve the same problem that is understandable in the public but hard to account for within the field.

The article is called "Computational Science: Same Old Silence, Same Old Mistakes". It appears in a volume on AMR. It is behind a Springer firewall, so if you can't get the PDF note that Google will display a scan of it.

(Remember, rationally, the less we trust the models the more severe our policy constraints on modifying the system should be.)

Why Pencil Science?

This blog has, at least for now, wandered from its original intent, but its original explanation of purpose is worth saving. I do think computer science is for everybody, not just middle school kids but even climatologists...

This is a blog about the nuts and bolts of computer literacy from a
Pythonic perspective.

I would like not merely to defend the position that computer
programming is "for everybody" (including children, intelligent
adults, and scientists) but also to examine the proposition that
Python is the appropriate vehicle for such a de-professionalization of
programming.

Of course, "computer science is not about computers" but about
formally expressing processes using symbols. The misnomer is
consequential in how most people think about the question of the role
of programming in education.

We don't teach children how to write because we expect them all to be
professional writers. Professional writers are not opposed to literacy
on the grounds that they will lose readership. We do not call the
ability to write "pencil science".

Is writing (pencil science) for everybody? That wasn't clear a
thousand years ago. Can "computer science" be for everybody too?

I'll be reorganizing my online presence soon. Still thinking about what to do...

Bad month index=36; help, you prawns!

Recall we are trying to get a sensitivity to 2xCO2; the public thinks of this as the primary use case.

Most of what was hanging me up for the past week was building a 6Mb one-time data file (containing climatological mixed layer depth, clamped at 200m; the magic number is somewhat arbitrary and hard-wired, but we suppose others have put thought into it.) It would have been nice if I could just download the file. It turned out I needed a c compiler compatible with the Fortran compiler that built netcdf. The magic word was "icc", specifically

setenv CC=icc

Once that was done, it turned out I had some other data file to dig up. I had trouble tracking down my ncar UID/password (as if this data needed to be kept under wraps for some reason). Then I found that the URL specified in the README was incorrect! Eventually tracked the file down with Google. Built (huzzah) the heat flux boundary condition. Thought I was done.

Today I tried to submit the job. Well, still a use case I;ve never run. I was hung up for some time on the error messsage "BNDDYI: Bad month index=36" which is not mentioned in CAM documentation but is mentioned in the CCM3.6 documentation! (an ancestor)

It seems the month number should be a number between 1 and 12 (reasonable). No idea where the number 36 came from. I look into my data file and see the following for dates:

time = 116.5, 215, 316.5, 416, 516.5, 616, 716.5, 816.5, 916, 1016.5, 1116, 1216.5 ;

There are twelve of them as needed, but...

Those are indeed months in a peculiar sense. Consider it a puzzle, and here is your clue:

double time(time) ;
time:units = "day as %m%d.%f" ;
time:axis = "T" ;

Go figure. No clue where the 36 comes from. Charles advises, though, that I should not specify a start and end date for a slab run. So I make no effort to surgically fix the month numbers and remove the start and end date, specifying a duration instead. (Because I specify in days rather than seconds, I use a negative number...)

The 36 BNDDYI message goes away! Apparently the error "month number is 36" really means "you can't specify dates on a slab run".

The model gets considerably further now. It starts to run the first time step and then proffers this droll message a few times

sublimate away all sea ice
something is probably seriously wrong
ice state at dh stop nstep = 0

amid some other unexpected output, and then aborts.

Three weeks and counting. We wanted a result by next Monday and this will take 48 hours to run. Groan.

Somebody else has come to this pass, describing it as "display the seriously error". Note the useful assistance provided by the community. There's also something in Chinese that Google translates as follows:

help you prawns, the slab ocean model CAM operations serious mistake!

-=-=-=-=-=> Turn CCM and CAM counterparts all know, by adding a simple ocean model (slab ocean model) running CAM, but regardless CAM3.0, or CAM3.1, there are the same mistakes can not continue to run, the Daxia Please help!

Any prawns out there have any advice for me?

Update: Running at last! The last hurdle was knowing that you have to replace the file

cami_0000-09-01_64x128_L26_c030918.nc

with the file

cami_0000-01-01_64x128_T42_L26_SOM_c030918.nc

Unlike the mixed layer depths, this is actually distributed in the initial conditions dataset. I am also using a topography file that differes from the standard one. Not sure why, this is based on lore and not knowledge. It's called

topo-from-cami_0000-01-01_64x128_T42_L26_SOM_c030918.nc

I don't know its provenance, but possibly the name is correct. If that's the case, though, it isn't clear why there are two different files at all. Some totally undocumented use case perhaps?

This is actually consistent with the message from "seriously message" guy. It is not documented as part of the procedure for the use case in the documents. There is a tiny clue in the redme for defineqflux:

Another executable available (but hopefully unnecessary) in the definesomic
module is (also) called definesomic. It adds necessary SOM fields to an
initial dataset if they are not already there.

If there are fields missing from the initial conditions, you'd think something better than "something is seriously wrong" and lots of numerical output would be possible.

Not sure how much further to investigate.

My Deep Naivetee

I've made a couple of comments on Bryan's blog early this morning. One is a continuation of my usual ranting, and another is an expression of confusion about what web services have to do with anything scientific coders care about.

Meanwhile I am amused to see my position described as "deeply naive" in a blog by Dan Hughes that I haven't spotted before. It seems worth following for the in crowd.

I don't, actually, think a ground-up rewrite would be easy, and I have some ideas for a less ambitious approach to prove the pudding for a higher level abstraction approach. We'll see what NSF says about it.

However, I think the tenacious attachment to the code which Dan himself goes so far as to describe as "very olde, very poorly constructed, undocumented, and yet very actively used" (much further than I would go) is damned peculiar. It seems to imply that what we are doing doesn't matter. Well, if it doesn't matter we should quit, and if it does matter we should bite the bullet and write something clean and maintainable and (dare I say it) even literate.

In the end this matter is too important for anything less than maximal transparency. It completely baffles me that this goal is so thoroughly outside the field of vision of practitioners. I think it's fundamental.

Update: Whoops. I mistook what Dan Hughes was about. He isn't attached to the code. He doesn't work with the codes, he just snipes at them from a distance. He's an "auditor". Likely, he doesn't want to believe that climate modeling is feasible, or at least is playing to an audience that thinks like that. One can hope he does not reach the illogical conclusion that climate protection is not a useful goal of policy pending the substantial improvement of such models.

Let me make it very clear from observing real productive climate scientists that extant models are useful tools for science, and that the scientists are well aware of their imperfections and flaws.

This doesn't mean they/we should be immune from criticism, nor that we have a clear sense of how useful the extant models are for projection. I have a lot doubts on that score, but I take the rational risk-weighted response of being more worried about our collective future, rather than less so, as a consequence.

Further Update: That said, the discussions on Hughes blog are not without value. I found this one especially interesting.

Further Update: While it might appear that there is no intent by Hughes to provide a constructive alternative (see comment #6 to this thread) he does in fact offer us up a strawman proposal for an alternative approach here (linked from #7 in the same thread). His server is slow and unreliable, but it's there on occasion. Reading...

Life in the trenches III

One of my ensembles has died. The run dropped an error file as follows (excerpt):

print_memusage iam 47 stepon after dynpkg. -1 in the next line means unavailable
print_memusage: size, rss, share, text, datastack= 52301 22403 4114 806 0
print_memusage iam 13 stepon after dynpkg. -1 in the next line means unavailable
print_memusage: size, rss, share, text, datastack= 52441 22616 4121 806 0
print_memusage iam 36 stepon after dynpkg. -1 in the next line means unavailable
print_memusage: size, rss, share, text, datastack= 52135 24788 5987 806 0
[0] [MPI Abort by user] Aborting Program!
done.
/opt/lsf/bin/ibrun: line 6: 545 Killed pam -g 1 mvapich_wrapper $*
[0] [MPI Abort by user] Aborting Program!
done.
/opt/lsf/bin/ibrun: line 6: 13870 Killed pam -g 1 mvapich_wrapper $*
[0] [MPI Abort by user] Aborting Program!
done.
/opt/lsf/bin/ibrun: line 6: 3813 Killed pam -g 1 mvapich_wrapper $*
[0] [MPI Abort by user] Aborting Program!
done.

The error file is 1132 lines long. Python would give me more legible information.

I am simply going to hope it has something to do with Lonestar's I/O problems. I need to rerun the exact case, of course.

The ensemble controller somehow knew something was awry. It wasn't written by a very seasoned programmer and I'm eager to rewrite it but the time never shows up. Fortunately this time it appears to have done the right thing.

Meanwhile

• Charles has convinced me that he really has something going with his current proposal so I am trying to help firm up the text (for practical purposes, on my own time)
  
• I still have yet to get sensible output from defineqflux, or understand how to build it for that matter (see Trenches II)
  
• I need to back up my expired files from Chicago
• I need to document and version control my December work
• I need to finish building the NCAR diagnostic toolkit now that NCL works
• round up speakers for scientific software day
  
• somehow we need a local MPI install; yow!
  
• eventually we need a CCSM build on a TACC platform, but don't hold your breath

This is on halftime! I'm not even supposed to be in on Mondays!

Will I ever write a line of my own code again?

Update: The error file says nothing whatsoever. Successful runs have the same errors, up to and not including the Abort messages, where instead they contain 64 lines consisting of "0".

Why Python?

Other people have other reasons, but mine is succinctly summarized by Paul Prescod in his article On the Relationship between Python and Lisp. Here's the core of the argument:

Paul Graham says that a language designed for "the masses" is actually being designed for "dufuses". My observation is that some of the smartest people on the planet are dufuses when it comes to programming, (or in some cases just dynamic programming languages) and I am pleased to share a language (and code!) with them.

I usually spend a big chunk of my day in Python. But most professional programmers will not be able to do that until Python is a dominant language. In the meantime they must switch back and forth between Python and the language of their day-job. Python is designed to make that easy. During the day they can pound out accounting code and at night switch to hacking distributed object oriented file systems. Every intuititively named function name makes it that much easier to "swap" Python back into your brain. After we have been away from a language for a while, we are all dufuses, and every choice made in favor of the duffers actually benefits even high-end programmers.

I get paid to share my code with "dufuses" known as domain experts. Using Python, I typically do not have to wrap my code up as a COM object for their use in VB. I do not have to code in a crappy language designed only for non-experts. They do not have to learn a hard language designed only for experts. We can talk the same language. They can read and sometimes maintain my code if they need to.

From a purely altruistic point of view, it feels good to me to know that I am writing code (especially open source code) that can become a lesson for a high school student or other programming beginner. I like to give programming classes to the marketing departments at the companies where I work.

Even hard-core geeks get a certain aesthetic pleasure out of using something that feels minimal and simple because most of the complexity has been hidden away or removed.

Code is a communication medium. The machine is not the only reader. Python is the only language explicitly designed with both beginners and experts in mind. This has direct benefits for the transition from beginner to expert, and it also has direct benefits for development collaboration among groups with distinct expertise.

Here is an example. I have never taken a compiler course and I still consider code compilation to be deep magic, though not as much as I did before Python began boosting my sophistication. Nevertheless, I can understand and appreciate the following.

---

# romanNumerals.py
#
# Copyright (c) 2006, Paul McGuire
#

from pyparsing import *

def romanNumeralLiteral(numeralString, value):
return Literal(numeralString).setParseAction(replaceWith(value))

one         = romanNumeralLiteral("I",1)
four        = romanNumeralLiteral("IV",4)
five        = romanNumeralLiteral("V",5)
nine        = romanNumeralLiteral("IX",9)
ten         = romanNumeralLiteral("X",10)
forty       = romanNumeralLiteral("XL",40)
fifty       = romanNumeralLiteral("L",50)
ninety      = romanNumeralLiteral("XC",90)
onehundred  = romanNumeralLiteral("C",100)
fourhundred = romanNumeralLiteral("CD",400)
fivehundred = romanNumeralLiteral("D",500)
ninehundred = romanNumeralLiteral("CM",900)
onethousand = romanNumeralLiteral("M",1000)

numeral = ( onethousand | ninehundred | fivehundred | fourhundred |
       onehundred | ninety | fifty | forty | ten | nine | five |
       four | one ).leaveWhitespace()

romanNumeral = OneOrMore(numeral).setParseAction( lambda s,l,t : sum(t) )

print romanNumeral.parseString("XLII") # prints "42"

---

Try doing that in a dozen or so lines of C++ or Java, mate, so that a random reader could get half a clue as to what was happening! Yes of course the "import" statement hides a great deal of magic, but that's the whole point, see?

Life in the Trenches: Part II

This story is probably more illustrative (than is Part I) of some of our productivity sinks:

We have a modified set of CAM parameters that demonstrably improves climate fidelity, and are looking to establish its CO2 doubling sensitivity. This is an atmosphere-only model with six of its magic numbers tweaked by our (UTIG's, i.e., Sen, Stoffa and Jackson's) smart ensemble search algorithm. Our suspicion is that we will see a sensitivity closer to 3 C (increasingly regarded as the most likely value) rather than NCAR's 2.4 C.

Now, CO2 sensitivity is a very common use case, perhaps the most famous of all. However, rather than a data ocean it requires an interactive ocean in order to make any sense.

This use case is so common it merits a section in the User Guide.

You will note that the description of what is going on is strikingly terse. If you haven't hung around climate modelers you will not be able to parse it at all, I'll wager.

So note if you can or take my word if you must that step 1 seems more or less model independent, though admittedly it is resolution dependent. However, since CAM only supports a few resoilutions, it's unclear why I should have to jump through the hoops.

Nevertheless, deep within the CAM file hierarchy you do find the directory in question with a perfectly fine Makefile to build definemldlxl, definemldbdy and defineqflux, which in my case naturally doesn't work.

You sort of have to know to tell Lonestar module load netcdf and then env to get the netcdf paths from the environment, which you can then use to create the environment variables the NCAR buld file expects. None of this is documented anywhere, and this is enough to get through the compile phase but still no joy at link time.

/opt/apps/netcdf/netcdf-3.6.1/lib/libnetcdf.a(attr.o): In function `dup_NC_attrarrayV':
/home/build/rpms/BUILD/netcdf-3.6.1/src/libsrc/attr.c:199: undefined reference to `_intel_fast_memset'

Scratching my head on this one. Maybe someone at TACC can help; will file a ticket and see what happens.

Meanwhile, we have tracked down an executable for the last one in a directory of a worker who has left and his data file for the first phase; we are hoping that is good enough. Note, though, that the docs do not say what fields need to be in the output of the file you pass to defineqflux. It turns out that the ones we normally save aren't the right ones.

No problem, you say. This is a runtime variable ('namelist' variable in fortranese) and I won't have to rebuild the whole model for that, will I? I'll just add the missing fields (rummage through the source code to find them... rummage, rummage...) and Bob will be my uncle.

Well actually this would have cost me two days, except that Lonestar lost two days to a panic and I was off Monday, so it cost me a whole week. You see, if you do a restart run, (i.e., a continuation of a prior experiment) the fortran will happily ingest your list of output variables and ignore them (without any message to that effect) in favor of the list you used in the spinup run. No, to have your namelist read and actually used, you have to to a "branch" run rather than a "restart" run, which requires significantly different set of namelist parameters that is essentially ill-documented. In fact if you try to do this based on the CAM manual, you will not find sufficient information. The clew, as Sherlock Holmes would say, is here

This appendix describes a small subset of the namelist variables recognized by CLM 3.0. For more information, please see the CLM User's Guide.

Aha; well that does help. After a while you understand that the namelist contains two (!) definitions of NREVSN (the restart file name) in "branch" mode (the namelist apparently can have multiple namespaces, though Fortran jargin calls them some other thing most likely), whereas in "restart" mode it suffices to use the default values for REST_PFILE which is a small text file containing what in the other case is the value of NREVSN.

So by close of business today we should be able to get the flux file we need to actually run the SOM version of CAM, the building of which was another story but it's in the past so I won't bother you with it.

Update: Still haven't pursued the build question.

After rerunning the spinup, the run of defineqflux fails, but later in the process, with the message:

nc_put_vara_double -1 failure: Variable not found.

So another 12 hours spinup run needed, once I figure out what it needs. This is a good example of a bad error message. In a real language it would not be too much trouble to report which variable wasn't found and where. In a reasonable document the requisiste fields would be listed. If I were more competent I'd have done a better job perusing the source. 0 for 3.

Update 1/28: Used default outputs and set it running on Friday. Trioed to run defineqflux today with the same error. I don't think it's a matter of which fields I use. Perhaps the version of defineqflux which I have is defective. Still don't really know how to build it so though it is only modestly complicated C code, I can't debug it.

...
Using SST mid-months 8 and 10 to compute QFLUX mid-month 9
dayfact=62
Using SST mid-months 9 and 11 to compute QFLUX mid-month 10
dayfact=60
Using SST mid-months 10 and 0 to compute QFLUX mid-month 11
dayfact=62
nc_put_vara_double -1 failure: Variable not found
Abort


Update 1/29: I have managed to build against a NetCDF library that I myself built from scratch. Still trying to resolve the error, but at least I can put some tracers in the code now.

Meanwhile, a ticket is pending at TACC for how to build against their library.

Update 1/31: Still no joy. Looks very much as if I need the other pieces of the tool chain. But they require the fortran as well as the C libraries. Linking against TACC's library fails as I reported some time ago. My own build of Fortran NetCDF yields

vcc works; but missing link magic for f77 I/O. - NO fort. None of gcc, cc or vcc generate required names. - f2c : Use #define f2cFortran, or cc -Df2cFortran - NAG f90: Use #define NAGf90Fortran, or cc -DNAGf90Fortran - Absoft UNIX F77: Use #define AbsoftUNIXFortran or cc -DAbsoftUNIXFortran - Absoft Pro Fortran: Use #define AbsoftProFortran - Portland Group Fortran: Use #define pgiFortran - PathScale Fortran: Use #define PATHSCALE_COMPILER"
make[3]: *** [fort-attio.lo] Error 1
make[3]: Leaving directory `/home/utexas/ig/tobis/CAMYA/cam1/models/atm/cam/tools/defineqflux/NCDF/netcdf-3.6.2/fortran'

so here I sit helpless. Two weeks since starting to try to run a standard use case and I am set back to trying to build system-supported libraries. TACC has been episodically helpful but is presently silent.

I am going to try from the beginning on a UTIG machine next.

The Cybernetic Perspective

In the original sense, "cybernetics" means optimal use of information in decisionmaking.

It emerges that to develop a statistics of information requires a change of perspective from the simple one where developing the model is informed by the system under study to one where the model and the system under study are two components of a larger system. The Wiener filter and its descendant the Kalman filter are the best known examples of this approach but it is more general.

I believe this use of the word "model" is not entirely coincident with the software engineering sense as in "model-driven development" though it does mesh more or less well with "Model-View-Controller". In our world the View is relatively trivial and usually done offline as postprocessing; it's the utter absence of the Controller that troubles me.

(Update: That last is really a stretch on second thought. Probably more sellable is the idea that the M in MVC corresponds to system state (variables) and the C corresponds to executable code (statements). In the end the word "model" is just as much a source of confusion as ever.)

Anyway I think the cybernetic perspective has value in getting climate modeling unstuck.

Quote of the Day

Chris McAvoy on the ChiPy list:

I don't know for sure, but from a "Zen of Python" perspective, when I
can't do something with Python it usually ends up that what I'm trying
to do shouldn't be done.

Life in the Climate Trenches: Part 1

I've got not one but two build problems going on. If the rest of my life weren't going well I'd be awfully depressed. I really hate wasting time on this sort of mundane frustration.

Lonestar (the supercomputer) is offline for alternate Tuesday maintenance, so for today I will share build Probleme Numero Un with you. I welcome any skill transfer that can help untangle all this; comments welcome. Rather than polluting the blogspace with innumerable followups I will just provide updates to this entry.

Tomorrow I'll start documenting Probleme Numero Deux: running a standard CAM use case.

It may be that we'll more be documenting my own ignorance than any fundamental problem with the software from some point of view; that will be a learning experience just the same.

You may object that this isn't the sort of thing you want your tax dollars paying PhD's to fumble around with; in that case I would be in agreement with you. In the private sector this would clearly be a systems task and not a scientist task. So I appeal to your mercy as a lousy admin and a potentially productive scientist, albeit of a somewhat eccentirc stripe. The sooner I get these things done the sooner I can be a scientist.

OK, here's the scoop (unix/posix skills required to make any sense out of this):

To participate in NCAR CAM development we need to use their diagnostic package which in turn depends on NCL. Having a legitimate connection to a US research institution, I have long since jumped through the hoops and am registered on the Earth System Grid. So I download the package and all its dependencies.

OK, so the UTIG disk systems are all cross-mounted and so, though I have root on the climate research machine I don't have write privileges on /usr . Our excellent systems folk have set it up so that climate users just define $CLIMATE in their .cshrc which will invoke anything I put (or conceivably somebody else puts) into /usr/local/climate so all I have to do that is nonstandard is insert "climate/" into the path; I make my own ../bin ../lib ../man in there; everyone interested in climate ends up with it in their path, nobody else has their namespace messed up, a very nice approach.

Alas, the lengthy but very clear build instructions do not work (this is standard practice with NCAR products alas; apparently they lack the skills and/or funding to actually make their products portable, though the build instructions are usually very hopeful.) In the present case the snag occurs even before I get to NCAR's stuff.

NetCDF is already available. I have acquired and build jpeglib and zlib as follows:

cd jpeg-6b
make clean
./configure --prefix=/usr/local/climate
make
make install

which puts five executables into /usr/local/climate/bin and

cd zlib-1.2.3
make clean
./configure --prefix=/usr/local/climate
make
make install

which puts a library into /usr/local/climate/lib

My current snag is building HDF-4, which in the configure phase always stops at:

checking zlib.h usability... yes
checking zlib.h presence... yes
checking for zlib.h... yes
checking for compress2 in -lz... yes
checking jpeglib.h usability... no
checking jpeglib.h presence... no
checking for jpeglib.h... no
checking for jpeg_start_decompress in -ljpeg... no
configure: error: couldn't find jpeg library

I have tried various combinations here. What seems to me ought to work is

./configure --prefix=/usr/local/climate --with-zlib=/usr/local/climate \
--with-jpeg=/usr/local/climate --disable-netcdf

but it seems to ignore the prefix altogether. I change --with-zlib to point to a totally bogus address, and yet it finds the zlib stuff anyway. It seems to be ignoring the flags I am setting.

Working hypothesis: I am building jpeg wrong, and some system setting is overriding my value of --with-zlib.

A less satisfying but easier solution could come from downloading binaries. At this point I simply have to swallow my pride and ask which if any of the binaries listed here might work.

Advice would be even more welcome than sympathy.

Update 1/24:

UTIG support advised option 4, which worked more or less smoothly.

Also, from UTIG support:

Oh I didn't see you had your own (up a directory). The command you
were looking for would be

./configure --prefix=/usr/local/climate \
--with-zlib=/usr/local/climate/NCLc3/zlib-1.2.3 \
--with-jpeg=/usr/local/climate/NCLc3/jpeg-6b --disable-netcdf

And I'm idiotically proud of this script, my first conditional in a shell script in a long time:

set hn = `hostname`
if ($hn == 'foo.whatever.utexas.edu') then
echo "climate paths added"
setenv NCARG_ROOT /usr/local/climate/NCLBin/
set path = ( /usr/local/climate/NCLBin/bin/ $path )
endif


So there's finally a happy ending on this one. TACC is still having hardware or configuration problems with Lonestar which is preventing any progress on the other front.

Language of Choice

Presumably everyone knows that Python is on the rise with a bullet. Of course, now that Python is sexy it will probably attract some weaker programmers, diluting its reputation. Such are the costs of great beauty. Right now, though, the upswing seems to be near its maximum rate.

In the Slashdot article discussing this I saw a ref to a lexicon definition the concept of Language of Choice in the Jargon File. FORTRAN is dismissed as a niche language for scientists, but nothing better is proposed.

So what should the language of choice be for high performance numerical computing?

There are teams working on replacements (Fortress, Chapel, some IBM thing with a forgettable name), but I think much of the high performance work of the future can be done in Python with a code generation strategy. I'm not sure what the target language ought to be for the code generation. Would there be advantages to Fortress or Chapel for this? For contemporary purposes even F77 would work (as I said in the PyNSol papers) but to be honest the F77 target is almost a stunt, just to make a point. I don't expect it would be useful in the long run.

I am currently thinking most of the system should be agnostic as to the target language. If done right, multiple targets can be supported in an architecture-dependent bottom layer. Even Verilog is a candidate. ( F90 and onward are not. I'd sooner use BF or Whitespace. )

Update: Bryan Lawrence has a nice summary of the issues.

Do NCAR or GISS Ask for This?

This crossed my path on a CS mailing list I follow. The context is about whether students need more software engineering or more computer science. It's a bit harsh on us older types but it certainly is an interesting list. I post it so scientific computing types can consider what they may or may not be missing. I should also note that there are practicing scientific coders for whom I have great respect who would find this list ludicrous.

As a software engineer in the industry and as a person who hires developers I have to say that 2 out of every 3 interviews I take from people with a CS or MS in computer science I do not hire because they are completely inadequate as software engineers. More often than not, they do not know about or have distaste for one or more of the following:

• Design Patterns
• Agile engineering / programming
• Extreme Programming
• Test Driven Development
• Composition vs. Inheritance

We also give a programming test as a part of the interview process, it is mathematically / algorithmically moderate in challenge and most interviewees (80%) can solve the problem, but only 15-20% solve it with good programming practices and maybe 5-10% write unit tests. Actually, if an interviewee who has good engineering, but does not solve the problem is more appealing than someone who solves the problem.

I do recall from my undergraduate program at Purdue that we did have a "Software Engineering" class that did try to drive home the points I mentioned, but I have to say that it did not go far enough.

I would never diminish CS courses, mathematics, or general studies programs ever. I would say that anyone who only takes their CS and never masters the engineering principles either through other course work (several hours worth) or through an internship does not stand much of a chance getting hired at the company I work for.

In the end, I would say that if I knew 6 years ago what I knew now, I would have gone for the software engineering program and tried for a dual major in CS if I had the bandwidth.

For what it's worth I believe that design patterns in practice are overstressed, and that the list isn't especially elegant or complete. That said I also have the impression that these ideas are only very dimly perceived in the climate modeling centers and a great deal of benefit could be gained from them.

Progress

OLPC is a great achievement if only for this story:

"Microsoft struggles to port Windows to a device originally conceived to run Linux."

Still waiting on mine...