Moving notes to self

Notes to self or those interested in my daily pain will be moved to http://aardvarknoplay.blogspot.com/ . Old notes being moved.

This blog will actually have content that others may find interesting.

Some Key Papers in Climate Modeling History

A first pass at a greatest hits list; updates will not necessarily be noted as such. I had a request for key papers in the history of modeling so the emphasis leans that way. Nominations welcome.

Charney, Fjortofft & von Neumann 1950: Numerical integration of the barotropic vorticity equation, Tellus 2:237-254

Lorenz 1963: Deterministic Nonperiodic Flow J. Atmos. Sci 20:130-141

Lorenz 1967: The nature and theory of the general circulation of the atmosphere. WMO. http://eapsweb.mit.edu/research/Lorenz/General_Circ_WMO_1967_Part1.pdf

Manabe & Bryan 1969: Climate calculations with a combined ocean-atmosphere model.

Bryan & Cox 1972: The circulation of the world ocean: a numerical study. Part I, a homogeneous model J Phys Oceanog
Arakawa & Lamb 1977: Computational Design of the basic processes of the UCLA general circulation model. Methods in Computational Physics 17:173-265

Charney et al 1979: Ad Hoc Study Group on Carbon Dioxide and Climate. http://www.atmos.ucla.edu/~brianpm/charneyreport.html

Manabe & Stouffer 1988: Two stable equilibria of a coupled ocean-atmosphere model

Ramanathan et al 1985: Trace gas trends and their potential role in climate change. J Geophys Res 90:5547-5566

Cess et al 1990: Intercomparison and interpretation of climate feedback processes in seventeen atmospheric general circulation models

Bryan 1991: Poleward heat transport in the ocean: a reviwe of a hierarchy of models of incerasing resolution. J Phys Oceanog 18:851-867

Cubasch et al 1992: Time-dependent greenhouse warming computations with a coupled ocean-atmosphere model

Hansen et al 1992: Potential climate impact of Mount Pinatubo eruption GEOPHYSICAL RESEARCH LETTERS, VOL. 19, NO. 2, PP. 215-218, 1992
doi:10.1029/91GL02788

Hasselmann 1997: Multi-pattern fingerprint method for detection and attribution of climate change. Climate Dynamics 13:601-612

Hansen et al 1998: Climate Forings in the Industrial Era. PNAS

Bengtsson 1999: From short-range barotropic modeling to extended-range global weather prediction. Tellus 51 A-B:13-23

Randall 2000: General Circulation Model Development: Past, Present, and Future (International Geophysics) [Hardcover]

Hansen et al 2005: Earth's Energy Imbalance: Confirmation and Implications. Science.

Hack et al 2006: Simulation of the Global Hydrological Cycle in the CCSM Community Atmosphere Model Version 3. J Clim Volume 19, Issue 11 (June 2006)

Schmidt et al 2006: Present-Day Atmospheric Simulations Using GISS ModelE: Comparison to In Situ, Satellite, and Reanalysis Data. J. Climate, 19, 153–192.

Rahmstorf et al 2007: Recent Climate Observations Compared to Predictions. Science 316:709 DOI: 10.1126/science.1136843

More Adventures in Climate Analysis Installs

So here I document getting Ferret to work on a Mac; getting it to run on the super is another problem. I assumed no privileges and more or less succeeded. However, the install instructions are (as is normal with scientist code) broken.

Here are the absurd installation instructions:

http://ferret.pmel.noaa.gov/static/Downloads/mac_intel_downloads.html

This is basically just three useful links to the actual binaries under "Environment and Executable files"; ignore the rest of it.

Then proceed to

http://ferret.pmel.noaa.gov/static/Downloads/ferret_installation_and_update_guide_v600.html

If you don;t knwo about environment variables in bash, go study up on them.

The first step 1 should be

tar xf $GET_LOCATION/fer_environment.tar

The second step 1 and step 2 [sic] work as noted. Step 3 should be

tar xf $GET_LOCATION/fer_dsets.tar

as the Mac ungzips behind your back. Your next step will have to be backwards, as the Finstall script expects the exectuables to be gzipped. So go back to your download directory and type

gzip fer_executables.tar

Also, you will have to find a place for an extra file in the middle of the next step. Be sensible, create a ~/.Ferret directory before doing the next step. Use that for your $SET_FER .

Now you can run the Finstall per instructions. But the script you ghet out will be a csh script not a bash script. The following edits worked for me:

export FER_DIR=/Users/tobis/Ferret
export FER_DSETS=/Users/tobis/FerSamp

#   if ( $PATH !~ *ferret* ) then
#       export PATH={$PATH}:$FER_DIR/bin
#   else
#        echo replacing ferret path
#      export PATH=`echo $PATH | awk -F: '{for (i=1; i<=NF; i++) {if ($i \!~ /ferret\/bin/) {printf "%s:",$i}}}'`
#      export PATH={$PATH}$FER_DIR/bin
#   endif
 
# =========== Initially make no modifications below this line ===========

#  Default Ferret document browser
#   export FER_WEB_BROWSER="netscape -ncols 60"
   export FER_EXTERNAL_FUNCTIONS="$FER_DIR/ext_func/libs"
   export FER_GO=". $FER_DIR/go $FER_DIR/examples $FER_DIR/contrib"
   export FER_DATA=". $FER_DSETS/data $FER_DIR/go $FER_DIR/examples $FER_DIR/contrib"
   export FER_DESCR=". $FER_DSETS/descr"
   export FER_GRIDS=". $FER_DSETS/grids"
   export TMAP="$FER_DIR/fmt"
   export PLOTFONTS="$FER_DIR/ppl/fonts"
   export SPECTRA="$FER_DIR/ppl"         # for old ferret versions
   export FER_PALETTE=". $FER_DIR/ppl"       # palette search list

So basically, change all the setenvs to exports, put in the equal sign, comment out the if/else block. If you are better with bash scripts, the if/else thing looks harmless.

I did not uncomment the reference to netscape. I don;t know where to get a netscape for os X 10.6 anyway. Sigh.

This actually works!

Meanwhile Chas has bunches of hoops for me to jump through. Will keep you posted, if you exist.

PyNGL Even worse than CDAT?

Undecided as yet. More to follow. But plenty of the same signs of cluelessness.

I am still hopeful that the interpolation routine I need will work. And the install is much more straightforward, there being suitable binaries for the machines I care about.

But the test routine starts with an absolute path to a python installation at NCAR!

And this is unpythonic, though perhaps this is a Fortran 77 constraint rearing its ugly head

import Ngl
import Nio
import numpy

# create an array and make a netcdf file

ff = Nio.open_file("foo.nc","c")
ff.create_dimension("x",10)
ff.create_dimension("y",10)

ff.create_variable("u","f",("x","y"))

z = numpy.zeros((10,10),"f")

# ff.variables["u"] = z doesn't work

ff.variables["u"].assign_value(z)

# assignment is by value, as the name indicates

z[3,3] = 3.3

ff.variables["u"][4,4] = 4.4

# but after assignment you can pick up a reference

u = ff.variables["u"]

u[2,2] = 2.2

"""
>>> ff.variables["u"][:]
array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  2.2,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  4.4,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]], dtype=float32)
"""

ff.close()

Anthony Doerr

Read the whole thing. It starts

During my sophomore year, 1992, 1,500 scientists, including more than half the living Nobel laureates, admonished in their Warning to Humanity: “A great change in our stewardship of the earth and the life on it is required if vast human misery is to be avoided and our global home on this planet is not to be irretrievably mutilated.”

So what have we done? Not much. From 1992 to 2007, global CO2 emissions from burning fossil fuels rose 38 percent. Emissions in 2008 rose a full 2 percent despite a global economic slump. Honeybees are dying by the billions1, amphibians by the millions, and shallow Caribbean reefs are mostly dead already.2 Our soil is disappearing faster than ever before, half of all mammals are in decline, and a recent climate change model predicts that the Arctic could have ice-free summers by 2013. Unchecked, carbon emissions from China alone will probably match the current global level by 2030.

“The god thou servest,” Marlowe wrote in Dr. Faustus, almost four hundred years before the invention of internet shopping, “is thine own appetite.” Was he wrong? How significantly have you reduced your own emissions since you first heard the phrase “climate change?” By a tenth? A quarter? A half? That’s better than I’m doing. The shirt I’m wearing was shipped here from Thailand. The Twinkie I just ate had 37 ingredients in it. I biked to work through 91-degree heat this morning but back at my house the air conditioner is grinding away, keeping all three bedrooms a pleasant 74 degrees.

My computer is on; my desk lamp is glowing. The vent on the wall is blowing a steady, soothing stream of cool air onto my shoes.

h/t Andrew Sullivan. Anthony Doerr, whom I had not heard of until today, lives in Idaho, writes "on Science" column for the Boston Globe, and is a 2010 Fellow of the John Simon Guggenheim Memorial Foundation.

Exploring NIO

The PyNIO install from binary went more or less without incident; I had to dig up a conformant numpy (1.5 is out but the binary required 1.4)

I am deferring a PyNGL install until I figure out what PyNIO can do.

So now I have py 2.5.1, numpy 1.4.1, Nio 1.4.0 and a doc.

The main doc is here

Not hard for me to dig up a netcdf file from the CAM distribution; let's take a 2 D dataset from the land surface model. No idea what is in it.

-rwxr-x---  1 tobis G-25522 7671124 Oct 22  2009 clms_64x128_USGS_c030605.nc

seems to work! The data object has some attributes set by the data file.

>>> import numpy as np
>>> import Nio
>>> data = Nio.open_file("nctest.nc")
>>> data

>>> dir(data)
['Conventions', 'Glacier_raw_data_file_name', 'History_Log', 'Host', 'Inland_water_raw_data_file_name', 'Input_navy_oro_dataset', 'Lai_raw_data_file_name', 'Logname', 'Revision_Id', 'Run_mode', 'Soil_color_raw_data_file_name', 'Soil_texture_raw_data_file_name', 'Source', 'Urban_raw_data_file_name', 'Vegetation_type_raw_data_filename', 'Version', '__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', 'close', 'create_dimension', 'create_variable', 'set_option']

but how to get at the actual contents?

Ah; the directory is incomplete. Apparently the extension module wasn't Pythonic. This can be patched.

So we can get at dv = data.variables() yielding

['PCT_CLAY', 'LATIXY', 'LANDMASK', 'LANDFRAC_PFT', 'PCT_LAKE', 'LANDFRAC', 'NUMLON', 'LONGXY', 'MONTHLY_HEIGHT_BOT', 'PCT_WETLAND', 'MONTHLY_SAI', 'PCT_URBAN', 'PCT_SAND', 'MONTHLY_HEIGHT_TOP', 'PCT_PFT', 'MONTHLY_LAI', 'SOIL_COLOR', 'PCT_GLACIER', 'PFT']

Now, just printing (__str__) data tells us

   dimensions:
      lsmlon = 128
      lsmlat = 64
...
   variables:
...
      float PCT_LAKE [ lsmlat, lsmlon ]

so dv['PCT_LAKE'] ought to be a 128 x 64 array

but dv[2000,0] works. Apparently an overflow is treated as a [-1] index and an underflow as a [0] index. Not very reassuring.

Exasperatingly, though PyNIO is available for py2.5 and numpy 1.4, PyNGL is not. Must get python 2.6... boring...

Install confusion

Trying to install PyNCL on a Red Hat machine without sudo priv's.

1) Py2.7 from source. Builds fine. Runs fine.

2) numpy from source using aforesaid python. atlas/blas instructions aren't too clear, proceed blindly ahead. build works fine.

3) import numpy or import numarray fails:

ImportError: Error importing numpy: you should not try to import numpy from
its source directory; please exit the numpy source tree, and relaunch
your python intepreter from there.

very little info on Google about this. Try various versions of current working directory. Failure all the same.

4) looks like this code:

if __NUMPY_SETUP__:
    import sys as _sys
    print >> _sys.stderr, 'Running from numpy source directory.'
    del _sys
else:
    try:
        from numpy.__config__ import show as show_config
    except ImportError, e:
        msg = """Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python intepreter from there."""
        raise ImportError(msg)
    from version import version as __version__

is getting invoked. Try setup command, but it tries to install in /usr/local/lib/python2.7

5) How to tell setup where to put the numpy?

setup.py install --help

helps, yielding

Options for 'install' command:
  --prefix            installation prefix
  --exec-prefix       (Unix only) prefix for platform-specific files
  --home              (Unix only) home directory to install under
  --install-base      base installation directory (instead of --prefix or --
                      home)
  --install-platbase  base installation directory for platform-specific files
                      (instead of --exec-prefix or --home)
  --root              install everything relative to this alternate root
                      directory
  --install-purelib   installation directory for pure Python module
                      distributions
  --install-platlib   installation directory for non-pure module distributions
  --install-lib       installation directory for all module distributions
                      (overrides --install-purelib and --install-platlib)
  --install-headers   installation directory for C/C++ headers
  --install-scripts   installation directory for Python scripts
  --install-data      installation directory for data files

It must be one of those!

6) Used "--prefix". setup.py runs without complaint.

Haha! New error. "{PATH}/multiarray.so: cannot open shared object file: No such file"

but the file exists! Now what?

7) Wait - no. I used the system python not the self-installed 2.7. It works! numpy is imported.

8) Now get PyNCL? (ncl is already working on this machine)

NO! Looks like I have to do the numpy thing again, because PyNGL recommends against the g77 compiler and requires gfortran. So my numpy, though working, is wrong.

9) While I'm at it, no BLAS/ATLAS? etc?