Now This Is Interesting: A Different, Larger Dataset at NCDC?

Today, while going over the Temple, Tx station data one more time to find where the 2003 to 2008 data comes from, I stumbled across a document that is very interesting in what it has to say. Before I get indepth on what is in it lets do a quick review on NCDC’s two “main” datasets: USHCN and GHCN.   

Now NCDC tells us there is about 1200 stations in the USHCN v2 dataset, they also have the GHCN dataset which is a worldwide dataset that back in the 50’s, 60′ and 70’s had a station high point of roughly 6,000 stations. Starting in the 80’s, with an acceleraton in the 90’s and all the way up to the mid 2000’s there has been a “dying of thermometers” from the GHCN dataset so that there is now only roughly 1500 stations reporting.     

The USHCN dataset is compiled from CO-OP stations and National Weather Service first order stations. The GHCN dataset is comprised of stations that were brought together back in the early 90’s but then updated by stations reporting in over the World Meteological Organization’s (WMO), Global Telecommunications System (GTS). What is sent across the GTS is something called a CLIMAT report. These reports are supposed to go to the NCDC in Asheville, NC (Not because its a US Government agency, but because it was selected as one of three world wide repositories of world climate data centers see: http://www.ncdc.noaa.gov/oa/wdc/index.php ).  

Now we have been told that the drop out of these stations over the years was due to various reasons and some of those reasons are perfectly reasonable and logical. Examples of this is the drop in stations in Central Africa or in Iraq, both areas of political unrest through the 1980’s, 90’s and up till today. So it is logical and reasonable that loss of reporting stations from those areas, however that never explained the “loss” of all those stations in Canada or Russia. Canada is the biggest elephant in this particular room, since there is no logical reason to have a drop off from there. The Canadian Government maintains over 1400 stations but somehow only about 35 (34 of which are either in the southern part of the country or on a coastline) of those stations find their way into GHCN. Trying to say that Canada doesn’t have the knowledge or techinical ability to have what NCDC calls “live” reporting is an insult to not just the intelligence of the American Taxpayer, but a bigger insult to our neighbors to the north. I am sure that the Canadian government knows how to operate satellite communications systems and about this new fangled invention called the “Internet” and can find enough non idiots that can operate  laptops to update the climate databases since AGW is such a threat.   

Now what if I told you that there is a different dataset, one much bigger and newer (2003), one that is not updated by NCDC but by the US Air Force and then it’s given to the NCDC, but you the American Taxpayer have to pay for access to this dataset (Thats if you somehow stumble accross it since it is not well known like USHCN and GHCN) would you believe me?   Well it’s true there is such a database and here it is:    

National Climatic Data Center     

DATA DOCUMENTATION  FOR     

DATA SET 9950 (DSI-9950)     

DATSAV2 SURFACE    

January 6, 2003    

1. Abstract: DATSAV2 is the official climatological database for surface observations. The database is composed of worldwide surface weather observations from about 10,000 currently active stations, collected and stored from sources such as the US Air Force’s Automated Weather Network (AWN) and the WMO’s  Global Telecommunications System (GTS). Most collected observations are decoded at the Air Force Weather Agency (AFWA) formerly known as the Air Force Global Weather Central (AFGWC) at Offutt AFB, Nebraska, and then sent electronically to the USAF Combat Climatology Center (AFCCC), collocated with NCDC in the Federal Climate Complex in Asheville, NC. AFCCC builds the final database through decode, validation, and quality control software. All data are stored in a single ASCII format. The database is used in climatological applications by numerous DoD and civilian customers.      

http://www1.ncdc.noaa.gov/pub/data/documentlibrary/tddoc/td9950.pdf  

Read that bolded part again: DATSAV2 is the official climatological database for surface observations, not USHCN and not GHCN. Notice how many stations are currently active in this database? 10,000. There was never 10,000 stations ever in the GHCN database and it’s now down to 1500. No wonder NCDC isn’t that concerned about this and brushing it off, GHCN isn’t the official database and the real one has 10,000 active stations in it. Now the question is why isn’t this one as well known as GHCN and why does it cost the US Taxpayer money for something he already paid for through his taxes.  

Now for those that ask obvious questions of “Is this an ongoing project?” and “What about coverage?” but don’t want to hit the link and read it for yourself, here you go:    

3. Start Date: 19300101  

4. Stop Date: Ongoing. 

5. Coverage: Global coverage. 

a. Southernmost Latitude: 90S

b. Northernmost Latitude: 90N

c. Westernmost Longitude: 180W

d. Easternmost Longitude: 180E

Now the question I have is why isn’t this “official” dataset well known like GHCN and why is it behind a paywall instead of being available like GHCN.

Advertisements

5 responses to “Now This Is Interesting: A Different, Larger Dataset at NCDC?

  1. Rod Smith February 20, 2010 at 7:45 pm

    VERY INTERESTING! Old AWN guy here… see: “http://gcmd.nasa.gov/records/GCMD_USAFETAC_SFFMG.html” and notice the distribution is on “11 magnetic tapes” which may explain some of the cost. But even a year’s surfaceobs (and this set doesn’t seem to have any upper-air either) is NOT FREE from NOAA. And 11 mag tapes is a TON of data — much more than one year’s worth.

    USAFETAC is the Environmental Technical Activities Center

    Check out: “http://gcmd.nasa.gov/KeywordSearch/Keywords.do?KeywordPath=%5BKeyword%3D%27DATSAV2%27%5D&Portal=GCMD&MetadataType=0

  2. boballab February 20, 2010 at 8:46 pm

    Thanks for the links. According to what NASA says the database is up to 13,000 active stations and they report Max and Min Temps. It makes one wonder why this dataset is not the one touted.

  3. Rod Smith February 20, 2010 at 10:07 pm

    Hmmm. Max/min temps can be reported in Synoptic code but are not to my knowledge ever seen in METAR.

    As I remember (and the ol’ memory ain’t what it used to be!) a year’s worth of all the world’s METAR and Synoptic observations on a zipped DVD is about $20 from NCDC. My 2008 sfc-ob DVD unzipped to more than 21 GB. It contains data from 13,400 stations but probably half are METAR. Airports could care less about max/min temps.

    And I don’t believe the USAF climo folks care one whit about UHI. They are are telling Generals that, yeah you can probably operate heavy tanks in April in country X, but airdrops may be problematic due high winds, etc. In other words, military climo has little relationship to the current academic versions.

  4. boballab February 20, 2010 at 10:48 pm

    I’m ex Navy and know what you mean about what the military needs compared to what academia needs, but there is something interesting about this dataset.

    The 2003 NCDC document says there is 10,000 active stations, but on the NASA site (which looks more current) they say 13,000 active stations. Now besides the military data they are also inputting data from the GTS(the UN’s climate network)system. Now GHCN is supposed to be made up of data that comes through the GTS system and we have seen a drop in stations between 2003 and now in GHCN. In the DATSAV2 dataset we see an increase of 3,000 stations. Now if my memory is serving me right there is no way there was an increase of 3,000 US military installations world wide so that big increase has to be stations reporting over the GTS system via CLIMAT reports. So if the US Air Force is able to get those WMO station data and incorporate it into it’s dataset, why can’t NCDC (who is suppose to be doing this) get those multiple thousands of stations into GHCN. According to NCDC the DATSAV2 dataset is a better QCed set then GHCN, so it can’t be because the data is of poor quality or to hard to work with.

    Something just stinks big time about this period.

  5. Rod Smith February 21, 2010 at 11:28 am

    I’ve looked on my NCDC 2008 Surface temperatures DVD, and there are only 9 stations with CLIMAT reports. Eight are in Canada and the ninth is on Tuvalu.

    I see no rational explanation for the increase of 3000 reporting stations. That amounts to an increase of 30%. I find that hard to believe, but then I also think NCDC is NOT as “professional” as should be expected. If I had to guess, I would blame sloppy management.

    I was under the impression that DATSAV2 was only CLIMAT data. Anyway, maybe that accounts for the 3000 extras all of a sudden. In other words, not new data — just that it is now being handled differently, i.e. by comm line instead of mail.

    In the days of the AWN (they don’t exist any longer) we freely exchanged data with the Weather Bureau WMO comm center in Suitland, Md. via a fairly high-speed (for that time) line. They also had a line connected to Moscow, but once it went down hard and they asked us to supply them with our Soviet obs. We did. My assumption is that Moscow was having local problems too, because they asked Suitland to send our data to them.

    We later heard from the WxBureau that Moscow told Suitland that they had supplied data from the USSR that they were not even aware of!

    I honestly believe that it is impossible to define climate by temperature alone, so that the idea that climate can be forecast using temperature changes is absurd on its face. Which leads me to the next conclusion that the whole thing is basically a way for academia to get grant money.

    Sorry, but I’m a cynic at heart.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: