Playing At Dice - What That "Weekend Exercise" Was All About
By Skanderbeg Posted in Archived — Comments (32) / Email this page » / Leave a comment »
'Morning, class!!
Prof. Skanderbeg was pleased to see that some of the readership actually took up the challenge of doing the homework assignment - and the exercise was valuable, since (judging by the comments) it did seem to be getting the point across.
Just to explain what that was all about, I thought I'd put some comments in a separate post.
There are basically three items to explain.
Why Dice?
The short take is that using a pair of dice and summing up the results hands you about the simplest possible physical "system" that actually presents the real behavior of a "natural system" that is behaving statistically (rather than deterministically) - that is, it gives you a statistically-correct probability density function.
If you have one die, the probability density function is boring and trivial - each value has an equal 1/6th probability. But with two dice and the addition of the results, you get a very good and properly-behaved statistical system. That is, if you histogram-out the 36 possible additive combinations, there is a stable statistical mean value (7.0), and the distribution of other probabilities (out to 2 and 12) follows a Gaussian distribution.... along with proper standard deviation behavior and so forth. So it's about the cheapest and easiest laboratory for examining the behavior of statistical systems (both natural and man-made).
Bulk Statistics
If a system is behaving "statistically," it will show a stable central mean, a Gaussian statistical distribution, and proper allocation of standard deviations. The two-dice system does this.
As "nilram" noted....
I wrote a quick perl program to simulate a dice roll and plugged the output into gnuplot. The average was 6.917, sd=2.37
As long as you have access to a good pseudo-random number generator, you can code this up. (It's more spectacular to flummox people by doing the by-hand version with real dice, of course, but we can save that for when we're doing Congressional testimony.) From what I can tell, nilram's experiment involved 1000 rolls of the dice. I had done this some time ago with my own code and had rolled only 100, but produced the same results (though I rounded it off a bit more strongly, producing a mean of 6.9 and a standard deviation of 2.3). I'm too lazy this morning to actually go directly compute the standard deviation from the system histogram, but given that we got the same results from the same approach, I'd say that these are the real results for this statistical system.
The "bottom line" here is that a statistical system must be described in terms of not just its mean (average), but also in terms of its standard deviations. This two-dice system is very well-behaved and comprehensible.... but note that the standard deviation is rather large. That's the rub about dealing with "climate" and such - if you go look at any measured temperature data, the standard deviations (of say the high temperature for a particular place on a particular calendar date over a period of say a century) are huge - one standard deviation is at least multiple degrees, and frequently reaches double digits. This calls into question the sanity of obsessing over various forms of "data processing" that boil down to blind panic over apparent variations of fractions of a degree.
Sequential Plotting - The "Trend Chart"
So we have shown that we have a nice, simple statistical system to study and benchmark against - the humble pair-of-dice. Nothing strange about it.
Or is there? This gets us into the slightly wacky world of statistical process control (SPC). This is important stuff, and we should give much credit to Walter Shewhart and his protege W. Edwards Deming for their insights in this field. The name "Deming" might ring a bell, even though (sadly) he is no longer with us; he is the famous "quality guru" who finally got his due in the 1980s. He was the first to realize the implications of SPC for manufacturing; he got brushed off in the U.S. (especially by Detroit), but got a ready audience in Japan, where his ideas were adopted enthusiastically by the nascent Japanese auto industry.... and the rest is history.
The short take (too short) is that in a manufacturing process, the first goal is to create a stable process - that is, one that is (of course) meeting the basic targets but which has been "stabilized" - that is, under monitoring, it is behaving statistically, with a stable central statistical mean, a Gaussian profile to the bulk data, and reasonable standard deviations. The first task in stabilization is to get the mean to the right place, and then to get the magnitude of the standard deviations down to an acceptable level - without, of course, dorking up the stability along the way. When that has been achieved, the process is healthy and ready for routine manufacturing use.
But here is where the Deming-based "fun" really starts. You need to have ways of deciding if the process is indeed stabilized - and, perhaps more importantly in the long run, if a "stabilized" process is actually remain stable (statistically), or if it is actually systematically drifting away from the desired stability - and heading toward moving away from acceptability.
This is where SPC comes into play. The simplest approach is to identify a few things that you can actually measure well and measure sequentially - the thickness of a thin film layer, the width of a metal rod, whatever. As "stuff" comes along, you sequentially measure a list of items, and plot the trend that they show.
When you make plots of this sort, you can start to see trends - and this is where Deming's brilliant insight came into play. A well-behaved, well-centered statistical system will, when examined "sequentially," appear to show trends that look like "drift." This is how nature behaves, but if people don't know that, they tend to panic. They see an apparent trend, think that the manufacturing process is drifting, and make changes to the process to correct the "drift." Now they really have changed the process, but they've basically dorked it and it then does become broken - to the great cost of the manufacturing, who suddenly can't make things - as the engineers involved run around in circles chasing the "trend" that really isn't there.
That's the challenge of SPC - the need to be able to determine if some apparent "trend" represents a real, systematic drift of the process - or if it's just a statistical fluctuation.
Does that challenge sound at all familiar in the "climate change" context?
As "bennjneb" asked,
What point are you trying to make by recording a bunch of dice rolls?
Prof. Skanderbeg gave out the homework assignment of basically creating a "trend chart" (or control chart) for the simple system of the two dice. If you do this, you will actually see sections that will seem to show a "trend" of some sort - an apparently significant trend of the numbers rising or falling. If you haven't tried this yourself, it's a highly recommended learning exercise - literally one of the most enlightening (about nature) that one can do.
So as "rdbwiggins" replied,
Given the degree of extrapolation required to determine the average temperature of the earth at any given point in time, random rolls of the dice would be about as accurate at predicting future climate change as the current computer models.
Yes, that's more or less the point. If the system is behaving statistically, it will show apparent sequential trends that in reality are mirages. The dice experiment demonstrates that - and if you look at statistical and sequential temperature data, you see the exact same behavior!
Now, "Neil Stevens" said something really interesting:
Red Paint causes global warming!
I'm serious! In clinical trials, red dice show HIGHER average climate temperatures than any other color!
I'm assuming this is actually a report of results. But it basically gets the point. If you take some red dice and also some green dice and roll them independently, they can both have the exact same bulk statistical behavior - yet they can show completely different apparent trends!
This is why we must be careful with any "data" regarding climate and/or temperature and/or whatever. We are looking at a system that is behaving statistically - not deterministically. Trying to impose determinism onto a statistically-behaving system leads to conclusions that are comical - except when they cause you to "intervene" and take the manufactured-product yield to zero.... or to destroy the global economy and global society.
(N.b. I'm aware of Nassim Taleb's book "Fooled by Randomness" - in fact, a copy of it is in my "travel reading stack" and will come to the top on some plane at some point. His "The Black Swan" seems to be showing up everywhere - I have that as well, but haven't read it yet either. The touchstone on this for Prof. Skanderbeg was Henry Neave's "The Deming Dimension," which looked at SPC in manufacturing and then went beyond to apply it to other things. Later in his long and productive life, Dr. Deming got interested in notions of how SPC principles were catastrophically not being understood in a variety of other settings. I really wish he were alive today to comment on SPC and "climate change." I honestly believe that he'd produce an analysis much like this one.)
Class dismissed!!
Glad to help - hopefully it was enlightening.
I seems to be germane to the problem at hand, but none of this ever seems to come up.
There's a great (but chilling) Deming story to add to this (the discourse was already too long) that I'll try to hang up here this evening. For the moment, when I finish my tea I have to go get some trees to carbon neutrality....
If you slip me a contact e-mail, I will cheerfully mail you some slides that flesh much of this out....
Worth reading, and perfect for a long flight.
Hi Blackie!
Glad you showed up - some of the mathematical/statistical approaches you folks use in the finance world actually can be applied to this "problem" as well.
"The Black Swan" seems to be showing up everywhere. There were people reading it on the flight to Munich a couple weeks ago, and it comes up frequently in conversation.
If you can find a copy of "The Deming Dimension" by Henry Neave (it was published c. 1990 and is still more-or-less in print), that actually summarizes much about SPC in non-obscure English. I suspect that "Fooled by Randomness" is in the same vein.
It's still stochastic, but we use a different density function: log-normal instead of normal. That's for the common-sense reason that the value of a capital asset can't become negative (unlike the deviation from ideal of a mechanical measurement). A security that has dropped in value by 10% has consumed much more of its probabilistic "range" than a security that has risen by 10%.
That said, you can't ignore the fact that the log-normal ("random walk") model does a very good job of predicting normal market behavior, but a terrible job of predicting market behavior at the extremes. If you plot a theoretical log-normal curve against real market data, you'll generally find that the "tails" at both ends are fatter in the real-world data than in theory.
I've been told that the 1987 stock market crash was a -27 standard-deviation event, about as likely as shattering a window pane and then seeing it jump back together again by itself. But it happened. And it happened again in 1989. And 1991. And 1998. And 2007.
Having said all that, there's no denying that the financial world has totally internalized the key results of mathematical finance. And I have no doubt that this has fundamentally changed the way financial markets operate. And this has happened only in the last twenty-five years.
Yes, figuring out the likelihood of drastic but rare events is something that statistics is often incorrectly used for. If you hear, for example, that a bridge is designed for a 100-year flood, be sceptical. They have probably fitted a normal or lognormal to a few decades of data, and reasoned from the distribution tail probabilities. But they have at best a measured basis only for the assumption about the main central moments (mean, variance). There is inadequate data to sustain the belief that the tails follow the normal distribution. You need to observe for centuries to know how the tails behave. BTW, this is not to criticise the engineers who use these criteria; a bridge is needed, and its probably the best design guide they could get.
This issue came up in a recent AGW diary. I didn't get to say much about it, because the thread veered of into ozone stuff. The topic was climate sensitivity, and a recent paper suggesting that it was an inadequate concept, because it could never be precisely estimated. The issue is that there is always a finite probability of something happening which just doesn't fit the model - in this case, thermal runaway. So when you try to get a probability-weighted sum over all the possibilities, it breaks down. The lesson to be drawn, btw, is not that the whole prediction enterprise is flawed, but that you needed a better criterion (which they supply).
Under extreme conditions, the models break down. That obviously means that the various theories that underlie the models (the efficient-markets hypothesis, the Capital-Asset Pricing Model, the Black-Scholes-Merton equation) don't perfectly model the real world.
Wall Streeters even make jokes about this. The assumptions under which everyone prices options are so wildly unrealistic that people used to talk about living in a "Black-Scholes world."
The irony is that the models we now use to price capital assets have shaped the real world to the extent that under normal circumstances, the real world models theory quite closely. It's only when everything falls apart (which it seems to do on cue about once a decade) that the strangeness of all this comes to the fore.
But Mother Nature isn't as malleable as human nature, nor as obsessed with and devoted to theory as we are.
Your response to all this is to point out that the temperature data are, if not probative, at least non-contradictory. And that what we should be paying attention to is the simple fact that the Earth is holding on to more heat, and it's going to come out somehow.
Let's grant that for the sake of argument (although not being a scientist I'm not qualified to grant or deny it). I don't think you can then make the leap, based on models of questionable correspondence with reality, that the effects of a warmer Earth are necessarily bad.
But even if you're completely right, you're ultimately left in the position of Cassandra, because the developing world will not forgo carbon fuels. The best you can hope for is to someday be able to say "I told you so."
I do see a trend now that I only used 10 points. With 1000, the graph looks flat but jagged. I guess I was just a little overzealous.
Try looking at c. 100 point intervals - since this corresponds to the kind of numbers of years we have for measured weather data. Inevitably you'll see sections where there appears to be a "trend" one way or another (not all of them, but the bouncing line seems to be pointing somewhat up or down).
More later re: the promised Deming story. Tea finished, trees await transition to carbon neutrality....
This is all well and good, and of course statistical interpretation of the data is necessary. However, we can use a variety of statistical tests to rule out your null hypothesis, of sampling from iid random variables. As an example, I think we would all agree there is a long-term trend in the stock market even though it looks jagged on short time scales.
You are making a great point and people constantly read information into things that don't really have it, but that's not the case for global temperature. There is a statistically significant trend.
/me Shakes Benj's hand
/me Pats him on the back
Good to see you.
Lets try this do you have Excel ?
Do You have works ?
Do you have open office ?
Open up a spreadsheet
Pick a cell
Fill the cell with the following formula
=(rand()*5+1)+(rand()*5+1)
Fill down for a 120 rows
fill the next column with the numeric sequence 1..120
Plot a graph of the results
Look at the statistics and trendline for the graph.
______________________________
"Those who expect to reap the blessings of freedom must, like men, undergo the fatigue of supporting it."
-Thomas Paine: The American Crisis, No. 4, 1777
It's such a fine line between stupid and clever. - David St. Hubbins
Let me sketch a simple model (even simpler than the dice!) that can possibly help illustrate the idea of statistical significance.
Consider a random variable that can be either high or low with 1/2 probability each. (Although extremely simple, any probability distribution can be transformed into this one by designating above the median as "high", and below the median as "low".) Now, let's say we look at a piece of data, such as the global temperature record, and ask whether or not that data could be explained by our null hypothesis of this simple model. By inspection is is clear that all of the last 20 years have been "high" temperatures. So, is this just randomness or is there more going on? We answer by considering the likelihood of randomness producing the observed data.
For the numerically minded, consider constructing a program that picks either high or low sequentially and running it until you observe 20 highs in a row, recording how many steps it takes to get there. For those who just want to move forward analytically, consider that given a 1/2 probability of high on each independent roll, the chance of 20 highs in a row is (1/2)^20 ~= 1/(1 million).
Let us not be rash about discarding our null hypothesis, however, for it is not so dire as it seems. We have to account for our multiple hypotheses, which in non-jargon is the fact that we would find 20 straight years of same-direction temperature interesting over any 20 year period, not just the last 20. So, given the instrumental record of about 130 years we can say we have (130-20) = 110 different hypotheses. (These hypotheses are not actually independent, so the effective number is much lower, but take this as a best case for randomness showing this behavior). Also lets throw in a factor of two to account for 20 years of lows also being interesting.
So, what is the probability of our random model producing the observed behavior of 20 straight years of high or low temperature?
p <= (110 hypotheses) * 2 * 1/(1 million) ~= 1/(5 thousand)
So we can say that random variables, as suggested, are not a good model of global temperatures, with (at minimum) 99.98% confidence.
Or do I get to pick where the samples are taken from ?
A priori vs A posteriori.
Take a look at the data from 1860 to 1880 where we had twenty years below average.
Or take a look at the data from late 20's to the early 40's.
You have the exact same things happening on both sides of the hypothesis.
If you look back at your data spot something you like and then decide to test you have invalidated yourself.
______________________________
"Those who expect to reap the blessings of freedom must, like men, undergo the fatigue of supporting it."
-Thomas Paine: The American Crisis, No. 4, 1777
My above post was pointing out how global temperature data cannot be described by just repeatedly sampling a random variable. The fact that there were also long stretches of "low" years only reinforces this. If you include that in your test of the null (random number) hypothesis, the probabilty (continuing my above anlaysis) of the null hypothesis producing the observed behavior are less than 1 in 1 trillion.
It is (statistically) impossible for a sequentially sampled random variable to produce the observed global temperature. If you want to convince yourself, try the non-parametric runs test of randomness on the global temperature data binned into high and low bins. Unless you think that was constructed a posteriori for the purpose of showing trends in global temperature data?
Look lets say we are talking about another physical system.
Something much simpler. The system I am thinking of is a circuit board assembly line consisting of an automated pick and place system, a wave soldering system, two human quality control people.
You will have trends, both linear and cyclical in the system. The pick an place systems will drift back and forth over their range. Boards will hit the wave solderer too high and too low. The humans will tire and suffer ennui, over time the whole system ages.
So you have non random but cyclical components, purely random components and finally the effects of wear and tear on the system.
The funny thing is the cyclical components look very much like linear trends if you only see half a cycle or less. The purely random components can and usually will apppear cyclical and high frequency. Finally the aging of the system will show up as a spread in the deviation of the system.
______________________________
"Those who expect to reap the blessings of freedom must, like men, undergo the fatigue of supporting it."
-Thomas Paine: The American Crisis, No. 4, 1777
Skandenberg suggested global temperature data can be explained by repeatedly sampling a random variable. This is not true.
If you want to make a thread about the new subject of general time series analysis, go ahead. I suggest incorporating Pliny's post below, in particular some background reading on ARIMA analysis.
Let me borrow a quote, this one isn't mine but I certainly want to steal it.
"The way psychics prophesies is they shoot an arrow at a wall and later people paint a target around it".
Thats what you were doing. You have painted a target around the data.
______________________________
"Those who expect to reap the blessings of freedom must, like men, undergo the fatigue of supporting it."
-Thomas Paine: The American Crisis, No. 4, 1777
This is a good exercise. I would have added interest by plotting moving averages and maybe some exponential smoothing to give an even stronger illusion that the data is going somewhere.
But seeing through this is what time series analysis is all about. With sophisticated methods like ARIMA, the null hypothesis is that the data was generated by a random process, with autoregression and smoothing, but no actual information. Just like your exercise. Then you test whether a hypothesis incorporating real information (eg a trend) explains significantly more variance.
But I think there is an underlying misconception that AGW is reliant on this kind of analysis. This is encouraged by the focus on temperature data on sites like Redstate, which suggests that the case for AGW is based on temperature-CO2 correlation. This is not so - the primary argument is mechanistic - the greenhouse effect, and all the features of IR radiation that can be measured and analysed.
AGW came to prominence between about 1975 and 1985, leading to the Kyoto conference, and the ratification by the Senate (on Pres Bush's referral) of the UNFCCC in 1992. At that stage there was not much more than a century of instrumental temperature readings, and the warming signal was quite ambiguous. So temperature was subsequently studied, not to "prove" AGW, but to confirm. If you have a greenhouse model showing heat coming in, and temperatures don't go up, then you have some explaining to do. But they did. That fits the picture, but doesn't become the proof.
Gee, I bug out to carbon-neutralize some trees and do other chores, then have a coffee and conjure up Sunday's traditional Mexican dinner, and hop back in to see that much fun has been had.
Well, I'm glad the discussion was actually pretty high-level. After all the tree-crushing and such, yours truly is really, really worn out, so I'll just scroll through the discussion in another tab and make a few woozy comments....
Blackie, I don't envy you trying to function at the intersection of mathematics and finance - because that's a street corner where something silly is impossible right until the moment it happens. :-) Remember Long Term Capital Management? This is what happens when non-physicists imitate how they think physicists work (rather than how they actually do work). Keep writing down equations until you can cancel the terms for "risk"? Gadzooks. That's pretty much a working definition of "hubris."
Having tilted at log-normal distributions myself over the years, I'm a little surprised that you didn't mention 1/f "noise" and such - and the probability behavior of very rare events. That seems to be closer to the observed reality.
Bennjneb, I really appreciate that you've taken such an interest in this. In school, for some reason we get loaded up with all sorts of stuff that gives us the impression that everything in the world is solidly deterministic - I really wish that even rudimentary statistics were pushed to the forefront, even in secondary school.
It's funny that you mentioned equity markets, since that's a good example of how long-term sampling can confirm a solid trend. If you take the DJIA from way back (pick a way back - 1910 or whenever) and do any sort of statistical analysis (just means and standard deviations, or moving averages) and compare with "measured data," you can quickly confirm that there has been a long term trend (mercifully, up).
But I'm afraid on that basis that I can't agree with the contention that the same can be said of temperature. A core problem with almost all of the analyses of "global temperature" and such is that.... there really isn't such a thing. The physics is that "temperature" can only exist for a particular location in three-dimensional space at a particular time. When any sort of "averaging" is done, things get unmoored. E.g., ponder even the simple question, "What is the average temperature in the state of Virginia right now?" Besides the mechanics of how you define that and then try to measure it, what does the result even mean?
I didn't try to "model" temperature - the goal was just to take a swag at the situation and see if something significant showed up. The best "real" data we have is measured temperatures (unfortunately (over the long haul) "only" high and low temperatures for particular locations for particular calendar dates). So the most basic (and relatively "pure" analysis we can do is to take a given place, a given calendar date, and the high (or low) temperature as measured. There are quibbles with even that ("high" can occur at any time of day, locations move, instruments change, etc.), but this seems to be about the cleanest stuff we have. The level of detail is important, but the contention is that if we are on the cusp of some large and unprecedented shooting-up of the temperature, there should be a clear and unambiguous "surge" upward in the trend for the measured data. But there just isn't. Surprisingly little is happening no matter how you look at it (more on that below). When you look at the bulk statistical behavior of the data, with the very large standard deviations, any apparent trends are comparatively very, very small. That's why the "dice-pair" provides a comparative - since the trend behavior you can see in a pair of dice is quantitatively about the same as you see in the trend behavior of measured temperature data.
The variations and trends that appear in the temperature data are very small (and also contradictory). Given this behavior and the small magnitude relative to the standard deviations, any apparent trends seem to be small enough to be explainable as statistical fluctuations rather than as a systematic trend.
Pliny, your suggestion about looking at moving averages is a good one. I actually have done this as well, and the results are not only contradictory but in some instances almost comical. For one place and date, the 15-year moving average in the high temperature data shows a shocking 12 degree (F) rise.... from 1934 to 1955! (Weren't nothing supposed to be happening then!!!). But the 15-year moving average for the same place and date but for the low temperature.... is basically flat. For some places, the 15-year moving averages appear to show a recent run-up, but the 30-year moving averages are much more subdued. Interestingly, if there's a trend in the 15-year moving averages (noting that given the number of places and number of days of year this is a very limited survey), it's to have peaked in the late 90s and they are mostly now trending down. Even funnier, if you look at the moving averages for the same place (this is one case) but for two dates two days apart.... in one the 15-year moving averages are heading up while for the other they are heading down!! Ugh. My suspicion is that the moving averages are just artifacts that aren't telling us anything - there's just not enough systematic signal coming through the statistical noise.... which may be the point.
Methinks that the bottom line for the discussion here is that there continues to be a lot to be discussed and argued over in the details. But I do feel comfortable from what I've seen in stating that the Al Gore heat-death-disaster scenario would show up strongly in an SPC-type of analysis.... and it just doesn't....
ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
To a large extent, modern finance is mathematics. There's no intersection.
Risk is fundamental to portfolio analysis. In fact, it's the most fundamental factor. Analysis seeks not to eliminate risk, because axiomatically you get no return without taking risk.
The goal is rather to construct exposures to multiple risks that are non-correlated. This actually works, until it stops working. When risks in unrelated asset classes become linked together, everything melts down, fast.
You mention LTCM, who of course had no fewer than three Nobel Laureates working for them, including Myron Scholes himself. I remember every minute of that situation, which is why I was among the first to compare this year's crisis with that one. (I did so posting under my pseudonym here at RS.)
It's pretty widely acknowledged now that this year's crisis was structurally similar to the Long-Term crisis, but far bigger, more widespread, and more damaging.
Your remark about financial engineers trying to think like physicists is actually resonant. I've often felt that the immense success of the "Knabenphysik" of the 1920s inspired envy in every other branch of intellectual pursuit, with often ridiculous results (the "social sciences" are the obvious example).
In the case of mathematical finance, what started as an attempt to model the real world has become the real world.
Yep, we reach.
It's always seemed to me that other disciplines think that physics consists of the massive scrolling on blackboards of completely ab initio differential equations (with no guesswork, no adjustable parameters, no etc.) and "the absolute truth" appears at the end. Perhaps the high-energy-theory types even encourage this perception. But it's nothing like the reality.
The "fun" starts when other disciplines start behaving that way. As you noted, the social scientists are the champion beclowners in that regard. But this infects many other disciplines as well. LTCM seemed to be the apotheosis of hubris-based physics envy.
(I even got dragooned on short notice on giving a lecture to a group of humanities professors some years back, and entitled it "Stop Imitating Us!" - even citing LTCM as a good example of the problem. :-) )
Methinks that at some general level, most of the world is still thinking at the level of the Newtonian worldview - very deterministic, very clear-cut differential equations to solve. Wow is that world long gone....
...knocks out a whole city and ends a world war. That sure gave the new physics quite a bit of respectability!
I hope that biology might finally be on the edge of some similar breakthroughs. As things stand today, we still create drugs largely by guessing and then seeing how they do, which is crazy.
You reminded me of something I once liked to say: modern finance is as different from the old-fashioned kind as quantum mechanics is from classical physics.
like physicists - they often had been physicists. One of my main activities was writing pde solving software for fluid dynamics (we call this CFD). About ten years ago, pde methods became popular for pricing exotic options. And sure enough, we found ourselves doing barriers, GARCH, stochastic volatility etc. And your suggestion that nature is imitating art here is very resonant. I encountered very few who really wanted to use our software to get a better price estimate that they could use to get ahead of the market. Instead, they wanted to be sure we could reproduce someone else's prices. Maybe to satisfy a regulator, or value their books for auditing, or whatever.
A core problem with almost all of the analyses of "global temperature" and such is that.... there really isn't such a thing.
What is generally referred to as the global temperature is, of course, not a temperature in the proper theoretical sense of the word. The earth is not in equilibrium and therefore does not have a temperature in this strict sense. The global temperature is usually the mean annual global surface temperature, roughly speaking this is an average of temperature readings over the course of a full year weighted by the land area represented. Though not temperature in the theoretical sense, this quantity is well-defined and informative.
The best "real" data we have is measured temperatures (unfortunately (over the long haul) "only" high and low temperatures for particular locations for particular calendar dates).
Attempting to draw conclusions based on analysis at single sites is when you will run into variance too large to draw conclusions. However, by summing many terms with "errors" we succeed in eliminating all uncorrelated errors. This leaves us with only large-scale, correlated changes in our end indicator (in this case global temperature) which gives us a much higher signal to noise ratio.
Your characterization of the instrumental record, particularly in more recent decades, is also inaccurate. The record is far better, with hourly (or more frequent) temperature recordings being the norm over large parts of the world. As an introduction to the data try this website, in particular click on the search by map link for a visual look at station coverage.
When you look at the bulk statistical behavior of the data, with the very large standard deviations, any apparent trends are comparatively very, very small.
I am possibly placing this statement in the wrong context, but I am going to reply to it in the context of mean global surface temperatures. First, you can read in the description that the trend in the 20th century was ~ 0.6 ± 0.2 °C, which is significant.
Additionally, there is a simple procedure to estimate the variation in data allowing you to consider if a trend is significant or not. The one year differences of this data will approximate the error distribution in these measurements. Looking at this data, the largest one year difference is ~.25°C, yet the trend is ~.7°C, well above the estimated variation.
The warming trend over the past century in the instrumental mean annual global surface temperature is statistically significant.
I'm sorry, but I am too tired (this is my fault, not yours - it's late) to continue, but I will note one thing.
You're treating the IPCC's "0.6 +/- 0.2" as though it's established and deterministic fact - like taking a micrometer and measuring the thickness of a piece of metal.
We had a long discussion here some months back about this. The 0.6 is an average computed over a lot of space and a lot of time - which is pushing into "sausage factory" territory unfortunately. Trying to draw solid conclusions from that much processing is risky - and it has been argued that with some slightly different averaging assumptions one can just as easily produce 0.0. The "0.2" really seems to come out of thin air, and given the wild statistical behavior that the temperature data show that doesn't seem terribly valid.
We can argue about the details of the methodology, but the 0.6 +/- 0.2 just isn't solid or scientific. Even if the 0.6 is somehow valid, I suspect that statistical reality would be something more like 0.6 +/- 4.2 (or whatever), which renders the 0.6 pretty much moot.
It was to avoid the sausage factory and interpretations and what-not that I went to the cleanest stuff we have and looked at that. It's the rawest of raw data we have, and if something genuinely significant is in progress, it should show up strongly in the data. It doesn't.
ThankZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ.
is real enough, or at least it's as real (IMHO) as the DJIA. In fact it plays a similar role. An index like the DJIA is an indicator of the total capitalization of some part of the market. It's a weighted sum of prices. Global temperature is an indicator of total heat content. It's a weighted sum of individual temperatures. Weighted according to some understanding of thermal inertia. The ratio between global temperature change and incoming heat is derived from the climate sensitivity. Of course, the heat content of what, exactly, is not so easy to pin down. But in fact the lower atmosphere and upper sea levels do form a reasonably bounded system. This was the basis of the Stephen Schwartz calculation, that we discussed a while ago.
My management team was unaware of the benefits of SPC – I convinced them to purchase InfinityQS SPC software and we have already begun to improve our processes and save money.
we just implemented SPC at my plant. we've already seen an increase in production as we can finally get good analysis of our processes. SPC works like a charm...i was blind but now i see!
That was one of the most lucid explanations of a statistical process I have ever seen.
______________________________
"Those who expect to reap the blessings of freedom must, like men, undergo the fatigue of supporting it."
-Thomas Paine: The American Crisis, No. 4, 1777