Gold Mining

[Published 6/18/2011 and originally posted for government leaders, July 6, 2009]

My last posting was about the “goldmine” that exists in the information your government collects every day. It’s a goldmine because this data can be analyzed to determine how to save money by learning what policies and programs work best. Some governments have the internal skills to do this kind of sophisticated analysis or they can contract for those skills. But no government – not even the US Federal government – has the resources to analyze all the data they have.

What can you do about that? Maybe there’s an answer in a story about real gold mining from the authors of the book “Wikinomics”[1]:

A few years back, Toronto-based gold mining company Goldcorp was in trouble. Besieged by strikes, lingering debts, and an exceedingly high cost of production, the company had terminated mining operations…. [M]ost analysts assumed that the company’s fifty-year old mine in Red Lake, Ontario, was dying. Without evidence of substantial new gold deposits, Goldcorp was likely to fold. Chief Executive Officer Rob McEwen needed a miracle.

Frustrated that his in-house geologists couldn’t reliably estimate the value and location of the gold on his property … [he] published his geological data on the Web for all to see and challenged the world to do the prospecting. The “Goldcorp Challenge” made a total of $575,000 in prize money available to participants who submitted the best methods and estimates. Every scrap of information (some 400 megabytes worth) about the 55,000 acre property was revealed on Goldcorp’s Web site.

News of the contest spread quickly around the Internet and more than 1,000 virtual prospectors from 50 countries got busy crunching the data. Within weeks, submissions from around the world were flooding into Goldcorp headquarters. There were entries from graduate students, management consultants, mathematicians, military officers, and a virtual army of geologists. “We had applied math, advanced physics, intelligent systems, computer graphics, and organic solutions to inorganic problems. There were capabilities I had never seen before in the industry,” says McEwen. “When I saw the computer graphics, I almost fell out of my chair.”

The contestants identified 110 targets on the Red Lake property, more than 80% of which yielded substantial quantities of gold. In fact, since the challenge was initiated, an astounding 8 million ounces of gold have been found – worth well over $3 billion. Not a bad return on a half million dollar investment.

You probably won’t be able to offer a prize to analysts, although you might offer to share some of the savings that result from doing things better. But, since the public has an interest in seeing its government work better, unlike a private corporation, maybe you don’t have to offer a prize.And there are many examples on the Internet where people are willing to help out without any obvious monetary reward.

Certainly not everyone, but enough people might be interested in the data to take a shot of making sense of it – students or even college professors looking for research projects, retired statisticians, the kinds of folks who live to analyze baseball statistics, and anyone who might find this a challenge.

The Obama administration and its new IT leaders have made a big deal about putting its data on the Web. There are dozens of data sets on the Federal site data.gov[2], obviously taking care to deal with issues of individual privacy and national security. Although their primary interest is in transparency of government, now that the data is there, we’ll start to see what people out there learn from all that information. Alabama[3] and the District of Columbia, among others, have started to do the same thing.

You can benefit a lot more, if you too make your government’s data available on the web for analysis. Then your data, perhaps combined with the Federal data and other sources on the web, can provide you with an even better picture of how to improve your government – better than just using your own data alone.

  1. “Innovation in the Age of Mass Collaboration”, Business Week, Feb. 1, 2007 http://www.businessweek.com/innovate/content/feb2007/id20070201_774736.htm
  2. “Data.gov open for business”, Government Computer News, May 21, 2009, http://gcn.com/articles/2009/05/21/federal-data-website-goes-live.aspx
  3. “Alabama at your fingertips”, Government Computer News, April 20, 2009, http://gcn.com/articles/2009/04/20/arms-provides-data-maps-to-agencies.aspx

© 2011 Norman Jacknis

Gold Mining

[Note: This was originally posted on a blog for government leaders, July 6, 2009]

My last posting was about the “goldmine” that exists in the information your government collects every day. It’s a goldmine because this data can be analyzed to determine how to save money by learning what policies and programs work best. Some governments have the internal skills to do this kind of sophisticated analysis or they can contract for those skills. But no government – not even the US Federal government – has the resources to analyze all the data they have. 

What can you do about that? Maybe there’s an answer in a story about real gold mining from the authors of the book “Wikinomics”[1]:

A few years back, Toronto-based gold mining company Goldcorp was in trouble. Besieged by strikes, lingering debts, and an exceedingly high cost of production, the company had terminated mining operations…. [M]ost analysts assumed that the company’s fifty-year old mine in Red Lake, Ontario, was dying. Without evidence of substantial new gold deposits, Goldcorp was likely to fold. Chief Executive Officer Rob McEwen needed a miracle. 

Frustrated that his in-house geologists couldn’t reliably estimate the value and location of the gold on his property … [he] published his geological data on the Web for all to see and challenged the world to do the prospecting. The “Goldcorp Challenge” made a total of $575,000 in prize money available to participants who submitted the best methods and estimates. Every scrap of information (some 400 megabytes worth) about the 55,000 acre property was revealed on Goldcorp’s Web site. 

News of the contest spread quickly around the Internet and more than 1,000 virtual prospectors from 50 countries got busy crunching the data. Within weeks, submissions from around the world were flooding into Goldcorp headquarters. There were entries from graduate students, management consultants, mathematicians, military officers, and a virtual army of geologists. “We had applied math, advanced physics, intelligent systems, computer graphics, and organic solutions to inorganic problems. There were capabilities I had never seen before in the industry,” says McEwen. “When I saw the computer graphics, I almost fell out of my chair.” 

The contestants identified 110 targets on the Red Lake property, more than 80% of which yielded substantial quantities of gold. In fact, since the challenge was initiated, an astounding 8 million ounces of gold have been found – worth well over $3 billion. Not a bad return on a half million dollar investment. 

You probably won’t be able to offer a prize to analysts, although you might offer to share some of the savings that result from doing things better. But, since the public has an interest in seeing its government work better, unlike a private corporation, maybe you don’t have to offer a prize.And there are many examples on the Internet where people are willing to help out without any obvious monetary reward. 

Certainly not everyone, but enough people might be interested in the data to take a shot of making sense of it – students or even college professors looking for research projects, retired statisticians, the kinds of folks who live to analyze baseball statistics, and anyone who might find this a challenge.

The Obama administration and its new IT leaders have made a big deal about putting its data on the Web. There are dozens of data sets on the Federal site data.gov[2], obviously taking care to deal with issues of individual privacy and national security. Although their primary interest is in transparency of government, now that the data is there, we’ll start to see what people out there learn from all that information.  Alabama[3] and the District of Columbia, among others, have started to do the same thing.

You can benefit a lot more, if you too make your government’s data available on the web for analysis. Then your data, perhaps combined with the Federal data and other sources on the web, can provide you with an even better picture of how to improve your government – better than just using your own data alone. 

  1.  “Innovation in the Age of Mass Collaboration”, Business Week, Feb. 1, 2007 http://www.businessweek.com/innovate/content/feb2007/id20070201_774736.htm
  2. “Data.gov open for business”, Government Computer News, May 21, 2009, http://gcn.com/articles/2009/05/21/federal-data-website-goes-live.aspx
  3. “Alabama at your fingertips”, Government Computer News, April 20, 2009, http://gcn.com/articles/2009/04/20/arms-provides-data-maps-to-agencies.aspx

© 2011 Norman Jacknis

Beyond The Inbox And Outbox

[This was originally posted on the web on June 15, 2009 for elected executives of governments.]

Every day, the employees of your government follow the same routine.

They have a stack of problems, applications, forms and the like in their inbox.  It may be a real, old-fashioned inbox with lots of paper or the computer-based equivalent. Doing the best they can, they then work through the pile and, we hope, with wisdom and efficiency, they process the incoming tasks and then move them to the outbox. As far as many employees are concerned, their work is done when the thing is put in the outbox.

However, for the people who run the government, this represents more than a ledger of what came in and what went out.  It is a gold mine of information.  Especially because of all the automation that has been put in place in government agencies, it is also an easily accessible gold mine.

Unfortunately, this gold mine is often ignored.  But if that data is analyzed, you will discover the patterns that can help you improve government programs and policies. Consider two examples, from very different areas, of what statistical analysis of that data can tell you:

What kinds of programs have worked best for which kinds of prisoners?  (This knowledge can be used to come up with better treatment and assignment of prisoners at intake.)

Who has used the public golf courses at what times of the week and day?  (This can identify where you might want to offer new programs targeted at particular groups of residents to even out usage during the day and get more golf fees.)

In 2007, Professor Ian Ayres wrote a book, “SuperCrunchers: Why Thinking-By-Numbers Is The New Way To Be Smart”, in which he described how various organizations are using statistical analysis to dramatically improve their performance. 

One of its chapters, “Government By Chance”, provides public sector examples and offers an interesting idea.

Imagine a world where people looked to the IRS as a source for useful information. The IRS could tell a small business that it might be spending too much on advertising or tell an individual that the aver age taxpayer in her income bracket gave mote to charity or made a larger IRA contribution. Heck, the IRS could probably produce fairly accurate estimates about the probability that small businesses (or even marriages) would fail. In fact, I’m told that Visa already does predict the probability of divorce based on credit card purchases (so that it can make better predictions of default risk). Of course, this is all a bit Orwellian. I might not particularly want to get a note from the IRS saying my marriage is at risk. But I might at least want the option of having the government make predictions about various aspects of my life. Instead of thinking of the IRS as solely a taker, we might also think of it as an information provider. We could even change its name to the “Information & Revenue Service".

This is yet another example, though, of moving the public sector from a transactional view of citizens to something more helpful.  While even the author admits the IRS example is a scary, there are other possibilities that are not scary and that your residents would like. 

The use of the data the government collects for better policy and better service to citizens is what I call “learning how to drive the government” because it is different from the usual fad and fashion approach to policy.

Too often policy debates are like a driver in a car who cannot see outside the windows.  So the driver keeps going until the car hits a wall, at which point the usual reaction is to go in the opposite direction until the same thing happens again.  This accounts for the feeling of a pendulum swinging in public policy debates, rather than real learning occurring.  

When everyday data is analyzed, it is like being able to look out the windows and figure out what direction to drive.

© 2011 Norman Jacknis

Beyond The Inbox And Outbox

[Re-published 5/18/2011.  This was originally posted on the web on June 15, 2009 for elected executives of governments.]

Every day, the employees of your government follow the same routine.

They have a stack of problems, applications, forms and the like in their inbox.  It may be a real, old-fashioned inbox with lots of paper or the computer-based equivalent. Doing the best they can, they then work through the pile and, we hope, with wisdom and efficiency, they process the incoming tasks and then move them to the outbox. As far as many employees are concerned, their work is done when the thing is put in the outbox.

However, for the people who run the government, this represents more than a ledger of what came in and what went out.  It is a gold mine of information.  Especially because of all the automation that has been put in place in government agencies, it is also an easily accessible gold mine.

Unfortunately, this gold mine is often ignored.  But if that data is analyzed, you will discover the patterns that can help you improve government programs and policies. Consider two examples, from very different areas, of what statistical analysis of that data can tell you:

What kinds of programs have worked best for which kinds of prisoners?  (This knowledge can be used to come up with better treatment and assignment of prisoners at intake.)

Who has used the public golf courses at what times of the week and day?  (This can identify where you might want to offer new programs targeted at particular groups of residents to even out usage during the day and get more golf fees.)

In 2007, Professor Ian Ayres wrote a book, “SuperCrunchers: Why Thinking-By-Numbers Is The New Way To Be Smart”, in which he described how various organizations are using statistical analysis to dramatically improve their performance.

One of its chapters, “Government By Chance”, provides public sector examples and offers an interesting idea.

“Imagine a world where people looked to the IRS as a source for useful information. The IRS could tell a small business that it might be spending too much on advertising or tell an individual that the aver age taxpayer in her income bracket gave mote to charity or made a larger IRA contribution. Heck, the IRS could probably produce fairly accurate estimates about the probability that small businesses (or even marriages) would fail. In fact, I’m told that Visa already does predict the probability of divorce based on credit card purchases (so that it can make better predictions of default risk). Of course, this is all a bit Orwellian. I might not particularly want to get a note from the IRS saying my marriage is at risk. But I might at least want the option of having the government make predictions about various aspects of my life. Instead of thinking of the IRS as solely a taker, we might also think of it as an information provider. We could even change its name to the Information & Revenue Service”.

This is yet another example, though, of moving the public sector from a transactional view of citizens to something more helpful.  While even the author admits the IRS example is a scary, there are other possibilities that are not scary and that your residents would like.

The use of the data the government collects for better policy and better service to citizens is what I call “learning how to drive the government” because it is different from the usual fad and fashion approach to policy.

Too often policy debates are like a driver in a car who cannot see outside the windows.  So the driver keeps going until the car hits a wall, at which point the usual reaction is to go in the opposite direction until the same thing happens again.  This accounts for the feeling of a pendulum swinging in public policy debates, rather than real learning occurring.

When everyday data is analyzed, it is like being able to look out the windows and figure out what direction to drive.

© 2011 Norman Jacknis