My Canadian Open Government Consultation Submission

Some of the questions posed in the Canadian Open Government Consultation Submission required written responses.  Here are my responses:

question 1: "What could be done to make it easier for you to find and use government data provided online?"
To make data easy to find I would like to see all of the available data along with descriptive text, preferably one dataset per page, published on government web sites that can be indexed by Google.

To make the data easy to use, I would like to see the data published in open non-proprietary formats. The data should be addressable by URL, without any requirement to sign in or identify or be required to click on anything extra in order to download the data as these sorts of mechanisms are obstacles to performing repetitive automated tasks.

The data should be either released as public domain, or if that's not possible then it should be licensed under a legal framework that conforms with the open data definition posted on opendefinition.org. The Open Data Usability Index (ODUI) document at OpenDataBC also provides helpful information about how to make data more usable. http://www.opendatabc.ca/odui.html

question 3: "How would you use or manipulate this data?"
I would use the data to create analysis and visualizations that would help Canadians better understand their country. I would create value added applications that would help people to create businesses and jobs using open data. I would also try to help people use the data, by providing online tutorials and holding events that would encourage people to get involved. I would like to help create the incentives for citizens of Canada get more engaged and participate more with their governments (local, provincial, federal).

question 4: "What could be done to make it easier for you to find government information online?"
Government information should be made available to search engines like Google by providing topic specific pages, in text that search engines can understand so that people can use the tools they are used to to get information. There is no need to create an elaborate portal and specialized search engine for this sort of effort, at least in the foreseeable future.

Specifically about Open Information, again, it should all be either released to the public domain or if that's not possible licensed using a license that conforms to opendefinition.org. It should not be given a different license than open data if you want people to be able to use it.

question 7: "Do you have suggestions on how the Government of Canada could improve how it consults with Canadians?"
I don't have a lot of ideas here other than make use of the available social media tools such as Twitter and Google+ to reach out to Canadians. I don't always agree with Tony Clement, but I do like the way he engages citizens. I would like to see more MPs do the same. Also, adopt social media guidelines like the Province of BC has, which fosters trust with its employees and allows them to engage directly with citizens.


question 8: "Are there approaches used by other governments that you believe the Government of Canada could/should model?"
As far as open data goes, the legal framework is a big deal and why I keep advising governments to work hard to get this part of open data right. Follow the example of the US or Australia or New Zealand or Surrey or Langley or Township of Langley or Winnipeg Transit.

In Australia agencies are supposed to go through a process in order to keep data closed. (There is currently no binding law to enforce this but the intent is there). This would be relatively easy to put in place in many jurisdictions as data is already routinely assessed for privacy concerns and treated accordingly. Determining if it should be closed would be just another check box, and since the vast majority of public data should eventually be open data, it makes sense to identify and document the much smaller set of data destined to be closed.

I think there is an unintended side effect of the way open data is being rolled out in some jurisdictions. Public data that was once generally considered usable ( financials, stats, directories, registries, codes, inspection data, schedules ) are now considered not usable because they don't have explicit permissions associated with them. When they do get an official license applied to them, their new explicit "open" status can be significantly more restrictive than the implicit understanding prior to the movement.

Also, governments often set up a new agency that identifies data citizens can use and place it in a catalogue, and anything that's not in the catalogue then has a mysterious cloud over it by contrast, even if it was formerly considered public. Depending on how fast this new group can get through the relevant data to catalogue it, citizens could feel they have significantly less data than they had before the official process.

Everything already on the internet should just be open licensed.

question 9: "Are there any other comments or suggestions you would like to make pertaining to the Government of Canada's Open Government initiative?"
Thank you for taking the time and putting effort in to address this opportunity and to consult with citizens. I think we, as a country, have a lot to offer the world in terms of leadership in this space.

2011 in Review

I find that the end of the year is a great time to reflect on the past 12 months and refresh the vision for the upcoming 12 months. It works well for me not only because of the changing year, but also because the holidays bring a brief window of downtime, which I like to use to, as Stephen Covey says, “sharpen the saw”.

Anyone who knows me knows that I am a avid supporter of open government and open data. My personal belief is that open government is what’s next for modern democracies. Public servants and citizens are starting to see for themselves the strategic advantage that openness represents, something that the open source movement has known for decades. As more politicians and high level public servants start to realize the positive effects that openness can have on their work and their personal careers we will see more adoption. Soon we will look back to this period and it will seem archaic for governments to keep critical decision making information from the people they serve.

My personal vision for 2011 was that by the end of the year people in BC would be more aware of open data. When we started 2011 most people I talked to had no idea what open data or hackathons were. Now, 12 months later, we have had 11 hackathons in 5 cities in BC, 350 hackathon attendees, several news paper and other media stories, we have a ministry with Open Government in its name, a provincial government that publishes open data and we've had 5,000 unique visitors to my OpenDataBC.ca site viewing 35,000 pages of content.

One of my heros, Seth Godin, popularized the term shipping which describes the act of completing a project, getting something out the door or delivering something. It doesn’t matter if it was a hit or not, or whether it was perfect or not. All that matters is that it is done.

Here’s a partial list of things I shipped in 2011:

• Designed a system to enable secure storage and sharing of health information
• Continued to build the OpenDataBC google group which now has 116 members
• Created the Open Data Usability Index with other open data experts
• Held the 2nd annual Victoria International Open Data Hackathon
• Facilitated 11 open data hackathons around the province of BC
• Added 170 datasets to the OpenDataBC catalogue
• Presented open data at several conferences
• Advised several governments in Canada on open data strategy
• Developed Municipedia, an app to improve Voter turnout in the 2011 Municipal elections
• Contributed to several other open data projects
• Proposed a change to the Open Definition
• Launched an HTML5, Python based data visualization platform
• Attended several conferences including Google I/O, EuroPython and OSCON
• Developed a tiny open source split testing framework for Python
• Purchased several Internet ready music albums
• Gave $500 to causes I believe in (EFF, FSF, Demand Progress)
• Moved 55 domains from GoDaddy to DreamHost
• Wrote 30,000 lines of code
• posted 1,900 tweets
• Wrote 9 blog posts

I was very honoured to be able to work on these projects with so many great people in 2011.  I worked on many other projects as well, some of which are not complete so they’ll be my 2012 list. :)

Now, I will get to work on my vision for 2012.  I will post here in the near future.

Happy New Year!

The Value of the Seal

The implementation of government open data initiatives presents a number of interesting challenges to governments on the leading edge of Gov 2.0 policy making. Established business practices invented in another age sometimes seem impervious to ideas like open data. One such practice is charging money for public data under the banner "cost recovery".

When looking at the bureaucracies and systems required to manage the authorized distribution of and the collection of payments for data (i.e. products whose marginal cost of distribution is effectively zero) it's easy to wonder if the operating costs are mainly due to the billing apparatus itself.

But let's take the case where there is in fact a cost to distributing this data. Perhaps there is a collection cost or some sort of initial investment cost. In some cases, governments have chosen to offset that tangible short term cost with a fee for access, to either the data itself or some byproduct of that data.

People or organizations are willing to pay money for this service of data retrieval. From this, one could deduce at least two things: 1) provision of data is a valuable service; and 2) the government is the considered the best or perhaps the only place where they can get this data.

Let's look at the BC Corporate Registry for example. To search for a registered company in BC I have to first become a member of BC Online. To become a member of BC Online I have to first deposit $100 into a BC Online account, and then to perform a search they will deduct $8.68 from my account. People do use this service, so there is some value. People are willing to pay for this data.

Or, are they?

What if BC Corporate Registry data was released as open data. The deal is, I would have access to that data to do what I want with it, including creating an app, and making money, the only condition being that I could not claim it was official.

So, I get the data, I offer the data to lawyers and corporate deal makers and entrepreneurs and whoever else I think can use this data and I offer to sell it to them for 1/2 of what the government currently charges for exactly the same thing except that my data comes with a clause that says something like "this data is 'not official'". My guess is that no one would take me up on that offer. In fact, my guess is that I wouldn't be able to give that data away for free. Why? Because it's not official. It turns out being official is pretty important.

I think that this whole model of people paying for public data is a collapse of two concepts and what's missing is a distinction. People do not pay for public data. People pay for data with an "official seal". I would go one step further. Often the seal is all they are paying for.

This is not a new idea. Just as I can get a certified engineer to come over to my house and talk with me for an hour about the structural changes I want to make to my house, and she'll give me all sorts of ideas and draw me a sketch and perhaps help me with my design ideas, so long as it's all "unofficial". Some will even provide me with an non-certified printout of my plans. To build my house though, I need the official certified plan, and for that there is a fee. I am not paying the engineer for the plan - I am paying her for the plan stamped by a certified engineer.

I suspect this won't even be a surprise to the folks involved in crafting the open data licenses that we use. The clue is in the licenses themselves. For example, if you look at the Government of B.C.'s new Open Government License you will see in clause 5. b) "ensure that you do not use the Information in a way that suggests any official status or that the Information Provider endorses you or your use of the Information".

That clause is there because someone recognized that that's where the value is. It's the official status, or said another way, the authenticity that the government brings to the data that people need and are willing to pay for and often for legal reasons must obtain.

My personal preference would be to have all publicly funded data free and available for any lawful purpose by default. The question of whether or not people should have to pay for the official status of anything is a separate and worthy question. In any case, fees can stay in place for "official" data as there really isn't any threat to existing business models. Releasing the data itself can happen using existing policy and the existing licenses. And, my guess is that in time governments will discover what open source and many other folks already know, that by opening the data, the demand for paid services will actually increase.

And for some young entrepreneur, who wants to build the next iPhone app to accelerate the process of creating a new business in BC, that in turn could create jobs and attract talent to our province, that's good news. Because his app doesn't actually require that the data have the official seal.

He just needs data he can use.

Internet Ready Music

I love music. I love listening to it, and I love creating it.

I used to buy Vinyl records, then switched to CDs and now I buy mostly online. I have created mp3 files of most of my CDs so I can listen to them in one big huge playlist without ever putting in a CD.

A couple of months ago, Google invited me to use their new Google Music Beta. It's awesome. With it I can easily place all of my music on their servers and then access that music from any of my computers or Android devices.

But I haven't done that. Why? Because the record companies have been attacking people use the internet to enjoy music. And, while I don't think it breaks any laws to use Google Music Beta, I am not 100% sure. Even if it's 100% legal today, with the music industry lobbying governments and influencing law makers and suing students, and now getting the laws changed so that ISPs are required to spy on us, it's not inconceivable that they could convince law makers to make my uploading of music to Google Plus illegal, retroactively. And since once I upload music, there is really no way to make sure it's removed from wherever I upload it, uploading is an action that I can't really undo.

It seems more and more that the vast majority of digital music that I have purchased thus far is becoming legally incompatible with internet technologies and it's actually becoming increasingly risky to own it at all. My iTunes purchases have declined as I have become more aware of the associated risks of owning music that's incompatible with the internet.

Just like driving a horse and buggy on a modern freeway today would be considered dangerous, using music locked into outdated copyright schemes that prohibit uploading on something designed for uploading and downloading is dangerous.

Some of the music I own, however, is compatible with the internet. Bands such as The Charlatans, Radiohead, Nine Inch Nails, and others have been making their music available for free on the internet for some time.

I have also recently (re-)discovered Jamendo which is a music site that contains 300,000+ songs from all over the world and in all genres, all of which are available for downloading and can be uploaded and shared freely with others using the Creative Commons license.

With Jamendo I have discovered some new music that I am totally hooked on, such as the fantastic Canadian artist Brad Sucks, who you can currently find me listening to most days.

Since the internet is all about sharing, I am starting to call this new music, "internet ready". I can upload it to Google Music Beta without worrying about any scary legal incompatibilities down the road, and I can share it with Friends.

Incidentally, I still pay for much of this "free" music. Usually $20 an album. I like that the whole $20 goes to the artists because I want them to be able to continue to create great music. But mostly, it's worth more to me than the risky legacy "recording industry" music because it's internet compatible and as such I am free do so much more with it.

About the License

Yesterday while we were all celebrating the awesome work of the BC Provincial government, a very well respected Harvard researcher, software developer and open data advocate was arrested by the US Federal government on charges related to computer hacking, based on allegations that he downloaded too many scholarly journals that he was entitled to get for free.  He now faces a possible 30 years in jail.

I don't know Aaron, but I feel as though I do.  He writes brilliant software code and releases it to the world as completely free public domain software.  He advocates for open data and transparency and democracy in the US and he is a founder of demandprogress.org an organization dedicated to progressive policy changes for ordinary people.

Please consider visiting demandprogress.org and reading about what's happening to Aaron.

How does this relate to open data and the new DataBC portal?

It's about the license.

License : official or legal permission to do or own a specified thing.

It's strange that we need permission to use our own data and information.  It's strange that we sometimes have to pay to use data that we have already paid to have created.

Being an open data advocate and application developer comes with a certain level of risk and anxiety.  We are actively trying to do things that have not been done before.  And unlike other areas where innovation takes place, we are innovating in an area involving strange legalities, usually as individuals with no corporate backing or protection, and the consequences of making a mistake can be severe.  There is a lot of uncertainty.  Although many of us are open source developers and thus are pretty familiar with copyright law, licensing and the jargon that goes along with them, very few of us are lawyers.

This is why we want governments to use standard licenses.

By choosing to invent a new license rather than use an existing one the BC Government has added to the uncertainty.  Yes, they based it on the UK license, but it's clearly not the same as the UK license, otherwise they could have just used that.

Because they chose to invent a license I spent several hours last night pouring over the license and comparing it to both the PDDL and the UK license to see where those differences are and to see what additional risk I might have to take on as a result.  Every developer I know will have to do the same thing now before they start using the data.

Many won't bother.

And that's the lost opportunity. People who get turned off by the custom license won't use the data, or won't bother coming to the hackathons, or won't bother creating that new app. It's just too risky. Sadly, we all lose. because as I understand it, the BC Provincial government is in this for the right reasons.  It's clear to me that they absolutely get it.  Innovative new ideas and applications will be generated as a result of this.  This increased transparency and engagement and collaboration with the citizens will build trust and goodwill and is good for the government and good for the people of BC.

From what I can tell, not being a lawyer, as a standalone license, the BC Open Government License is actually mostly good (check out unrest.ca for some of the details on issues with the license).  And, there are a handful of us, that will push through this licensing thing, grumble a bit, say "it's pretty good" because it is, and weigh our risks and move forward with our apps and visualizations and innovations.

BC is seen as a leader in citizen engagement and open data by local governments, other provinces and internationally.  Looking at what's going on in the rest of the world, particularly in the US, we are really very fortunate to live where we do and to have the public service and leaders that we have.

I will encourage others to take the time to read the BC license rather than blowing it off because it's not standard.  And I will continue to urge local governments and other provinces to use a standard license rather than invent their own.

This is a first great first attempt, and as Christy Clark said in her excellent and encouraging video, this is very much a work in progress.  The license does have a version number, which to me implies that they are open to input, discussion and changing it if necessary, which is awesome.

Remembering "Open"

Today the British Columbia Provincial Government launched a new Open Data Portal, making thousands of our publicly owned datasets available to us including everything from employee salaries to historical school locations to Local Government Incorporation Dates to the data catalogue itself. As a citizen and taxpayer in BC and an open data advocate I am very excited to see my own Provincial Government take these steps toward innovation and transparency. I congratulate those public servants within the BC Government that understood the opportunity, recognized value and championed the cause.

These days, governments all over the world are starting realize the value of "open" but it wasn't so long ago that we were at the opposite end of the spectrum. As a public servant in the 1980's, employed as a junior data analyst, I personally produced the monthly report for the minister and deputy minister showing the basic metrics of the ministry I was working for. As part of that job I was required to produce 5 copies of that report and place them in brown paper bags and tape them closed. I would then personally attempt delivery to the recipients office. If the recipient or their assistant weren't there to receive the report, I was required to take the report with me and try again later.

Those same metrics are still being used today and were released today as part of the Provincial Open Data portal. It's striking how far we have come in such a short amount of time.

Although we have gone through a very opaque phase with our governments, the idea of governments being open and transparent is not actually new. Our own BC Government has been publishing the public accounts and other financial information for decades. They also produce a monthly publication called the British Columbia Gazette and have done so since the 1920's that is teeming with useful information about our province, from disposition of Crown Lands, to election results, to tree farm licenses to road name changes.

Technology has evolved since these publications were originally developed. Where at one time publishing this type of data on paper was about as usable as one could have imagined, these days its available electronically, and hopefully soon it will be included as part of the open data portal.

So, although openness and transparency aren't new, they were definitely forgotten, and as I like to say, we are now remembering the value of "open".

Congratulations and a big "Thank you!" to our public service employees and political leaders who are helping to make this happen.

Now, I need to go to the portal and look for some data. :)

Government as Platform - An Example

Tim O'Reilly uses the words "government as platform" to describe an interpretation of what Government 2.0 is. I am often asked what "government as platform" means. I think the question arises because much of government already operates as a platform. People don't distinguish it as something new because it's routine.

To explain what government as platform is, you can look at an example where it's already the norm such as our public roads. Our three levels of government are involved in the construction, maintenance and regulation of roads. Together they deliver infrastructure upon which we ride our bicycles and drive our cars.

We are used to the idea that we can hop on our bicycle or get in our car and and transport our selves on public roads. As a platform we are left to decide where we are going, when and how to get there. The outcomes of the platform, the actual transportation conducted, is not determined beforehand. The roads are built to certain standards and the drivers are left to figure out how to get from point A to point B.

And because the system is open, people and companies can and do invent innovative new ways to use the platform. Because they are free to choose their route, people optimize the routes they use to get from point A to Point B.  Inventions like automobile GPS are created to help with navigation.  Every year the entire vehicle industry (bicycles, cars, buses) releases new versions of their transportation products and introduces innovative new ones. Several huge industries can rely on the fact that our roads systems is a platform that they can build on.

And for travelers, the system works so well we can travel almost anywhere and we're usually not even aware of which level of government is responsible for which roads. We can even travel to different countries and travel, because all of the public roads work pretty much the same way. Done well, the platform becomes transparent. We don't even notice it.

I am convinced that this what we need with our public data. Our data – all of it – made freely available on the internet, with standard licensing, in formats we can use - would provide a platform for innovation like we've seen with transportation. Entire industries could grow on such a platform, providing jobs and value we can barely imagine right now.

And though only a few people will use the data in the beginning, like the few people building cars and bicycles, those people will create huge value for their fellow citizens all built on a platform managed by our governments.