Open Data Learning Summit

Two weeks ago I had the privilege of speaking at two different conferences in Vancouver held in celebration of Right to Know Week.  I previously shared some of my thoughts about the first conference and these are my thoughts about the second one which was the Open Data Information Summit held by the Office of the Information & Privacy Commissioner for BC, the BC Libraries Cooperative and the BC Ministry of Education.

The day started with an engaging and inspiring talk by David Eaves (@daeaves).  David covers a lot of interesting topics in his talks but the one that stood out for me the most was his discussion about the political nature of data.  As more public data is made available to the public, decisions have to be made about what to release and what data to hold back.  Currently, those decisions are typically made one by one for each individual dataset, and there is a risk that those decisions might be made according to the interests of status or authority within an organization rather than on matters of principle.  What David pointed out though is that this political nature is not unique to open data.  It’s a property of data itself, whether or not it’s open, and it’s a challenge worth thinking about.

As I prepared my talk for this conference, I knew many of the participants would be professional librarians and archivists.  It occurred to me how important libraries and archives are to our societies, and how the challenges we face in the open data movement are the very same challenges that the proponents of the original library idea must have faced.  

I imagined the response that the first poor soul who came up with the idea must have faced.  “You know all these books we have?  I have this great idea.  Let’s build a great big building and take all of these books and make them available to the public, for free!”  

It’s easy to see why at the time this may have seemed like a crazy idea:

• Most citizens can’t even read.
• The vast majority of people don’t even care about books.
• Who is going to pay for this?
• Who is going to read these books?
• They will interpret the books incorrectly.
• If everyone learns how to read there will be chaos!
• What if someone uses a book to do something evil?
• How will authors make a living?
• How will publishers make a living?
• Who is going to pay for a book if we make them available for free?

Or, even after the library was formed:

• We published those books a year ago... and still only a few people are reading!

As it turns out, we now recognize and value libraries and literacy and it’s hard to imagine that it wasn’t always like this.

The challenge of data literacy is one that came up again and again during the conference,  and is a challenge I think we would do well to more rigorously address as an important skill for the future, particularly for our kids.  

It's here that open data presents another valuable opportunity.  I have had the privilege of leading many open data hackathons attended by young people and I have seen how much fun they can have working with data when given the opportunity.  I think would be great to see courses offered in schools that address data literacy topics.  It also be great to see kids empowered to do school projects using open data about their own community or municipality or province.

I was reminded of the talk that Danny Hillis presented at OSCON 2012 in which he describes the contribution his school librarian made to his life when he was a 4th grader.   Librarians in general, but school librarians in particular, are perfectly poised to introduce kids to a wide variety of fields of learning that they may find interesting and useful but that perhaps aren’t yet offered by the school system generally.

And, while I am optimistic about the classroom and think teachers will increasingly find opportunities to weave data literacy into their work, much as happens with reading today, I think that librarians are well positioned today to kick start the process and to help our kids discover the fun and usefulness of data.  

By the end of the day the summit was buzzing with inspiration and excitement.  It was clear to me that the attendees saw the opportunities for learning and sharing that open data offers.  I look forward to working with librarians and educators and to seeing where they take it from here.

BC Information Summit Thoughts

Last week I had the privilege of speaking at two different conferences in Vancouver held in celebration of Right to Know Week.  I want to share some of my thoughts about the first one here.  I will post another article to cover my thoughts about the second conference.

The first event I spoke at was the BC Information Summit held by the BC Freedom of Information and Privacy Association (FIPA).  FIPA is a non-partisan, non-profit society founded in 1991 to promote and defend freedom of information and privacy rights in Canada.  The event focused on open data and Government 2.0 as well as current privacy issues arising out of current provincial IT efforts.  I believe some of the conference was captured on video.  If I find out where it's posted I will let folks know here.

Many interesting topics were discussed but the two that resonated most with me with respect to open data were, 1) that open data stands on the shoulders of many people who have worked hard for us to have the right to access government records; and 2) open data is no substitute for FOI.  In other words, while open data represents a significant opportunity and can and will save money by governments being proactive in making public data available readily available, the right to access records held by our governments is a fundamental right and we should promote and exercise that right.  

It's with some embarrassment that I admit I have never filed an Freedom of Information (FOI) request to access public information.  This is a fundamental thing to be able to do in a free and democratic society and I live in a free society and I haven't used that right, not because I haven't wanted to, but partly because of what I perceive as the work involved and partly because I have somehow thought that it's wasteful for me to do.  Like my questions are not important enough or generally applicable enough to exercise this right.  That's just not true.

With respect to the BC provincial government we have the option to ask for data to be released as open data via the Data BC site, but additionally, if we just want access to some data we have the right to make an FOI request for datasets.  Of course data obtained this way will not fall under the BC Open Government License, but for some public data that may be sufficient for now.  The folks releasing data have limited resources and are prioritizing releases of data the best they can with the resources available.  FOI is one more way we can help figure out what's important.

I see the Freedom of Information community and the Open Data community as having little overlap so far in terms of people and membership, yet much in common in terms of interest and opportunities.  I will be looking for ways for the two communities to complement each other and help our public servants with their open data efforts.

I will also be filing an FOI request sometime in the near future to learn more about how it works.

Why Supporting the CIRA Election Matters

Canada is a world leader in internet technologies.  We have some of the brightest and most innovative people and companies in the world inventing new technologies that are shaping the future and giving us a competitive edge.

We typically think about the Canadian Internet Registration Authority (CIRA) in terms of the .CA Internet Domain.   CIRA is the organization given authority by the Government of Canada to act as the registry for the .CA Internet Domain and to provide professional registry services.  However, CIRA also has the responsibility to develop, carry out and support other Internet-related activities in Canada and to do things to attain all of these objectives.  Thus, it’s a wider scope than name registration.  While name registration is important, other factors also are becoming increasingly important for the Canadian Internet.  

The internet is becoming integral to how businesses and governments operate as well as how citizens of Canada interact with each other and with organizations.  In addition, a highly effective, functioning and open internet is critical to supporting the flow of open data and information needed to solve many of the challenges facing governments and citizens.  

As a platform for innovation and job creation in Canada, the stability, sovereignty and integrity of our Canadian Internet is critical for Canadian businesses to be able rely on it for their business needs and continue to invest in research and technology development that will keep us at the forefront of the internet frontier.

The internet is vastly different than it was when CIRA was created in 1998.  Back then, .ca name registration was the primary objective and other internet related issues were secondary.  Today it is imperative that the Canadian Internet be recognized as key infrastructure and as something that Canadians rely upon and are key stakeholders in.  To date, Canada has taken a largely “go with the flow” approach to how we operate our part of the Internet.  We’ve had a lot of success, but increasingly we are seeing challenges that we need to address to ensure continuing success.  Challenges include technical challenges such the deployment and adoption of of IPv6 and DNSSEC, integrity challenges such as corporate influence, and political challenges such as interference by foreign governments.  Meeting these challenges requires concerted effort by Canadians who are dedicated and passionate and also up to technical demands of the problems.  

For these reasons and others, I am a member of CIRA (any Canadian who owns a .CA domain name can apply for a CIRA membership).  Being a member enables me to have a say in how the Canadian Internet is run, as well as vote for the candidates that I believe are best able to meet our upcoming challenges in the Board of Directors elections.  

I will be voting for Kevin McArthur in the upcoming election.  Kevin has many skills and talents.  In this context, the ones that I see as critically important are that he understands the Internet, the technologies and what is at stake for us as Canadians; he is a successful business person, entrepreneur and innovator; he is a supportive and outspoken community leader; and he is passionate about what’s best for Canadians.  You can find out more about him here.

If you take a moment and think about how the internet has affected your life and your work, and how much you rely on it I hope you’ll agree that it’s worth your time to take part in this election.  I invite you to register with CIRA, become a member and cast your vote.

OSCON 2012

Last week I attended OSCON in Portland Oregon, to sip from the open source software firehose, as I have for the past 5 years.  If you are not familiar with OSCON, it stands for "Open Source CONference" and it is the premier open source software conference attended by about 3000 people each year.  It's hosted by O'Reilly.  You can find out more about it here.

The Clothesline Paradox
Tim O'Reilly, the founder and CEO of O'Reilly Media, talked about what he called the "clothesline paradox".  The clothesline paradox points at the fact that if you dry your clothes in an electric dryer, the energy used is measured and accounted for, but if you hang them on the clothesline to be dried by the sun, the energy saved disappears from our accounting!  

He related this to open source software and crowd sourced content.  The enormous amounts of value created by open source software developers and people using that software to contribute interesting content on sites like Wikipedia, Blogs, YouTube, Facebook, Twitter and Google+ goes completely unaccounted for.  We have done a great job of creating value, but we haven't yet told the story.  When we do think about how open source software is doing economically, we think of the few companies that have monetized open source directly, like Red Hat, MySQL or WordPress as the size of the open source economy.  What most don't realize is that it basically powers everything on the internet and many of the devices used to access it.

Talking about the value of open source in terms of licenses sold is like talking about the value of solar energy in terms of sales of solar panels without taking into account that it actually grows our food, provides warmth, is the source of many other energy sources used on the planet including oil and generally just supports all life on the planet.

DevOps on the Fly
Mark Shuttleworth, the CEO of Canonical (the company that produces Ubuntu) delivered a keynote address where he demonstrated their new juju DevOps suite.  Juju is designed to deploy and maintain cloud systems.  It dynamically creates VMs, installs and configures software on those VMs, and even sets up the relationships between the VMs so they automatically start talking to each other.  It's magical.  While some have a hard enough time showing off pre-installed operating systems without unexpected "blues", Mark on the other hand created, configured and deployed a private cloud on his laptop in front of 3000+ attendees.  That takes something.   Five percent of all new computers that ship next year will ship with Ubuntu.

Simple Inventions that Change the World
I also attended a talk by Ward Cunningham the inventor of the Wiki.  Ward invented the Wiki in 1994 and made it freely available to the world shortly thereafter.  By 2001 Wikipedia was created based on the Ward's Wiki concept and it now contains 22 million articles.   Ward also is one of the founders of agile software development and co-author of the agile manifesto.  One of my favourite quotes by Ward is "Do the simplest thing that could possibly work".  It's hard to exaggerate the influence Ward has had on technology.

Ward demonstrated one of his current projects, a collaborative writing tool called the smallest federated wiki.  Although it's very much a work-in-progress it was simply amazing in many ways, so much so that it's hard for me to describe in a few words other than to say it's sort of like a mashup between wikis, GitHub and Diaspora.  I don't know of a working demo of the software at this point but will keep my eyes open for it and in the meantime I will be checking out the source to see if I can put up an instance.

OSCON is my favourite conference of the year.  Not only is it highly professional, inspiring and visionary, it's only a few hours away from us here in BC.  I highly recommend to developers and anyone else interested in where IT is going to attend this conference next year.  If you would like to learn more about OSCON or open source software and how it can be used in your organization, please feel free to get in touch.  I hope to see you there next year!

ThinkHealthBC

Last week I attended the launch of a new initiative for British Columbia by the BC Ministry of Health, called ThinkHealthBC.   As part of the initiative the BC Ministry of Health has launched site  (http://www.thinkhealthbc.ca/) with a forum where citizens can discuss health care issues, including care and policy issues and participate in the process of health care delivery in BC.  I see this as part of the provincial government's continuing effort to engage citizens in discussions about their province and the decisions affecting them.

The morning started with a discussion led by Minister Michael de Jong.  Though I had never met him in person, I was already a fan of Minister de Jong because of his early support and adoption of open data and his stand on being accountable to citizens.  He led an engaging discussion and impressed me as an effective leader, as well as pragmatic, sincere and generally very likeable.  I believe he is creating an environment that fosters discussion and innovation where the best ideas and solutions can emerge.

Here's what I got from the general discussion with Minister de Jong:

• Healthcare costs are the number one issue facing [provincial] governments and may well be the issue of the decade;
• Costs have doubled in the past ten years to the point where nearly ½ of every tax dollar is spent on healthcare;
• 1% of the population consumes ⅓, and 5% consumes ⅔, of healthcare costs;
• Individual costs are decreasing (e.g., pharmaceuticals cost less, surgeries cost less than ten years ago), but expectations are rising and costs of new treatments higher; and
• Our generation may be the first generation in history that does not live as long as our parents.

My first reaction to the issue of escalating health care costs is resignation.  What can I do?  I am not a health care expert and I am not included in the discussion.  I just pay for it and it continues to consume more and more of my money.

This thinking clearly flies in the face of my role as an open government and open data enthusiast.  With governments making more and more of our data available to us, we are being presented with more of an opportunity to help out.  Further, with new inclusive initiatives like ThinkHealthBC, the "I'm not included in the discussion" argument holds less water.  I expect that this trend of governments making our data available and finding ways to engage with citizens will continue.  Our role as citizens is expanding from merely voting once every four or five years, to having the data and tools to analyze problems, propose solutions and participate in policy making.

Viewed simply, the health care cost issue occurs as a huge problem.  But, given that it consumes 42 percent of our provincial budget, an alternate point of view could be to see it as a huge opportunity.  Small positive changes here can have a huge impact on the cost and delivery of health care.   As a result of attending this discussion and my involvement in various projects,  including my role at MD DataBank and my open data projects and advisory roles, I will be thinking of ways that I personally can contribute to solving tough problems and making our country and provinces even better.

As I can often be heard saying, we can no longer expect governments to do all the thinking and all the doing.  Not only can we not afford it, but more and more there is recognition that good ideas can come from anywhere, including from you.  

Join me in visiting http://www.thinkhealthbc.ca to see how you can get involved.

My Canadian Open Government Consultation Submission

Some of the questions posed in the Canadian Open Government Consultation Submission required written responses.  Here are my responses:

question 1: "What could be done to make it easier for you to find and use government data provided online?"
To make data easy to find I would like to see all of the available data along with descriptive text, preferably one dataset per page, published on government web sites that can be indexed by Google.

To make the data easy to use, I would like to see the data published in open non-proprietary formats. The data should be addressable by URL, without any requirement to sign in or identify or be required to click on anything extra in order to download the data as these sorts of mechanisms are obstacles to performing repetitive automated tasks.

The data should be either released as public domain, or if that's not possible then it should be licensed under a legal framework that conforms with the open data definition posted on opendefinition.org. The Open Data Usability Index (ODUI) document at OpenDataBC also provides helpful information about how to make data more usable. http://www.opendatabc.ca/odui.html

question 3: "How would you use or manipulate this data?"
I would use the data to create analysis and visualizations that would help Canadians better understand their country. I would create value added applications that would help people to create businesses and jobs using open data. I would also try to help people use the data, by providing online tutorials and holding events that would encourage people to get involved. I would like to help create the incentives for citizens of Canada get more engaged and participate more with their governments (local, provincial, federal).

question 4: "What could be done to make it easier for you to find government information online?"
Government information should be made available to search engines like Google by providing topic specific pages, in text that search engines can understand so that people can use the tools they are used to to get information. There is no need to create an elaborate portal and specialized search engine for this sort of effort, at least in the foreseeable future.

Specifically about Open Information, again, it should all be either released to the public domain or if that's not possible licensed using a license that conforms to opendefinition.org. It should not be given a different license than open data if you want people to be able to use it.

question 7: "Do you have suggestions on how the Government of Canada could improve how it consults with Canadians?"
I don't have a lot of ideas here other than make use of the available social media tools such as Twitter and Google+ to reach out to Canadians. I don't always agree with Tony Clement, but I do like the way he engages citizens. I would like to see more MPs do the same. Also, adopt social media guidelines like the Province of BC has, which fosters trust with its employees and allows them to engage directly with citizens.


question 8: "Are there approaches used by other governments that you believe the Government of Canada could/should model?"
As far as open data goes, the legal framework is a big deal and why I keep advising governments to work hard to get this part of open data right. Follow the example of the US or Australia or New Zealand or Surrey or Langley or Township of Langley or Winnipeg Transit.

In Australia agencies are supposed to go through a process in order to keep data closed. (There is currently no binding law to enforce this but the intent is there). This would be relatively easy to put in place in many jurisdictions as data is already routinely assessed for privacy concerns and treated accordingly. Determining if it should be closed would be just another check box, and since the vast majority of public data should eventually be open data, it makes sense to identify and document the much smaller set of data destined to be closed.

I think there is an unintended side effect of the way open data is being rolled out in some jurisdictions. Public data that was once generally considered usable ( financials, stats, directories, registries, codes, inspection data, schedules ) are now considered not usable because they don't have explicit permissions associated with them. When they do get an official license applied to them, their new explicit "open" status can be significantly more restrictive than the implicit understanding prior to the movement.

Also, governments often set up a new agency that identifies data citizens can use and place it in a catalogue, and anything that's not in the catalogue then has a mysterious cloud over it by contrast, even if it was formerly considered public. Depending on how fast this new group can get through the relevant data to catalogue it, citizens could feel they have significantly less data than they had before the official process.

Everything already on the internet should just be open licensed.

question 9: "Are there any other comments or suggestions you would like to make pertaining to the Government of Canada's Open Government initiative?"
Thank you for taking the time and putting effort in to address this opportunity and to consult with citizens. I think we, as a country, have a lot to offer the world in terms of leadership in this space.

2011 in Review

I find that the end of the year is a great time to reflect on the past 12 months and refresh the vision for the upcoming 12 months. It works well for me not only because of the changing year, but also because the holidays bring a brief window of downtime, which I like to use to, as Stephen Covey says, “sharpen the saw”.

Anyone who knows me knows that I am a avid supporter of open government and open data. My personal belief is that open government is what’s next for modern democracies. Public servants and citizens are starting to see for themselves the strategic advantage that openness represents, something that the open source movement has known for decades. As more politicians and high level public servants start to realize the positive effects that openness can have on their work and their personal careers we will see more adoption. Soon we will look back to this period and it will seem archaic for governments to keep critical decision making information from the people they serve.

My personal vision for 2011 was that by the end of the year people in BC would be more aware of open data. When we started 2011 most people I talked to had no idea what open data or hackathons were. Now, 12 months later, we have had 11 hackathons in 5 cities in BC, 350 hackathon attendees, several news paper and other media stories, we have a ministry with Open Government in its name, a provincial government that publishes open data and we've had 5,000 unique visitors to my OpenDataBC.ca site viewing 35,000 pages of content.

One of my heros, Seth Godin, popularized the term shipping which describes the act of completing a project, getting something out the door or delivering something. It doesn’t matter if it was a hit or not, or whether it was perfect or not. All that matters is that it is done.

Here’s a partial list of things I shipped in 2011:

• Designed a system to enable secure storage and sharing of health information
• Continued to build the OpenDataBC google group which now has 116 members
• Created the Open Data Usability Index with other open data experts
• Held the 2nd annual Victoria International Open Data Hackathon
• Facilitated 11 open data hackathons around the province of BC
• Added 170 datasets to the OpenDataBC catalogue
• Presented open data at several conferences
• Advised several governments in Canada on open data strategy
• Developed Municipedia, an app to improve Voter turnout in the 2011 Municipal elections
• Contributed to several other open data projects
• Proposed a change to the Open Definition
• Launched an HTML5, Python based data visualization platform
• Attended several conferences including Google I/O, EuroPython and OSCON
• Developed a tiny open source split testing framework for Python
• Purchased several Internet ready music albums
• Gave $500 to causes I believe in (EFF, FSF, Demand Progress)
• Moved 55 domains from GoDaddy to DreamHost
• Wrote 30,000 lines of code
• posted 1,900 tweets
• Wrote 9 blog posts

I was very honoured to be able to work on these projects with so many great people in 2011.  I worked on many other projects as well, some of which are not complete so they’ll be my 2012 list. :)

Now, I will get to work on my vision for 2012.  I will post here in the near future.

Happy New Year!

The Value of the Seal

The implementation of government open data initiatives presents a number of interesting challenges to governments on the leading edge of Gov 2.0 policy making. Established business practices invented in another age sometimes seem impervious to ideas like open data. One such practice is charging money for public data under the banner "cost recovery".

When looking at the bureaucracies and systems required to manage the authorized distribution of and the collection of payments for data (i.e. products whose marginal cost of distribution is effectively zero) it's easy to wonder if the operating costs are mainly due to the billing apparatus itself.

But let's take the case where there is in fact a cost to distributing this data. Perhaps there is a collection cost or some sort of initial investment cost. In some cases, governments have chosen to offset that tangible short term cost with a fee for access, to either the data itself or some byproduct of that data.

People or organizations are willing to pay money for this service of data retrieval. From this, one could deduce at least two things: 1) provision of data is a valuable service; and 2) the government is the considered the best or perhaps the only place where they can get this data.

Let's look at the BC Corporate Registry for example. To search for a registered company in BC I have to first become a member of BC Online. To become a member of BC Online I have to first deposit $100 into a BC Online account, and then to perform a search they will deduct $8.68 from my account. People do use this service, so there is some value. People are willing to pay for this data.

Or, are they?

What if BC Corporate Registry data was released as open data. The deal is, I would have access to that data to do what I want with it, including creating an app, and making money, the only condition being that I could not claim it was official.

So, I get the data, I offer the data to lawyers and corporate deal makers and entrepreneurs and whoever else I think can use this data and I offer to sell it to them for 1/2 of what the government currently charges for exactly the same thing except that my data comes with a clause that says something like "this data is 'not official'". My guess is that no one would take me up on that offer. In fact, my guess is that I wouldn't be able to give that data away for free. Why? Because it's not official. It turns out being official is pretty important.

I think that this whole model of people paying for public data is a collapse of two concepts and what's missing is a distinction. People do not pay for public data. People pay for data with an "official seal". I would go one step further. Often the seal is all they are paying for.

This is not a new idea. Just as I can get a certified engineer to come over to my house and talk with me for an hour about the structural changes I want to make to my house, and she'll give me all sorts of ideas and draw me a sketch and perhaps help me with my design ideas, so long as it's all "unofficial". Some will even provide me with an non-certified printout of my plans. To build my house though, I need the official certified plan, and for that there is a fee. I am not paying the engineer for the plan - I am paying her for the plan stamped by a certified engineer.

I suspect this won't even be a surprise to the folks involved in crafting the open data licenses that we use. The clue is in the licenses themselves. For example, if you look at the Government of B.C.'s new Open Government License you will see in clause 5. b) "ensure that you do not use the Information in a way that suggests any official status or that the Information Provider endorses you or your use of the Information".

That clause is there because someone recognized that that's where the value is. It's the official status, or said another way, the authenticity that the government brings to the data that people need and are willing to pay for and often for legal reasons must obtain.

My personal preference would be to have all publicly funded data free and available for any lawful purpose by default. The question of whether or not people should have to pay for the official status of anything is a separate and worthy question. In any case, fees can stay in place for "official" data as there really isn't any threat to existing business models. Releasing the data itself can happen using existing policy and the existing licenses. And, my guess is that in time governments will discover what open source and many other folks already know, that by opening the data, the demand for paid services will actually increase.

And for some young entrepreneur, who wants to build the next iPhone app to accelerate the process of creating a new business in BC, that in turn could create jobs and attract talent to our province, that's good news. Because his app doesn't actually require that the data have the official seal.

He just needs data he can use.

Internet Ready Music

I love music. I love listening to it, and I love creating it.

I used to buy Vinyl records, then switched to CDs and now I buy mostly online. I have created mp3 files of most of my CDs so I can listen to them in one big huge playlist without ever putting in a CD.

A couple of months ago, Google invited me to use their new Google Music Beta. It's awesome. With it I can easily place all of my music on their servers and then access that music from any of my computers or Android devices.

But I haven't done that. Why? Because the record companies have been attacking people use the internet to enjoy music. And, while I don't think it breaks any laws to use Google Music Beta, I am not 100% sure. Even if it's 100% legal today, with the music industry lobbying governments and influencing law makers and suing students, and now getting the laws changed so that ISPs are required to spy on us, it's not inconceivable that they could convince law makers to make my uploading of music to Google Plus illegal, retroactively. And since once I upload music, there is really no way to make sure it's removed from wherever I upload it, uploading is an action that I can't really undo.

It seems more and more that the vast majority of digital music that I have purchased thus far is becoming legally incompatible with internet technologies and it's actually becoming increasingly risky to own it at all. My iTunes purchases have declined as I have become more aware of the associated risks of owning music that's incompatible with the internet.

Just like driving a horse and buggy on a modern freeway today would be considered dangerous, using music locked into outdated copyright schemes that prohibit uploading on something designed for uploading and downloading is dangerous.

Some of the music I own, however, is compatible with the internet. Bands such as The Charlatans, Radiohead, Nine Inch Nails, and others have been making their music available for free on the internet for some time.

I have also recently (re-)discovered Jamendo which is a music site that contains 300,000+ songs from all over the world and in all genres, all of which are available for downloading and can be uploaded and shared freely with others using the Creative Commons license.

With Jamendo I have discovered some new music that I am totally hooked on, such as the fantastic Canadian artist Brad Sucks, who you can currently find me listening to most days.

Since the internet is all about sharing, I am starting to call this new music, "internet ready". I can upload it to Google Music Beta without worrying about any scary legal incompatibilities down the road, and I can share it with Friends.

Incidentally, I still pay for much of this "free" music. Usually $20 an album. I like that the whole $20 goes to the artists because I want them to be able to continue to create great music. But mostly, it's worth more to me than the risky legacy "recording industry" music because it's internet compatible and as such I am free do so much more with it.

About the License

Yesterday while we were all celebrating the awesome work of the BC Provincial government, a very well respected Harvard researcher, software developer and open data advocate was arrested by the US Federal government on charges related to computer hacking, based on allegations that he downloaded too many scholarly journals that he was entitled to get for free.  He now faces a possible 30 years in jail.

I don't know Aaron, but I feel as though I do.  He writes brilliant software code and releases it to the world as completely free public domain software.  He advocates for open data and transparency and democracy in the US and he is a founder of demandprogress.org an organization dedicated to progressive policy changes for ordinary people.

Please consider visiting demandprogress.org and reading about what's happening to Aaron.

How does this relate to open data and the new DataBC portal?

It's about the license.

License : official or legal permission to do or own a specified thing.

It's strange that we need permission to use our own data and information.  It's strange that we sometimes have to pay to use data that we have already paid to have created.

Being an open data advocate and application developer comes with a certain level of risk and anxiety.  We are actively trying to do things that have not been done before.  And unlike other areas where innovation takes place, we are innovating in an area involving strange legalities, usually as individuals with no corporate backing or protection, and the consequences of making a mistake can be severe.  There is a lot of uncertainty.  Although many of us are open source developers and thus are pretty familiar with copyright law, licensing and the jargon that goes along with them, very few of us are lawyers.

This is why we want governments to use standard licenses.

By choosing to invent a new license rather than use an existing one the BC Government has added to the uncertainty.  Yes, they based it on the UK license, but it's clearly not the same as the UK license, otherwise they could have just used that.

Because they chose to invent a license I spent several hours last night pouring over the license and comparing it to both the PDDL and the UK license to see where those differences are and to see what additional risk I might have to take on as a result.  Every developer I know will have to do the same thing now before they start using the data.

Many won't bother.

And that's the lost opportunity. People who get turned off by the custom license won't use the data, or won't bother coming to the hackathons, or won't bother creating that new app. It's just too risky. Sadly, we all lose. because as I understand it, the BC Provincial government is in this for the right reasons.  It's clear to me that they absolutely get it.  Innovative new ideas and applications will be generated as a result of this.  This increased transparency and engagement and collaboration with the citizens will build trust and goodwill and is good for the government and good for the people of BC.

From what I can tell, not being a lawyer, as a standalone license, the BC Open Government License is actually mostly good (check out unrest.ca for some of the details on issues with the license).  And, there are a handful of us, that will push through this licensing thing, grumble a bit, say "it's pretty good" because it is, and weigh our risks and move forward with our apps and visualizations and innovations.

BC is seen as a leader in citizen engagement and open data by local governments, other provinces and internationally.  Looking at what's going on in the rest of the world, particularly in the US, we are really very fortunate to live where we do and to have the public service and leaders that we have.

I will encourage others to take the time to read the BC license rather than blowing it off because it's not standard.  And I will continue to urge local governments and other provinces to use a standard license rather than invent their own.

This is a first great first attempt, and as Christy Clark said in her excellent and encouraging video, this is very much a work in progress.  The license does have a version number, which to me implies that they are open to input, discussion and changing it if necessary, which is awesome.

Remembering "Open"

Today the British Columbia Provincial Government launched a new Open Data Portal, making thousands of our publicly owned datasets available to us including everything from employee salaries to historical school locations to Local Government Incorporation Dates to the data catalogue itself. As a citizen and taxpayer in BC and an open data advocate I am very excited to see my own Provincial Government take these steps toward innovation and transparency. I congratulate those public servants within the BC Government that understood the opportunity, recognized value and championed the cause.

These days, governments all over the world are starting realize the value of "open" but it wasn't so long ago that we were at the opposite end of the spectrum. As a public servant in the 1980's, employed as a junior data analyst, I personally produced the monthly report for the minister and deputy minister showing the basic metrics of the ministry I was working for. As part of that job I was required to produce 5 copies of that report and place them in brown paper bags and tape them closed. I would then personally attempt delivery to the recipients office. If the recipient or their assistant weren't there to receive the report, I was required to take the report with me and try again later.

Those same metrics are still being used today and were released today as part of the Provincial Open Data portal. It's striking how far we have come in such a short amount of time.

Although we have gone through a very opaque phase with our governments, the idea of governments being open and transparent is not actually new. Our own BC Government has been publishing the public accounts and other financial information for decades. They also produce a monthly publication called the British Columbia Gazette and have done so since the 1920's that is teeming with useful information about our province, from disposition of Crown Lands, to election results, to tree farm licenses to road name changes.

Technology has evolved since these publications were originally developed. Where at one time publishing this type of data on paper was about as usable as one could have imagined, these days its available electronically, and hopefully soon it will be included as part of the open data portal.

So, although openness and transparency aren't new, they were definitely forgotten, and as I like to say, we are now remembering the value of "open".

Congratulations and a big "Thank you!" to our public service employees and political leaders who are helping to make this happen.

Now, I need to go to the portal and look for some data. :)

Government as Platform - An Example

Tim O'Reilly uses the words "government as platform" to describe an interpretation of what Government 2.0 is. I am often asked what "government as platform" means. I think the question arises because much of government already operates as a platform. People don't distinguish it as something new because it's routine.

To explain what government as platform is, you can look at an example where it's already the norm such as our public roads. Our three levels of government are involved in the construction, maintenance and regulation of roads. Together they deliver infrastructure upon which we ride our bicycles and drive our cars.

We are used to the idea that we can hop on our bicycle or get in our car and and transport our selves on public roads. As a platform we are left to decide where we are going, when and how to get there. The outcomes of the platform, the actual transportation conducted, is not determined beforehand. The roads are built to certain standards and the drivers are left to figure out how to get from point A to point B.

And because the system is open, people and companies can and do invent innovative new ways to use the platform. Because they are free to choose their route, people optimize the routes they use to get from point A to Point B.  Inventions like automobile GPS are created to help with navigation.  Every year the entire vehicle industry (bicycles, cars, buses) releases new versions of their transportation products and introduces innovative new ones. Several huge industries can rely on the fact that our roads systems is a platform that they can build on.

And for travelers, the system works so well we can travel almost anywhere and we're usually not even aware of which level of government is responsible for which roads. We can even travel to different countries and travel, because all of the public roads work pretty much the same way. Done well, the platform becomes transparent. We don't even notice it.

I am convinced that this what we need with our public data. Our data – all of it – made freely available on the internet, with standard licensing, in formats we can use - would provide a platform for innovation like we've seen with transportation. Entire industries could grow on such a platform, providing jobs and value we can barely imagine right now.

And though only a few people will use the data in the beginning, like the few people building cars and bicycles, those people will create huge value for their fellow citizens all built on a platform managed by our governments.

Selling Data - Privacy Is Not The Issue

When data can't be released it's usually for one of two reasons, privacy or cost. Leaving cost for a future post, let's focus on the privacy issue.

We don't want our personal medical records released to the public for free or for cost, for example. Most people I talk to can agree on that. In fact, in BC, we have laws that state that personal information collected for one purpose cannot be subsequently used for a different purpose. There are exceptions, but that's the general idea.

Sometimes governments sell data to the public. Sometimes that data contains people's contact information. Sometimes it's about organizations or places. But, before it can be sold, this data typically undergoes rigorous processes and checks to ensure that no personal data is compromised.

For example, for $32 anyone can request a business name search from BC Registry Services to find out if a company exists . For "as low as $5,250" anyone can purchase a business site license for postal code address data from Canada Post. And, anyone can download a wide variety of key socioeconomic data from Statistics Canada - for a price.

How does this relate to open data? Well, sometimes, privacy is held up as a reason for why some data that is currently sold by governments, cannot be freely released as open data. As if, somehow, paying money for that data alleviates any privacy concerns. It doesn't, because there aren't any. If there were, not only could it not be made available as open data, it also could not be sold.

While there may be very good reasons why data that is currently for sale by governments cannot be made available for free, privacy isn't one of them.

Requirements of Open Data

As more and more governments start to realize the benefits of opening up their data, sometimes I hear chatter about enterprise solutions and though I understand the logic of hitching the wagon the the latest thing, I think there is a lot to gain by taking a strategic and pragmatic view of things. Open Data does not have to be an expensive exercise. In fact it can be very inexpensive from the data publisher's point of view.

There are three main requirements that have to be satisfied before data is considered "open data". They are:

1. Legal Framework - Anyone can use it for any legal purpose (PDDL or CC0 license)
2. Accessible - I can download it on the internet free of any mechanisms of control
3. Readable - it's published in a non-proprietary format

Of these three, the first requirement is most important and costs the least. And, in fact, without the first one there is really no point in doing the other two. If it's not legal for me to use the data then it doesn't matter what format it's in or whether I can get my hands on it... I won't use it.

The great thing is though that governments in some cases have already done the other two steps so the Legal Framework is all that's left to do. And, if they do that, then instantly and without any expensive technology, a raft of published data becomes "Open Data", ready to use.

For example, the City of Courtenay publishes a great RSS feed of "Surplus Equipment for Sale" but nowhere on their site does it say that I can use it. It's both accessible and readable so it's satisfying two of the three requirements but the first requirement isn't met so developers would likely shy away from using it. That's too bad because an app that went around and gathered up this type of information from all local governments and made it available as an mobile app would be pretty cool and would help the local governments sell their surplus equipment.

The BC Government on the other hand has this page which looks pretty much what it looked like over 10 years ago, and pages like this that basically say you can't use this data to make anything.

Here again, there are many examples of data that is both accessible and readable, but because of these pages, we can't use it. And that's unfortunate for both the government organizations that could benefit from the huge talent pool outside of government and for the citizens who pay for the data.

The good news is that governments already have a large amount of data online, and by getting the legal framework sorted out, citizens will instantly be able to use it to create innovative solutions and tools to help themselves and others.

Book: Rework

I just finished reading Rework by the 37 Signals team (Jason Fried and David Heinemeier Hansson), a Christmas gift from my youngest son.  If you are not familiar with 37signals and their work I recommend checking them out.

The book provides an outline of the 37signals philosophy including tips and opinion.  It is aimed at small businesses, however, I think everyone from small business to large business to government organizations should read this book and think about how it might be applied to their work.

They don't say this explicitly but what the book is really about is rethinking some of the dogma of business-as-usual.  I have often thought that any organization that has an "innovation" branch is already in trouble.  If you read this book you'll find out why.  An innovative culture isn't something you can install, or force directly.  Innovative cultures happen by consistently rewarding innovation.  Sharing cultures happen by consistently rewarding sharing.  Organizations that consistently treat employees as untrustworthy, end up with a culture of fear and lack of trust.

For examples of how to do it right in a government context, I think the Province of BC is on the right track with their new B.C. Government Social Media Guidelines and their public statement of Open Data as Defining Principle No. 1 of their Citizens @ the Centre: B.C. Government 2.0 strategy. The very fact that these documents exist sends a signal to employees of a relatively large organization that they are trusted and empowered to engage citizens and empower citizens to create value from open government data.  This kind of positive reinforcement will go a long way to creating a culture of learning and trust as people come up to speed with the new tools of social media and open data.  Kudos to the folks that made this happen and the executives that supported them.  I look forward to seeing what they do next.

Creating an environment where innovation happens, where sharing is rewarded, where great work is recognized and where trust is leveraged is the hallmark of an organization that gets it.  37signals definitely gets it.  Rework is an easy and worthwhile read.  If you're interested in innovation in the workplace, I recommend you read it.

Shipped in 2010

One of my favorite authours of all time is Seth Godin.  I purchase and read every book he writes and often give them as gifts.  One of the things Seth talks about is "shipping".  We use this term to describe the act of completing a project, getting it out the door.  It doesn't matter if it was a hit or not, it just matters that it's done.

In a recent blog post Seth encourages people to publish their list of things they shipped last year, because it's not something we often do.  I encourage everyone to 1) make your own list; and 2) read Seth's blog.

Here is a list of things that I shipped last year:

• Launched OpenDataBC.ca
• Identified 149 BC datasets from all levels of government
• Created OpenDataBC Google Group which now has 70+ members
• Established Open Government conference for BC
• Held two hackathons for provinicial Apps for Climate Action Contest
• Created Waterly.ca which won two awards including Best in BC (Yay!)
• Participated in Google IO and OSCON conferences
• Spoke on Open Data at the Ideawave conference
• Sat on an Open Data panel at the Global Knowledge eGov conference
• Accepted a CTO position with an exciting new Canadian startup
• Held the first annual Victoria International Open Data Hackathon
• Developed DataZoomer version 4
• 25 blog posts
• 500+ tweets

This isn't the entire list of things I worked on.  I worked on many other things that either failed or that I didn't ship (yet).  I also didn't do this alone.  I was fortunate to be able to work with a bunch of dedicated and talented people this year.

2010 was a great year of learning and I look forward to an exciting 2011.

Terms of Use

I am not a lawyer, and I do not give legal advice. I am a developer who uses open data on a regular basis and as such I spend a lot of time, but probably not nearly as much as I should, trying to understand open data licenses and terms of use that governments post on their open data portals.

The whole point of open data is to liberate data so it can be used. An organization's open data strategy then should be working toward that end result, encouraging and making it as easy as possible for people to use the data.

One of the first things that developers think about when contemplating writing an application using government data is, "am I going to get in trouble". This question, however absurd it may seem is very real to developers. If developers think they are going to get into some sort of trouble using a data source they will usually not create the app, which means governments and the citizens they serve lose an opportunity.

If governments are going to release data the most important thing is to release data in a way that is easy to understand from a legal perspective, preferably in a way that developers are already familiar with. There are already many licenses in use so inventing new licenses rather than releasing data under commonly understood mechanisms is a waste of effort on everyone's part.  As Chris Rasmussen spoke about at the recent OpenGovWest BC Conference, "We all think that our data is unique.  It's not true."

Unfortunately, many custom licenses currently in use today are often full of things that don't need to be there like, "you cannot break the law with this data" or "you can't say you're us".  I already know I can't break the law.  A disclaimer makes sense - but it doesn't need to be part of a license.  Being clear about preferences around attribution make sense, but these can go in a policy statement offered for clarity rather than in a license.

In my opinion, the best license is no license at all. It's just public domain. Many government organizations consider their open data as public domain but don't go that extra step and actually state it on their web site. That's unfortunate because it's by far the simplest and easiest and least expensive way to release data and by not stating it explicitly on the web site, developers are still left wondering if they'll get sued.

  Herb's Ideal Open Data Declaration
   * This data is in the public domain.
   * It comes with no guarantees.

Please consult with lawyers that "get" open data.  See if you can go public domain explicitly rather than implicitly and or consider using the Creative Commons Zero tool, before liberating the data and see if you can work together to make data we can all use.

The Power of Open

The OpenGovWest BC conference is now complete. The day was filled with amazing speakers, amazing speaking formats and amazing topics.

One of the highlights of the conference was the talk given by Nick Charney (@nickcharney) and Walter Schwabe (@fusedlogic) where they talked to the audience about participation, and as part of their talk unveiled a blog where folks in the conference were encouraged to participate in real time, right there, while they were talking. Now, days later, blog posts are still being generated on http://www.opengovnorth.ca by individuals and the enthusiasm is still present.

Nick and Walter took a risk. They put the idea out there, provided a place for it to happen and then made a simple request for participation. Though they are both accomplished bloggers they didn't tell people what to write, or how to write it, and they didn't try to control the conversation. They shared their ideas generously and provided a space for expression.

When skilled speakers like Nick and Walter encourage audience members share their ideas with each other in real time, while the talk is going on, they are engaging the participants in a vastly larger conversation. And when the talk they are giving happens to be about encouraging this type of engagement then they are really leading by example, in an almost recursive way.

They also weren't trying to promote themselves, or their organisations, or trying to take credit for anyone else's work or building their brand.

No, they were just there as Nick and Walter, a couple of guys encouraging us to take a chance and move a little closer to the edge. Giving us a gift, expecting nothing in return. Within minutes the site was crashing because the server had exceeded its capacity.

In a closed model, people focus on controlling the message and dictating top down what is supposed to happen. This model is built on fear and lacks trust and although many results can be and are generated this way, communities are not. Contrast this with what Nick and Walter created.

I agree with others who have remarked that this particular conference has taken us from a great idea to a movement. I think there are several reasons for this and I want to acknowledge the lead organizer, Donna Horn (@inspiricity), who just like Walter and Nick, used an open approach with her expertise in community building and leadership to support, encourage and then trust the conveners and speakers to create their own parts of the conference. And the result was a level of enthusiasm from the conveners and the speakers that spilled over to everyone else in the conference.

That’s how a community is created and that's the power of open.