GovHack 2016

GovHack 2016 (29 - 31 July)





GovHack overview

GovHack is an annual event held simultaneously around Australia to provide an opportunity for teams to explore, mashup and use open government data over a 46 hour period.

For more information see GovHack.org. To register visit the GovHack Eventbrite page

LINC Tasmania datasets

LINC Tasmania has a total of 20 datasets available via data.gov.au.

The 20 datasets comprise of:
  • 856,915 records from the popular Tasmanian Names Index
  • 55,354 meta-data records of digitised item from the Tasmanian Archives Collection including references to their digitised resources.
  • 20,896 meta-data records for series of items sored with in the Tasmanian Archives Collection.
  • Geotagged location records for 74 LINC Tasmania locations

Here is a full list of all our open-data resources and their retrospective IDs:
  • LINC Tasmania locations

    Tasmanian LINC, Library, Archives & Online Access Centre (OAC) locations by name, address, latitude and longitude. (http://www.linc.tas.gov.au/locations)

    ResourcesFormatID
    LINC Tasmania Locations - 12/04/2016CSV8318261a-0fcc-4794-9435-c0f76765d08a
    LINC Tasmania Locations - 12/04/2016XMLe957f95d-9f24-48b4-9c44-b11c9d6bbd1d
    LINC Tasmania Locations - 12/04/2016geoJSONd0c08118-92b5-41ae-9cef-d12f1a1ca616
    LINC Tasmania GeoJSON Locations - 26/04/2016geoJSON05eda516-8afa-421c-a523-eb8e1fd4a212
    LINC Tasmania GeoJSON Locations - 27/06/2016geoJSONb5552515-5e65-4a03-81a4-37d9988e5bbf
  • Tasmanian Hotels & Property Records (1818 - 1958)

    Tasmanian Hotels & Property Records available via the Tasmanian Names Index. Over 5,000 records.

    ResourcesFormatID
    Hotels & PropertiesCSV4152dd68-88df-4134-9a9a-47e142442762
    Hotels & PropertiesXMLf891b16b-f864-47a2-8ef4-4ecc3e6c3130
    Hotels & PropertiesJSONc392c2fe-ef5e-4ea3-a642-b66bbfdfcd3a
  • Tasmanian Births (1803-1933)

    Over 197,759 records (as at 14/4/2016) of people born in Tasmania including some baptisms collected by the Registrar General (1900-1933 baptisms only) and available from the Tasmanian Names Index.

    ResourcesFormatID
    Tasmanian BirthsCSV8d710758-26ce-46ca-972d-5067db3a75d2
    Tasmanian BirthsXMLa7c05fba-f620-4380-9a5e-b34bbb327fb2
    Tasmanian BirthsJSON4e30dd59-f38d-489a-8ee5-817ac70e5f1a
  • Tasmanian Marriages (1803-1899)

    Over 103,524 records of people who married in Tasmania. Available via the Tasmanian Names Index.

    ResourcesFormatID
    Tasmanian MarriagesCSVfabcdd72-4410-4d56-8abc-799fd33280c8
    Tasmanian MarriagesXML37ce8ddb-858e-468e-a407-fcf9f54a873a
    Tasmanian MarriagesJSON48867082-bfca-44d4-a5ef-4a99152bc426
  • Tasmanian Departures (1817-1867)

    Over 112,500 records of people who people leaving Tasmanian ports, mainly Launceston. Available via the Tasmanian Names Index.

    ResourcesFormatID
    Tasmanian DeparturesCSV264a2c46-70a3-4dc7-80f4-13639014874a
    Tasmanian DeparturesXMLe8b4a2b7-0d77-458a-8526-ebee523280de
    Tasmanian DeparturesJSON3c4c1621-5919-471b-82a0-d8ac33825d45
  • Tasmanian Deaths (1803-1933)

    People who died in Tasmania including some burial records collected by the Registrar General (1900-1933 burials only). Available via the Tasmanian Names Index.

    ResourcesFormatID
    Tasmanian DeathsCSVd6c716f2-b66d-4096-bb8e-fccc454c49a3
    Tasmanian DeathsXMLf4825fdb-17a5-47ab-bb4e-6b9b21bfe6b5
    Tasmanian DeathsJSONbcbed7d3-e0ad-4006-b9da-b4828a83268f
  • Tasmanian Convicts (1803-1893)

    Convicts transported to Tasmania and those convicted locally through the convict system. Available from the Tasmanian Names Index.
    Over 78,504 records (as at 14/4/2016).

    ResourcesFormatID
    Tasmanian ConvictsCSV8d3e12fa-a9fd-4268-9164-6240a97ff44d
    Tasmanian ConvictsXML0107a43d-c30b-4398-9cbf-d44008ef8101
    Tasmanian ConvictsJSON36a08ac4-c7a8-4f29-b383-161408a70ecb
  • Tasmanian Wills (1824-1989)

    Wills and letters of administration of estates registered for probate. Over 104,122 records (as at 14/4/2016). Available via the Tasmanian Names Index.

    ResourcesFormatID
    Tasmanian WillsCSV3e4a55d2-554f-4577-a530-7fcb2d255060
    Tasmanian WillsXML880d70fb-10d3-4982-bf9c-313ac9f774c7
    Tasmanian WillsJSON752573d2-3c05-44c3-b8fa-fcc4b79ac4db
  • LINC Tasmania Digitised Archives

    LINC Tasmania's collection of digitised archives including files, letters, manuscripts, maps, plans, photographs films and more.

    ResourcesFormatID
    Digitised ArchivesCSVee8ec788-77ed-4faf-8ceb-17f046ab2a4f
    Digitised ArchivesXMLb3bbc43c-d8ed-4515-8d13-a925a2e3bb0b
    Digitised ArchivesJSON2eb9e1bd-6e6c-4541-be0d-e3166b5bd84a
  • Tasmanian Convicts - permission to marry (1829-1857).

    Convicts applying to marry free people or other convicts. Over 280,000 records (as at 14/4/2016). Available via the Tasmanian Names Index.

    ResourcesFormatID
    Tasmanian Convicts - permission to marryCSVfd0f27a5-e2af-4ec7-b782-ee791a56acd6
    Tasmanian Convicts - permission to marryXMLc94a7da4-b2e9-411b-b091-015d3a348ca8
    Tasmanian Convicts - permission to marryJSON886d14db-11a9-40ed-9295-e35d267b9fb3
  • LINC Tasmania Archives Series

    Descriptive records for series held by LINC Tasmania

    These records can also be viewed online here:
    https://linctas.ent.sirsidynix.net.au/client/en_AU/tas/search/results?te=ARCHIVES_SERIES

    ResourcesFormatID
    Archives SeriesCSV4ca7e432-e3f6-411d-adf3-1eab2efe2147
    Archives SeriesXML1b03a062-d376-4304-be64-125efc180539
    Archives SeriesJSONec6d7702-ea1b-44ba-a85a-3b61d6082e68
  • Tasmanian Arrivals (19th Century)

    Passengers and ships arriving, mainly in Hobart. Contains surnames A-K, L-Z to be added. Available from the Tasmanian Names Index. Over 52,918 records (as at 14/4/2016).

    ResourcesFormatID
    Tasmanian ArrivalsCSV7d2c5531-94ec-474e-b60c-744b9d0cbbb8
    Tasmanian ArrivalsXMLbf619908-593a-44f2-9d7f-e14629d323a8
    Tasmanian ArrivalsJSON6864c5f2-3489-4ef3-acb7-0b353e2ab5c1
  • Tasmanian Newspaper Index

    The Tasmanian Index provides references to articles in the three metropolitan Tasmanian newspapers; the Mercury, the Examiner and the Advocate, and to most other Tasmanian journals and regional newspapers. The index was maintained from 1966 to 2010. This online version of the Tasmanian Index covers 1994 to 2010 with approximately 20,771 records. The records from 1966 to 1994 are available online as PDF files

    ResourcesFormatID
    Tasmanian IndexCSV973d05c4-2f17-4609-bd98-1b96135d69c4
    Tasmanian IndexXML37c714be-8ddd-4069-b131-bda352979a7f
    Tasmanian IndexJSONf0feae8b-d02a-467c-9329-e37a9c2f38ac
  • World War One Tasmanian photographs (1914-1919)

    Photographs and articles from the Tasmanian Mail and the Weekly Courier. Data indexed by Wendy Knolle and kindly supplied by the Tasmanian Family History Society, Hobart Branch The dataset is available via the Tasmanian Names Index.
    Over 8,250 records (as at 14/4/2016).

    ResourcesFormatID
    World War One Tasmanian PhotographsCSV2c27bb2e-c55f-4606-af72-85d3ed69049c
    World War One Tasmanian PhotographsXML4c2a9c85-12dc-401a-a5b5-07d39f95dbae
    World War One Tasmanian PhotographsJSONe747d968-905c-4b92-9910-95cc824a0fdf
  • Tasmanian Naturalisations (1835-1905)

    People applying to become citizens. Available via the Tasmanian Names Index. Over 1,190 records (as at 14/4/2016).

    ResourcesFormatID
    Tasmanian NaturalisationsCSV6e4a4678-9845-42da-a7e1-99f00aee2028
    Tasmanian NaturalisationsXMLa871cd24-9518-4768-9a44-4731cb582a10
    Tasmanian NaturalisationsJSONe2060a5c-9b42-4f14-9a1b-bcc7bef88fe7
  • Tasmanian Prisoners (1895-1902)

    Over 450 records (as at 14/4/2016) of prisoners in the Hobart gaol. Available via the Tasmanian Names Index.

    ResourcesFormatID
    Tasmanian PrisonersCSV2dc53c91-40a6-4eac-8b12-949bf0278bf2
    Tasmanian PrisonersXML1e1ef73d-b096-455b-95f1-0147549f9bb6
    Tasmanian PrisonersJSON441035ce-d276-4112-a871-94d51bb8714f
  • Tasmanian Inquests (1828-1930)

    Inquests into people's deaths. Available via the Tasmanian Names Index.

    Over 13,750 records (as at 14/4/2016).

    ResourcesFormatID
    Tasmanian InquestsCSV59ca1b56-51a6-47ef-b1be-1d5064697599
    Tasmanian InquestsXML42e0e4a6-3539-48ea-bdb7-40165ab8dcc5
    Tasmanian InquestsJSON980aa7af-45ce-49a8-a813-013d20c52011
  • Tasmanian Health & Welfare Records (1829-1873)

    Includes records from the New Norfolk Colonial Hospital and the Queen's Orphan School. Available via the Tasmanian Names Index.

    Over 386 records (as at 14/4/2016).

    ResourcesFormatID
    Tasmanian Health & Welfare RecordsCSVba3d9d0f-df12-4016-b3b4-23fc84fb77b8
    Tasmanian Health & Welfare RecordsXML35ce31a8-2298-4085-bdcd-cef2f454d4ff
    Tasmanian Health & Welfare RecordsJSONee76eb11-739e-4775-8186-33c7d1022427
  • Tasmanian Divorces (1861-1920)

    Petitions for divorce in the Tasmanian Supreme Court. Available via the Tasmanian Names Index. Over 448 records (as at 14/4/2016).

    ResourcesFormatID
    Tasmanian DivorcesCSV90996377-6eac-4d90-b72c-2eacde42a7d8
    Tasmanian DivorcesXML7db2ad9b-5cd2-420f-83aa-c3b71a6d20e7
    Tasmanian DivorcesJSON278fcf1b-5a21-48eb-82bf-7e7f1ddd3ef2
  • Tasmanian Census (1837-1857)

    Householders in the 1840s and 1850s, not complete for all districts. Available from the Tasmanian Names Index. Over 13,750 records (as at 14/4/2016)

    ResourcesFormatID
    Tasmanian CensusCSV3e8ac94c-0e5a-44dd-9a61-a8a08707650d
    Tasmanian CensusXML80ea84b3-2983-41ec-9489-843f4d338c61
    Tasmanian CensusJSON5ae7c468-b57f-4b8d-a057-94d749fe5042

Data formats

LINC Tasmania provide our open data in to 3 main formats.

  • JSON

    JSON (JavaScript Object Notation) is a syntax for storing data and works well with opendata sets. It's very easy for the user to fetch and manipulate and is easily parsed in many different programming languages. JSON is a good choice for processing data at both the server side and with javascript on the client side.

    Note: Our datasets are very large and are probably too large to be loaded at the client side so we recommend only using JSON at the client side if you are pulling smaller sections of a dataset via the data.gov.au API.

    Here is an example of some JSON data that we provide:
    {
        "1449570": {
            "CATKEY": "(Sirsi) 1449570",
            "FORMAT": "VIEW",
            "INDEX": "Prisoners",
            "LINC_TAS_AVAIL": "Online",
            "NAME": "Allen, William",
            "NAME_FULL_DISPLAY": "Allen, William",
            "PERMA_LINK": "https://linctas.ent.sirsidynix.net.au/client/en_AU/all/search/detailnonmodal/ent:$002f$002fNAME_INDEXES$002f0$002fNAME_INDEXES:1449570/one",
            "PUBDATE": [
                "1900",
                "1901",
                "1902"
            ],
            "PUBDATE_RANGE": "1900-1902",
            "REFERENCE_URL": {
                "URL_TEXT": "GD128/1/1",
                "URL": "https://stors.tas.gov.au/GD128-1-1$init=GD128-1-1p155j2k"
            },
            "TASMANIAN": [
                "Published in Tasmania",
                "About Tasmania",
                "By a Tasmanian"
            ],
            "YEAR": "1900-1902",
            "YEAR_DISPLAY": "1900-1902"
        }
    }
    
  • XML

    XML is a mark-up language and works very well at storing our open datasets due to their hierarchical nature. XML and JSON provide a basically the same data but you will most likely require an XML parser to navigate the XML document or possibly use regular expressions to extract the required data. Being that XML supports attributes the data may be represented in a slightly different way in XML than the same data in JSON.

    Here is an example of the same record in XML:
    <?xml version="1.0"?>
    <records>
    	<record id="1449570">
    		<PERMA_LINK>https://linctas.ent.sirsidynix.net.au/client/en_AU/all/search/detailnonmodal/ent:$002f$002fNAME_INDEXES$002f0$002fNAME_INDEXES:1449570/one</PERMA_LINK>
    		<REFERENCE_URL>
    			<URL_TEXT>GD128/1/1</URL_TEXT>
    			<URL>https://stors.tas.gov.au/GD128-1-1$init=GD128-1-1p155j2k</URL>
    		</REFERENCE_URL>
    		<TASMANIAN>Published in Tasmania</TASMANIAN>
    		<TASMANIAN>About Tasmania</TASMANIAN>
    		<TASMANIAN>By a Tasmanian</TASMANIAN>
    		<INDEX>Prisoners</INDEX>
    		<PUBDATE_RANGE>1900-1902</PUBDATE_RANGE>
    		<NAME_FULL_DISPLAY>Allen, William</NAME_FULL_DISPLAY>
    		<LINC_TAS_AVAIL>Online</LINC_TAS_AVAIL>
    		<NAME>Allen, William</NAME>
    		<FORMAT>VIEW</FORMAT>
    		<PUBDATE>1900</PUBDATE>
    		<PUBDATE>1901</PUBDATE>
    		<PUBDATE>1902</PUBDATE>
    		<YEAR>1900-1902</YEAR>
    		<YEAR_DISPLAY>1900-1902</YEAR_DISPLAY>
    		<CATKEY>(Sirsi) 1449570</CATKEY>
    	</record>
    </records>
    
  • CSV

    CSV is just simple comma-separated values. We provide all of our data in CSV because data.gov.au is able to produce an API queryable dataset from CSV records. The issue with CSV is it's difficult to represent hierarchical data structures so we have used delimiters like pipe to separate values within a single value. Due to this we recommend using XML or JSON when possible for a more complete representation of the data.

    Here is an example of the same record in CSV:
    CATKEY,FORMAT,INDEX,LINC_TAS_AVAIL,NAME,NAME_FULL_DISPLAY,PERMA_LINK,PUBDATE,PUBDATE_RANGE,REFERENCE_URL,TASMANIAN,YEAR,YEAR_DISPLAY,NAME_SEE_ALSO
    "(Sirsi) 1449570","VIEW","Prisoners","Online","Allen, William","Allen, William","https://linctas.ent.sirsidynix.net.au/client/en_AU/all/search/detailnonmodal/ent:$002f$002fNAME_INDEXES$002f0$002fNAME_INDEXES:1449570/one","1900|1901|1902","1900-1902","GD128/1/1|https://stors.tas.gov.au/GD128-1-1$init=GD128-1-1p155j2k","Published in Tasmania|About Tasmania|By a Tasmanian","1900-1902","1900-1902",""

Using the open datasets

There are countless different ways of using the open datasets but here is an example of how to get started.

For this example I will use PHP to calculate the top 10 most popular baby names from 1803-1933.
I will start off by writing a function for fetching json data from data.gov.au

/*
Function for fetching JSON from CKAN installations
- LINC Tasmania (www.linc.tas.gov.au)
*/
function fetchJSON($id) {
	//CURL options
	$ckanApiAddress = 'http://data.gov.au/api/';
	$proxyHost = 'proxy.education.tas.gov.au'; //insert proxy host or false to not user a proxy
	$proxyPort = '8081';
	$proxyUser = false; //insert proxy host or false to not user a proxy
	$proxyPassword = '';
	
	//CURL init
	$ch = curl_init();
	curl_setopt($ch, CURLOPT_HEADER, 0);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
	if ($proxyHost !== false) {
		curl_setopt($ch, CURLOPT_PROXY, $proxyHost);
		curl_setopt($ch, CURLOPT_PROXYPORT, $proxyPort);
		curl_setopt($ch, CURLOPT_PROXYTYPE, 'HTTP');
		if ($proxyUser !== false) {
			curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxyUser . ':' . $proxyPassword);
		}
	}
	
	//Fetch download URL
	curl_setopt($ch, CURLOPT_URL, $ckanApiAddress . 'action/resource_show?id=' . $id);
	$data = curl_exec($ch);
	
	$data = json_decode($data);
	$dlUrl = $data->result->url;
	
	//Downloading JSON
	curl_setopt($ch, CURLOPT_URL, $dlUrl);
	$data = curl_exec($ch);
	$data = json_decode($data);
	curl_close($ch);
	
	return $data;
}
Using this function I will fetch the births dataset and count the occurrence of every name.
$birthRecords = fetchJSON('4e30dd59-f38d-489a-8ee5-817ac70e5f1a');

$names = array();
$count = 0;
foreach ($birthRecords as $record) {
	$givenName = $record->NAME_FULL_DISPLAY;
	if (strpos($givenName,', ') === false) continue;
	if (strpos($givenName,'Not Recorded') !== false) continue;
	$count++;
	$givenName = substr($givenName, strpos($givenName,', ') + 2);
	if (isset($names[$givenName])) $names[$givenName]++;
	else $names[$givenName] = 1;
}
//Sort name data desc
arsort($names);

echo 'Top 10 most popular baby names from 1803-1933', "\r\n";
$i = 0;
foreach ($names as $name => $num) {
	echo $name, ' (', $num, ' of ', $count, ') ', number_format( $num/$count * 100, 2 ), '%', "\r\n";
	$i++;
	if ($i == 10) break;
}
After running our code we get the following output:
Top 10 most popular baby names from 1803-1933
John (3800 of 191615) 1.98%
William (3568 of 191615) 1.86%
James (2616 of 191615) 1.37%
Mary Ann (2359 of 191615) 1.23%
Thomas (2104 of 191615) 1.10%
Elizabeth (2024 of 191615) 1.06%
Mary (2021 of 191615) 1.05%
George (1827 of 191615) 0.95%
Henry (1365 of 191615) 0.71%
Charles (1207 of 191615) 0.63%
As you can see it's relatively easy to perform reports on JSON datasets using PHP.

Using the data.gov.au API

data.gov.au provides an API for querying datasets within their collection. Unfortunately only our CSV datasets are currently available via this API.
The API URL for Data.gov.au is http://data.gov.au/api/ and you can view the full API guide here.

The CKAN API is useful for front end ajax style calls.
One of the abilities made available by the API is the ability to perform SQL syntax queries against datasets
You just use the ID as the table name and field names from the first row of the CSV to perform your SQL query.
Note: you must wrap the table and field names in double quotes.

Here is an example of a SQL statement that I can use to query a dataset:
SELECT count("DEPARTURE_PORT") as count, "DEPARTURE_PORT" from "8d3e12fa-a9fd-4268-9164-6240a97ff44d" WHERE "DEPARTURE_PORT" != '' GROUP BY "DEPARTURE_PORT" ORDER BY count DESC LIMIT 10
To perform that SQL query I simply encode my SQL statement and pass it via a query string parameter like so:
https://data.gov.au/api/action/datastore_search_sql?sql=SELECT%20count(%22DEPARTURE_PORT%22)%20as%20count,%20%22DEPARTURE_PORT%22%20from%20%228d3e12fa-a9fd-4268-9164-6240a97ff44d%22%20WHERE%20%22DEPARTURE_PORT%22%20!=%20%27%27%20GROUP%20BY%20%22DEPARTURE_PORT%22%20ORDER%20BY%20count%20DESC%20LIMIT%2010

The API will respond in JSON which makes it very easy to parse the response.

Here we have an example using AJAX to fetch the JSON from the API and parse it in to a response

$.ajax({
    url: 'https://data.gov.au/api/action/datastore_search_sql?sql=SELECT%20count(%22DEPARTURE_PORT%22)%20as%20count,%20%22DEPARTURE_PORT%22%20from%20%228d3e12fa-a9fd-4268-9164-6240a97ff44d%22%20WHERE%20%22DEPARTURE_PORT%22%20!=%20%27%27%20GROUP%20BY%20%22DEPARTURE_PORT%22%20ORDER%20BY%20count%20DESC%20LIMIT%2010',
    dataType: "jsonp",
    success: function( response ) {
		var output = "Top 10 departure ports\n";
		$.each(response.result.records, function(k,v) {
			output += v.DEPARTURE_PORT + ": " + v.count + "\n";
		});
		alert(output);
    }
});
You can execute the example code with the button bellow:


Licence information

Our open data collection is covered by the Creative Commons Attribution 4.0 International licence.
More information on this licence can be found here.