Author Archive
Accurate Postcode data for the UK and Northern Ireland
by Mike on Jun.06, 2011, under Knowledge Base, Technology
Getting accurate postcode data for use in my programs has proved to be an interesting technical challenge, despite the availability of free services such as those from Google.
I needed a UK postcode to Latitude/Longitude conversion tool that would work offline, which made the Google API’s completely useless.
A freely available dataset was made available from the Ordnance Survey under the moniker of “Code-point open”, however this is missing some data (notably Northern Ireland) so can’t really be called complete, and additionally it does not provide locations in Latitude/Longitude, but instead it uses Ordnance Survey Grid references.
If you need to do this, then hopefully the tools here will be of some help.
First we need to grab the Ordnance Survey Code-point open dataset. They don’t provide a straight link, so you will need to register to actually get access to the data. Once downloaded and extracted you will have a folder full of CSV files.
Secondly we need to convert the grid references to more useable LatLong values. Chris Veness has written some Javascript to do just that – as well as writing some incredibly informative articles about how this stuff actually works (there’s some horrible maths involved!). I spent a couple of evenings converting his scripts to Python and wrote a little data extract utility which put all the data into a SQLite3 database.
Another couple of hours to write a set of wrapper functions around the database and include the validation rules from the UK government data schema and hey presto! We have a usable postcode_utils module.
This just leaves Northern Ireland…
It turns out that the Northern Ireland Statistics and Research Agency (NISRA) have some GIS data available in ESRI ShapeFile format. There is a useful Python library to work with ESRI Shapefiles and it took a matter of minutes to extract the coordinates from the NISRA data, that provides postcodes in… bah! another Grid System, but this time it’s the Irish Grid!
Some more hunting online to get the right values for the grid transformations and the Geoid dimensions used for each grid system and a bit of tinkering with Chris Veness’ code and we are done! Here is a Sqlite3 database containing all the postcodes in Great Britain and Northern Ireland with accurate Latitude and Longitudes (more accurate than some Web Services I could mention) and some Python to use with it. Also, this information is all freely available and free for use commercially and non-commercial use (provided you give attribution as mentioned on the various sites linked to above).
If you find this useful, please let me know by using the comments feature. Also, if anybody has a source of data for the Channel Islands or the Isle of Man I would be very please to find them!
Generating HTML5 using XSLT
by Mike on Jan.28, 2011, under Technology, Tutorials
Recently, I have been updating some of my HTML generation tools to output valid HTML5, rather than the XHTML 1.0 standard I have been using for the last few years. The main advantage from my perspective is the ability to use the more semantic block elements, such as the nav, section and article elements.
In general this is a fairly straightforward task, as I am generating clean XHTML using XSLT and my template library works pretty well, but I ran into some problems whilst validating the output using the W3C Validator.
The first issue is to sort the DOCTYPE out. The XHTML doctype looks like this:
<!DOCTYPE html
PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
This is easy to generate in XSLT using the following output element.
<xsl:output encoding="UTF-8" indent="yes" method="xml"
omit-xml-declaration="yes"
doctype-public="-//W3C//DTD XHTML 1.0 Strict//EN"
doctype-system="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd" />
This unfortunately forces the document to validate against the XHTML 1.0 specification which does not include all the lovely new semantic elements – which means that my new documents are suddenly invalid!
We need to generate:
<!DOCTYPE html>
which is really hard to do using XSLT. I have read a number of articles that suggest you output the element as text, however this is extremely ugly and as it turns out, incorrect.
The correct XSLT incantation is:
<xsl:output
method="xml"
doctype-system="about:legacy-compat"
encoding="UTF-8"
indent="yes" />
This forces use of a dummy DTD (about:legacy-compat), which is the W3C recommended way of not using a standard DTD URI.
Now the W3C validator will happily validate against the HTML5 specification rather than the XHTML 1.0 specification.
The Strange (Knighton Heath Demo) – giveaway
by Mike on Jan.15, 2011, under Free downloads, Music, The Strange
Although I don’t seem to have any early recordings of The Strange, I managed to locate these which were made by the last incarnation of the original line-up (Dave, Richard and myself with Graham White on drums) some time in the late 90′s. The demo was made at Knighton Heath Music Centre in Poole and was engineered by Dave Thomas (of Manitou fame).
This era was probably the last time that The Strange was a going concern as a live band – once Graham quit (for personal reasons unrelated to the band) we migrated to my flat, and although did lots more interesting experimentation, the music would have been harder to perform live relying as it did on drum loops, samples and sequencing as much as on drums, guitar, synth and bass.
Anyway, enough rambling, here are the tracks.
ASCII me!
by Mike on Jan.13, 2011, under News
...................................................................................................................................................... ...................................................................................................................................................... ...................................................................................................................................................... ............................................................................................................................,,........................ ...................................................................................................................................................... ...................................................................................................................................................... ...................................................................................................................................................... ...................................................................................................................................................... ...................................................................................................................................................... ...................................................................................................................................................... ...................................................................................................................................................... ...................................................................................................................................................... .......................................................................................................................................,.............. ...................................................................................................................................................... ...................................................................................................................................................... ................................................~==:.................................................................................................. .............................................,~++??+:.....................................................................,,............,............. ..........................................,::=+?????=:....................................................................,,............,............. ..........................................~=+???????+=~~:.................................................................,:,...........,............. ..........................................:=?+???????+==~,................................................................,~:,,,........,............. ...........................................=+?????????+===~,..............................................................,~:.,..........,............ ...........................................~+++????????+++=:,.............................................................,~:.,....................... ............................................:+++????????++++=~,............................................................~~,........................ .............................................=++??????????++?+~::,,........................................................:~,.,...................... ..............................................=++?????????+~~~=++====~:,...................................................:~,........................ ...............................................~++=+=~:,..,,:=?+???=~==~,..................................................~=~,,,..................... .................................................~~........:+?????+=~~~:,..................................................:==::,..................... ...................................................???I??+==??????++++=::,..................................................==,,,..................... ...................................................+?I?I???+?????+++=++~:,..................................................~=:,,..................... ....................................................???I??+????+?++??++=~:..................................................~=~,,..................... ....................................................?II?????+????????++==~,.................................................=+=:,..................... ....................................................+IIII??=???????????+=~:,................................................~+=,...................... ....................................................~II?+.=++?+??????+++==~:................................................:++,,..................... ............................................................,:::::~=++====~~,................................................+?:,,.................... .............................................................:=+?+==,.,::,,.......,..:,,,....................................++~...................... .............................................................=?++~==:.............,,,::,,..........................=:,.......+?=...................... ...............,::~~~:,..........................................,=+~.............=+~~~~~........................,=+~,.......=?+...................... .............,~=+++++++=~:,.......................................,,,.............~++=~:.........................~=+:........~+=...................... ..........,:=?????????????+~,....................................................~?+,..........................~.~++,........:++,...........,......... .........,~+???????????????~,...................................................,=............................~?,,++=:........++:...........~,........ ......,~+????????????????=~,.................................................................................,,I:.+??=,.......=+~,,,,.......::........ ....,~+????????????????+=:..................................................................................:,,+=.+??=,.......~=~,,,,,.....,:~........ .,:=???????????????+=~,....................................................................................,,.,,?.,,..........:==:::,,..,:,..~........ =+??????????+++=~:,......................................................................................,.,..................:~=::~~~:,,::..,........ +????????++==:,,..........................................................................................,,...................~=~:~~~:,.,,..,........ ??II?+~:...............................................................................................,.,.....................~+=:::,,............... ???+~,.....................................................................................,..........,.,......................=+=:::,,............... ??~.........................................................................................:+,......:.:.......................:==,,..........,....... ?+:...,...............................................................................+?+..,+?+.....,.,.,......................,~~,,,,........,....... ??+,..................................................................................?I?+.,???,...:.,.,........................~~,.,,........:,...... ??I+,.................................................................................+?I?,.???~..,.,,..........................~=:,,,........:,...... ?????=.................................................................................?II?,?II?~++=.,.,........................~+=:::....,..,::...... ???+:...............................................................................:?+,?II???I?+??+=...........................:=~,,,........:~...... +=:,................................................................................=I?.III???I?????+,..........................,~~,,,,.,...,,:=,..... ....................................................................................~I??:IIII??I???I?=:.....~~,..................:=:,,,,....,.,=:..... ....................................................................................~?I?.??I??II????++=:...=++=..................~=:,,,....,,,,=:..... .....................................................................................III+.??II?I???++++=~=:,.=~..................:=~,,,,...,,,:=~,,... .....................................................................................+II?.:?II?I?II?++++==~:..,..................:==:::,,..,,,:=~,.... ......................................................................................?I?I:=??I?IIIII???+++~.....................,~=~::,.,,,,:~==:,,.. ......................................................................................:?II?.?IIIIIIII?????~~.....................,:=:,:,,,,..:~==:,... ................................................................................:......,?IIIIIIIIIII??+++=,,......................:~:,,,,,,,.,::+=:,.. ...............................................................................,~.....~.~??IIIIIII?++=~~~~:,,.....................,~:,::,,,,,,::=~::,. ...............................................................................::......,,+I?II?????+===~~~:,,.....................,::,:::,,,,,:~==:,.. ..............................................................................,,....~.~.:.,???+====~~~~~~:::,......................:~:::,,,,,,,,~+=:,. ..............................................................................,....,.:,,..,.~++===~~~~~==:::,.,....................:~,::::,,,,::~+~:,. ...........................................::,.............................,.,....~.:,,......,:====~~~=~~:~~:,,,...................,:::::,,,,,,,~++~,. ..................................,,,,::~~~~=,...................................:.::.,.......,~~==~~~~~~:~~::,....................,~~:::::,:,:::=+:,. ......................,,,:~==++++==========~,.........................???..,I+..+.~,:.,...........,~~~~==~~~~:,:,...................:~,:,,,,.,,,:~=:.. .................,,,:~~==++++???+++==++==+=:........................,?III,,+,..~.:,,..,............:~========~:,....................::,,,,::,,::,:~,.. ...........,:~=+++++++??????????++++++====~........................?I?III?+:..+.~.,.,................:==+==~:.......................,:,,,,,,,,,,,,,,.. .,:::~~~~~==++++++++++??????????+++=+===~.........................+??I????+?+:.~...,,..................:::==:,......................,:,,,,,.......,,.. :~~=========+++++???????????????+++++++=.........................+I??II?+?+++.=.,......................,::,,,........................,,........,...,.. ~==========++????????????????????+++++=,.,......................+I??II?++++:.:,:.,......................,:::,.,,.....................,,............... ,~========+++????????????????????++?+=~:~==+~..................,???III++++?.+.:..,.......................:~:::,,......................,............... ...,:~====++++????????????????????+=:~=~=+++?++?,..............I??III?+++:,:.~.,..........................::,,..,,....................,,.............. .....,:~==+++???????????????+?????~,::~~==+++?++?~............~???III??++,=.~..,..........................,::,,,.,.................................... ,,....,..:=++??????????????????+,,,,,,,::~~=+++++++=,........,?I?I77???:,~.~,,.............................:::,.,,,,,................................. ::,,..,...,~=+++?+???????+???+~:~~~:..,.,:::=+????++=:.......~II?III??+,=.~,,,.............................,,,:,...................................... :,:,,.........:~++????????+=~?IIIIII~......,:=+????+==~+?..:...~7II??:,~.~,,,..............................,::,.,......................,:::,,,........ :,::,,,...,,.....,:~::,,,.:?II7777II+,.......~++??++==~=+..~...?77II.=:.~.:.................................,:,,........................,,,,,......... :,,:,,,,..................=I77777777I~.......~+++++=~~~=:..:...I7I?~.~,+,,,.,................................,,,........................,,,........... ..~......................?II777777777I+.......:~=~~,.:~:......~I7I,=:.~.:,.,...................................,...................................... ........................,?I777777777I7I:.......,:,,..,,...I...I7I=~~,=,,,............................................................................. .........................?I777777777777I:................IIIII7?,+~:~.:............................................................................... .........................?7777777777777I?...............=IIIII?=:~:+,:,,.............................................................................. ........:...........,,::,+7777777777777II?.............IIIIIII:+~:~,:,.,.............................................................................. .......,I.....,,::~====~:+7777777777777II?:...........?IIIIII=:=:+,::,,:.............................................................................. ++??++=:+::==~=========~~~I77777777777IIIII,......:?IIII7III~+~:=:~:,==............................................................................... ~??????+=77,~~~~~~~~~::::,I7777777 777I??77I....?IIII7III??~+:=::~,.??+............................................................................... .~?I????~==..,:::::::::::,I77777 7 77I?I77I....II?77I7II?~?~:?,~,:=???.,?~........................................................................... ...,=??=........::::,::::~~I777 7777 7I?+I7I,,..IIIIIIII?~?~~~:~,,?????I?I=........................................................................... ....,~++,,~++...,,:,,::::::777 7 777I?~I77+...IIII777I:?~~==::.~???????+............................................................................ .......,..,.,....,,,,,,,::~77777777 7I+.+777,..IIIII7I++~=~:~,.?????+++=............................................................................. .........:,:7=.......,,,:,~77777777 777?.= 7 I..III77II?==+:~~,~I?I??+++,............................................................................. ..........,~?I...........,,.,+??7777 77?,,I 777I:II7I?II+::+:,?II??I+?+......,........................................................................ ...........,,~................,?II77777I::+I 77777?III7I=::~,=IIII???+:............................................................................... ..............::.................,=77 77?=::=??7++,~IIII,=:,?77III???,................................................................................ ...................................~77777II7I7I=7.~?7?II+~,II777IIII=.................::,............................................................. ....................................777III7II=,?..?=+7?II,?77777IIII:,................::~:~~.....~=,.................................................. .......................................+??,:~+I.==,+?IIIIII77777II??..............,,...::~~==+=....................................................... ....................,~=,...............??II?I7I,I,:=IIIIIII77777III?....+.........,,...,::::.......................................................... ......................:+=:............?II~I=I77=I?.:?IIIIII777777??+...:=..,.......,,................................................................. .......................~+=~...........?II.:?.:I:?,.~:IIIIII7777II??I++~:~,,........................................................................... .........................~++=........+II..I.~??II~+..IIIIIIIIIIIIII7I????~............................................................................ ..........................:?=~.......?II,?..?:+II+.,:II?IIIIIIIIII77I????~............................................................................ ....................................=III?.~:,?IIII..,IIIIIIIIIIIIIIII????............................................................................. ...............................,,...??:+.?..?:+II?...IIIII??I??III??+++=.............................................................................. ..............................~~77~:II?=~..+,,?III?=?IIIIIIII?????+~~=~,.............................................................................. ................................,,.+II.I:.?..+:?IIIIIIIIII???++~,..................................................................................... ................................,=,.?+~:,=..=.,II7IIIIIII???+~:....................................................................................... .................................:=.:~+:+..=.:~IIIIIII?III?+=~=:...................................................................................... .................................:.~.::+,.:..,,IIIIIIIIIII?++++,...................................................................................... ...............................,.,.+~~+.,+.....?IIIII?I77????+~....................................................................................... ...............................,...?:~?.=,.,..=IIIIIIII7I?????........................................................................................ .....................................,I+=:...IIIIIII?I77II???:........................................................................................ .................................,,...,~~....I?III??II77????+......................................................................................... ................................,.....,:,~.=,?II??I??77I??I?~......................................................................................... ...........................,........,,,~:..?I~..,=??7III???+.......................................................................................... ..............................,..?.,...,.=???,..,+?I7II????~.......................................................................................... ..............................,,=?......?I?II+=~=:?III????+........................................................................................... ...............................,+I:....~IIIII?I==??I7I????~........................................................................................... ................................I7=::..IIIIII??I??I7I????+............................................................................................ ................................7I:,..:+I+==+?I??I77I????~............................................................................................ ................................+......,I.=,,~?I?III????=............................................................................................. ...............................~:.....,,,=,,+:?IIII?+++=.............................................................................................. ...............................,........~??~.???III?+++~.............................................................................................. ...............................,.....,.,:????IIII??+++:............................................................................................... ...............................+.......::?I????II++=+~................................................................................................ ...............................,.......=???I?I??++==~................................................................................................. ...................,....................+????I?+=~~:,................................................................................................. ..................:::,...........:,.......,~:~...,.................................................................................................... .................,:,,,.......................?........................................................................................................
Setting up a web project environment in Visual Studio 2010 to allow debugging using both IIS7 and the development Web server
by Mike on Jan.13, 2011, under Knowledge Base, Technology, Tutorials
We have run into an issue recently when importing a number of our legacy web projects into Visual Studio 2010. It seems that some projects don’t provide the options to allow us to run and debug them on the local version of IIS. For some reason, when a project is initially configured in VS2010 it sets a flag that determines whether the project properties page is shown inside the project window (where it will contain the necessary configuration options) or the project properties page is shown as a pop-up dialog window (where it doesn’t contain the options to allow debugging on a local IIS).
Here are the notes you will need when setting up a new project to guarantee that you have access to the required configuration page.
IIS 7 Configuration
The first step is entirely optional, as it is perfectly reasonable to run multiple Web Applications within a singe Web Site, however I prefer having separate Web Sites for different projects (or clients) as this allows me to control individual settings for each of them (for example host and port bindings) without me worrying about breaking a different project.
If you will use “localhost” to access the server, then you will only be able to have one Web Site instance running at a time, but this is rarely an issue, and you can easily run more than one site at a time by using a unique hostname or port for each.
Create the site in IIS Manager:
Where you create the Web Site mostly doesn’t matter, Visual Studio will add your project to the site as a virtual directory and the physical files can exist anywhere on your filesystem.
Once the site has been created, stop any other site that is using port 80, and start the Web site from the actions panel at the right of the window.
Because you want Visual Studio to modify the IIS configuration, you will need to run Visual Studio as Administrator.
Create an empty Web Application
Create a new project in Visual Studio (File > New > Project…).
Select: Visual C# > Web > ASP.NET Empty Web Application.
If you leave the “Create directory for solution” checkbox selected, then all your project files will end up one directory below the solution file. Personally I don’t mind having the .sln file in the same folder as my project files so I uncheck it.
If you click OK to create the various project files, then immediately compile and run the application you will see that by default Visual Studio has configured the development web server.
Reconfigure the project to use IIS
Now whilst this is useful for small standalone projects, we would ideally like to be able to switch development to the full version of IIS7, and to do this, we need to perform another couple of steps.
Back in Visual Studio, stop the project and in the Solution Explorer, right-click on the project and select “Properties”.
In the window that appears, select the Web option and scroll down until you can see the Servers section. Change the selection from “Use Visual Studio Development Server” to “Use Local IIS Web server”.
Click the “Create Virtual Directory” button to complete the IIS configuration. If you see the dialog below then everything has gone as expected and you can now run (and debug) your web application in IIS. You can switch between IIS and Cassini (the development web server) simply by changing the radio button back.
Using the same web.config for both IIS and Cassini
Because IIS7 uses a new integrated pipeline, and Cassini uses the classic pipeline, some of the settings in the web.config are not compatible. Whilst it is possible to use two web.config files (and move them in and out of the application directory when you switch webservers), it is much more convenient to use a single web.config with both webservers.
.NET will complain about the legacy settings if you are using the integrated pipeline (which we normally will want to do) unless you have the following included in the system.webServer section in the web.config:
<validation validateIntegratedModeConfiguration="false" />
This line instructs the .NET not to validate any settings it finds in the web.config for the integrated pipeline – this allows you to have settings that only work in classic mode (such as httpHandlers and httpModules co-existing with the handlers and modules sections that are required by IIS7.