Learning to Not Reinvent the Wheel

It’s been a while since I’ve blogged about this project- mostly because I’ve been doing other things, like going to conferences. However, recently there have been a number of advances. In this post, I’ll be talking about how I’ve been making things complicated for myself, and about how things can change.

First- making things much more complicated for myself? Well, I’ve been trying to code/write a front-end webpage from scratch. This has involved PHP & HTML, but also trying to find a way to do it that didn’t involve PHP. Further, I’ve been trying to figure out how I could insert my custom code into either a custom WordPress page, or an adjacent page that uses much of the CSS that the rest of this site uses. I’ve finally come to accept (after four or five emails) that this isn’t needed- and that there already exist plugins for WordPress that will allow users/researchers to access the contents of the database, and do all kinds of neat queries. Further, it shouldn’t be difficult to find yet another plug to allow geographic data to be displayed on a map. I think the most important lesson I’ve learned during this project is that it’s a total waste of time to try to code something from scratch and reinvent the wheel when somebody else has already done it and it works well. And yet, I keep on learning this lesson over and over again with every part of the project.

Of course, not every part of this project can be done using a tool that somebody else has created, and that’s where I’ve been making a lot of progress. ADM 8 contains more than just lists of where the ships are deployed; so far I’ve found lists of officers, categorized according to the positions they were employed, and also two different types of reports that list ships, how long they’ve been in service, and how much money has been paid to them or owed to them. I’m sure there’s more different types of reports that I’ve not seen yet. That’s fine too- the real strength of using a relational database for this project is that the ships and officers can be linked to so many reports, and then the queries can easily pull it all back together in various arrangements.

What I’ve been doing, and where I’ve been making the progress is in the back end of things.

1. Identifying the important parts of a report when I come across a new type – can it fit into one of the existing tables?
2.  Creating parts of the database- this usually involves three distinct sets of tables.
   1. A list of the reports as a whole
   2. A list of each line in each report- (each of which is associated back to an entry in the previous table)
   3. A third table that accommodates multiple associations within each line above- for example, in the lists of officers, for those that served aboard more than one ship. Having a table for this means any number of these can all be incorporated- it’s more work up front but the most flexible.
3. Creating a template for a CSV file, so that all the reports of the same type can be transcribed in the same way, and the data organized in the same way
4. Creating a PHP upload script, so that each report can be uploaded to the database. This has been incredibly complex because with each report, each line of data, I need to check whether the ship or officer or whatever already exists, or whether there is a possibility for confusion. And there is a lot of possibilities for confusion, whether it be ships or officers with the same name.

The front-end is really interesting- whatever plugin we end up using, it has to be able to accommodate both pre-written queries to generate a standard menu of reports, and have a more advanced interface so that users can ask more interesting questions of the database. But that’s not the most important work- the more important task is in the database design- and in parsing the documents so that really interesting questions can be asked of them. And that’s the task I’m focusing on at the moment.

I have also somewhat impetuously launched a similar project for the Canadian Nautical Research Society, where the Society will be transcribing ships’ logs from the Hudson’s Bay Company archives, and creating a research tool from them. This project will be announced/discussed in the upcoming edition of Argonauta. I am convinced that these digital efforts are going to be much more important to academic (and other) historical research than my PhD or anything else I’ve done so far. These kinds can provide incredible opportunities to really study historical documents that are so far mostly neglected. I look forward to bringing these projects online- and doing what I can to help other similar projects as well.