Get support with problems at work, and join 157,000 others to fight for better pay, conditions, and job protection.

GLAM Labs Futures ● 25-26 June 2026 ● Scotland

I often describe what I do as GLAM data plumbing. Most of the time I’m not creating new tools, I’m figuring out what data is available and how I can connect it up to _existing_ tools. It’s rarely straightforward, but if I can get all the pipes connected and data flowing in the right direction, suddenly new things become possible. **Things like turning all the State Library of Victoria’s digitised maps into data.** I’ve just created a workflow that uses AllMaps and IIIF to georeference the SLV’s digitised maps. There’s some technical details below, but the idea is pretty simple. A userscript links the SLV image viewer to Allmaps – so you just click on a button, and the digitised map opens, ready for georeferencing. Why is this useful? Georeferencing relates a digitised map to real world geography. It describes the map’s position and extent using geospatial coordinates – turning historic documents into geospatial data that can be indexed, visualised and manipulated. Georeferencing opens digitised maps to new research uses. So, how many maps we can georeference before my residency finishes in December? Hundreds? Thousands? If you like maps and want to help, head to the documentation page to find out how to get started. And if you want to see how things are progressing, have a look at the project dashboard. View the documentation to get started A few technical details follow… Early on in my time as Creative Technologist-in-Residence at the State Library of Victoria, I started playing around with Allmaps for georeferencing digitised maps. It’s a great tool (really a suite of tools and standards) because instead of constructing a whole new platform it integrates with existing IIIF services. The SLV provides digitised images through IIIF, so I thought it should be possible to use Allmaps to georeference the SLV’s map collection. But I struck a problem that took some time to unravel. The IIIF urls in the SLV manifests include port numbers and that confused Allmaps. The manifests also sometimes contained references to image formats that weren’t actually accessible, generating errors when they were loaded. Hopefully these problems will be fixed by the SLV, but in the meantime I’ve created a proxy service that edits the manifest on the fly. The proxied urls can be loaded into the Allmaps Editor without errors. Pipes fixed, data flowing! Using the manifest proxy To generate a link to a proxied manifest, first grab the item's `IE` identifier from the url of the digitised item viewer. For example, the identifier in this url `https://viewer.slv.vic.gov.au/?entity=IE15485265&mode=browse` is `IE15485265`. Once you have the identifier, add it to the end of the url `https://wraggelabs.com/slv_iiif/`. For example, https://wraggelabs.com/slv_iiif/IE15485265. You can then supply this url to the Allmaps editor. But having to fiddle around with proxies didn’t make a great user experience. I needed some way of integrating the two services, so that a user could just click a button in the SLV website and start editing in Allmaps. Userscripts to the rescue! I wrote recently about hacking GLAM collection interfaces using userscripts. Since I started my residency at the SLV, I’ve also created a userscript to display the IIIF manifest url in the SLV image viewer, and run a Code Club workshop where we played around with an assortment of SLV website hacks. As in a number of these examples, the georeferencing userscript adds new features to the SLV website, but there’s a fair bit more going on under the hood. It runs automatically every time you load the SLV image viewer, and then: * it checks the metadata of the digitised item to see it it’s a map (or something that contains maps, like an atlas or street directory) * if it looks like a map, it generates an Allmaps identifier using the item’s IIIF manifest url and checks with Allmaps to see whether the item has already been georeferenced * it adds a ‘Georeferencing’ section to the page, with a button to georeference the item (or edit the existing georeferencing) * if the item has already been georeferenced, it adds a button to view the item in the Allmaps Viewer, and embeds a live preview Accessing metadata The userscript gets the item metadata from a JSON file that's loaded by the image viewer. The JSON file includes a lot of extra, useful information about the digitised item. To access the JSON file, you just construct a url like this: `https://viewerapi.slv.vic.gov.au/?entity=[IE identifier]&dc_arrays=1`. The IE identifier is in the url of the image viewer. Allmaps identifiers Allmaps creates its identifiers by hash encoding the IIIF urls. The userscript borrows some code from the Allmaps id module to generate the ids, then sends a HEAD request to the Allmaps API to see whether an entry for the current manifest exists. Example of an item that hasn't been georeferenced yet Example of an item that has been georeferenced, displaying an embedded version of the Allmaps viewer I’ve also created a GitHub repository to save copies of the data. Every two hours this notebook is run to query the Allmaps API for newly georeferenced maps. These are added to a dataset which is saved in three formats: * a CSV file * a CSV file that includes thumbnails and links for viewing in Datasette-Lite * a GeoJSON file, that can be viewed in services like geojson.io At the same time, the data for each individual map is downloaded and saved as IIIF annotations (in JSON) and GeoJSON. Finally, this notebook is run to generate a dashboard that provides an overview of the project’s progress. The project dashboard is updated every two hours One of the Allmaps developers described all my plumbing and workarounds as a ‘very cool lofi example of how you can set this up with little means’, and I think that’s pretty apt. It’s really just an experiment to demonstrate the possibilities, but by connecting up existing services it’s generating real data of long term value.

Interested in testing a new Wikimedia Commons batch upload tool? Going to GLAMwiki Conference? https://meta.wikimedia.org/wiki/GLAM_Wiki_2025/Hackathon#Hack_projects #WikimediaCommons #GLAMWiki

I like urls. They take you places. And if you know how to read them, they can tell you things about the systems that created them. One of the first things I did when I started my residency at SLV LAB, was to try and understand how their collection urls work. There’s a couple of well-worn methods I use when digging into a new site. The first is url hacking – this involves fiddling around with the parameters in a url and submitting the result to see what happens. The Trove Data Guide includes some examples of hacking Trove urls to change the delivery of search results. The second method involves opening up the developer console in your web browser and watching the activity in the network tab as you click on links. This tells you where the information that gets loaded into your browser actually comes from – sometimes exposing handy urls that you can use to shortcut access to useful data. ## Permalinks The SLV uses Primo for its public-facing catalogue, as well as other systems such as Rosetta and IIIF to deliver digitised content. I’d noticed that Zotero gets some useful data from the catalogue using the default ‘Primo 2018’ translator, however, important things like the item url aren’t captured. The problem is that Primo’s ‘permalinks’ are generated as required by a browser click – they’re not embedded anywhere on the page. This makes it hard to Zotero to grab them. So I started wondering how Zotero could construct short, persistent(ish) links to items. Here’s a link to an item in Primo: https://find.slv.vic.gov.au/discovery/fulldisplay?vid=61SLV_INST:SLV&search_scope=slv_local&tab=searchProfile&context=L&docid=alma9941325055707636 It looks pretty long and messy, but if you start deleting parameters and resubmitting, you’ll find that only two parameters are essential, `vid` and `docid`. This means we can rewrite the url as: https://find.slv.vic.gov.au/discovery/fulldisplay?vid=61SLV_INST:SLV&docid=alma9941325055707636 Much nicer. The ‘permalink’ for the same item is: https://find.slv.vic.gov.au/permalink/61SLV_INST/1sev8ar/alma9941325055707636 If you look closely at the url path and compare it to the example above you’ll see the path is constructed from `/vid/[some other id]/docid`. One of the librarians explained to me that the other identifier in the permalink is an encoding of the view type, but given that the ‘fulldisplay’ view is the default, we don’t really need it. So the shortened url seems fine for use in Zotero and is easy to generate from the current url. Nice. It’s also worth noting that the `vid` value doesn’t seem to change, so to construct catalogue urls in your code, all you really need is the ALMA identifier that’s in the `docid` parameter. ## Structured data Item pages in Primo include a link labelled ‘Display source record’. If you click on this you’re taken to a representation of the item’s metadata in MARC. Here’s what the urls look like: https://find.slv.vic.gov.au/discovery/sourceRecord?vid=61SLV_INST%3ASLV&docId=alma9941325055707636&recordOwner=61SLV_INST Notice that the ‘fulldisplay’ in the url path above has changed to ‘sourceRecord’. There’s also a new `recordOwner` parameter, but it seems you can delete this and still get the same result. Having access to the MARC record is handy, because it delivers the metadata in a simple, structured plain text format. But while the ‘source record’ page looks like a plain text file, it’s actually a HTML page that embeds a plain text record. If you open up the network tab of your browser’s developer console and reload the ‘source record’ page, you’ll see a different url is loaded under the hood: https://find.slv.vic.gov.au/primaws/rest/pub/sourceRecord?docId=alma9941325055707636&vid=61SLV_INST:SLV&recordOwner=61SLV_INST&lang=en See how the url path has changed from `/discovery/` to `/primaws/rest/pub`? This url _does_ deliver a plain text version of the MARC record. Once you have the plain text version you can parse the contents to extract the structured data. There are tools that can probably do this automatically, but it’s also pretty easy using regular expressions. Here’s an example of some code I used to parse map records. def get_marc_value(marc, tag, subfield): """ Gets the value of a tag/subfield from a text version of an item's MARC record. """ try: tag = re.search(rf"^{tag}\t.+", marc, re.M).group(0) subfield = re.search(rf"\${subfield}([^\$]+)", tag).group(1) except AttributeError: return None return subfield.strip(" .,") You can also access a JSON representation of the record by adding the parameter `&showPnx=true` to the catalogue url: https://find.slv.vic.gov.au/discovery/fulldisplay?vid=61SLV_INST:SLV&search_scope=slv_local&tab=searchProfile&context=L&docid=alma9941325055707636&showPnx=true Once again, this is a JSON representation embedded in a web page. Using the same developer console trick, you can identify the direct url is: https://find.slv.vic.gov.au/primaws/rest/pub/pnxs/L/alma9941325055707636?vid=61SLV_INST:SLV&lang=en&search_scope=slv_local&showPnx=true&lang=en You should be able to parse the response from this url as JSON and use it in your code. I think the Zotero translator makes use of this `pnx` data. If you want to download the MARC or JSON representations in your code, all you really need is the `alma` identifier. Just use it to construct one of the direct urls, such as this: https://find.slv.vic.gov.au/primaws/rest/pub/sourceRecord?docId=alma9941325055707636&vid=61SLV_INST:SLV The `recordOwner` and `lang` parameters are not needed, and the `vid` parameter doesn’t change. Librarians using Primo have documented a number of tricks like this and shared handy bookmarklets to rewrite urls and get catalogue data in different forms. ## IIIF and images SLV delivers digitised images using IIIF. The IIIF manifest urls are not directly exposed through the web interface, but you can construct your own. IIIF manifest urls look like this: https://rosetta.slv.vic.gov.au/delivery/iiif/presentation/2.1/IE24074939/manifest.json All we need to construct them is the `IE` identifier, in this case `IE24074939`. But where do you find this identifier? If you’re looking at an image in the SLV’s image viewer, the url will be something like this: https://viewer.slv.vic.gov.au/?entity=IE24074939&mode=browse Yep, the `IE` identifier is right there in the url. Just extract it from the viewer url, and plug it into the manifest url! If you’re looking at a catalogue record, or starting with one of the `alma` identifiers, you can get the `IE` identifier from the `956$e` field of the MARC record. The IIIF manifest will, in turn, provide identifiers for individual images that can be requested using the standard IIIF syntax. To save myself a bit of fiddling about, I created a userscript that exposes the IIIF manifest url within the image viewer. If you install it you’ll see something like this: ## Handles Links to digitised items sometimes come in the form of ‘handles’: http://handle.slv.vic.gov.au/10381/4338980 These urls are redirected to the image viewer. If you want to construct one of these handles, the identifier can be found in `956$a` field of the MARC record. ## From old to new I was looking at the datasets created about 8 years ago in the SLV open data repository and noticed they included urls from the previous catalogue. Fortunately, the old urls redirect to the new system. For example, this url: http://search.slv.vic.gov.au/MAIN:Everything:SLV_VOYAGER1842440 Redirects to: https://find.slv.vic.gov.au/discovery/fulldisplay?context=L&vid=61SLV_INST:SLV&search_scope=slv_local&tab=searchProfile&docid=alma9918424403607636 If you look closely at the urls you’ll see that the identifier from the old system is embedded in the new identifier: `1842440` is in `9918424403607636` – `99_1842440_3607636`. This means if you have a lot of old urls, such as in the open datasets, you can easily rewrite them in your code. ## The process of GLAM hacking No doubt a lot of this is well-known to librarians, and there’s probably many subtleties or complexities that my poking about has missed. But I wanted to document the process as much as the results – to give an idea of what I do when I approach a new GLAM collection online. I suppose this is GLAM hacking 101.

If you missed my talk on benchmarking automated indexing methods: the slides and recording are now available on the webinar Info page: https://www.cenl.org/network-group-ai-in-libraries-webinars-2025/

I’m very excited to be the new Creative Technologist-in-Residence at the SLV LAB. For the next few months I get to play around with metadata and images, think about online access, experiment with different technologies, and build things to help people to explore the State Library’s collections. In other words, I get to be in my happy place! My group at the recent SLV WikiFest was thinking about ways of helping researchers find resources relating to particular locations – how do I find material about my suburb, or my street? Coincidentally, the main focus of my residency will also be place-based collections, so I get to really think through some of the possibilities. SLV staff have already pointed me to some amazing maps and photographs, such as the Committee for Urban Action collection, the Mahlstedt fire survey maps, the MMBW plans, and the Victorian parish maps. At the same time, I’ll be using my usual GLAM hacking approach to poke around in the SLV website to try and understand what data is currently available, identify any roadblocks, and document opportunities for computational research. The results of my residency will be shared on the SLV LAB site, in GitHub, in the SLV section of the GLAM Workbench, and of course here. As usual, I’ll be working in the open, documenting things as I go along, so please join me on the journey! Although the residency was formally announced today, I’ve actually been working with SLV data for the last couple of weeks and I’ve already got a backlog of stuff I need to blog about. Here’s a taster – what happens when you generate bounding boxes for thousands of parish maps from the available metadata and throw them on a map…?