The search engine recently exposed a fair number of new tools for custom filtering to the API consumers and users of the new UI. This was originally going to be an incredibly chaotic update, both annuncing the new features and doing a technical walkthrough of the changes but that ambition turned out a bit too chaotic, so let’s split them up and focus on the feature announcement bit today. New Search Filtering GUI It’s now possible to define a custom filter in the GUI, on the marginalia-search.

One of the big ambitions for the search engine this year has been to enable searching in more languages than English, and a pilot project for this has just been completed, allowing experimental support for German, French and Swedish. These changes are now live for testing, but with an extremely small corpus of documents. As the search engine has been up to this point built with English in mind, some anglo-centric assumptions made it into its code.

I’ve been using the CoPilot plugin for IntelliJ on and off for the last few years, and while initially pretty enthusiastic, I’ve come to first disable it and then delete it altogether along with JetBrains’ local AI-completions, and generally felt this has been an improvement in productivity and a reduction of frustration. CoPilot is pretty good at taking things that are already pretty fast, such as monotonous code transformations like mapping an object to a SQL statement, and then making that even faster.

How you engage with the world changes how you experience the world, and how the world experiences you. A snarky and cynical approach, by its default assumption that things are shit, or if they are not yet shit will inevitably turn to; such an approach will give your world a malodorous brownish tint. Granted, snark gives you plausible deniability, a motte-and-bailey that protects you from direct criticism, encountering backlash you can always backpedal and say it was just a joke that you accidentally took a bit too far.

The Marginalia Search index has been partially rewritten to perform much better, using new data structures designed to make better use of modern hardware. This post will cover the new design, and will also touch upon some of the unexpected and unintuitive performance characteristics of NVMe SSDs when it comes to read sizes. The index is already fairly large, but can sometimes feel smaller than it is, and paradoxically, query performance is a big part of why.

As some of the work planned for Marginalia Search this year has been progressing a bit faster than anticipated, there was time to implement an unplanned change. This post details the implementation of a system for detecting when servers are online, to avoid serving dead links and improve data quality, and for detecting when websites have significant changes including ownership transfers and parking. Table Of Contents Feature Rationale Data Representation Live Data Event Data Change Detection Details Availability Detection Ownership Changes DNS Implementation Hurdles Scheduling Certificate Validation Conclusions Feature Rationale Availability detection is useful not just for filtering out dead links in the search results, but for informing the crawler that it should stop trying to reach a dead domain, as well as a host of other things.

The most recent change to the search engine is a system that profiles websites based on their rendered DOM. The goal is identifying advertisements, trackers, nuisance popovers, and similar elements. The search engine already tries to do this, but isn’t very good at it because it’s only looking at static code. It turns out to be somewhat difficult to determine what a website that has non-trivial javascript will look like based its source code alone, as this would require us to among other things solve the halting problem.

You wake at 05:30 in the morning, feeling somewhat groggy. Instead of the alarm clock ringing like it normally does, a cheerful hologram appears: “Hi! I’m Kyle, your new alarm clock assistant!” You get dressed as Kyle explains all of the fantastic things he is capable of. You head over to the coffee machine. “Hey there! I’m Evan! Are you ready for AI in your coffee? But first - tell me about yourself!

The search engine has recently gained the ability to index the PDF file format. The change will deploy over a few months. Extracting text information from PDFs is a significantly bigger challenge than it might seem. The crux of the problem is that the file format isn’t a text format at all, but a graphical format. It doesn’t have text in the way you might think of it, but more of a mapping of glyphs to coordinates on “paper”.

Some time ago, I migrated the crawler off the okhttp library, to use Java’s builtin HTTP client. This seemed like a good idea at the time, but has led to a fair number of headaches. Java’s HttpClient has one damning flaw, and that that it doesn’t support socket timeouts. Its only supported timeout values are time to connect, and time until first byte of the response. This means the client can get stuck on a read call if a server stops responding, potentially for a very long time!

A problem the search engine’s crawler has struggled with for some time is that it takes a fairly long time to finish up, usually spending several days wrapping up the final few domains. This has been actualized recently, since the migration to slop crawl data has dropped memory requirements of the crawler by something like 80%, and as such I’ve been able to increase the number of crawling tasks, which has led to a bizarre case where 99.

I’m happy and grateful to announce that the Marginalia Search project has been accepted for a second nlnet grant. All the details are not yet finalized, but tentatively the grant will go toward addressing most of the items in the project roadmap for 2025. I’ve already been working full time on the project since summer 2023, and this grant secures additional development time, and extends the runway to a comfortable degree.

This text is satirical in nature. Tech news is abuzz with rude AI crawlers that forge their user-agent and ignore robots.txt. In my opinion, if this is all the AI startups can muster, they’re losing their touch. wget can do this. You need to up your game, get that crawler really rolling coal. Flagrant disregard for externalities is an important signal to the investors that your AI startup is the one.

This update is a few days late, the canonical birth date of the project is Feb 26. It has been another year of Marginalia Search. The project is still ongoing, still my full time job, although the project is entering a somewhat more mature phase of development, most of the big pieces are in place and do a decent job at what they do. The roadmap for the project is available on GitHub.

A while back an update went live that, with some caveats, changes the time it takes for an update on a website to reflect in the search engine index from up to 2 months to 1-2 days. Conditions being if the website has an RSS or Atom feed. The big crawl job takes about two months, and is run partition by partition, meaning there’s typically a slice of the index that is two months stale at any given point in time.

I recently put together a small library called Slop, for intermediate on-disk data representation for the search engine, replacing a few ad-hoc formats I had in place before. This post isn’t so much an attempt to convince anyone else to use this library, as it makes trade-offs catering to a fairly niche use case, but to explore some of its design ideas, as it all came together very nicely, in the hopes that other libraries can draw ideas from it.

This is a new major release of marginalia search, mostly leaning toward the technical side. Emphasis has been on ensuring the search engine has the technical capabilities to serve more types of queries, especially longer queries which it previously did not handle very well. Effort has also been put toward making sure it’s possible to install and run outside of docker. There is still some work to be done to streamline the installation process, but we’re getting there.

Marginalia Search now properly supports phrase matching. This not only permits a more robust implementation of quoted search queries, but also helps promote results where the search terms occur in the document exactly in the same order as they do in the query. This is a write-up about implementing this change. This is going to be a relatively long post, as it represents about 4 months of work. I’m also happy and grateful to announce that the nlnet people reached out after the run of the grant was over and asked me if I had more work in the pipe, and agreed to fund this change as well!

I’ve been on a bit of a frustration-driven quest to solve a problem I frequently encounter working on the search engine, that is, reading data from disk. You’d think this would be a pretty basic thing, but doing this in a way that is half-way performant is surprisingly hard and requires avoiding basically all the high level tools at your disposal. There’s a common sentiment that modern hardware is fast, so this may not matter, but we aren’t speaking a 30% performance hit, the the question is how many orders of magnitude you’re willing to forego.

As an experiment, I’ve reduced my coffee-intake to a single cup a day for about a week now. It’s made an enormous difference in sleep, mood and energy. I get tired at night, fall asleep quickly, and wake up refreshed. As mentioned previously in the context of morning sunlight exposure—another thing that’s aided my sleeping habits, but is somewhat less practical to sustain as it requires fair weather—I’ve always been slow to get going in the morning, active at night, bad at getting to bed at sane hours.

A year ago I walked out of the office for the last time. I handed in my corpo laptop, said some good-byes, and since then I have been my own boss. This first year has been funded by an NLnet grant, which I’m in the midst of wrapping up. As of now, the work is all done, the final request for payment has been sent. There’s a similar last-day-of-school levity to both these events.

The best description of my problem solving process is the Feynman algorithm, which is sometimes presented as a joke where the hidden subtext is “be smart”, but I disagree. The “algorithm” is a surprisingly lucid description of how thinking works in the context of hard problems where the answer can’t simply be looked up or trivially broken down, iterated upon in a bottom-up fashion, or approached with similar methods. Feynman’s thinking algorithm is described like this:

I’ve experimentally replaced some of the Java implementations of quicksort and binary search with calls to C++ code, and saw huge benefits for the sorting code but the same or worse performance for binary search. The Marginalia Search engine is mainly written in Java, which is language that is good at many things, but not particularly pleasant to work with when it comes to low level systems programming. Unfortunately, a part of building an internet search engine involves database-adjacent low level programming.

A neat property of the parquet file format is that it’s designed with block I/O in mind, so that when you are interested in only parts of the contents of a file, it’s possible to some extent to only read that data. Many tools are aware of this property, and DuckDB is one of them. Depending on which circles you run in, a lesser known aspect of HTTP is range requests, where you specify which bytes in a file to be retrieved.

Been working on improving Marginalia Search query parsing and understanding. This is going to be a pretty long update, as it’s a few months’ work. Apart from cleaning up the somewhat messy query parsing code, a problem I’m trying to address is that the search engine is currently only good at dealing with fairly focused queries, they don’t need to be short, but if you try to qualify a search that is too broad by adding more terms, it often doesn’t produce anything useful.