Dense block of math

I am incredibly skeptical of the AI hype. I am curious and interested in the longer tail dragging in its wake. Setting the skepticism aside and focusing on the curiosity and the interesting bits can be useful. First, we can give the skepticism some air and then we should talk about big balls of math.

LLMs are interesting. They are also expensive, exploitative, biased, environmentally bad, unwieldy, unreliable and in many ways primitive. Whether valuable functionality you can wrest from them will ever efficiently separate from the way they will be abused, the distractions they are, the misinformation and misunderstandings they will perpetuate, that's unclear. We know they will perpetuate bias. We know they will spread false information. We know they will be applied to tasks they are woefully unsuited to perform. They are also here and given capitalist incentives enshittification of all spaces by the power of AI will commence post-haste.

Setting that big ball of not-sure-I-can-do-much-about-it aside. What are the technically interesting bits, what are useful perspectives, what are wins we can wrest from this hype-cycle and if we are unfortunate, accelerating dystopia?

It seems that we are closer to being able to operate on semantics of language than ever before. That is meaning, content, intent and style of expression. As well as generating good enough keys for spoken language. I keep coming back to Whisper because it is a machine learning model that really tackles something we cannot program rules for. The specification for spoken language only exists in the way that language is spoken. Taking a massive amount of examples and feeding those into the math machine and getting something out that can take chunks of audio and produce the likeliest textual representation is a massive win. And it allows you to do fun things, sure. It also makes accessiblity much easier, at scale, with limited resources and so on.

The open source models, may they stay mostly competitive, are in some ways libraries for things that we cannot write code for. Much like generating API clients from specs when dealing with major cloud providers is really the only feasible option, I think generating a fuzzy and dense ball of math is probably the only solution to wrangling language.

If you've poked around with vector embeddings and all that you might have experienced the weirdness of it. I think they clicked for me listening to Chris Grainger talk about the vector space and searching for similar texts based on how close they are in language. The word algebra example in simonw's post is also incredibly enlightening. Germany + (Paris - France) -> Berlin. These models are relatively small and run reasonably quick, even on CPU. Sentence Transformers lands at 134 Mb if I'm checking correctly. For an LLM reference Llama2 is about 10 Gb and 3 Gb for some other part needed. So you can generate vectors for your data and then you can perform searches by generating vectors for the query. What's good about it?

A few things scratch my brain on this. Natural language search. You write what you want or you provide some text that is similar to what you want more of. And it won't match just on exact words. It will generally match on related topics and overall ends up giving you semantic relevance. Hopefully. I am not sure how sensitive it is to writing style, LLMs certainly can mimic styles, so my assumption is that to some extent you will get matches kind of based on writing style. Which is more interesting than necessarily useful in most cases.

But this type of language-based retrieval is near trivial now where I hadn't even heard of people doing it a couple of years back. Every database is shipping some vector solution now. We have a static chunk of math that will let us do math on the language of chunk of text.

A thing I haven't tried doing with embedding but which I hear LLMs do quite well is also to provide a snap-to-grid solution for taking natural text and behind-the-scenes mapping it to a limited set of possible commands. I think you could do a leaner version of this with embeddings. Everything will be chat bots now I guess. Your chat bot can perhaps only do a few different things for the user. But it can select the options based on natural language. Some will be voice-operated of course. Because that's easy now. We will probably hate it. And then someone starts trying to save a buck by pulling back on the hardware so everything becomes slower. Oh no. Back to dystopia.

In some ways I consider the embedding models almost UNIX-like tools in their relative simplicity. They just deal in a domain where we can't write lean code and instead need to pack in a bunch of dense math. 134Mb is kind of lean for tackling most of the english language. I guess.

I hope the current crop of massive LLMs are the primitive progenitors of more useful slimmed down models. I hope we can do a lot more with less training and less resource expenditure. Unfortunately the only place where that will be a focus is obviously mobile. Because optimization and restraint for the general public only ever happens on mobile devices and in massive data centers.

I don't think LLMs are clearly the path to general artificial intelligence. I do think they've proven to be more capable, more quickly than most of us expected. I don't think the fact that they are heavily bell-curved machines that produce averagely truthy language necessarily makes them useless. I think they are extremely poorly suited for some tasks. And should be forbidden to perform many actually important tasks. But as mechanisms to do more with language and software they are really powerful and interesting.

I have no trust for the AI-industrial complex. I don't expect this to go well for the world in the grand scheme of things. I also don't think it will leave me alone as a generalist developer and I do find parts of it quite interesting. So I try to engage with it in ways I think make sense.

How do you approach the ML spectacle as a developer? Not at all? Carefully? Diving in? You can reply to this email or poke me on the fedi @lawik@fosstodon.org, I enjoy hearing your thoughts.

Thanks for reading. I appreciate your attention.