Ben Levin, Experience Strategy and Design

Table for 1 (Part 1)

Tue, 27 Jan 2026 06:26:35 -0800

Over the weekend, my son asked if there was any work he could do to earn some money. I’ve got a few side projects running, including a Job Board for UX Researchers and Designers, and he sometimes helps to manage it: Cleaning up data, finding new jobs, getting them queued up for me to review. Just a few hours of work to buy presents, Chipotle, Tendy’s: all the things a teenager uses money for.

But instead of giving him the usual data cleanup task, I had a different idea.

Earlier that day, I’d been experimenting with an AI coding tool called Lovable, trying to re-platform the jobs board. I wanted to make some functional changes and though I’m stronger in IA/UX/IxD than development, I thought I’d use a few “vibe coding” tools to experiment and see what I could hack out.

I gave it a prompt, pointed it at the existing site, and said “build me a replacement for this.” In about four minutes, it had built a fully functioning site — complete UI, data model, everything you’d need to get the thing up and running, including one-click deploy to the cloud.

On Payola and X-Rays, Part 2

Fri, 26 Jan 2024 03:50:53 -0800

In Part 1 I talked a little about the difference between probability and severity in how we assess and manage risk, and how misalignments between our estimates of the two lead us to take actions that are at a minimum sub-optimal, and at worst decidedly against our own interests.

Using the example of how the American managed healthcare industry works, with its tendency to over-emphasize data tracking and documentation, I identified three risks in such a system:

Inadequate Accounting of Costs, such that providers of healthcare services might go unpaid.
Mis-Tracking of Interventions to Outcomes, possibly resulting in unnecessary procedures being performed on patients (or necessary procedures not being performed.)
Confusion Leading to Avoidance, where the system as a whole becomes so daunting, patients avoid it until it's impossible to do so.

There are far from the only risks...Read More

On Payola and X-rays: Part 1

Tue, 09 Jan 2024 05:01:50 -0800

It’s worth thinking about the many ways in which incentives drive behaviors in unexpected ways.

Recently I was trying to digest four cataclysmic AI scenarios and pondering for a bit the difference between severityand probability, as we apply it to risk.

Probability is the easiest to paraphrase, though the statistical definition quickly gets wonky. But to simplify: given a period of time over which you might make separate observations, divide the number of times you observe an event by the number of observations. That’s your probability.

Severity is generally more loosely measured and less well understood: What impact might an event have on those exposed to it? Somewhere between negligible and catastrophic captures most events, but usually these are estimates and subject to dispute about the particulars.

Risk, then, is the combination of both factors: a catastrophic scenario with a 1% probability of occurring in the next 100 years/hands...Read More

Good Design is an Act of Love

Mon, 27 Nov 2023 05:05:06 -0800

My Thanksgiving preparations were a laid back this year; Hosting was not my job this time around, and I gleefully took responsibility for making “pies”, plural. Not promising anything too specific left me with an opportunity to improvise, which in my case means “procrastinate and then dash to the Acme ('Ack-ah-ME') for last minute ingredients.”

This year did not disappoint, though my trusty backup - a $5.99 pumpkin pie from Costco that is easily the best on the planet (fight me), and measures approximately 2 1/2 feet across - keeps me out of too much trouble if things go sideways.

Tuesday night had me in a bit of a panic, as I’d not yet chosen a recipe. But I hauled out The Joy of Cooking, and turned to page 670.

A photograph of page 670 of The Joy of Cooking.
Photograph Only ©2023, Ben Levin

Now, I have a number of go-to cookbooks. The Silver Palette, The Book of Jewish Food, The Frugal Gourmet, etc. My favorite is an anthology of MFK Fisher’s - The Art of Eating - which I re-read almost every year. It has recipes, sure, but also Fisher's stories of cooking and eating and living...Read More

Hoping for Benign Invisibility

Mon, 20 Nov 2023 03:30:01 -0800

Of last week's late-breaking news of the firing of Sam Altman from OpenAI, much has already written, I think only two points matter for the moment:

It is quite remarkable to watch the speed with which a Board of Directors can act in a non-profit context vs. what happens at most publicly and privately owned companies (at least from an outsider's view.) Rarely are CEOs held to account at all, even for a company's poor financial performance.

Boards are often populated with people friendly to the CEO, and there is, structurally, a lot of incentive for everyone to get along on the big-picture items. But in this case, four people with no financial ownership stake took quick and decisive action over a matter of maybe 2-3 months (the "firing" seems to have coalesced between Thursday and Friday afternoon, but that was just the visible portion. The maneuvering may have begun some weeks ago.)
Apparently, at least if you're Sam Altman,Read More

Whither AI "Users"?

Thu, 31 Aug 2023 07:10:00 -0700

Lately, there's been some brouhaha about the idea of using Generative AI as an "artificial user".

At the outset, I’ll say I’m highly skeptical of the utility of such a thing, in most contexts. But let's keep an open mind.

Automated testing tools are not new, but they have relied on the ability of a QA engineer, for example, to write a test script, which an automated set of software can interpret and execute.

So as an extension of that kind of “user”, there’s a possibility that an “AI user“ could have some utility. As I’ve discussed in the past, this really is more akin to asking a text generation tool to complete the sentence, “a user would accomplish [specific task] by…“ and waiting for the completion.

In fact, in Paixon.io, we use similar contructs to help solve a "blank slate" problem around Customer Journey Mapping. It's particularly useful when you - the map maker - know a little bit about the domain in question, but want something to react to. It's easier to point out a Generative AI's mistakes when you already know a bit about the subject matter area.

So there's a world in which a Generative AI tool could speed up some aspects of usability testing, and maybe even preference testing for a well-defined use case (i.e., one for which you are not developing a completely novel solution.) Again, remember that LLM‘s are text...Read More

Where is AI Most Compatible with UX Research?

Wed, 30 Aug 2023 06:50:52 -0700

To write this post, I spoke into my phone and watched with no small bit of amazement how quickly my device was able to translate my individual voice into written text, albeit with an aggressive use of punctuation.

When we talk about AI, and how it will impact the work of UX, we really do need to consider the fact that in many ways, the impact is already there - so much so that we don’t even notice it anymore.

In that regard, AI's impact on UX Reseach will probably center around transcription and summarization - if we can get the abelist, English-first, western-centric bias out of the system.

And yet, the evidence that tools can do this well at the moment (at least in so far as Chat-GPT 3.5, llama, and Bard are concerned) is still uncertain. To understand why that's the case, we need to think about what LLM‘s are, and what they can and cannot do.

LLM‘s are quite good at reducing the relationships between seemingly unrelated things, and following linkages that may not be apparent on a surface level. Think of a simple spreadsheet, listing types of animals as columns headers (mammals, birds, reptiles, invertebrates, etc.), and a list of characteristics as rows (average height, average weigh, number of legs, etc.)

An extremely simplistic LLM, looking at the data filling in this spreadsheet, will "notice" that mammals tend to have two or four legs, and run larger and heavier than...Read More

Can UX Researchers Remain Viable Without Integrating AI Into Their Practice?

Sat, 19 Aug 2023 07:25:03 -0700

What’s a UX researcher to do, when faced with a environment in which many of their day-to-day tasks seem, at least on the surface, susceptible to automation?

In some ways, this isn’t really a question about AI. It’s more a question about our daily workload, and whether how it fundamentally contributes to a deeper understanding of human computer interaction and UX design.

For example, when we are looking through a session transcript and trying to pull out common threads across multiple participants, the semantic analysis may seem easy to speed along with some automated help: finding common phrases, analyzing completion times, and looking at how participants wend their way through structured tasks, the recording of which efficiently marked up. A “fully instrumented” usability test, for example, already lends itself to automated analysis.

For UX researcher to remain viable in this market, she’s going to have to make use of whatever tools make her job more efficient. But this doesn’t necessarily mean diving headfirst into using a large language model AI to analyze her research sessions.

In fact, that could be quite counterproductive. Many of the current models show, strong biases, lack context, awareness, and are prone to creating summaries of text one submits for analysis that are semantically logical, but contextually incoherent.

As a result, you may end up spending more time “debugging“ any AI-generated analysis then you would spend just performing it on your own.

Having said that, there are areas where, especially for individual, independent practitioners, automation tools can make a team of one far more efficient. Participant recruitment and compensation, screening, scheduling, panel management,...Read More

Is the Rise of AI Good, or Bad, for UX Research?

Tue, 15 Aug 2023 12:25:59 -0700

Like any new technology, AI is going to substantially impact all of the fields of the touches.

But to talk about this intelligently, we need to distinguish between different types of AI, and think deeply about what they are fundamentally capable of.

That’s difficult to do, because at this point we don’t actually know what AI is capable of, nor do we fully understand why the capabilities it does seem to exhibit, exist.

The best example I can think to explain this phenomenon was provided in a recent article by Simon Willison, who posited:

As example of this note that some of the most interesting behaviors of large language models in particular were only discovered years after those models were released to the public.

This probably outlines the way in which AI is going to impact UX research the most substantially. If we, as researchers, are worth our salt, we’ll quickly dive into exploring how users interact with AI, because doing so will put...Read More

How (Some) AI Will Crash (Maybe.)

Tue, 25 Jul 2023 06:14:41 -0700

I’m writing this mostly as a note to myself, though if it does get published and read by anyone, maybe some feedback will help me work out the thinking a little more clearly. If so, email me @ ben AT Ben Levin dot com.

An image of a robot pushing rod against a flat-bottomed boulder in an attempt to move it up a hill, generated by OpenAI's DALL-E with the prompt, ""a pencil sketch of a robot Sisyphus pushing a boulder up a hill"

Part 1

AI has moved Crypto into the background, and the only news stories you hear these days are of Crypto company leaders being arrested and smacked with securities and fraud allegations, from a jumble of three-letter (mostly) US government agencies. The SEC wants you for taking people’s money without filing disclosures; CFTC wants you for market manipulation; DOJ wants you for just stealing money.

These are not exhaustive descriptions of what’s currently rendering the crypto ecosystem into mulch, but they are illustrative, and I think fairly common. And you can argue whether they are the righteous justice of a public latterly moved to uncover and prosecute fraud perpetrated on the weak by the wealthy or the swift; or you can grumble that it’s more of the same establishment trying to knee-cap the up-and-comers in an attempt to defend an equally-fraudulent, but far more politically-connected and societally embedded system. Both, if not right, are also somewhat probably not-wrong.

And I should say, I think what follows applies AI LLMs being developed by for-profit companies who aim to make some kind of financial return on their development. I don’t think the mechanics I’m describing here apply to, for...Read More