Lately, there's been some brouhaha about the idea of using Generative AI as an "artificial user".
At the outset, I’ll say I’m highly skeptical of the utility of such a thing, in most contexts. But let's keep an open mind.
Automated testing tools are not new, but they have relied on the ability of a QA engineer, for example, to write a test script, which an automated set of software can interpret and execute.
So as an extension of that kind of “user”, there’s a possibility that an “AI user“ could have some utility. As I’ve discussed in the past, this really is more akin to asking a text generation tool to complete the sentence, “a user would accomplish [specific task] by…“ and waiting for the completion.
In fact, in Paixon.io, we use similar contructs to help solve a "blank slate" problem around Customer Journey Mapping. It's particularly useful when you - the map maker - know a little bit about the domain in question, but want something to react to. It's easier to point out a Generative AI's mistakes when you already know a bit about the subject matter area.
So there's a world in which a Generative AI tool could speed up some aspects of usability testing, and maybe even preference testing for a well-defined use case (i.e., one for which you are not developing a completely novel solution.) Again, remember that LLM‘s are text completion engines; they are searching and information space for an “best match“ for the query that you’re presenting Think of them as “best practice evaluators.”
For example, let's ask ChatGPT: "Imagine you are a 40 year male shopping for new sneakers. You encounter a website with a list of styles and sizes. What problems might you have navigating such a website?" Our "virtual user" lists of 15 fairly-generic, but not wrong, answers:
1. Complex Navigation: If the website has a complex or confusing navigation structure, it might be difficult for me to quickly find the section for sneakers. I'd prefer a simple and intuitive menu that clearly directs me to the sneaker collection.
2. Small Font Size: If the font size of the text on the website is too small, it could strain my eyes and make it hard for me to read product details, especially if I'm browsing on a smaller screen like a smartphone.
3. Lack of Filtering Options: I would appreciate filtering options that allow me to narrow down my search based on size, style, color, and other preferences. Without these filters, I might have to scroll through a lot of options that aren't relevant to me.
4. Limited Size Availability: It can be frustrating if the website doesn't have my size available in the sneakers I like. I would prefer a feature that only shows me styles available in my size.
5. Complicated Checkout Process: If the checkout process is overly complicated with multiple steps and forms to fill out, it might discourage me from completing the purchase. A streamlined and user-friendly checkout is essential.
6. Insufficient Product Information: Clear and detailed product descriptions, including information about materials, comfort, and durability, are important for me to make an informed decision.
7. Missing High-Quality Images: High-resolution images from different angles help me get a better sense of the sneakers' design and details. Blurry or limited images make it hard to assess the product.
8. Inadequate Sizing Information: Accurate sizing information is crucial. If the website lacks detailed sizing guidelines or information about how specific styles fit, I might hesitate to buy due to concerns about fit.
9. Unorganized Styles: If the sneakers are listed without clear categories, such as casual, athletic, or formal, it can be time-consuming to scroll through various styles that don't match my preferences.
10. Slow Loading Times: Slow-loading pages can be frustrating and may even lead me to leave the website if it takes too long to browse and select products.
11. No Reviews or Ratings: Reviews and ratings from other buyers can provide valuable insights into the comfort and quality of the sneakers. Their absence might make me hesitant to make a purchase.
12. Hidden Shipping Costs: Unexpected shipping costs during checkout can be a turn-off. Clear information about shipping fees should be provided upfront.
13. Limited Payment Options: If the website only offers limited payment methods that I'm not comfortable using, I might abandon my purchase.
14. No Easy Returns Policy: A clear and easy-to-understand returns policy is important. If it's not clearly stated, I might worry about the hassle of returning or exchanging sneakers that don't fit properly.
15. No Guest Checkout: For privacy reasons or just convenience, having the option to check out as a guest without creating an account would be appreciated.
If I were to ask Google for "15 things every eCommerce website should consider", I'd probably get a few thousand results for websites with similar lists. There are not hugely insightful areas for investigation (though I am surprised and delighted to see "small font sizes" called out at #3; ChatGPT clearly "understands" that an "over 40" male might need glasses.
The more you are working on a novel problem or use case for which there is no known ideal solution, the less useful a artificial user will be.
Prompt: Imagine you are a 75 year old woman trying to decide if an Apple VisionPro is for you. What questions might you have?
Again, ChatGPT's answer is not exactly wrong - but not insightful either:
Of course, dear! I'd be happy to help you think through whether the Apple VisionPro would be a good choice for you. Technology can be a bit overwhelming, especially as we get older, but let's take it step by step. Here are some questions you might want to consider:
Of course, dear! I'd be happy to help you think through whether the Apple VisionPro is the right choice for you. It's important to consider how this technology fits into your lifestyle and meets your needs. Here are some questions you might have:
1. What exactly is the Apple VisionPro? It's good to start with understanding the product itself – what it does, what it looks like, and how it works.
2. How would the Apple VisionPro benefit me on a daily basis? Think about your daily routines and activities. Would it help you stay connected with family and friends, manage your health, or assist with other tasks?
3. Is the Apple VisionPro easy to use? Technology can sometimes be overwhelming. You might want to know if it's user-friendly and if there's someone to help you set it up.
4. What features does it offer that are specifically designed for seniors? Are there any accessibility options or features that make it more suitable for older individuals?
5. How does it handle health and safety concerns? Are there any features that can help monitor health conditions or provide assistance in emergencies?
6. What's the cost, both upfront and ongoing? Consider the price of the device itself as well as any subscription fees or additional costs for services.
7. Can I adjust the text size and other settings for better visibility? Vision changes as we age, so being able to customize the display can be important.
8. Does it connect me with medical professionals or caregivers if needed? Some devices offer ways to contact medical personnel or family members in case of emergencies or health concerns.
9. How does the privacy and data security work? You'd likely want to know how your personal information and data will be handled.
10. Are there any tutorials or support options available? Learning to use a new piece of technology can be a bit challenging, so having access to tutorials, customer support, or local assistance could be reassuring.
11. Can I use it to see photos and videos of my loved ones? Sharing and enjoying memories can be a wonderful use of technology.
12. What happens if the device malfunctions or needs repairs? Knowing about warranty, repair, and replacement options is important.
13. Is it portable? Can I take it with me when I visit family or go on trips? Depending on your lifestyle, portability could be a big factor.
14. Are there any ongoing fees or subscriptions? It's good to understand the long-term costs associated with the device.
15. How do software updates work? Technology evolves, so knowing how updates are handled can help ensure your device remains useful.
Remember, dear, there's no rush. Take your time to research and maybe even talk to someone who has experience with the Apple VisionPro. It's all about finding a technology that fits into your life and brings you joy and convenience.
Again, helpful if obvious considerations. (Also, it's interesting that ChatGPT has decided that I am a 70-year old woman, as this list of considerations is now addressed to me, "dear".
But ChatGPT shows no intuitive understanding of what this device is, what it's for, and what might be different about it. The more novel your problem space, the more generic your "virtual user" is going to be.
In a nutshell, this is a sort of Mobius strip kind of problem: the closer we get to having a general intelligence AI that can serve as a useful Artificial User, the closer will be to having that same AI design good solutions to user problems that won’t need testing in the first place.