Self-Checkout Assistant
A self-checkout that answers back
A GPT-4 voice assistant at self-checkouts: it tells customers where a product is, what the promotion is and what to do when a barcode will not scan. The project focused on self-checkouts in brick-and-mortar stores. A self-checkout speeds up shopping but shifts some of the work onto the customer. Customers have to find products themselves, understand promotions and deal with barcodes that will not scan. We built a voice assistant that answers these questions at the checkout, before the customer calls for staff.
At the checkout, the customer is left alone with a question.
- Work without knowledge. The checkout shifts service onto the customer but does not give them a sales assistant's knowledge. Customers are left alone with questions the checkout cannot answer.
- Three reasons to call staff. Most often it is about where a product is, how a promotion works and what to do when a barcode will not scan. Every such call holds up the queue, even though the answer is usually simple.
- One employee, several stations. Usually one person watches several checkouts at once. While they help one customer, the others wait. At peak hours, the queue grows faster than the employee can help.
- Customers do not read instructions. On-screen text and stickers at the checkout are rarely read. Customers want to ask and get an answer, just as they would ask a sales assistant.
- A model without data makes things up. A language model without access to store data will invent an aisle or a promotion. At the checkout, a wrong answer is worse than no answer.
A conversation instead of another instruction screen.
- A voice interface. Customers ask by voice because it is the fastest way. They do not have to hunt through menus or type a question on screen. Their hands are busy with shopping, so voice is the natural channel.
- Push-to-talk microphone. The assistant only listens after a button is pressed. This solves two problems: shop floor noise and other customers' privacy.
- Store context. The assistant answers based on the specific store's data: assortment, availability, aisles and promotions. The model does not guess, it works from facts.
- Answers by voice and on screen. Customers hear the answer and see it on screen: the aisle, the price and the next step. In a noisy store, the screen matters as much as the voice.
- A clear boundary of competence. The assistant hands returns, complaints and payment issues over to an employee. In matters that require a human decision, it does not try to replace one.
- Speed as a requirement. The voice loop has to respond before the customer decides the device has frozen. We treated response time as a requirement, not an extra. Too long a silence after a question makes customers give up and call staff anyway.
- Product location. Customers ask where to find a product and get the aisle number by voice and on screen. They do not have to leave the checkout to look for an employee.
- Explaining promotions. The assistant explains how a promotion works and whether it covers a given product. Customers understand the price before they pay. They do not have to guess whether the discount will apply at payment.
What we built it with.
How it came together.
Customer journey analysis
We mapped the moments when customers at the checkout need help. This analysis revealed the three main reasons for calling staff.
Voice loop
We combined speech recognition, GPT-4 and speech synthesis into a single flow. The most important criterion was the time from question to answer.
Store data
We built a context layer with the store's assortment, availability, aisles and promotions. The assistant uses it for every answer. A promotion change or a product moved to another aisle reaches the assistant along with the store data.
Checkout system integration
We connected the assistant to the checkout so its answers relate to the current situation at the station.
Testing in shop floor noise
We tested it in real store conditions, with noise and conversations in the background. A lab cannot replace this, because speech recognition always performs better in a quiet room.
What changed.
See other projects.
Have self-checkouts and queues to handle?
We will design an assistant that answers customers based on your store's data. If you are rolling out self-checkouts, count how many times an employee walks over to a customer with the same questions. These repetitive situations are the easiest for an assistant to take over. A language model at the checkout must use store data, because without it, it will guess. It is also essential to define what the assistant does not do and hand those matters over to a person.
Let's talk about an AI assistant →