A Talkdesk consumer survey reveals the ethical considerations of AI in retail

Artificial intelligence (AI), undoubtedly, is helping build stronger connections for brands and create easier customer experiences. It is increasingly used to personalize product recommendations and power voice/chatbots. However, with all emerging technologies, including the most recent development of generative AI (GenAI), it’s all about the data — “garbage in, garbage out” remains as accurate as ever. Consequently, if AI models are not monitored and kept up to date, incorrect responses, biases, and hallucinations can creep in. 

Retail brands that want to take advantage of AI are wrestling with keeping their products accurate and unbiased, as evidenced by the recent Talkdesk Bias & Ethical AI in Retail survey. We asked 1,000 consumers about current personalization experiences and their expectations for retailers to deploy ethical and transparent AI applications. The results of this survey should be an eye-opener for every retailer, whether they use AI or not.

Why consumers have issues with AI in retail.

Biases, stereotyping, and inaccurate personalization are some common threads of frustration among the survey respondents:

• 64% have received an AI-powered product recommendation that did not match their preferences, interests, or previous shopping behaviors. The number is even higher for Hispanic (72%) and Black (69%) respondents. 

• 79% refrain from purchasing product recommendations because they are not tailored to their interests but appear to be the top-selling products that retailers are pushing for sales purposes. 

• 60% avoid AI recommendations because they believe recommended products are biased or stereotypical. Hispanic (75%) and Asian (67%) consumers are most concerned about the problem. 

Shoppers want AI in retail to reflect DEI principles.

Developing, assessing, and deploying responsible AI involves ensuring safety, trustworthiness, and ethics while including people from diverse ethnic, racial, and cultural backgrounds. A perceived neglect of diversity, equity, and inclusion (DEI) in AI has many consumers suspicious and concerned. Less than half (46%) of shoppers believe AI-powered recommendations include diverse shoppers, and only 39% believe retailers consider diverse communities when using or developing AI products. As a result:

• 53% believe that AI-powered facial recognition software will lead to increased instances of racial discrimination. This concern is not unfounded when you consider the example we shared earlier regarding AI image generators perpetuating dangerous race and gender stereotypes. 

• 44% worry that AI will make the shopping experience less inclusive because brands aren’t deploying it responsibly, with Hispanic (50%) and Asian (52%) consumers being the most concerned. 

• 45% are fearful their online shopping experience will no longer be personalized because retailers are not practicing responsible AI. For example, they worry that they’ll no longer receive personalized products inclusive of all demographics or that AI will recommend irrelevant products.

• It is important for brands to note that 31% would stop shopping with a brand if they felt their AI products did not represent or include diverse communities, with 34% of Gen Z and 35% of Millennials willing to go elsewhere

• 69% of men (compared to 53% of women) avoid AI recommendations because of bias and stereotypes. 

• 71% have never purchased a recommended product because it makes them feel like a brand is tracking them — i.e., listening to their in-person conversations, tracking their browsing history, and other concerns. 

Customers expect transparency and accountability.

Regarding AI and customer data, customers are abundantly clear about their expectations. Brands must pay attention to the fact that customers expect transparency at every point of the journey. The report found that:

• 90% of shoppers believe retailers should be required to openly disclose how they use customer data in applying AI usage

• 87% believe customers should have the right to access and review the data retailers have collected about them.

• 80% of shoppers want retailers to seek explicit consent to use their data for AI.

• 46% review a retailer’s customer data privacy statement before shopping.

Sadly, only 28% of shoppers are confident retailers have the proper security measures to protect data used by AI technology. As AI is quickly becoming a pillar of modern customer experience, brands must ensure they launch safe and ethical AI solutions and be transparent in how their AI models leverage and use customer data.

How can retailers promote the ethical use of AI?

Consumers believe the responsibility for improving AI experiences falls on the brands deploying the technology. They highlighted three main focus areas for brands:

1. 88% agree that brands must develop internal policies to guide the ethical use of AI. 

2. 86% state that brands are responsible for monitoring, controlling, and mitigating AI bias.

3. 83% express that brands should develop proactive communications to customers on their use of AI and any potential biases in their services. 

Data governance teams are essential to ensure the data used to train AI systems is consistent, accurate, trustworthy, and up to date. Do you know what your brand is doing? 

Ethical considerations of AI in retail: a top priority for 2024.

Addressing the ethical considerations of AI in retail will be an ongoing activity. Some technologies can monitor and refine AI models to identify and remove discriminatory patterns. This safeguard helps detect and mitigate instances where AI may generate inaccurate or misleading information. However, this checks and balances system shouldn’t entirely rely on technology. Humans should also monitor AI systems to track for biases or inaccuracies, as people can do something even the most powerful AI technology can’t: review information with context and empathy.