When gen AI can help, and when it can't

Given the hype surrounding generative AI since ChatGPT burst onto the scene, one could be forgiven for thinking it's the panacea to our woes. Indeed, its capabilities are so pronounced that many have warned about the looming destruction of jobs (again).

However, as research from Harvard demonstrates, generative AI is likely to be helpful in some tasks and harmful in others.

The researchers worked with over 750 consultants from the Boston Consulting Group, each of whom was randomly assigned to work on a task designed to be as realistic as possible. One of the tasks was more analytic in nature, asking the consultants to analyze a company's performance before providing the CEO with recommendations.

The second task was much more creative. The consultants were tasked with developing a new footwear product to serve a fashion brand's as-yet-underserved segment of the market.

The researchers divided the consultants into three groups. The first group used AI without any instructions. The second group used AI after watching a short training video. The third group had no access to AI.

AI support

Those using AI finished both tasks faster. However, the quality of their work varied significantly between the two tasks. In the creative footwear task, those using AI outperformed those without it by 40%. The lower-performing consultants saw the biggest improvements, catching up with their stronger peers.

However, those using ChatGPT-4 did worse in the strategic decision-making task than those without AI. AI users were 20% less likely to find correct solutions. Still, their recommendations were rated higher because they were more persuasive and well-written.

In short, consultants using AI were more likely to be wrong but sounded more convincing. This creates a significant issue for businesses.

Where the limits lie

The results highlight one of the core challenges with generative AI. Given the extreme amount of hype surrounding the technology, it's difficult to know where its capabilities end. Indeed, Sam Altman has even tried to claim that hallucinations are a feature rather than a flaw.

You can trust AI to help with tasks within its "jagged frontier" and get high-quality results. But you're more likely to make mistakes if you use it for functions beyond this frontier.

This is tricky because large language models (LLMs) are still fundamentally opaque. Sometimes, they produce incorrect results that look plausible and convincing, making it hard to predict their failures.

Even if you accurately identify AI's current limits, the technology is evolving so quickly that these boundaries could shift tomorrow.

Decreasing diversity

Another challenge posed when people use generative AI en masse is that it can reduce cognitive diversity. The researchers noticed that the group's pool of creative ideas became smaller when generative AI was used to help them.

In the footwear task, for instance, those using AI often came up with strikingly similar ideas. Consultants without AI worked more slowly, and their ideas were generally of lower quality. However, they produced a more diverse range of proposals.

Companies could benefit from relying more on human creativity to generate unique ideas. This is especially important when radical innovation is needed, and competitors heavily use AI.

There is no simple answer to the question, “For which tasks should companies use AI?” Instead, businesses should experiment systematically and strategically with different uses.

Finding your way

The study is a timely reminder that the best approach is not to blithely take the marketing hype at its word and assume that generative AI will be the answer to all of your problems. Instead, employ a more experimental approach, testing it to see where it helps and where it doesn't.

Your aim should be to properly understand where the technology might be best suited and the various risks and unintended consequences associated with it.

If you simply deploy AI haphazardly, it could reduce productivity, diminish human creativity, and harm the accountability essential to the most important tasks.

Of course, this isn't to say that AI shouldn't be used, as ignoring it could also affect your competitive edge. There are likely instances where the technology helps and instances where it doesn't. It's your job to experiment so you fully understand which is which.

The best way to use AI will remain uncertain in the near future. Therefore, managers will need to keep experimenting with the technology as it evolves.

Even if companies don't question whether to adopt AI, they should consider how to use it responsibly to make work more productive and meaningful for employees working at the jagged frontier of their abilities.