In summary, our paper provides concrete evidence of the risk of using LLMs in creative activities and a route toward studying its influence.

Project: joongishin.github.io/problemRefra...

Authors: @joongishin.bsky.social, Anna Polyanskaya, @andrikos.bsky.social, @oulasvirta.bsky.social

(8/8)

What do these all mean?
These mean that using LLMs does not help designers generate more creative perspectives or help less experienced designers to become experts, different from earlier beliefs that using LLMs benefits creative practitioners.
(7/8)

Of greater concern, using LLMs may increase the competence gap between experts and novices. More experienced designers generated more novel problem frames than less experienced ones, which contradicts the optimistic view that LLMs help novices perform similarly to experts.
(6/8)

Rather, we found that using LLMs can make designers have a false belief about the helpfulness of LLMs. While our participants perceived that LLMs were useful, they perceived a similar level of effort in reframing with and without LLMs.
(5/8)

Similarly, using LLMs did not help generate more useful problem frames. These findings suggest that using LLMs does not help, at least in problem reframing, opposite to the earlier belief that LLMs help people to be more creative.
(4/8)

Firstly, we found that using LLMs does not help increase the quality of problem frames. Participants who used LLMs generated problem frames that were as novel as those generated by participants who did not use LLMs.
(3/8)

We observed how expert and novice designers (N = 280) generate alternative ways of defining design problems (i.e., problem reframing) without LLM and three conventional ways of using LLMs. From this, we contribute novel findings on understanding the influence of using LLMs.
(2/8)