 Large Language Models (LLMs) have revolutionized natural language processing but can exhibit biases and may generate toxic content. While alignment techniques like Reinforcement Learning from Human Feedback (RLHF) reduce these issues, their impact on creativity, defined as syntactic and semantic diversity, remains unexplored. We investigate the unintended consequences of RLHF on the creativity of LLMs through three experiments focusing on the Llama- series. Our findings reveal that aligned models exhibit lower entropy in token predictions, form distinct clusters in the embedding space, and gravitate towards "attractor states", indicating limited output diversity. Our findings have significant implications for marketers who rely on LLMs for creative tasks such as copywriting, ad creation, and customer persona generation. The trade-off between consistency and creativity in aligned models should be carefully considered when selecting the appropriate model for a given application. We also discuss the importance of prompt engineering in harnessing the creative potential of base models.  Keywords: Large Language Models (LLMs), Reinforcement Learning from Human Feedback  (RLHF), AI Alignment, Recommendation Systems, Diversity and Creativity . **Introduction** Large Language Models (LLMs) have demonstrated remarkable capabilities in generating human-like text, with applications spanning various domains, including marketing. However, LLMs have also been shown to exhibit biases and generate toxic or inappropriate content (Bender et al., ; Gehman et al., ), prompting the development of techniques such as Reinforcement Learning from Human Feedback (RLHF) to *align* LLMs with human values and preferences (Ouyang et al., ; Stiennon et al., ), aiming to mitigate these issues.  While RLHF has proven effective in reducing biases and toxicity in LLMs, our work suggests that this alignment process may inadvertently lead to a reduction in the models' creativity and output diversity. In the context of this paper, we define "creativity" as the model's ability to generate outputs with high syntactic and semantic diversity. Syntactic diversity refers to the variety in the choice of words, sentence structures, and other linguistic elements, while semantic diversity pertains to the range of meanings, sentiments, and ideas expressed in the generated text1.  The potential trade-off between safety and creativity is particularly relevant in the context of marketing, where generating diverse and engaging content is crucial for various applications, such as customer persona generation, ad creation, writing product description, and customer support. One specific application of LLMs in marketing is the generation of simulated customers or personas with diverse preferences and backgrounds.   These personas can be used for various purposes, such as training bank associates to better communicate with actual customers or providing business school students with more engaging alternatives to traditional case studies.  When generating customer personas using LLMs, there are two primary approaches:  generating multiple personas simultaneously2 or generating them one at a time. While creating multiple personas simultaneously might seem more efficient, it has limitations due to the context size3 constraints of LLMs and the causal attention mechanism in these models (Vaswani et al., ). The causal attention mechanism means that the distribution of the generated personas will not be independent, as each new persona would depend on the previous generations. Therefore, generating personas one at a time is a more suitable approach, as it solves both the context size and independence issues. However, when using this method with an aligned LLM, an unexpected challenge arises: the generated personas often exhibit striking similarities in their preferences and characteristics, lacking the desired heterogeneity. This lack of heterogeneity in the generated personas is problematic as it limits the ability to capture the diverse preferences and behaviors of real-world customers, potentially leading to less effective marketing strategies and suboptimal user experiences.  This observed lack of creativity in the outputs of aligned models led to the suspicion that the RLHF process itself might be the underlying cause. We investigate this problem by taking a foundational approach and examining the issue at both the semantic and syntactic levels. Our study comprises three experiments that aim to provide a comprehensive understanding of how the alignment process affects the diversity of LLM outputs.  Experiment  serves as a concrete example of the impact of RLHF on creativity in a practical marketing context. We generate customer personas and their corresponding product reviews using both the base and aligned models, comparing the diversity of the generated attributes, such as names, demographics, and review content. The results reveal significant differences in the variety of outputs between the two models, with the aligned model exhibiting less diversity and more repetitive patterns.  Experiment  investigates the semantic diversity of the models' outputs by examining their ability to recite a historical fact about Grace Hopper1 in various ways. The generated outputs are encoded into sentence embeddings and visualized using dimensionality reduction techniques. The results reveal that the aligned model's outputs form distinct clusters, suggesting that the model expresses the information in a limited number of ways. In contrast, the base model's embeddings are more scattered and spread out, indicating a higher level of semantic diversity in the generated outputs. These results are further supported by the cosine similarity analysis which shows the aligned model's outputs are more semantically similar to each other compared to the base model's outputs.  An intriguing property of the aligned model's generation clusters in Experiment  is that they exhibit behavior similar to *attractor states* in dynamical systems. We demonstrate this by intentionally perturbing the model's generation trajectory, effectively nudging it away from its usual output distribution. Surprisingly, the aligned model gracefully finds its way back to its own attractor state and in-distribution response. The presence of these attractor states in the aligned model's output space is a phenomenon related to the concept of *mode collapse* in reinforcement learning, where the model overoptimizes for certain outputs, limiting its exploration of alternative solutions. This behavior contrasts with the base model, which exhibits greater flexibility and adaptability in its outputs.  Experiment  delves into the syntactic diversity of the models by analyzing the entropy2 of generated tokens and the probability distributions over the top predicted tokens at each step. The results show that the base model exhibits significantly higher average entropy, assigning more spread-out probabilities to different tokens, while the aligned model has a more skewed probability distribution, favoring certain tokens over others.  The findings from these experiments suggest that the RLHF process, which aims to reduce biases and toxicity in LLMs, may transform them into more deterministic algorithms that lack the capacity to explore diverse sets of token trajectories, leading to reduced semantic and syntactic diversity in their outputs. In other words, aligned models exhibit higher *confidence* in their outputs, providing consistency and predictable behavior. However, this confidence comes at the cost of lowered creativity, as the models tend to stick to a limited set of outputs.  In marketing, this trade-off between consistency and creativity has far-reaching consequences. Applications such as copywriting, writing scripts for ad clips, and customer persona generation all require a high level of variation and diversity in the generated content. If an aligned model is used for these tasks, the resulting outputs may lack the necessary heterogeneity and novelty to effectively engage the target audience. Similarly, in the domain of *recommendation systems*, LLMs that lack diversity in their outputs may struggle to recommend a diverse set of products to users, potentially leading to suboptimal user experiences and reduced customer satisfaction.  It is important to note that base models, while more creative, are not directly usable in applications like chatbots. As a result, techniques such as *prompt engineering* (also known as prompt programming) (Sahoo et al., ) become even more crucial when working with base models. These techniques can help guide the models' outputs and make them more suitable for specific applications while still leveraging their creative potential. Contrary to the belief that prompt engineering may become obsolete, our findings suggest that these techniques will be more important than ever in harnessing the power of base models.  Consequently, the choice between base and aligned models should be carefully considered based on the specific requirements of the task at hand. For applications where creativity is paramount, such as marketing, fiction writing, and other areas where novelty and diversity are valued, base models may be more suitable. On the other hand, aligned models may be preferred when safety and consistency are the primary concerns, such as in customer support or content moderation.  The remainder of this paper is structured as follows: Section  provides background information on LLMs, RLHF, and their applications in marketing. Section  presents our experiments comparing the behavior of base and aligned models, followed by a discussion of the results and their implications in Section . Finally, Section  concludes the paper and outlines future research directions.  . **Literature Review** RLHF has emerged as a promising technique to align LLMs with human preferences and values. However, recent research has highlighted several limitations and potential unintended consequences of RLHF, including scalability and efficiency concerns due to its reliance on human annotators (Lee et al., ; Yuan et al., ), variability and potential bias in human feedback affecting the quality and consistency of the process (Yu et al., ), vulnerability to manipulation by adversarial annotators leading to security breaches and ethical concerns (Wang et al., ), and alignment challenges such as objective mismatch and length bias (Lambert and Calandra, ; Shen et al., ).  Despite these limitations, LLMs have shown significant potential in transforming various aspects of marketing and business. They can automate and accelerate timeconsuming tasks such as text generation, summarization, and content creation, leading to increased productivity and efficiency in marketing and business operations (Head et al.,  ). LLMs also enhance customer interaction by providing personalized and contextaware responses, which can improve customer satisfaction and engagement, particularly in customer service and support functions (Franceschelli and Musolesi, ). Furthermore, by analyzing large volumes of data, LLMs can generate valuable market insights, helping businesses understand customer preferences, market trends, and competitive landscapes, which can inform strategic decisions and marketing campaigns (Eloundou et al., ).  However, while the literature has explored the applications and limitations of LLMs in marketing and business contexts, there is a notable gap in understanding how the RLHF process affects the creativity and variation in the models' outputs. This is a crucial aspect for marketers and professionals who rely on LLMs for creative tasks. Understanding the trade-off between alignment with human preferences and the preservation of creative diversity in the generated content can have significant implications for the effectiveness and engagement of marketing initiatives.  . **Experiments** Remember that our goal is to evaluate and contrast the diversity of texts produced by base models and their aligned counterparts. We focus on both the short-term (syntactic) and long-term (semantic) variations in model outputs using the Llama- language models.   Meta has made both the base models1 and their corresponding aligned versions2 publicly available, making them an ideal choice for this study. Therefore, comparisons are made between Llama--B-text (the "base" model) and Llama--B-chat (the "aligned" model) where B refers to the parameter size of the LLM. These models are currently highly favored within the open-source community1. Their widespread use is partly attributed to the affordability of finetuning them.  We conduct three experiments to examine the effects of the alignment process on model creativity and diversity. Experiment  serves as a concrete example of the differences in creativity between the base and aligned models, while Experiments  and  investigate the underlying mechanisms that contribute to these differences.  Each LLM is given an initial prompt which must be completed for a maximum2 of predict tokens3. LLMs typically generate one token at a time. At each step, the LLM  produces a set of logits over the potential next tokens. These logits are then normalized to sum up to  using the softmax function, and one token is sampled randomly according to its probability:  $$\mathrm{Pr}(t o k_{i})={\frac{\exp(\mathrm{logit}(t o k_{i})/T)}{\sum_{i}\exp(\mathrm{logit}(t o k_{i})/T)}}$$  $$3.1$$  where  is token number  in the LLM's vocabulary4 of tokens and  ∈ (0, 1] is a parameter called *temperature* which controls the "softness" of the probability distribution5.  In our experiments we choose  = 1.0 for maximum response variation. 