When Your Teammate Is a Bot: What 2,310 People Taught Us About Working With AI

Artificial intelligence isn’t just a tool you ask for help. It’s becoming a teammate. But what actually changes when your collaborator is an AI agent that can chat, edit your work, and create images alongside you in real time? 

New research from MIT, using a novel platform called MindMeld, dives deep into this question. In a large-scale experiment involving over 2,300 participants, researchers put humans into teams, some with other humans, some with AI agents, to create marketing ads. The findings offer a fascinating glimpse into the future of human-AI collaboration, revealing surprising shifts in teamwork, productivity, and even the subtle art of personality dynamics. 

Want a quick and easy overview of the MIT paper? View the infographic, created with my experimental AI infographic generator.

Imagine a teammate who cuts straight to the chase, handles the grunt work, and boosts your individual output. That’s essentially what this study found with AI collaborators: 

• Communication Explosion (but different): Human-AI teams sent a whopping 137% more messages than human-human teams. But this wasn’t more chitchat; it was more task-oriented communication. Humans with AI focused 23% more on content generation messaging (e.g., suggestions, instructions) and 20% less on direct text editing. They also sent 23% fewer social messages, allowing them to bypass much of the “social coordination costs” typical in human interactions. 

• Productivity Soars for Individuals: While both team types produced a similar number of ads, individuals in human-AI teams were significantly more productive, submitting 60% more ads per person. This boost was particularly pronounced for lower-performing users, suggesting AI can level the playing field. The AI agents took on many of the direct editing tasks, letting humans focus on guiding the creative process. It’s like switching from being the car’s driver to being the navigator with a very capable autopilot. 

• A Multimodal Trade-off: Here’s where it gets interesting: 
  
  • Text Quality Wins: Ads created by human-AI teams were rated as having significantly higher text quality by human evaluators (+0.324 on a Likert scale). AI (specifically GPT-4o) excels at crafting compelling copy. 
     
  • Image Quality Loses: Conversely, ads from human-AI teams had lower image quality (-0.134 on a Likert scale). This highlights a current limitation: general-purpose AI models, while multimodal, aren’t yet as fine-tuned for visual aesthetics as they are for language.  In short, AI shines on text today; humans still have the edge on visuals—unless you bring in better visual tooling. 
  • Real-World Validation: The ads were then put to the test in a field experiment on X (formerly Twitter) with nearly 5 million impressions. The overall performance (Click-Through Rate and Cost Per Click) of ads created by human-AI teams was like those made by human-human teams. Crucially, higher text quality predicted higher CTR and longer view duration, while higher image quality predicted lower CPC. This confirms that the AI’s text strengths translated to real-world impact. 

Perhaps the most intriguing discovery is how AI’s “personality” can complement, or clash with human traits – just like with human teams, the “fit” between human and AI personalities significantly impacted collaboration. The researchers used prompt engineering to give AI agents different Big Five personality traits (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism).  Here’s what they found: 

• Complementary Traits Boost Image Quality: A conscientious human paired with an open AI agent led to significantly improved image quality. It seems the human’s diligent focus combined with the AI’s imaginative approach works wonders for visuals. 
   
• Mismatch Can Harm Output: On the flip side, an extraverted human paired with a conscientious AI agent actually reduced the quality of text, images, and click performance. This counterintuitive result highlights that simply having an AI isn’t enough; the right AI for the right human is key. 
   
• Niche Benefits: An agreeable AI could reduce advertising costs (cost per click), but this benefit was reversed for extraverted humans. And in another surprising twist, a neurotic human collaborating with a neurotic AI actually saw an increase in click-through rates (CTR)! This suggests that in some contexts, two “neurotic” partners might drive each other to optimize performance through meticulous iteration. 

• The Experiment: Over 2,300 U.S.-based participants were randomly assigned to work with either another human or an AI agent on a real-time platform called MindMeld. They had 40 minutes to create as many ads as possible for a real organization, using chat, editing tools, and AI-powered image generation. The AI agents weren’t just passive tools—they could actively edit text, select images, generate new visuals, and communicate strategies, mimicking a real collaborative partnership. 
   
• Massive Data: The study tracked 183,691 messages, 1.96 million text edits, and over 10,000 AI-generated images, all meticulously logged for analysis. 
   
• Real-World Test: After the lab experiment, a sample of 2,000 ads was run as actual campaigns on a major social media platform, measuring clicks, costs, and how long people engaged with the ads. 
   
• Personality Matchmaking: Both humans and AIs took “personality tests,” and the AI’s style was tweaked to see how different combinations affected teamwork and results. The AI agents weren’t all the same. Researchers randomly assigned them different “personalities” based on the Big Five personality traits (openness, conscientiousness, extraversion, agreeableness, and neuroticism). Some AIs were prompted to be highly open to new ideas, others highly conscientious, and so on. 

Why this approach? By combining controlled experiments with real-world field tests, the researchers could see not just what should happen in theory, but what actually happens when AI and humans team up on creative work. 

While robust, the study isn’t without its caveats:

• The task was ad creation; results may differ in coding, analytics, or other domains. 
• Collaborations lasted 40 minutes; longer-term trust and learning effects weren’t measured. 
• Image generation used a single image model; more specialized visual tools could change outcomes. 
• Some labeling and AI evaluations used models from the same vendor as the AI teammate, which could introduce bias. 
• The ad platform used auto-bidding, so field results are conditional on how the platform optimizes. 

The study reveals that AI agents are already sophisticated enough to serve as true collaborators, not just assistants. They change how we communicate (more task-focused, less social), how we work (more instructing, less hands-on editing), and what we produce (better text, weaker visuals). 

As AI continues to evolve, the art of successful collaboration will increasingly involve understanding and harnessing the unique “personalities” and capabilities of our intelligent digital partners. 

Would you prefer an AI colleague who matches your personality or one who complements it? And what aspects of human collaboration do you think we should fight to preserve as AI partners become more common?