Why Project Copilot Continues Using GPT-4o (for Now)

After evaluating multiple scenarios using OpenAI’s 01-preview model, we’ve decided to continue using GPT-4o for most content generation tasks. Why? Because GPT-4o combined with an agentic plan based on predefined rules has consistently delivered better results in these tasks. However, this is a moving target, as each new model release brings potential changes.

GPT4o

The Balance Between Intelligence and Rules

One of the key insights from our evaluation is that for some tasks, a certain level of intelligence suffices. Beyond that, more intelligence doesn’t necessarily add value. In fact, in some cases, predefined rules bring more value than relying solely on model intelligence. This realization has shaped our decision to use GPT-4o for high-intelligence tasks and GPT-4o-mini for tasks requiring less complexity. This approach ensures that our system is efficient and fits the task at hand.

Enhancing Flexibility in User Story Generation

A challenge we encountered was that some of our prompts were too rigid, often assuming that user stories would only be created for standard software projects with typical development teams (e.g., frontend and backend developers). This rigidity led to user stories that weren’t useful in certain scenarios. To address this, we’ve introduced more flexibility into our prompts, enabling them to better capture user intent—specifically how users want their user stories written. This shift allows us to generate more relevant and tailored content for diverse project requirements.

Improving Agentic Behavior

Our focus is also on enhancing agentic behavior. We are exploring ways to integrate other large language models (LLMs), including local models, to provide more flexibility. As part of this effort, we are evaluating IBM’s Bee Agent framework. This framework supports multiple models and offers a well-defined design that aligns with our goals. By migrating to this framework, we aim to further improve the agentic behavior of Project Copilot and deliver more value to end-users in the near future.

A Practical Example

To illustrate these concepts, we will soon share a comparison between Project Copilot’s current capabilities and outputs generated using OpenAI’s 01-preview model. This example will highlight the benefits of our approach and demonstrate the improvements we’ve made.

Prompt:

Create a backlog with the user stories to create a series of youtube video tutorials with an introduction to project copilot, an AI assistant for projects. Include an introduction video, a simple tutorial and an advanced tutorial. On each user story add a section with the title “Business Value” describing the business value of the user story:

Output Project Copilot:

(Only the user stories for advanced tutorial)

User Story 3: Advanced Tutorial for Project Copilot

Description: As a user, I want an advanced tutorial that covers in-depth features of Project Copilot so that I can maximize its capabilities.

Business Value: Providing advanced insights will empower users to leverage the full potential of Project Copilot, leading to better project outcomes.

Acceptance Criteria:

  • The tutorial covers advanced features of Project Copilot, including but not limited to AI-driven project management, advanced task automation, and integration with other tools.
  • The tutorial is structured in a clear, step-by-step format, making it easy to follow.
  • The tutorial includes practical examples and use cases to demonstrate the advanced features in action.
  • The tutorial is available in both video and written formats to cater to different learning preferences.
  • The tutorial is accessible from the Project Copilot help section and the official YouTube channel.

Technical Reference:

  • Content Creation (Content Creator)
    • Research and document the advanced features of Project Copilot.
    • Develop a detailed script for the tutorial, including practical examples and use cases.
    • Create a storyboard for the video tutorial.
  • Video Production (Video Producer)
    • Record the tutorial video based on the script and storyboard.
    • Edit the video to ensure clarity and engagement.
    • Add captions and annotations to highlight key points.
  • Documentation (Technical Writer)
    • Write a comprehensive written version of the tutorial.
    • Ensure the written tutorial is easy to follow and includes screenshots where necessary.
  • Integration (Web Developer)
    • Embed the video tutorial in the Project Copilot help section.
    • Ensure the tutorial is easily accessible and searchable.

Implementation Steps:

  • Content Creation:
    • Research and document advanced features (sequential)
    • Develop script and storyboard (sequential)
  • Video Production:
    • Record and edit video (sequential)
    • Add captions and annotations (sequential)
  • Documentation:
    • Write and format written tutorial (sequential)
  • Integration:
    • Embed video and written tutorial in help section (sequential)

Scenarios:

  • Happy Path:
    • User accesses the advanced tutorial from the help section and follows the video and written instructions to learn about advanced features.
  • Multiple Formats:
    • User prefers written format and follows the written tutorial with screenshots to understand advanced features.
  • Practical Examples:
    • User follows practical examples in the tutorial to apply advanced features to their own projects.
  • Accessibility:
    • User with hearing impairment uses captions and annotations in the video tutorial to understand the content.

Output ChatGpt using o1-preview:

(Only the user stories for advanced tutorial)

Advanced Tutorial

  • User Story: As an experienced user of Project Copilot, I want an advanced tutorial that delves into complex features and use cases, so that I can maximize my productivity. Business Value: Advanced tutorials keep experienced users engaged and help them extract more value from the product. This deepens user loyalty and can lead to upselling opportunities or premium subscriptions.
  • User Story: As a subject matter expert, I want to compile advanced tips and best practices for Project Copilot, so that I can educate users on how to leverage the tool fully. Business Value: Sharing expert knowledge positions the brand as a leader in the field and fosters a community of power users. This community can provide valuable feedback and advocate for the product, enhancing its reputation.
  • User Story: As a video creator, I want to structure the advanced tutorial logically with clear explanations of complex concepts, so that users can understand and apply them effectively. Business Value: Effective communication of advanced features ensures that users can adopt these capabilities, increasing the overall utility they derive from the product. This can differentiate Project Copilot from competitors and justify premium pricing.

Looking Ahead

We are already working on proof-of-concept versions of Project Copilot powered by the Bee Agent framework. These new developments promise even greater flexibility, improved agentic behavior, and the ability to harness the strengths of multiple LLMs to better serve our users.

Stay tuned for updates as we continue to refine and enhance Project Copilot’s capabilities!

Matías Molinas
CTO, Project Copilot