I’m a product manager by trade and very much enjoy the process of working with human colleagues to bring an idea to market. Over the last few weeks, I ran an extensive impersonation exercise, getting ChatGPT, Bing, Bard and autonomous AI agents like AutoGPT, BabyAGI to be my colleagues to collaboratively design, build, and deploy a fully-functional web application that teaches prompt engineering to knowledge workers who are absolute beginners to ChatGPT.
🚀 This post wasn’t generated by ChatGPT so let’s get straight to the verdict:
- No, ChatGPT or autonomous AI agents will not eliminate entire software teams, but evidently make humans more productive at work. The time freed from getting AI to do some of the heavy-lifting will leave room for creative and critical-thinking for humans.
- Humans need to figure out the context(what) and intent(why) for the AI to perform well at work and that requires collaborating with other humans and AI. Additionally, curated and thoughtful decision-making at each step of product development will still require human input.
- Humans are better at putting together new perspectives or drawing new meaning from previous outcomes. It will take years before AI can independently conceptualize a groundbreaking new invention like an iPhone, all on its own.
What did I build?
- www.promptsimulator.com - A way for a knowledge worker with minimal to no knowledge of AI tools. They can practice asking effective questions to an AI Language Model to get the desired results, in the context of their job or profession aka prompt engineering for work.
- I did not hire a single human. My AI team consisted of: ChatGPT, Bing, Bard, AutoGPT and more, referred to as “AI” here. The impersonation exercise was simple: I asked AI to be a Product Manager, Software Engineer, Machine Learning Engineer, QA Engineer, UX Designer, Marketing Manager and various other job roles to collaboratively build a web application with me.
How long did it take me to build it?
- Less than 3 days to get a fully-functional prototype designed, built, and deployed with no human help. Additionally, I got about 25+ users to complete a user study that week.
- I’ve not written a line of code nor designed an app in years. This does not use a no-code platform rather frameworks that real-world software developers at tech companies use to build web applications.
Why did I build this application?
- I attended a ChatGPT event along with 115+ people and the survey said that over 62% of the attendees were new to ChatGPT. They were fascinated by prompt engineering and were willing to pay money to learn to use ChatGPT at work.
- While talking to some people at the event, I was surprised to discover the fundamental lack of understanding that achieving objectives with ChatGPT is much more efficient when using a clear, concise, and direct prompt.
🤖 Here’s a quick summary of what AI did well:
Time-consuming tasks: 4.5/5
- AI can generate a pitch deck, debug code and whip up a self-review for performance in a few seconds. If you’re a knowledge worker with no experience with NLP or LLMs, there will still be several use-cases in your domain where an AI can help.
Example: “My firm trains people to become prompt engineers. Build me a 30-60-90 day onboarding plan for a new product manager to my team”
Creation tasks: 4/5
- At a high level, AI can create boilerplate documents and code. AI can help eliminate writer’s block and brainstorm your idea with you. However, the context and intent must be created by the human.
Example: Here’s a prompt to get an AI to brainstorm with you: Generate 10 reasons on WHY would someone want to build a web application like the prompt simulator for beginner prompt engineers?
Administrative or time-consuming tasks: 4/5
- AI can eliminate hours of updating comments in code, reduce the learning curve duration for existing tools and chatbots can answer common customer complaints.
Example: Here’s a prompt for a junior designer that prefers a kinesthetic learning style - As a UX designer, how do I create and export a series of icons for a mobile app in Figma?
Analytical tasks: 4/5
- AI can analyze large datasets, identify patterns and anomalies efficiently. A lower score because AI is still not capable of understanding the context, given the data.
Example: Analyze this prompt: “Help a consultant predict market trends for an e-commerce product”. The output generated is accurate. The interpretation will depend on the context and timing of this prompt.
Problem-solving tasks: 3/5
- Even if it’s pre-2021 data, AI was able to suggest a solid HuggingFace model and the dataset to train. AI can suggest 3 sentence similarity algorithm alternatives, if one of them isn’t working as desired. But if you had a machine learning engineer or data scientist on your team, they’re capable of re-investigating mistakes they made on their own. Furthermore, AI is still struggling at math, which justifies deducting a point for problem-solving.
Example: “Identify a pre-trained model on HuggingFace to evaluate a sentence against a given context”
👩🎨 Here's where human intervention was necessary to steer AI towards the intended direction:
Collaborative tasks: 2/5
- Building a collaborative, diverse and honest culture at work will still be important to bring consensus among humans and AI agents.
Example: Enter this prompt: “Build an app to teach prompt engineering” to AgentGPT or AutoGPT, almost always the first step will be to create a user login feature. Although the foundations for an app are similar, a skilled human software architect or a tech lead will not blindly follow a template. There will be a thorough evaluation of each component against time, stage of product development, business logic and more. This will require hours of collaboration with several stakeholders.
Strategic tasks: 1/5
- When a CEO hires a senior leader to set a strategic decision for a product, part of the responsibility is shared by the human leader. Would the CEO hold an Autonomous AI sales agent responsible for not hitting quarterly goals? Would the AI be held responsible for publishing an incorrect statement in the blog post? Ultimately, strategic decisions require human judgement with deep knowledge of context and ethics.
Example: “My company has $100K in funding, what should we do next?” will yield a variety of tried and tested strategies and this can be enhanced with the right prompt that includes context, constraints and trends. But in a culture where people are held responsible for their impact to society, strategic business decisions will squarely fall on a human.
Creative and Innovative tasks: 1/5
- Data is fascinating but also limited. AI can expedite learning, execution and even scale to millions of users. However, we have not yet reached the point where AI is capable of conceptualizing and building foundational platforms, such as Figma, Vercel, OpenAI, or the next iPhone. While AI can assist in copying or optimizing these platforms, a unique creation still requires human expertise and creativity.
It was fascinating to witness an AI Product Manager convince me of a product direction and talk through the trade-offs, given the context and metrics but the result wasn’t anything revolutionary. Even with a Jailbreak(DAN) prompt, the ‘new feature list’ suggested by DAN and ChatGPT was pretty standard for any software application.
Social tasks: 1/5
- While building a product is easy, achieving product-market fit or successfully marketing the product to real users necessitates grasping the ever-changing patterns of consumer behavior. Even though there is a rise in synthetic data and AI imagined a few user persona’s, the honest feedback from a real user did not match the fake user that was created by an AI.
I was fortunate to get a couple of users to test the initial prototype for 20+ minutes and answer 5 questions as part of a user study. While the input that AI gave me was useful, it was similar to a product review as opposed to a brutal and honest feedback from a real user.
In conclusion, the only way to outshine AI at work is to be more human. Domain knowledge has become accessible at a rapid pace but cultivating empathy, creativity, critical thinking, compassion and social intelligence will take a significant amount of time. A collaborative work environment that leverages human creativity and contextual understanding along with AI's information retrieval and processing capabilities will drive the next breakthrough innovation.