The Hidden Costs of AI Development Tools: What CTOs Need to Know

As a CTO who has spent decades in the trenches, I've seen countless "revolutionary" tools promise to transform how we build software. Today, AI coding tools like GitHub Copilot and Cursor are making similar promises.

Recently, I had an conversation with Joe Giglio on my Product Driven podcast about his extensive testing of these tools. What we discovered might surprise you.

The Allure vs. Reality

The marketing hype around AI coding tools is intense.

If you believe the headlines, we're all about to be replaced by AI. Microsoft, GitHub, and others are publishing carefully selected case studies showing dramatic productivity gains.

But here's the reality I've observed: Most developers spend only about 20% of their time actually writing code.

The average developer writes just six lines of code per day.

Why? Because the real work is in reading code, debugging, attending meetings, and understanding requirements.

The Integration Challenge

Joe's experience testing various AI coding tools revealed a critical insight: while these tools excel at basic tasks, they struggle with complex, real-world applications.

During our discussion, Joe shared his attempt to build a help desk system using AI tools.

"The basics work, but as the project becomes more complicated, it starts to go off the rails," Joe explained. He encountered issues with circular imports, server startup failures, and what he calls "hallucinated code" – where the AI generates code that looks good but doesn't actually work.

The Debugging Dilemma

One of the most concerning findings from our discussion was the maintenance burden these tools create.

Joe attempted to use AI to generate automated tests using Playwright and Selenium. The results were sobering.

The AI would create test scripts that looked perfect on the surface but were fundamentally flawed:

• Made up locators that didn't exist

• Failed to understand hidden elements

• Generated tests for deprecated code

• Created unrealistic test scenarios

The kicker? This was happening with code the AI itself had generated.

Imagine the complexity when dealing with legacy codebases.

The Real Cost of "AI-First" Development

A story that resonated with me was about a non-technical friend who tried using Cursor to build an application. Initially impressed, he quickly discovered that each AI-suggested fix broke three more things.

After three days, he gave up entirely.

This highlights a crucial point: AI coding tools might actually slow down development in complex projects. When you factor in:

• Time spent writing detailed prompts

• Debugging AI-generated code

• Fixing cascading issues

• Managing technical debt

The productivity gains can quickly turn into losses.

Looking Forward

Does this mean AI coding tools are worthless?

Absolutely not. I see them as productivity enhancers, not replacements.

In our conversation, Joe and I agreed that if these tools can make developers even 50% more productive – moving from six lines of code per day to nine – that's a significant win.

But CTOs need to be realistic about implementation. These tools work best when:

• Used by experienced developers who can spot issues

• Applied to well-defined, contained tasks

• Integrated into existing development workflows

• Supported by proper testing and review processes

The Bottom Line

The future of AI in software development is promising, but we're not at the point where it replaces human developers.

As I told Joe during our discussion, programming is essentially English – we're writing human-readable instructions. At what point is it easier to write the code than to write the prompt?

For CTOs and technical leaders, the message is clear: AI coding tools can be valuable additions to your development stack, but they're not magic bullets. The key is understanding their limitations and implementing them strategically.

Did someone forward this email to you? Sign up here!