AI projects vs traditional IT projects: key challenges
for Business Analysts and Project Managers

This blog article follows the previous  one to look into the depths of transformations to expect in AI projects. Here we will cover the most critical aspects of project inception, development, pitfalls and swamps - just to remind and to be careful while navigating the difficult road to AI project success.

 

Let’s start with the ordinary financial ratio – Return on Investment (ROI) into AI project. You may ask me why? I have three reasons for that:

 

• Having many years of experience as a Financial Director of a large corporation, I must say that this ratio is one of the most important for making the right decision about whether to start/accept a particular project or reject it;
• As for the AI projects – this ratio is undervalued / misunderstood all the time when the Big Beautiful AI Project is being considered;
• Far too often promising AI systems are not worth the time and resources to be spent, given the cost, complexity, and difficulty of implementing in comparison to the benefits expected.

 

To put it simply – the number of failed AI projects is times bigger than the number of successful ones as of today. Due to what – misaligned ROI? Any more reasons? Yes, agree – calculation of ROI under uncertainty is not an easy task. Very often companies started AI projects without proper assessment whether the project implementation will assure ANY positive return. So, how to calculate ROI under uncertainty in practice?

 

There are several ways to forecast, measure (and manage) return on investment, not just by dividing the expected return by the cost of the financial resources to be invested. Let me remind you a famous quotation of Management Guru Peter Drucker: “you can't manage what you don't measure”, so the problem is in measuring of return (benefits) from the project to launch. If you think about benefits, take into account non-financial issues, like accelerating main production, increasing reliability and safety, improving quality control, which in turn generates extra income. Enhancing technology (data processing), reducing the number of errors may bring you a lot of intangible benefits / assets, worth every dollar spent on improvements. What you need to do is actually plan your activities ahead of time and thus manage that ROI.

 

The next point to address is the reality of expectations. I mean the gap between what you promise and what you can actually deliver. I don’t stress on the level of human decency of those people who offer the AI project. At the kick-off meeting the whole set of stakeholders usually vote unanimously to accept the project. Everyone is happy that in a few months they will have a think machine, which is times faster, works perfectly well and allows to cut wages of a few very expensive analysts. What happens next? Weeks turn into months, months turn into quarters, and what's available right now? A smart engine that produces almost the same results as those analysts, sometimes unexpected and hard to explain.

 

Think about IBM’s Watson revolutionary AI for cancer diagnosis and treatment recommendations (hospitals quietly scaled back or discontinued its use), or Amazon’s internal AI system to screen job applicants (shut down due to discrimination against women), or Tesla’s FSD (Full Self Driving) system (still ongoing & not yet fully autonomous as of mid-2025). Inadequate or non-representative training data in critical domains, algorithmic decisions affecting lives, AI inherited bias from historical data, overpromising AI capabilities before technological maturity often endangers public trust—and lives. 

 

So, what was wrong with all these mentioned above projects?

 

There is no simple answer, but I’ve got a few thoughts. Probably, something about core of AI.  All AI projects face similar specific challenges compared to traditional IT projects. They indeed have unique complexities, such as:

 

• Data-Driven Nature: Unlike traditional software (rule-based, deterministic), AI models learn patterns and behaviors from data to make predictions or classifications. Thus:

 

• Data is the foundation of any AI system. If data is poor, insufficient, or irrelevant, even the most advanced AI architectures will fail to deliver meaningful results. Models may learn spurious patterns from noisy/inaccurate data.
• Performance of AI system, fairness, and trustworthiness, the whole AI project success depend heavily and directly on the quality (completeness, accuracy, consistency, validity and timeliness), quantity and relevance (related to the problem, collected from the right context and period) of data used for model training and evaluation. Inconsistent data confuses models. It is not just coding known functionality of features in traditional IT projects.

 

2. Iterative and Experimental: Unlike deterministic systems, AI development is inherently iterative, often unpredictable due to data dependency, uncertainty in achievable metrics, and the need for experimentation to find optimal models. Thus, it differs fundamentally from standard Agile delivery of deterministic software features.

 

3. Model Uncertainty: Unlike traditional systems with fixed rules, AI models:

 

• Make probabilistic decisions (outputs vary even for similar inputs near decision boundaries).
• Depend heavily on training data distribution: if production data shifts from training data, model performance degrades.
• Require continuous monitoring and recalibration.

 

More than that, AI models may behave inconsistently over time (e.g., model drift), which leads to:

 

• decreased accuracy or reliability of predictions,
• silent failures (the system continues producing predictions with high confidence but poor correctness) or even
• misaligned business decisions based on outdated model assumptions.

 

4. Cross-Functional Requirements: AI projects require close collaboration across data science, business, and IT — with different vocabularies and mindsets. Here is the point where Business Analyst could demonstrate creativity in collaboration with teams and excellence in communication with stakeholders!

 

5. Specific AI Testing: You can’t test an AI system like software; its behavior depends on evolving inputs and data. That is why:

 

• AI systems require new dimensions of testing: explainability, fairness, robustness to adversarial input, and continual learning validation.
• Testing an AI system is not a one-time effort. It needs ongoing monitoring and validation as data and environments evolve.
• Traditional QA (Quality Assurance) engineers need to upskill in data science concepts to be effective in AI QA roles.
• AI testing is highly context-dependent, requiring domain knowledge, data understanding, and model behavior insight.

 

That is why simple (from the first sight) projects become very complicated in a few months after the kick-off meeting. They repeat the same developmental pattern of evolving: overpromise at the start, underdelivering at the middle stage of project development (executing, monitoring & control), project failure as a result.

 

So, what AI project needs to become successful?

 

• Status of Readiness to start AI project;
• The right set of skills for AI project managers;
• Structured approach to AI project implementation.

 

That will set your organization up for success in AI project management. It’s essential — Project Manager in AI project is supposed not just to launch the model, but to guide it through the real-world pitfalls, swamps and continuous integration/testing for AI.

 

We will further discuss this hot topic in the next publication of GrafFIN’s blog.