AI has changed software development quickly.
Tools such as ChatGPT, GitHub Copilot, Cursor, Lovable and Replit can turn an idea into a working prototype in a fraction of the time it once took. You can describe what you want, answer a few questions and watch an application appear. That is genuinely useful. We use AI tools at Evergreen too.
But there is an important difference between using AI to help an experienced developer and asking AI to act as the developer. The risk is not the technology. The risk is putting software into use when nobody qualified has reviewed what sits behind the screen. A polished demonstration can create a dangerous sense of confidence. The buttons work. The information saves. The reports appear. Everything looks finished. That does not tell you whether the software is secure, reliable or capable of supporting a real business.
Working software is not necessarily safe software
AI coding tools are very good at producing code that looks convincing. They are less reliable at understanding the complete system in which that code will operate. This matters because many security problems do not stop an application from working.
A login page can accept the correct password while also containing a flaw that lets an attacker bypass it. A customer portal can display the right records while failing to check whether one customer should be allowed to see another customer’s data. The danger is hidden behind a system that appears to be doing its job.
Research published by Veracode found that only 55% of AI-generated code passed its security tests. The study covered more than 100 large language models, 80 coding tasks and four common types of vulnerability. Some of the individual results were particularly poor. AI models generated insecure code in 86% of the tests involving cross-site scripting and 88% of those involving log injection. These are known security problems. They are not obscure theoretical risks.
An experienced developer knows to question how information enters a system, where it travels and who is allowed to access it. AI usually sees the immediate request. It does not always understand the wider security context. That is how your customer data ends up exposed even though the application seemed to be working perfectly.
AI can confidently invent things
AI tools sometimes recommend software libraries or packages that do not exist. The name sounds plausible. The instructions look genuine. A non-technical user has no obvious reason to question it. This is known as a hallucinated dependency. It can become a serious supply-chain risk.
Attackers can create a malicious package using the invented name. The next person who follows the AI’s instructions may then install the attacker’s code into their application. The false package appears legitimate to both the developer and the automated build system.
Even genuine dependencies need checking.
They may be out of date, no longer maintained or affected by a known vulnerability. They may also bring dozens of additional packages into the project. Each one creates something else that must be monitored and updated. AI is usually focused on completing the task in front of it. It is not making a commercial judgement about whether you want your business to depend on that particular package for the next ten years.
Who owns the code?
There is another question that is easy to overlook. Where did the generated code come from?
AI models have been trained using very large volumes of existing material, including publicly available software. The output can sometimes closely resemble code from an open-source or proprietary project. That can create licensing and intellectual property problems.
Some open-source licences allow code to be used freely if certain conditions are met. Others require attribution, disclosure of changes or the release of related source code. If nobody checks, you may not know that those obligations exist.
One industry source we came across in researching this blog gave an example of AI-generated authentication code that contained comments and variable names from an existing MIT-licensed library. The business had used it in a way that breached the licence terms and later had to refactor the code.
That sort of discovery can delay an investment, acquisition or compliance audit. It may also force a costly rewrite at the worst possible moment. “I didn’t know the AI had copied it” is unlikely to be a satisfactory answer to a client, auditor or lawyer.
A demonstration is not a load test
A prototype might work very well with five users and a handful of records. What happens though when 500 people log in on Monday morning? What happens when the database contains a million records? What happens if a connected service responds slowly or stops responding altogether?
AI-generated applications often lack the less visible parts of good software engineering. Timeouts, rate limits, retries and safeguards against one failing service taking everything else down may simply be missing.
Performance problems can be equally difficult to spot.
A common example is one in which AI replaces one efficient database request with a separate request for every individual record. The result looks exactly the same during a small test. Under real traffic, the number of database requests multiplies and the application slows down or fails.
Then you have downtime, frustrated customers and an urgent repair bill.
You may also find yourself paying more for cloud hosting because inefficient software needs far more processing power than it should.
AI tends to test the “happy path”
Ask an AI tool to create a feature and it will often produce something that works when everything goes to plan. As we all know real users do not always follow the plan.
They leave fields blank. They paste unexpected characters into forms. Internet connections disappear halfway through a transaction. Payment services time out. Two people edit the same record at once.
Good testing deliberately looks for ways to break software.
AI-generated test suites can give false reassurance because they often test the same assumptions used to write the original code. I like to think of this as “asserting the same mistake twice”.
A test may confirm that the software behaves as its own code expects. That does not prove it behaves as your business needs.
Without proper error handling, one unexpected event can corrupt information, duplicate a payment or leave a process half-finished. And without a meaningful test suite, every later change risks breaking something that used to work.
Early shortcuts can become expensive later
AI is excellent at solving a tightly defined problem.
It is less capable of deciding how today’s feature should fit with the application you may need in three years.
That is an architectural decision. It affects how information is stored, how different parts of the system communicate and how easily the software can be changed. Poor architecture does not always hurt immediately. The first version might be fast and inexpensive. The cost arrives later.
A simple change takes weeks because the same business rule has been copied into several places. One small amendment breaks an unrelated process. A new developer is afraid to touch the code because there is no clear structure and no reliable test coverage.
This is often called technical debt. I prefer to describe it more plainly: you saved money at the start and are now paying for the same work again.
Rapid code generation can encourage people to skip modular design, documentation and other basic engineering practices. The result is software that costs more to maintain and takes longer to improve.
Your data may already have left the business
There are two separate data risks to consider here.
The first is how the finished application stores and processes personal information. AI can build a database and connect it to a form, but it does not automatically know your retention policy, access rules, contractual obligations or GDPR responsibilities.
The second is what was entered into the AI tool while the software was being created. People paste customer records, database structures, passwords, API keys and confidential business logic into AI systems because it helps produce a more relevant answer.
Depending on the tool and account settings, that information may be processed or stored outside your organisation.
The result can be a data protection problem before the application has even gone live.
It can also cause trouble during an ISO 27001 assessment, a customer security review, or a funding due diligence exercise. If you cannot explain where sensitive information went, which dependencies you used and who approved the code, you may fail the audit, even if no breach has yet occurred.
The biggest warning sign is “nobody understands it”
If your team has working software but nobody can explain how it works, you do not really control it. What happens when the AI tool cannot solve the next problem? What happens when an update breaks the application? What happens when the person who created the prototype leaves? Or if it conflicts with other software that you use.
Someone will eventually have to understand the code.
The longer that is delayed, the more expensive the job becomes. Developers may have to reverse-engineer the software before they can safely change it. In serious cases, repairing the existing system costs more than rebuilding the affected parts properly.
This all does not mean every AI-built application should be thrown away.
Some are sound prototypes with a good idea at their centre. They simply need an experienced team to check the foundations and complete the engineering work.
Start with a diagnosis
If you have built an application using AI, the first step should not always have to be automatically a rebuild.
We can review what you have and tell you what condition it is in.
At Evergreen, that can include a detailed review of the code and its architecture, security testing, penetration testing and realistic load testing. We can examine the third-party packages, check their licences and identify known vulnerabilities.
We can also create a proper test suite, improve error handling and document how the system works. Where the underlying idea is sound, but the implementation is weak, we can refactor the code and correct the architecture. If parts of the system cannot be made safe or maintainable, we will explain why and set out a practical route for rebuilding them.
The aim is to give you evidence before you make a larger investment.
You might discover that the prototype needs a few focused improvements. You might find that it is suitable as a demonstration but should never hold live customer information. Or you may learn that rebuilding now will cost far less than repairing it after launch.
All three answers are useful, even if not totally welcome.
Use AI, but keep an experienced person in the room
I remain positive about AI-assisted software development. It can help experienced developers work faster, explore options and reduce time spent on routine tasks.
But speed does not remove the need for judgement. Someone still needs to understand your business, challenge the assumptions, test the awkward cases and take responsibility for what goes live.
If you have used AI to build an application or prototype and want to know whether it is ready for real customers, talk to us. Evergreen can review what you have, explain the risks in plain English and help you take it through to secure, production-grade software. Book your free software review and let’s talk.