Mergen Demo Tech Neon 37/60
Skip to content
AISoftware

We pitted AI coding assistants against each other on the same project

We used different AI coding assistants to build a web app from scratch. We share the results for speed, accuracy and security in the video.

SLSophie LaurentArts & Entertainment Editor
8 min13.6B0
YouTubeYouTube content loads when you press play
Table of contents
  1. The test project and the rules
  2. Speed and code quality
  3. Watch out for security flaws
  4. Key takeaways

AI coding assistants have become an integral part of developers' daily workflow. In this test, we built a small web app from scratch with each assistant, following the same steps. All of them produced fast and accurate results for simple functions, but clear differences emerged in complex business logic and debugging. Noticing that some suggestions contained insecure code showed that human oversight is still essential. Software developer Ryan Douglas says the healthiest approach is to treat the assistants like an intern and review every line.

The test project and the rules

Our test project was a small web app with user login, a task list and a simple reporting screen. We gave every assistant the same written task description and started each one in the same empty folder. We followed a twenty-two-step plan in order, running the code and recording the result at the end of each step. A reference version we wrote by hand served as the yardstick for comparison.

Our rules were simple: we could fix the assistant's code ourselves only if there was a build error; for logic errors, we had to ask the assistant. We measured the time taken, the number of fixes requested and the number of working features at the end. Software developer Ryan Douglas acted as referee, reviewing the code and scoring it for security and readability. That way, the results didn't hinge on a single person's opinion, and the criteria were open to everyone from the start.

The test project and the rules — We pitted AI coding assistants against each other on the same project

The assistant writes fast, but the responsibility still lies with the developer.

Sophie Laurent

Speed and code quality

For simple functions such as data validation, date formatting or sorting a list, all of the assistants wrote correct code in seconds. The user interface skeleton was similarly quick. The differences emerged as the business logic got more complex: on queries joining multiple tables, two assistants produced the wrong result on the first try. The gap became more pronounced as the project grew.

During debugging, the assistants that took in the full error message together with the relevant code snippet performed best. On code quality, we saw differences in variable naming and comments; one assistant produced needlessly long, repetitive code, while another wrote short but hard-to-follow solutions. On average, the assistants let us finish the project 40 percent faster than the reference version we wrote by hand.

Watch out for security flaws

Our most striking finding came on the security side. Two assistants suggested code that inserted user input into a query without validating it, opening the door to a classic database vulnerability. Another suggestion stored passwords in plain text. These are bugs that are hard to spot at first glance because the code appears to work. A careful eye can catch them, but someone in a hurry can easily miss them.

Topic Software
Highlights AI, comparison, review
Reading time 8 minutes
Written by Sophie Laurent

Key takeaways

  • Different coding assistants were tested with the same task and the same steps.
  • All are fast at simple tasks; differences are clear in complex logic.
  • Some suggestions contained security flaws, so human review is essential.

Ryan Douglas's advice is clear: treat the assistant like a capable intern, read every line it produces, run security scans automatically and keep your tests independent of the code the assistant writes. It's also important not to paste secret keys or customer data into the assistant. In short, the tool saves time, but it doesn't take over responsibility; the final call and oversight still rest with the developer.

Share this post

What do you think about this post?

SL
Arts & Entertainment Editor

Sophie Laurent

A former film, music and performing arts reporter, Sophie Laurent covers festivals and exhibitions alongside the colorful world of entertainment news. In her interviews, she likes to put the story the artist is telling front and center, rather than the artist.

Leave a comment

Share your thoughts

Your email address will not be published. Respectful, on-topic comments are approved.

Neon Pixel

Welcome back

Log in to your account; your saved posts and comments are waiting for you.