Isaac Newton once quoted “If I have seen further, it is by standing on the shoulders of giants”. LLM and agentic coding technologies are the giants that are now roaming in our software engineering domain. Use of LLMs and agentic coding tools to improve software engineering is a strongly debated subject with plenty of strong arguments on both sides. For curious readers, there are plenty of examples of successes, failures, and mixed results are available in the blogs and online discussions.

Our engineering team has been using agentic coding tools for the last six months to streamline the day-to-day software engineering tasks, e.g., code review, root-cause analysis of production incidents, writing unit tests, onboarding new team members. We have improved the sprint velocity, written better code with higher code coverage, and were able to solve production incidents quickly. Here is a collection of the tips we have found useful and use cases that showcase the usefulness of agentic coding in writing and maintaining software. The Claude code commands used in these use cases can be found in this github repo.

Tips to help with Agentic Coding

• If you do not have a good grasp of the language and the framework in use, you will likely not be effective or efficient with the agentic coding tools.

• Always review all changes made by the agent - you are the owner.

• Direct the agent with very specific and detail instructions/prompts

• Create a feedback loop for the agentic tool: Analyze → make changes → write tests to verify the changes.

• For complex tasks and changes, ask the agent to think hard and justify its solution.

• Watch out for the cost (use a cheaper model) for long-running tasks

Use case 1: Reviewing Pull Requests

def convert_data(df: DataFrame) - > bytes:
    buffer = bytearray(b"")
    for row in df._data:
      row_dump = dumps(dict(zip(self._columns, row)))
      buffer.extend(f "{row_dump}\n".encode("utf-8"))
    return gzip.compress(bytes(buffer))

session = aioboto3.Session()

async with session.client(
    "s3",
    region_name = "ap-east-1",
    config = get_aioboto_config(),
  ) as s3_client:
  await s3_client.put_object(
    Body = convert_data(self),
    Bucket = bucket,
    Key = key,
  )

Agent recommendations

• Reuse the session and possibly use a connection pooling approach

• Improve the performance of convert_data function:

  • eliminate 1) multiple memory allocations, 2) encoding, 3) buffer extension and manipulations

  • add a shortcircuit

   if not df._data:
       return gzip.compress(b"")
   columns = df._columns
   json_lines = [dumps(dict(zip(columns, row))) for row in df._data]
   res = "\n".join(json_lines) + "\n"
   return gzip.compress(res.encode("utf-8"))

Use case 2: Root-cause analysis of a production incident

We have been experiencing some type of memory leak in our heavily-used Java webservice running on EKS in AWS. The symptoms of the leak were java connection timeouts to the storage services. The web service is designed using Java Spring and ORM. By enabling spring.datasource.hikari.leak-detection-threshold in the deployment, we were able to collect a detailed stack trace of the error in the production.

Exceptions

LOGGING_FORMAT: json
   caller_class_name: com.zaxxer.hikari.pool.ProxyLeakTask
   caller_file_name: ProxyLeakTask.java
   caller_line_number: 84
   caller_method_name: run
   level: WARN
   level_value: 30000
   logger_name: com.zaxxer.hikari.pool.ProxyLeakTask
   message: Connection leak detection triggered for org.postgresql.jdbc.PgConnection@123456c on thread http-nio-8080-exec-5, stack trace follows
   stack_trace: java.lang.Exception: Apparent connection leak detected
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
	at org.hibernate.engine.jdbc.connections.internal.ConnectionProviderImpl.getConnection(ConnectionProviderImpl.java:122)
	at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
...

Agent recommendations

Once the exception details and the stack trace is fed into the Claude code (Opus 4.1) fixed the issue:

• Identified the root cause of the issue (using transactions for S3 read/write operations)

• Performed the code change - using transactions only for DB access and removing it for S3 access and added unit tests to verify the changes

• Ran the unit tests and verified its change

Cost

Total cost:            $65.09
    Total duration (API):  25m 28.1s
    Total duration (wall): 17h 32m 53.9s
    Total code changes:    1012 lines added, 36 lines removed
Usage by model:
claude-3-5-haiku:  70.5k input, 3.8k output, 0 cache read, 0 cache write
claude-opus-4-1:  1.2k input, 48.6k output, 23.4m cache read, 1.3m cache write
claude-sonnet:  168 input, 7.7k output, 2.1m cache read, 164.0k cache write

Both sides have valid points, delivering features quickly seems to be a key component of capturing market share on the other hand, service outage causes the users real pain and translates into revenue loss, and in the worst case, loss of market share or user base.

Well, how about we hire an army of amazing engineers who write flawless code with 100% unit test coverage? - we can just push that code to production quickly, right?, Wrong. Testing (unit, manual, etc.) can only ensure the presence of bugs, not the absence

During a service incident, several engineers in one of my previous teams had to spend days debugging an issue caused by a single instance of case-sensitive string comparison.

Ring-based deployments

Over time, all software components increase its interdependencies with other components within the architectures. When we deploy a sizable change in a component or a new component, the probability of the introduction of bugs increases. These could be caused by logical errors, by faulty code, and or integration issues. Given this high probability, there needs to be a way to mitigate any negative impact on the customer bases while we deploy code to the production environment.

Ring-wise continuous deployment (CI) is one such method to mitigate the risk of customer impact while achieving high velocity of feature delivery.

• Divide all environments where the software component is available into several rings of availability.

• Starting from the developer's machine (Ring 0), to the final production environment (Ring 3). Continually deploy and test software features from one ring to the next until it is available to all users.

The exact implementation of the rings depends on the type of software, the team, and the target user base. A sample implementation:

• Ring 0 Developer's machine

• Ring 1 Staging environment

• Ring 2 Production preview or Beta Customers

• Ring 3 Production or Real Customers

The key point to remember is:

The infrastructure and the deployed code in a ring should closely (as closely as possible) represent the ring# after it

The underlying need of deploying software w/o causing disruptions in the production environment arises from the customer-first/user-first thinking. In that case, the tech, product, and operations teams all work towards achieving the optimal experience for their customers. The additional work for ring-wise deployment and testing is part of that cost.

Every few years, the technology innovations continue to change the way we write, use, and maintain software solutions. In the last decade, the rise of AI, cloud computing, and open source technology solutions completely transformed the information technology. Software can now be written with less effort, deployed without building any physical infrastructure, and delivered to millions of users with relatively lesser effort.

The ever changing technology landscape continually obligates the technology professionals, to keep up with the pace and modernize our existing software and services. An organization’s need to change its software and services could be driven by its business need improving economy, create new product, etc.. The need could also be a matter of survival for the organization as the underlying software or hardware technology ages toward sunset.

Migration is the process through which an organization updates the code, data, or infrastructure.

Migration could be as simple as upgrading the database to a more recent version or as complex as moving the infrastructure from on premise datacenter to public cloud such as AWS. For complex scenarios, completing the migration without having any negative impact to the ongoing business or operation is by no means, an easy affair. The cost and effort often puts enormous strain on the profitability of an organization, disrupts their daily operations, and in the worst case end could up destroying the organization. There are three main sources of the high risk and challenges that are associated with the migrations:

Techical Debt

Martin Fowler defines cruft as the “deficiencies in internal quality that make it harder than it would ideally be to modify and extend the system further“. Ward Cunningham describes Technical debt as a mindset of thinking cruft as a financial liability that needs to be paid over time. Examples of technical debt could be lack of unit tests, missing documentation, customized code that can only run on a specific piece of hardware, or a bloated database that have not had a cleaning done for a long time and cannot be backed up.

Case study: I have worked in a migration project where the technical debt was not calculated accurately ahead of time. This caused the project to take three times the time and cost and resulted in the technical lead leaving the organization.

The Three Cs - communication, collaboration, and coordination

Unless the engineering team is in a small flat organization, the software migration will often require support from other teams and needs to coordinate their efforts closely with those other teams. The priorities and the goals of each team is different, so, unless all the managers and upper management is fully focused on making the migration a success, the migration project is likely to fail or face significant hurdles.

Case study: I have worked in a migration project where the success of the migration project required a different team to run a re-indexing process to which they initially agreed but refused to do it at the end of the project because of the cost. This caused the migration project to almost fail and caused various career consequences for the members of the team who was in charge of the migration.

TCO - total cost of ownership

The total cost of ownership of completing a migration is notoriously difficult to measure. The technical, the product, and the business wings of any organization need to work hand in hand to achieve the common goal of that organization. Every stage of the migration process has associated cost such as time and resources and has opportunity cost. Since there are uncertainties that plague the planned work, the total cost cannot be accurately measured ahead of time.

Case study: I have worked in an on-prem to cloud migration project which was delayed by 1 year because of inaccurate accounting of the total work that needed to be completed. The project kept going for ever, the upper management could not get an approximate date of completion, and this migration delay caused several other projects to get delayed for unforeseen amount of time causing significant customer dissatisfaction and negative economic impact.

But, why do you, the reader would care about these nuances? Surely, you have a team of very strong engineers and run a tight ship. Hear me out, like the sailors sailing through uncharted seas, be afraid, be very afraid. You may end up paying more than you have bargained for. To be continued …

As the complexity of the software projects grow, so does the difficulty of testing those big software projects. With the growth of cloud technologies, the number of users who use a software now reaches hundreds of millions and even billions. When mistakes are made, the impact of that mistake now potentially be catastrophic. Here are a few software testing practices that I have found useful in assuring high QoS for the software and services.

Wait, AI could Test Everything!

It is true that AI agents have been making great progress in automating the process of writing and execution of software tests, it still has a long way to go. A lot of that has to do with pure combinatorics. Let’s assume we have have a program that has 10,000 functions, each of those functions has 5 input parameters and we use the AI agent to write unit tests that execute in 10 ms. The total time required to write unit tests would be 8 Hrs and that is just unit tests. Executing other types of tests would take even longer. So, sheer number and brute force execution will be too slow to be practically used in a CI/CD pipeline. We need engineers who can judiciously test the software without taking too much time1.

Types of Software Tests

Unit Test

Unit testing is the first line of defense against those creepy crawlies Once I did not add enough unit tests in my code, a senior developer asked me "How do you know that your code works?"When practicing TDD (test driven development), you would write a test for your new piece of code, run the test which will fail, and then write code until that failing test passes. The unit tests are usually white-box tests - tests that know how a unit/function operates. While building a car, a unit test would be measuring and verifying the current produced by the alternator.

Functional tests

The functional tests takes on the dependencies of external libraries and services to test the functionality of an entire computer program or module. While building a car, functional testing would be measuring the torque while running the engine with the input of gas and battery.Tools such as Sonarqube can help you run all unit and functional tests on demand or as part of the CI/CD pipeline.

Integration Tests

These are the heavyweights of the testing world. These are designed to test the software end-to-end with in a real-world environment. You can think of these as driving the automobile on a road to test how well it drives. In this test all components of the car are working in unison to produce the driving outcome. Integration tests are usually the most time-consuming tests to write and are the most difficult to maintain.

Performance Tests

So, the car drives fine in a plane obstacle-free road, but what about when the going gets tough? Performance tests are all about testing the program as a whole under load conditions. You can think of this as driving the car in with maximum load (passengers or goods).Performance tests are usually done after completing all other testing has been done and the infrastructure and code have reached a certain level of stability. Tools such as Jmeter are useful for running performance/load tests on APIs.

Monitoring

In cloud computing, the major components/modules of a piece of software is stored in the cloud and is updated frequently by a moderately large group of engineers. To ensure the QoS, the critical components of a software are now monitored continuously. The monitor is usually a program that measures the health of some service or executes a set of defined steps over and over. When the health metric of the service falls below threshold or one of those steps fail, the monitor alerts the supporting engineering team. In our car example, it would be like checking the battery power and the engine temperature while the car is being driven.Finding the right balance between effective monitoring and alerting noise will take continual testing and re-adjustment of alert threshold.

Optics

While monitors detect instantaneous failures issues, optics are a way of looking at the performance and history of the software in the real world over a longer term. Optics for a software may provide insights into its usage (growth of user base), performance (response time during peak and off-peak hours), etc. For our car example, optics would be like collecting and analyzing gas mileage data for the car over a month to verify its actual mileage. A successful optics infrastructure should contain a small set of basic reports and the ability to export collected data.

Testing in Production (TIP)

Replicating the scale and complexity of the production environment in a dev or testing environment is notoriously difficult. So, functional tests run in the dev or test environment may fail to surface the subtle bugs. To overcome this, engineers often isolate a smaller set of users within the production environment and test the software release there by running manual, automated tests and by reviewing the monitoring and optics.

Time-Tested Best Practices

• Unit tests should complete in milliseconds

• Code coverage should be fairly high >= 95% to be effective against regressions or bugs.

• Unit tests are the best tool to avoid regressions when refactoring your legacy code.

• When integrated with the build pipeline, unit tests are the best type of documentation for your code.

• Incorporating the unit tests in the CI/CD pipeline usually can be helpful in preventing regressions.

• Functional, Performance, and UI tests all have a significant cost of development and maintenance

• Manual testing still remains useful for catching complex software solutions and services

The fast continuous integration and deployment cycles adopted by the majority of the software organizations mean greater code churn - new features are being released and old features are being deprecated on a regular basis. Although there are usually multiple testing environments where engineers could run and test their code, they often cannot match the scale (# of users, requests, etc.) of the production environment. As a result, the true test of the new software occurs only when it is run in the production environment.

So, we need a mechanism to test new code and features in the production environment without impacting the user experience. We could first segment the users according to some criteria such as geolocation, type, etc. Once divided, we could choose one or more of those user segments to use the new app and test all functionalities of the new feature or app in the production environment. This mechanism is called feature flighting. A more complete discussion on feature flighting can be found here.

The implementation could be following. When flighting an API, we could add logic that checks the context (location id in this example) and then routes the code to legacy or new function according to that context.

Feature flighting Example

from fastapi import HTTPException

location_whitelist = ['x','y']

@app.get("/items/{location_id}")
def read_item(location_id: int):
    if location_id in location_whitelist:
        return new_read_item()
    else:
        return legacy_read_item()

def legacy_read_item():
    pass

def new_read_item():
    pass

The following feature tools are popular choices for flighting features in the production environment.

• Github scientist

• Unleash

Best Practices to Avoid Incidents

• Your feature flighting logic should use a whitelisting approach and route the code to the new feature if the context or the user is in the whitelist.

  • The default/legacy code path should continue to work if the flight check fails

• Always create tasks to remove the added flighting logic when all users have been migrated to the new feature and the testing is complete

Use case: Food Delivery App Migration using Geospatial Feature Flighting

In 2022, I was leading the engineering team of a tech startup in Bangladesh. One of the services offered by this startup was app-based food delivery services to ~ 10M customers in the city of Dhaka, Bangladesh. Customers would order food through its app and the delivery drivers would get the order, pickup the food from the restaurant and deliver it to the delivery address listed on the order for a fee. After rewriting the legacy food app, the management team of the startup wanted to migrate the users from the old app to the new app while maintaining a high QoS and avoiding service interruptions. The issue was, the refactored architecture, code, and storage systems of this food app could not be tested for a large scale user base and had the risk of encountering service disruptions and outages.

Problem

How can we choose a smaller set of users and delivery person to use our new version of the food delivery app so that the impact from any bugs remain isolated to that small set of users (w/o impacting the rest of the users). As the newly designed service gets stabilized, we would like to gradually onboard all users to the new app.

Solution: Strangler Pattern and Geographical Zones

The novel solution was to divide all users of the Dhaka city into several smaller subgroups based on their geographical locations (latitude, longitude): Uttara, Mirpur, etc. Then we could choose one or more of those smaller zones and onboard them to the new app one zone at a time. This gradual migration will allow us to test and stabilize the new app while avoiding any major service outages.

The solution steps were as follows:

• Continue running the legacy and the new app side by side in the production environment

  • Required us to onboard the support and delivery personnel to ensure they can use the new and legacy backend dashboards

• Design a geolocation webservice to map a user’s geolocation to one of the geographical zones

• Use feature flighting to route all network traffic from the user app (order, payment, etc.) to either the legacy app or the new app based on a user’s geographical zone

• Use strangler pattern to gradually transition the geographical zones from the legacy system to the new system

• When all users’ zones have been migrated to the new services, sunset the legacy services

End Result

Using the geographical feature flighting, our engineering team successfully migrated the users one zone at a time and stabilized the new app without any service outages.