Managing performance for small and middle-sized web application

Sometimes in software development, it is a little challenging to achieve excellent performance from your application.

It doesn’t matter if you work on an internal product or an application for your client. It would help if you considered how to manage the performance of the overall system. So you probably need some process to achieve it.
It is easy in the heat of rapid development to forget that performance is a critical part of your product. In addition, on the same side, it is tough to get it right.
Unfortunately, performance is truly a “hidden feature”. You can’t easily spot the bottleneck on a single PC with a few records in your development database.

This blog post is about web development. However, I believe you can use the same concept and advice for desktop or backend development as well. I would rather talk about the management of performance than about the performance in detail: such as the speed of response or throughput. It would take another post, and also there are many excellent resources on the Internet.

I think about performance before I start a new project or module.
In general, it is about balancing the effort and the budget. Therefore, you need to know what the clients’ requirements are and after that, you can make some estimation of the cost to achieve it.

You want to find the overall plan and the client’s vision. It is hard to go without knowing where you want to get. It is essential to find out customers’ expectations and their needs. Additionally, please bear in mind, these can be two very different goals.

Expectations:
Every client wants to have a performant application. However, it can be pretty hard to define what the performant application is. It is usually a little vague idea.

Needs:
Every application usually has a business purpose. Moreover, often, it is different from your clients’ expectation! You have to find their needs and, if it is necessary, give a reality check to the client’s expectations.

To find out the needs and to perform a reality check, I have a preparation phase.

Preparation phase:

First of all, I want to learn specific things about the usage of the new application or module. It is a crucial part of my analyzing software process.

I have great experience in using checklists whether I can. So obviously, I have a one for the performance of an application.

My checklist for almost all of my project:
- how many users will use the application/module
- how critical software it will be
- the budget of the client
- where a page can be slow and what its consequences
- general information about where the application will run (cloud, web hosting)
- find easy-to-spot tasks which can be generally slow. For example, generating many pdf files, processing complicated mathematical equations.

Note:
In this phase, I don’t take into account the final detailed production server configuration. Usually, the production server configuration is much more powerful than your computer for development. The problem can be on the client-side. I work on a web application, so I have to be aware that people using my web application might not have such a performant computer like me. I don’t have any general advice for you. You need to know your tech stack and its limitations and performance profile.

After I have found the answers to the questions in my checklist, I need to identify the critical place of my application and to classify these places into 2 or 3 categories representing the estimated performance profile.
For a medium-size project, I usually have only two:
1) Standard pages and actions — should be as fast as possible
2) Slow ones which need a progress bar indicator — slower with visualization

Maybe for some more extensive project, it is good to have a more precise setting like:
1) Normal pages — time under 1s
1) Slower — under 5s
1) Demanding pages — over 5s with a progress bar.

But I tend to have only two main types — the standard and the slow ones with the mentioned progress bar.

Realization phase

In the heat of the building of my software, it is not common for me to think about speed. It is just something I have at the back of my mind, but my priorities are different. I try to deliver a working solution, not the perfect one.

I try to anticipate where the performance problem can occur, but it is mostly more of a gut feeling than a focused work.

Except for the places which I marked as problematic during the preparation phase. I stick with the famous rule “Premature optimization is the root of all evil.”

Testing phase
In this phase, my colleagues or I test the software manually. We run a functionality test and check if our application sticks to the client’s requirements. The most visible performance problems can be found there.

In the realization phase, I try to prevent the situation mentioned above, where some super-speed pages get slower after the first few thousand records in the production environment. However, it is tough. So I am honest with my client. Together we prepare a pilot phase where some of my client’s users or their employees are willing to participate in pilot testing.

Pilot phase:
With respect to quality, the pilot phase is the most important to fix significant performance issues. Real data is steadily flowing from well-chosen users while bringing us valuable insights into the performance profile of our application. Users also report any slow pages or actions to my client or me. In the pilot phase, you usually discover many performance issues. I would recommend to let users run the application on a slower mobile internet connection.

Note: In the pilot phase, I heavily depend on “system monitoring/observability”. Later, I will explain what it is.

After the pilot phase, it is time to go public to all users.

Production phase and performance regression
In the production phase, some performance problems are likely to occur as well. It is natural. I know we all hate regression because the discovery of bugs(in this case slow pages) in working software are so annoying. However, we are all affected! There is a simple rule of thumb. The longer and more detailed pilot phase, fewer bugs and performance regression occurs in the production phase. However, of course, like everything, it depends on a budget of the client.

Now I would like to discuss one essential tool to manage the speed of the system.

Performance monitoring or observability:
The problem with performance is that it is not so easy to spot the problem. You can have some testing user account with a small set of data, and your application runs correctly. We need a way how to observe internal state in production server with real data.

I get used to handling it by tools named monitoring and observability.

Performance monitoring is a set of processes and tools which can determine the condition of our application.
It is a kind of pinging the website to check if it lives and maybe measure the response time. It can give you overall information about the liveness of the application.

The term observability comes from the world of engineering and control theory.
As the definition [https://en.wikipedia.org/wiki/Observability] states, it’s a measure of how well internal states of a system can be inferred from knowledge of its external outputs.

Observability and monitoring are different concepts. Considering monitoring as just superfluous techniques, I prefer observability more because it gives you more insight. It is good, but we can do better.
Observability brings you more information about what it is going on under the hood.

Example of my observability:
I have a custom performance event logger. It means that on some critical places of my application, I publish specific events describing the current state or measuring particular actions in my system.

Then I have a web page where I can see the performance of the whole system using the apply some maths on my logged events.
I visualize all the essential information about the system: like memory consumption, CPU, disk, and also other internal statistics.

For example

- Average of SQL query
- Average of SQL command
- Number of exceptions
- Slow queries
- Moreover, any other information you might find interesting.

These numbers give me an overall view of my application, and they can help me identify slow queries.

Additionally, it is good to publish some of the information to your client in order to be more transparent. It can be an internal thing, even misleading or confusing so you should create a good report for your clients with some general statistics and don’t forget to teach your client what these numbers mean.

So this is my summary of how I handle performance.
Let’s recap:

- In the first stage of development, it is not so useful to think about performance.
- The testing and pilot phase are your best friend to tackle the slowness of your application. Many bugs can be found here.
- Monitoring or observability are great tools, and definitely, you should have one.