An application works fine with a hundred users. Then comes a campaign, a deadline or a live event, and suddenly there are thousands of users at the same moment: pages slow down, requests fail, and the problem is not a single function but the way the whole system is built.
Scalability is a system’s ability to handle more load (more users, more data, more requests) without losing performance or quality of service, ideally with costs that grow in proportion. In this article we explain how we design for it in our projects: the techniques we use, in what order, and why measuring comes before optimising.
Scaling up or scaling out
There are two ways to give a system more capacity. Scaling up (vertical scaling) means using a more powerful machine, with more CPU and more memory. It is simple and often the right first step, but it has physical and economic limits, and that machine remains a single point of failure.
Scaling out (horizontal scaling) means adding more copies (replicas) of the same service and sharing the work between them. It lets you grow much further and also improves reliability, but the application has to be designed to run as several copies at once. This is where architecture choices matter.
Stateless services, load balancing and caching
The first requirement for scaling out is stateless services: no replica should keep in memory data that later requests depend on, such as the user’s session or uploaded files. Sessions and files go into shared storage, so any copy can answer any request.
In front of the replicas sits a load balancer, which spreads requests across them and stops sending traffic to copies that don’t respond. Next to them there is almost always a cache, a fast memory for data that is read often and changed rarely: with a tool like Redis, an answer that would need a complex database query can be served in a fraction of the time.
The database, often the real bottleneck
Multiplying the application is relatively easy; the database is more delicate, because it is where all the copies meet. These are the main techniques:
- Indexes: the first thing to check and often the most effective; a slow query on a large table is usually missing a suitable index.
- Read replicas: copies of the database that serve reads only, while writes stay on the primary database.
- Partitioning: the largest tables are split by date, customer or region, so each query works on just part of the data.
- Managed databases: services such as Amazon RDS, or their equivalents on Microsoft Azure and Google Cloud, take care of backups, updates and replicas.
Queues, CDNs and background work
Not everything has to happen while the user waits. Heavy operations, such as converting a video, building a report or sending emails, go into a queue (for example Amazon SQS or Apache Kafka) and are handled by dedicated workers, which can be multiplied when the queue grows. The user gets an immediate confirmation and the system absorbs peaks without stalling.
For static content and media, a CDN (Content Delivery Network, a network of servers spread across many countries) brings files close to the people requesting them, taking load off the servers and cutting waiting times.
Artika, our theatre streaming platform, is built this way on AWS managed services. Videos and librettos go to Amazon S3 straight from the browser, through short-lived signed links, without passing through the server; AWS Elemental MediaConvert creates the HLS stream; Amazon Transcribe and Amazon Rekognition handle subtitles and moderation, and the API collects their results every minute; Amazon CloudFront delivers videos and thumbnails to viewers.
Containers, Kubernetes and managed cloud
Containers make horizontal scaling practical: each service becomes a standard package, with its code and everything it needs, that starts quickly and runs the same everywhere. Kubernetes, the most widely used open source platform for orchestrating containers, can add or remove replicas automatically based on metrics such as CPU usage (autoscaling) and restart the ones that get stuck.
You don’t always need to run Kubernetes yourself. Managed services from AWS, Microsoft Azure and Google Cloud (containers, databases, queues, CDNs) hand part of the operational work to the provider. In 2VitaBCognitive for Atlantia, for example, the 6 microservices run in Docker containers on Amazon ECS, and content reaches users from Amazon S3 through CloudFront.
Measure before optimising, and keep costs in check
Scalability is not guesswork: it is measured. Before changing the architecture we run load tests that simulate expected and peak traffic, and we collect metrics on response times, errors, resource usage and queue length. Often there is a single bottleneck, and it is where nobody expected it: a query, a call to an external service, a file served by the server instead of the CDN.
- Clear targets: for example, the time within which the vast majority of requests must be answered.
- Repeatable load tests, run again after every significant change.
- Production monitoring, with alerts that fire before the problem reaches users.
- Cost control: limits on autoscaling, resources switched off when not needed, budgets and alerts on cloud spending.
Unlimited autoscaling solves traffic peaks, but it can turn them into an unexpected bill. That is why, for each component, we agree with the client how far it should be able to grow and how much they are willing to spend.
If your application is growing, or needs to get ready for it, we can start by measuring where it stands today and point you to the changes with the best balance of cost and benefit. Contact us to talk it over.
