DEV Community: Rizky Darmawan

How to Handle Time Zones and Sync Your Software on the server side using Go

Rizky Darmawan — Tue, 01 Oct 2024 02:08:22 +0000

When your application starts on a large scale, the increase in users will increase. What is very likely to happen is that the user's location is not only in the same area, it could be in another area that has a different time zone. So as a Backend developer, things related to handling time zone differences are very important to think about.

I recently came across an issue involving time zones. Let's be honest, dealing with dates and times is one of the most complicated areas a human being has to deal with. And this was an opportunity for me to learn how to properly handle dates and times on the server side.

In this article, I will share my experience on how I handle time zone differences on the server side as a Backend developer. Maybe if someone is willing to correct and provide additional input that would be valuable for me.

About Time Zone

Time zones are a standard time division system used throughout the world to organize and standardize the measurement of time. This concept emerged as a response to the need for global time coordination, especially along with developments in communications and transportation technology.

The basic principle of time zones is the division of the earth into 24 zones. Each time zone is generally one hour different from the zone next to it. The main reference for time zones is Greenwich Mean Time (GMT) or Coordinated Universal Time (UTC), which is at the zero degrees longitude line that passes through Greenwich, England.

Illustration by Hellerick from Wikimedia Commons

A small example is when the clock shows 12:00 noon in Jakarta, in New York the time shows 00:00 or midnight. This means that while Jakartans are enjoying lunch, New Yorkers are just starting their new day. From here you can certainly imagine the importance of handling time zones correctly in building an application.

Timezone handling on the server side

After we have seen the explanation above, now we will go into the points that can be done when our server application receives a request from a client that accesses our API to handle its time zone.

In this article, I will discuss several approaches to handling time zones on the server side. Here I will use code examples in the Golang (Go) language. Golang has a time package for working with time-related data which is considered quite complete. Here are some points that we will discuss:

How to save date to DB
Conversion of user's local time
Testing and Validation

How to save date to DB

The first thing we will discuss is which time we will save in the database, for example we have an ecommerce application that does flash sales, where our application is already on an international scale.

When a user processes a purchase transaction in America or if the user is in Indonesia, the user will send their different local time to the server. The question is, will our database store time data according to the user's local time? If the answer is yes, it is likely a complicated problem when we want to retrieve the data or for example the admin wants to do data processing, which user makes the earliest transaction.

To overcome this, the best practice is to store transaction times in the UTC (Coordinated Universal Time) time zone which is the primary time standard for clocks and time settings. Here is the application of the time to UTC.

package main

import (
    "fmt"
    "time"
)

func main() {

    now := time.Now()
    fmt.Printf("Local time: %s\n", now)

    nowInUTC := now.UTC()
    fmt.Printf("UTC time: %s\n", nowInUTC)

}

Let's see the meaning of the code above.

First, in the now := time.Now() code line, this line uses the Now() function from the time package to get the current time based on the system's local time zone. The result is stored in the current variable.

Then, in the nowInUTC := now.UTC() line, here we convert the local time (now) to UTC time using the UTC() method. The results are stored in a new variable nowInUTC and this time can be stored on the server, where it is hoped that developers can avoid ambiguity and errors that may arise due to time zone differences between servers and users in various regions with different time zones.

Here are the results if we run the code above:

But this is not always the best solution that you should use. There may be some points you can keep in mind in certain use cases, one of which is is it true that our users come from different time zones? If it's not possible, perhaps storing the time in UTC will add complexity to your code.

Change time to user locale

At the discussion point above, we have agreed to store user time data in one location, namely UTC. Now how users can see accurate time according to their location. An example of the discussion above is a flash sale on an e-commerce application that we have, where users also want to know information about which user made the first transaction. So at this point, converting the time we store in the database to the user's local time is another important thing that we should not ignore.

The approach I take is that the server side always asks the client to send the timezone on the user side. This can be done on the request side where the client sends a header with the key timezone and has the user's timezone value. For example, Indonesia has 3 time zone divisions, namely WIT(+9), WITA(+8), WIB(+7). Where each zone has a difference of 1 hour. If previously on our server we stored UTC time at 00.00, then on the WIT side it was at 09.00, then on the WITA side it was at 08.00 and WIB at 07.00.

Here's an example code to handle the above case:

package main

import (
    "fmt"
    "time"
)

func ParseTimezoneFromString(tz string) *time.Location {

    if len(tz) > 0 {

        t, err := time.Parse("2006 -07:00", fmt.Sprintf("2007 %s", tz))

        if err != nil {

            panic(err)

        } else {

            return t.Location()
        }
    }

    return time.Now().Location()
}

func main() {

    timeServerInUTC := "2024-08-04 00:00:00"
    nowInUTC, err := time.Parse("2006-01-02 15:04:05", timeServerInUTC)
    if err != nil {
        panic(err)
    }

    fmt.Printf("UTC time: %s\n", nowInUTC)

    witLocation := ParseTimezoneFromString("+09:00")

    nowInWIT := nowInUTC.In(witLocation)
    fmt.Printf("WIT time: %s\n", nowInWIT)

    witaLocation := ParseTimezoneFromString("+08:00")

    nowInWITA := nowInUTC.In(witaLocation)
    fmt.Printf("WITA time: %s\n", nowInWITA)

    wibLocation := ParseTimezoneFromString("+07:00")

    nowInWIB := nowInUTC.In(wibLocation)
    fmt.Printf("WIB time: %s\n", nowInWIB)

}

credit to dikac for create this function ParseTimezoneFromString

Let's understand the meaning of the code above:

First, we create a function ParseTimezoneFromString, where this function is used to find the time location based on the argument tz or timezone of the given user location. If the string value tz is valid, we will convert the string's timezone using the time.Parse function to convert the string to a time object, then extract the location (timezone) from that object. And we also handle if the string is empty or parsing fails, the function returns the local time zone of the system.

func ParseTimezoneFromString(tz string) *time.Location {

    if len(tz) > 0 {

        t, err := time.Parse("2006 -07:00", fmt.Sprintf("2007 %s", tz))

        if err != nil {

            panic(err)

        } else {

            return t.Location()
        }
    }

    return time.Now().Location()
}

Next we also define time data in the following string format:

timeServerInUTC := "2024-08-04 00:00:00"

nowInUTC, err := time.Parse("2006-01-02 15:04:05", timeServerInUTC)
if err != nil {
    panic(err)
}

You can think of this as the timing data that we get from the server. And parse it into a time object.

Next, we try to find the user's accurate location based on the ParseTimezoneFromString function that we previously created based on the string argument that we defined. What needs to be paid attention to is that this string argument is what is meant by the value of the timezone header sent by the client via the incoming request.

We can use the location we get from the ParseTimezoneFromString function to convert or shift the time we get from the server to the user's local time. We can do this using the In function which is also included in the time package which we can see in the following code snippet:

nowInWIT := nowInUTC.In(witLocation)
nowInWITA := nowInUTC.In(witaLocation)
nowInWIB := nowInUTC.In(wibLocation)

If we run it, we will get the time that corresponds to the timezone location that we defined.

Testing and Validation

The final point is no less important, namely testing and validation. When the development process often causes developers to make unexpected mistakes, testing and validation are always important.

In the discussion of point 2 above, the ParseTimezoneFromString function has been important in handling our time zones. Repeated testing and validation are important to make our applications get results that meet our expectations.

For testing, it is recommended to use unit tests, where testing will be carried out on the smallest unit with several scenarios that can be added. The more scenarios there are, the less likely it is to handle these time differences.

Conclusion

Handling time zones can indeed be tricky for backend developers. However, it's important to remember that every challenging task we overcome contributes to our growth and skill improvement. Properly managing time zones is not just a technical necessity, it ensures accuracy in scheduling and provides a smooth experience for our application users across different regions.

The points shared in this article about storing times in UTC, converting to user local times, and implementing robust conversion functions are starting points in tackling this complex issue. However, I acknowledge that there may be shortcomings or areas for improvement in the approaches discussed. This is why additional input and suggestions from the developer community are invaluable.

I sincerely hope that the insights and code examples provided in this article will be helpful to you in the future when you encounter time zone-related challenges in your projects. Remember, the goal is to create applications that work seamlessly for users, regardless of their geographical location.

Let's continue this discussion in the comments section below. I'd love to hear about your experiences with handling time zones, any challenges you've faced, or alternative approaches you've found effective. Your insights could be extremely valuable to me and other readers facing similar issues.

Thank you for reading, and I hope this article proves useful in your development journey. Let's keep learning and improving together! 👋

Reading References

Observability - Why logging its important

Rizky Darmawan — Fri, 28 Jun 2024 09:39:28 +0000

In an increasingly complex digital era, observability is the main key in managing modern software systems. One of the most important pillars of observability is logging. Let's explore why logging is so important and how to make optimal use of it.

What is Logging?

Logging is the process of recording activities and events in a system. This includes a variety of information, from error messages, user activity, to system performance. Think of logging as an airplane 'black box' for your system - always recording what's happening, ready to provide insights when needed.

Why is Logging So Important?

Here are some points that can be considered why logs are important:

Faster Problem Solving
With good logs, development teams can identify root causes without guesswork. It's like having a treasure map when looking for bugs!
Security Improvements
Logs can be your 'spy' in detecting suspicious activity. Security teams can respond to threats more quickly, such as having a fire department always on standby.
Performance Analysis
Through logs, you can identify bottlenecks in the system. It's like having a personal doctor for your app's health.
Understanding User Behavior
User activity logs provide valuable insight into how the product is used. It's like having a personal assistant constantly observing and reporting customer preferences.

Best Practices in Logging

To maximize the benefits of logging, below are some of the best practices that can be carried out:

Determine the Appropriate Log Level

Using these appropriate log levels can help you filter information quickly, such as sorting logs by urgency.

The following is an example of displaying logs using the Golang language with various levels. Here we use the Logrus.

package main

import (
    "github.com/sirupsen/logrus"
)

func main() {
    log := logrus.New()
    log.SetLevel(logrus.DebugLevel)

    log.Debug("Starting app..")
    log.Info("User has successfully logged in")
    log.Warn("CPU usage exceeds 80%")
    log.Error("Failed to save data to database")
    log.Fatal("A critical error occurs, the application will stop")
}

The following is an explanation for the several log levels above:

DEBUG: Detailed information for debugging, usually only enabled during development.
INFO: General information about the normal flow of the application.
WARNING: For situations that have the potential to become problematic in the future, but do not stop the application.
ERROR: An error that causes a specific function to fail, but the application is still running.
FATAL: Serious error that may cause the application to stop.

Include relevant contextual information

Each log entry should provide enough context to understand what happened. This could include:

Timestamp.
Transaction or session ID.
User ID (if relevant).
Function or module name.
Relevant input data (be careful with sensitive data).
Stack trace for errors

This is an example of code when printing a log, including context information that will help us trace.

package main

import (
    "github.com/sirupsen/logrus"
    "time"
)

type UserAction struct {
    UserID    int
    Action    string
    Timestamp time.Time
}

func main() {
    log := logrus.New()
    log.SetLevel(logrus.DebugLevel)

    // Use format json
    log.SetFormatter(&logrus.JSONFormatter{})

    // Dummy data
    action := UserAction{
        UserID:    12345,
        Action:    "checkout",
        Timestamp: time.Now(),
    }

    // Print log
    log.WithFields(logrus.Fields{
        "user_id":    action.UserID,
        "action":     action.Action,
        "timestamp":  time.Now().Format(time.RFC3339),
        "session_id": generateSessionID(),
        "module":     "payment_processor",
        "ip_address": "192.168.1.100",
    }).Error("Payment failed")

}

func generateSessionID() string {
    return "sess_abc123"
}

We can see that we have included several elements of context information that can make it easier for us to carry out tracing in the future. What are the conveniences in question, namely that we can search logs based on level, for example the error level in the code example above, and also based on time and others based on the information we enter.

Use consistent formatting

A consistent log format makes parsing and analysis easier, especially if using automated tools (regarding tools, will be discussed below). Formatting also makes it easier for us to search logs based on criteria, for example log level, message, or time. Example format:

[TIMESTAMP] [LEVEL] [MODULE] [MESSAGE]

Or JSON format for easy parsing like the results in the code example above:

{
    "action": "checkout",
    "ip_address": "192.168.1.100",
    "level": "error",
    "module": "payment_processor",
    "msg": "Payment failed",
    "session_id": "sess_abc123",
    "time": "2024-06-26T20:59:02+07:00",
    "timestamp": "2024-06-26T20:59:02+07:00",
    "user_id": 12345
}

Implement log rotation to manage file size

Log rotation prevents log files from becoming too large and difficult to manage. This involves:

Limits the size of log files.
Create new log files periodically (e.g. daily or weekly).
Archive or delete old log files.
Using tools such as logrotate on Linux or a logging framework that supports rotation.

Consider privacy and security in logged information

Security and privacy are very important in logging:

Do not log sensitive data such as passwords or credit card information.
Mask or encrypt personal data if necessary.
Ensure access to log files is restricted to authorized personnel only.
Implement a retention policy to delete old logs according to company policies and regulations.

Tools for Monitoring and Analyzing Logs

As system complexity increases, the need for sophisticated tools to monitor and analyze logs also becomes increasingly important. Here are some popular tools that can help with observability and log analysis:

Grafana
Grafana is an open-source platform for visualizing our log data. These tools can be integrated into various data sources including logs. Enables the creation of customized and interactive dashboards. Suitable for real-time visualization of metrics and logs.
New Relic
New Relic is an all-in-one observability platform
Provides log analysis, tracing, and metrics in one place. There are also AI features to detect anomalies and correlate problems.
Suitable for monitoring large-scale applications and infrastructure.
Loki
Loki is a lightweight and cost-effective log aggregation system. Loki is designed to work well with Grafana
Uses label-based indexes, similar to Prometheus
Ideal for organizations already using Prometheus and Grafana.
AWS CloudWatch Logs Insights
This integrated log analysis service from AWS enables querying and analysis of logs from various AWS services.
Feature to detect slow queries in RDS and other database services
Easy integration with other AWS services.

Conclusion

Logging is not just an additional feature, but a vital component in building a reliable system. With proper implementation, logging can become your supersensor - providing full visibility into system operations, helping prevent problems before they occur, and speeding resolution when problems arise.

So, start investing in good logging practices today. Remember, in the world of complex technology, good logs can be a guiding light in the midst of a storm!

If you have additional information, please enter it in the comments column below.

Reading References

Overcoming the Thundering Herd Problem in Go Applications with Circuit Breaker Pattern

Rizky Darmawan — Tue, 11 Jun 2024 03:07:00 +0000

Imagine you are on your way to the office during the morning rush hour. The highways were filled with slow-moving vehicles due to heavy traffic jams. When the traffic light turns green, thousands of vehicles try to drive at once, creating even worse traffic jams. This situation is similar to what is called the "thundering herd problem" in the world of computing.

What is the Thundering Herd Problem

Thundering Herd is a term in distributed computing and large-scale systems that refers to a situation where multiple processes or threads try to obtain the same resource simultaneously after a certain delay or event.
This issue often occurs in scenarios where a large number of processes are waiting for a limited resource, such as a database, shared cache, network service, or remote service.

When this resource finally becomes available, all waiting processes will try to access it simultaneously, causing a spike in activity that can overload or even disrupt the service.

The effects of Thundering Herd can cause performance degradation, delays, or even service failure. To overcome this problem, mechanisms such as scheduling, throttling, or using design patterns such as Circuit Breaker are needed to limit the number of requests at once to limited resources.

Case Example

Suppose you have a web application that fetches user data from a MySQL database and stores it in the Redis cache to improve access speed. When user data is not found in the cache, many requests will try to retrieve data from the database simultaneously, causing excessive load on the database and reducing overall application performance.

Look at the code below:

package main

import (
    "database/sql"
    "encoding/json"
    "fmt"
    "github.com/go-redis/redis"
    _ "github.com/go-sql-driver/mysql"
    "log"
    "math/rand"
    "time"
)

type User struct {
    Id       int
    Username string
    Email    string
}

type Config struct {
    redisClient *redis.Client
    dbClient    *sql.DB
}

func NewUser(dbClient *sql.DB, redisClient *redis.Client) *Config {
    return &Config{
        dbClient:    dbClient,
        redisClient: redisClient,
    }
}

func (e *Config) getDataFromMysql(username string) (*User, error) {
    row := e.dbClient.QueryRow("SELECT * FROM users WHERE username = ?", username)

    user := &User{}
    err := row.Scan(&user.Id, &user.Username, &user.Email)
    if err != nil {
        return nil, err
    }

    err = e.saveToRedis(username, user)
    if err != nil {
        return nil, err
    }

    return user, nil

}

func (e *Config) getDataFromRedis(username string) (*User, error) {
    val, err := e.redisClient.Get(username).Result()
    if err != nil {
        log.Printf("failed to get redis with key [%s], err: %v", username, err)

        log.Println("get data from mysql")
        user, err := e.getDataFromMysql(username)
        if err != nil {
            return nil, err
        }

        return user, nil

    }

    user := &User{}
    err = json.Unmarshal([]byte(val), &user)
    if err != nil {
        return nil, err
    }

    return user, nil
}

func (e *Config) saveToRedis(key string, data *User) error {
    jsonData, err := json.Marshal(data)
    if err != nil {
        return err
    }

    ttl := 10 * time.Second
    err = e.redisClient.Set(key, jsonData, ttl).Err()
    if err != nil {
        return err
    }

    return nil

}

func main() {
    mysqlConn, err := sql.Open("mysql", "root:root@tcp(localhost:3306)/employees")
    if err != nil {
        panic(err)
    }

    redisConn := redis.NewClient(&redis.Options{
        Addr:     "127.0.0.1:6379",
        Password: "",
        DB:       0,
    })

    client := NewUser(mysqlConn, redisConn)

    username := "jhon_doe"

    // Simulate get cache miss as asyncronous
    for i := 0; i < 100; i++ {
        go func() {
            time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
            val, err := client.getDataFromRedis(username)
            if err != nil {
                fmt.Println(err)

            } else {
                fmt.Printf("Got value: %v\n", val)

            }
        }()
    }

    select {}
}

Now let's try running the code above and in the middle of the process we will try to turn off the MySQL database that we have.

In the above process we can see, when data is not found in the Redis cache, all requests will try to retrieve data from the MySQL database simultaneously. Imagine that when you try to retrieve data from MySQL there are lots of requests. This can cause a thundering herd problem and overload the database with excessive requests and also when the database recovers it immediately gets a lot of requests that continue to be sent.

Circuit Breaker Pattern

There are many solutions that can be used to deal with this thundering herd problem. One way is to use Circuit Breaker Pattern. Before we go into the solution to the problem in the code above, let's first find out what a Circuit Breaker Pattern is. In the world of software, the Circuit Breaker Pattern is a design pattern used to prevent system failure due to unresponsive or failed external components. This pattern is similar to how an electrical Circuit Breaker works at home.

In everyday life, we often use electrical appliances at home, such as irons, toasters or washing machines. If too many appliances are used at the same time, this can place an excessive load on your home's electrical system. To prevent damage or fire, an electrical safety device called a "Circuit Breaker" is installed.

Likewise, in a distributed system, components such as databases, web services, or other external systems can fail or become unresponsive for various reasons, such as excessive load, network problems, or internal errors. If your application continually tries to access unresponsive components, this can lead to excessive resource usage, a backlog of requests, and ultimately a complete system failure.

Circuit Breaker Pattern addresses this problem by monitoring the success and failure of requests to external components. If too many failures occur within a certain time period, the Circuit Breaker will "open" and temporarily prevent new requests to external components. As long as the Circuit Breaker is open, new requests will be rejected or redirected to a safe fallback mechanism, such as returning cached data or a default response.
After a certain time delay, the Circuit Breaker will try to close itself again and allow new requests to external components. If the request is successful, the Circuit Breaker will remain closed.

However, if the failure occurs again, the Circuit Breaker will open again and the cycle will repeat. By using the Circuit Breaker Pattern, your application becomes more resilient to external component failures, preventing excessive resource usage, and giving external components time to recover before accepting new requests. This improves overall system stability, reliability and robustness.

Implementation of solutions with Circuit Breaker Pattern

Now we will use the Circuit Breaker Pattern to overcome the thundering herd problem above. Circuit Breaker acts as a safeguard that limits access to a resource (in this case, a MySQL database) when a series of failures occurs. If too many failures occur within a certain time period, the Circuit Breaker will "open" and prevent new requests from the failed resource. Here's the complete code:

package main

import (
    "database/sql"
    "encoding/json"
    "fmt"
    "github.com/go-redis/redis"
    _ "github.com/go-sql-driver/mysql"
    "log"
    "math/rand"
    "time"
)

type User struct {
    Id       int
    Username string
    Email    string
}

type Config struct {
    redisClient *redis.Client
    dbClient    *sql.DB
    circuit     *CircuitBreaker
}

type CircuitBreaker struct {
    failureThreshold   int
    consecutiveFailure int
    open               bool
    openedAt           time.Time
}

const circuitBreakerResetTimeout = 2 * time.Second

func NewCircuitBreaker(failureThreshold int) *CircuitBreaker {
    return &CircuitBreaker{
        failureThreshold:   failureThreshold,
        consecutiveFailure: 0,
        open:               false,
    }
}

func NewUser(dbClient *sql.DB, redisClient *redis.Client, failureThreshold int) *Config {
    return &Config{
        dbClient:    dbClient,
        redisClient: redisClient,
        circuit:     NewCircuitBreaker(failureThreshold),
    }
}

func (cb *CircuitBreaker) IsOpen() bool {
    if cb.open {
        // Check last time opened
        if time.Since(cb.openedAt) >= circuitBreakerResetTimeout {
            cb.open = false
            cb.consecutiveFailure = 0
            log.Println("Circuit Breaker closed")
        } else {
            return true
        }
    }
    return false
}

func (cb *CircuitBreaker) IncrementConsecutiveFailure() {
    cb.consecutiveFailure++
    if cb.consecutiveFailure >= cb.failureThreshold {
        cb.open = true
        cb.openedAt = time.Now()
        log.Println("Circuit Breaker opened")
    }
}

func (e *Config) getDataFromMysql(username string) (*User, error) {
    row := e.dbClient.QueryRow("SELECT * FROM users WHERE username = ?", username)

    user := &User{}
    err := row.Scan(&user.Id, &user.Username, &user.Email)
    if err != nil {
        return nil, err
    }

    err = e.saveToRedis(username, user)
    if err != nil {
        return nil, err
    }

    return user, nil

}

func (e *Config) getDataFromRedis(username string) (*User, error) {
    if e.circuit.IsOpen() {
        return nil, fmt.Errorf("circuit breaker is open")
    }

    val, err := e.redisClient.Get(username).Result()
    if err != nil {
        log.Printf("failed to get redis with key [%s], err: %v", username, err)

        log.Println("get data from mysql")
        user, err := e.getDataFromMysql(username)
        if err != nil {
            e.circuit.IncrementConsecutiveFailure()
            return nil, err
        }

        return user, nil

    }

    user := &User{}
    err = json.Unmarshal([]byte(val), &user)
    if err != nil {
        return nil, err
    }

    return user, nil
}

func (e *Config) saveToRedis(key string, data *User) error {
    jsonData, err := json.Marshal(data)
    if err != nil {
        return err
    }

    ttl := 2 * time.Millisecond
    err = e.redisClient.Set(key, jsonData, ttl).Err()
    if err != nil {
        return err
    }

    return nil

}

func main() {
    mysqlConn, err := sql.Open("mysql", "root:mysqlsecret@tcp(localhost:3306)/employees")
    if err != nil {
        panic(err)
    }

    redisConn := redis.NewClient(&redis.Options{
        Addr:     "127.0.0.1:6379",
        Password: "",
        DB:       0,
    })

    failureThreshold := 3
    client := NewUser(mysqlConn, redisConn, failureThreshold)

    username := "user20001"

    // Simulate get cache miss as asyncronous
    for i := 0; i < 100; i++ {
        go func() {
            time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
            val, err := client.getDataFromRedis(username)
            if err != nil {
                fmt.Println(err)

            } else {
                fmt.Printf("Got value: %v\n", val)

            }
        }()
    }

    select {}
}

Now we will see any changes to the code above. The first thing we do is modify the truct in the Config struct by adding a circuit field that holds a CircuitBreaker object:

type Config struct {
    redisClient *redis.Client
    dbClient    *sql.DB
    circuit     *CircuitBreaker
}

With the addition of this circuit field, the Config object now also has a CircuitBreaker object that will be used to apply the Circuit Breaker pattern. Next, there is the addition of a new CircuitBreaker struct and a NewCircuitBreaker function:

type CircuitBreaker struct {
    failureThreshold   int
    consecutiveFailure int
    open               bool
    openedAt           time.Time
}

The CircuitBreaker struct has three fields:

failureThreshold: The maximum number of failures allowed before the Circuit Breaker opens.
ConsecutiveFailure: The number of consecutive failures that have occurred.
open: Status whether the Circuit Breaker is currently open or not.
openedAt: Records when the Circuit Breaker opened.

func NewCircuitBreaker(failureThreshold int) *CircuitBreaker {
    return &CircuitBreaker{
        failureThreshold:   failureThreshold,
        consecutiveFailure: 0,
        open:               false,
    }
}

The NewCircuitBreaker function is a constructor to create a new CircuitBreaker object with the given failureThreshold, consecutiveFailure set to 0, and open set to false (the Circuit Breaker is in the closed state initially).
Finally, there is a modification to the NewUser function to register the circuit breaker so that it can be used in the function that initializes the Config object:

func NewUser(dbClient *sql.DB, redisClient *redis.Client, failureThreshold int) *Config {
    return &Config{
        dbClient:    dbClient,
        redisClient: redisClient,
        circuit:     NewCircuitBreaker(failureThreshold),
    }
}

In the code above we also add the Is Open function:

func (cb *CircuitBreaker) IsOpen() bool {
    if cb.open {
        // Check last time opened
        if time.Since(cb.openedAt) >= circuitBreakerResetTimeout {
            cb.open = false
            cb.consecutiveFailure = 0
            log.Println("Circuit Breaker closed")
        } else {
            return true
        }
    }

    return false
}

The IsOpen function will first check whether the Circuit Breaker is open (cb.open). If open, the function will check whether the time that has passed since the Circuit Breaker opened (time.Since(cb.openedAt)) has reached or exceeded the circuitBreakerResetTimeout.
If the elapsed time has reached the waiting time limit, the Circuit Breaker will be closed again (cb.open = false), and consecutiveFailure is set back to 0.

The function will also record a log message "Circuit Breaker closed" to notify you that the Circuit Breaker has closed again. If the elapsed time has not reached the waiting time limit, the function will return true to indicate that the Circuit Breaker is still open.
If the Circuit Breaker is not open, the function will return false.

And don't forget to add the constant circuitBreakerResetTimeout to provide an automatic checking time for the circuit breaker to reopen in the IsOpen function above.

const circuitBreakerResetTimeout = 2 * time.Second

And then the IncrementConsecutiveFailure function is also a method of the CircuitBreaker struct. This function is responsible for counting the number of consecutive failures that occurred when trying to access an external resource.

func (cb *CircuitBreaker) IncrementConsecutiveFailure() {
    cb.consecutiveFailure++
    if cb.consecutiveFailure >= cb.failureThreshold {
        cb.open = true
        cb.openedAt = time.Now()
        log.Println("Circuit Breaker opened")
    }
}

First, this function will increase the consecutiveFailure value (the number of consecutive failures) by adding it by 1. Then, the function will check whether the consecutiveFailure has reached or exceeded the failureThreshold (the specified maximum failure limit).

If the consecutiveFailure reaches or exceeds the failureThreshold, the Circuit Breaker will be changed to an open state (open = true) and record the time when the circuit breaker was opened for comparison in the IsOpen function above. When the Circuit Breaker opens, all new requests to external resources will be rejected or redirected to a secure fallback mechanism. In addition, the function will also record a log message "Circuit Breaker opened" to notify that the Circuit Breaker has opened.

After all the necessary configurations and functions have been created, now we will carry out the implementation to handle the thundering herd problem. First, in the main function, we will determine the initial value of failureThreshold, which in this case is the limit of 3 times for failure.

func main() {
    // Others code ...

    failureThreshold := 3
    client := NewUser(mysqlConn, redisConn, failureThreshold)

    // Others code ...
}

The last thing we do in implementing the circuit breaker is the getDataFromRedis function, let's discuss it.

func (e *Config) getDataFromRedis(username string) (*User, error) {
    if e.circuit.IsOpen() {
        return nil, fmt.Errorf("circuit breaker is open")
    }

    // others code ...

First, in the getDataFromRedis function we add checking whether the Circuit Breaker is open or not by calling e.circuit.IsOpen(). If the Circuit Breaker is open, the function will immediately return the error "circuit breaker is open" and will not continue the process of retrieving data from Redis or MySQL. This is done to prevent new requests to external resources while the Circuit Breaker is open.

If Circuit Breaker is not open, the function will try to retrieve data from Redis using e.redisClient.Get(username).Result(). If an error occurs while retrieving data from Redis, the function logs the error and tries to retrieve data from MySQL by calling e.getDataFromMysql(username).

// others code ...

val, err := e.redisClient.Get(username).Result()
    if err != nil {
        log.Printf("failed to get redis with key [%s], err: %v", username, err)

        log.Println("get data from mysql")
        user, err := e.getDataFromMysql(username)
        if err != nil {
            e.circuit.IncrementConsecutiveFailure() // <-- call function IncrementConsecutiveFailure 
            return nil, err
        }

        return user, nil
    }

// others code ...

If an error occurs while retrieving data from MySQL, the function will call e.circuit.IncrementConsecutiveFailure() to increase the number of consecutive failures in the Circuit Breaker. If the number of consecutive failures reaches a certain limit (failureThreshold), then the Circuit Breaker will open. After that, the function will return error and nil as user values. As in the example below when we run the code above and our MySQL database dies.

After several consecutive failures to retrieve data from MySQL, the Circuit Breaker will open and there will be no more attempts to retrieve data from MySQL. At this stage all incoming requests will be rejected or we will provide an appropriate fallback for the user to obtain. And the Circuit Breaker will close again until the time limit we specified in the constant circuitBreakerResetTimeout.

With the implementation of the Circuit Breaker Pattern, the getDataFromRedis function will prevent new requests to external resources (MySQL) while the Circuit Breaker is open. In addition, the function will also monitor failures when retrieving data from MySQL and open the Circuit Breaker if a certain failure limit is reached. This helps prevent thundering herd problems and gives external resources time to recover before accepting new requests again.

Conclusion

Thundering herd problem is a common problem in distributed systems and can cause significant performance degradation. By using design patterns like Circuit Breaker, you can overcome these problems and ensure your applications run smoothly and efficiently. Always consider the potential thundering herd problem when designing systems that depend on limited resources and implement appropriate solutions to prevent it.

Reading References

Optimizing Relational Databases for Best Performance in MySQL

Rizky Darmawan — Tue, 07 May 2024 09:08:14 +0000

Introduction

In today's data-driven world, relational databases are the backbone of countless applications. They store and manage critical information, but their performance can significantly impact user experience and overall system efficiency.

This blog post dives into key strategies for optimizing relational databases and ensuring they run at peak performance.

🔍 Common Query Mistake That Lead Bottleneck

Before optimizing, in this writing the discussion only covers Unnecessary Full Table Scans, Inefficient Queries, Denormalization, Insufficient Hardware Resources. It's crucial to identify performance bottlenecks. Here are some common causes:

1. Unnecessary Full Table Scans

For example, I have table products with a total 4000000 rows of data. If your query doesn't have proper indexing, the database may be forced to scan the entire table for each lookup. For example how we can know a table caught the table scan:

Here's, I can try to select the products with name condition



SELECT * from products WHERE name = "Produk 300665";

And this result I have a total time 0.628s, it's not a long time.

But the surprising thing is when you run explain to see what happens:



explain SELECT * from products WHERE name = "Produk 300665";

See you have scanned a total of 3052749 rows just to find 1 row of data. And look at the type column here it has the value ALL, meaning you are scanning row by row looking for data. This will be a nightmare when the total data keeps getting bigger.

2. Inefficient Queries

Poorly written SQL queries can take longer to execute than necessary. Here's an example of a poorly written SQL query:



SELECT * FROM orders WHERE YEAR(order_date) = 2024;

This query retrieves all orders from 2024. However, using the YEAR() function in the WHERE clause makes it inefficient because it prevents the query from using an index on the Order Date column.

Instead, this forces the database to apply the YEAR() function to every row in the Orders table, potentially resulting in a full table scan. For example, the above is a query snippet from the orders table, now let's look at the table.

My orders has an index on the order_date column. Run this query to see a list of available indexes:



show indexes from order_table;

And see I do have an index for the order_date column.

Now, when you run explain to see what happens:



explain SELECT * FROM orders WHERE YEAR(order_date) = 2024;

It can be seen in the results below, using functions such as YEAR(), can cause the available indexes to not be used.

3. Denormalization (Careful Approach)

In some situations, denormalizing your database schema (introducing controlled redundancy) can improve query performance by reducing the need for complex joins. Here's a simple example to illustrate denormalization in MySQL:

Let's consider a hypothetical scenario where you have two tables: users and orders. Below are the Original Normalized Tables:

users table with structure:



user_id (Primary Key)
username
email

orders table with structure:



order_id (Primary Key)
user_id (Foreign Key referencing user_id in the users table)
order_date
total_amount

Now, let's say you frequently need to retrieve a user's orders along with their username and email. The most common way is to always join the orders table to the users table. Here's an example query:



SELECT
    orders.order_id,
    users.name,
    users.email,
    orders.order_date,
    orders.total_amount
FROM
    orders
    JOIN users ON orders.user_id = users.user_id;

Joining these tables every time you need this information could become a performance bottleneck, especially if the tables are large.

We will discuss how to denormalize it in the solutions chapter below.

4. Insufficient Hardware Resources

Database performance is highly dependent on factors such as CPU, RAM, and storage capacity. This is very difficult to see without the help of supporting software. We will discuss several solutions below.

🎯 Solution

Once you've identified the bottlenecks, you can employ various techniques to streamline your database:

1. Unnecessary Full Table Scans Solution

Indexes act like shortcuts for the database, allowing it to quickly locate specific data. Strategically creating and maintaining indexes on frequently used columns can significantly improve query speed.

Diving Deeper into Indexing Types

In the field of database management and optimization, indexing plays an important role in improving query performance. Understanding the different types of indexing is critical to efficient data retrieval and manipulation.

Let's dive deeper into two fundamental types of indexing: single index and compound index, and explore their function with illustrative examples.

Single Index:

A single index is a straightforward mechanism in which an index is created on a single column of a database table. It speeds up the search process by organizing data in indexed columns in a structured format, often a B-tree or hash table, allowing for faster retrieval of specific records.

In the example above, we are looking for a name in the products table with a large amount of data in the user table. Remember, to search for 1 row of data, we are required to scan each row. Now let's try adding a single index to the name column with the following query:



CREATE INDEX product_name_idx ON products (name);

To see if an index has been created, run this query:



show indexes from users;

Yeah, index has created:

Now, try again to select the user with name condition:



SELECT * from products WHERE name = "Produk 300665";

And it's amazing how quickly you can get data when you only take 2 second to do it, compared to the previous time of 0.628s seconds.

And the important thing is that we noticed many rows were scanned. Please rerun this query again.



explain SELECT * from products WHERE name = "Produk 300665";

There is only 1 row scanned and you can also observe that we are using an index. Small notes, if you want to remove index can run query:



DROP INDEX product_name_idx ON product;

Composite Index:

Unlike single index, a composite index involves indexing multiple columns within a database table. This type of indexing is particularly advantageous when queries involve conditions on multiple columns or require sorting based on a combination of fields.

Expanding on the previous example, let's say we frequently run a query to retrieve products based on their name, category, and price.

In such a scenario, creating a combined index on the name, category, and price columns will optimize the search process by storing and sorting the data based on these combined criteria.

For example, if we want to search for users with a specific name, category (gadget), and price greater than 400.000, the query would look like this:



SELECT *  from products WHERE name = "Produk 300665" AND category = "gadget" and price > 400;

As a result we get 25 data.

See the single index that we previously had has no effect on queries with multiple conditions.

If we explain with the single index we previously created, we scan 28 rows:



explain SELECT *  from products WHERE name = "Produk 300665" AND category = "gadget" and price > 400;

Obviously 25 lines are obtained but by scanning only 28 lines it is not a big problem. But what we need to remember is what happens if we have a large amount of data. Certainly the number of rows we scan will be much larger than the data we get. And now we try to create a composite index, by combining several columns in 1 index, a query like this:



CREATE INDEX product_name_category_price_idx ON products (name, category, price);

Now looks, a composite index has created:

To differentiate between single index and composite index, refer to the column key_name. A composite index is an index with multiple column_name values that share the same name as another index.

And now, we trying run explain query in above for see different result:

Voila, we only scanned 25 rows for 25 results using the composite index. And the composite index that we have is also included in the list of indexes that we will use.

2. Inefficient Queries Solution

Writing clean and optimized SQL queries is very important. Techniques such as avoiding complex functions in the WHERE clause and using appropriate JOINs can make a big difference. Or as in the example above, use the YEAR function to sort the years.

A more efficient version of this query would directly compare the order_date column to a date range in 2024:



SELECT * FROM Orders WHERE order_date >= '2024-01-01' AND order_date < '2025-01-01';

See nothing happens and nothing seems strange. But let's try running an explain query:



EXPLAIN SELECT * FROM Orders WHERE order_date >= '2024-01-01' AND order_date < '2025-01-01';

Now the index is used, and this is very important when the data we are scanning is very large.

3. Denormalization Solution

To optimize this scenario using denormalization, you might add redundant columns from the users table into the orders table:

Denormalized Orders Table:

orders table:



order_id (Primary Key)
user_id (Foreign Key referencing user_id in the users table)
order_date
total_amount
name (Redundant column from the users table)
email (Redundant column from the users table)

By adding name and email columns to the orders table, you eliminate the need for frequent joins when querying user-specific order data. This can significantly improve the performance of queries that retrieve orders along with user information.

Now the query we have only deals with the orders table, without joining the users table and has the same results.



SELECT
    orders.order_id,
    orders.name,
    orders.email,
    orders.order_date,
    orders.total_amount
FROM
    orders;

However, it's important to note that denormalization comes with trade-offs.

It can lead to data redundancy, increased storage requirements, and potential data inconsistency if updates are not properly managed. Therefore, denormalization should be used judiciously and in situations where the performance benefits outweigh the drawbacks.

Additionally, denormalized data should be kept consistent through proper data management practices such as triggers or application logic.

4. Insufficient Hardware Resources

To monitor Insufficient Hardware Resources performance, consider using a third-party monitoring tool designed specifically for MySQL performance monitoring.

Tools such as MySQL Enterprise Monitor, Percona Monitoring and Management (PMM), New Relic, or open source solutions such as Prometheus with MySQL Exporter can provide comprehensive monitoring capabilities.

📋 Conclusion

Optimizing relational databases is critical to ensuring optimal performance and providing a smooth user experience. By overcoming common bottlenecks such as unnecessary full table scans, inefficient queries, denormalization trade-offs, and insufficient hardware resources, you can significantly improve the speed and efficiency of your MySQL database.

Key strategies include intelligent indexing techniques such as single and composite indexes, writing clean and optimal SQL queries, and judicious application of denormalization when the performance benefits outweigh the disadvantages. Additionally, investing in adequate hardware resources and utilizing monitoring tools can help identify and resolve performance issues proactively.

By implementing these optimization strategies, you can unlock the full potential of your relational database, ensuring lightning-fast data retrieval, efficient data manipulation, and a responsive application experience for your users.

Thank you, hope it helps. 👋