As a Staff Engineer at Livestorm, I’m writing this article to describe how we reduced our test run time by 2 hours by just removing the unnecessary, without deleting a single test.

One of the most common reasons of slow test suites, especially in larger sized applications, is the amount of database interactions, in particular the amount of created records. We use RSpec and FactoryBot, probably the most common setup in Ruby on Rails application. FactoryBot is an amazing library and I can’t remember a single application I worked on in the past that was not using it. It provides a DSL for defining and using test factories, and it makes it very easy to build test objects for your specs. Unfortunately, it makes it also very easy to define factories, that with time will generate a lot of unnecessary database calls and make your specs slow, and that is exactly what happened in our project.

In the following paragraphs I will show you:

• What is the problem of factory cascade
• How to check if you have a problem of factory cascade
• How to fix a problem of factory cascade in an iterative way
• How to avoid generating factory cascade, without noticing

What is the problem of “Factory Cascade”

Factory cascade is a very common, but rarely addressed problem that can bring your whole test suite to a crawl. In short, it is an uncontrollable process of generating excess data through nested factory invocations.

evilmartians.com — TestProf: a good doctor for slow Ruby tests

Let’s look at it with a simplified example of the Livestorm app. When you sign up to Livestorm you create a new organization, which can have multiple members and for which you can create events, each of which can have multiple sessions. If we draw the relationship tree starting from a session we will have something like this:

session
├─ participants
│  ├─ account
├─ event
│  ├─ organization
│  │  ├─ owner
│  │  │  ├─ account
│  │  ├─ members
│  │  │  ├─ account

This could be translated in the following model and factory definitions:

Of course this factory definition has some problems, but it is not uncommon to find similar definitions in most of our rails applications. Now let’s see what happens when we create a session object in our specs using the factory. How many database records are we going to create each time we create a session object?

As you see from the console output, when creating the session, we also create a lot of other records by default. In particular we are going to create

• 1 organization
• 1 owner
• 3 members
• 4 sessions
• 12 participants
• 16 accounts

A total of 38 records, and all of this hidden behind create(:session).

This is the problem of “Factory Cascade”.

How to check if you have a problem of “Factory Cascade”

In the previous paragraph we saw how we can check the number of records created by a factory using the rails console. This is not very handy, isn’t it. Fortunately we don’t need to do that manually, we can use evilmartian’s Test Profiler, which prints out at the end of each test run all information we need. After adding test-prof to your gems, you can simply use FPROF=1 in front of your RSpec command, and you will see the factory usage of your test run.

This table shows you all factories that have created objects, how many instances they created and how many of them were called directly (top-level). In this case only the session factory was called directly once, all the other instances were created as a side effect. In this case we have 37/38 (~ 97%) of instances created on cascade.

Now that we have a tool to print the factory usage and the amount of instances created on cascade, how can we use it to measure the health of a factory? Simply follow the example above. Create a new spec file and add a single factory instance. Then run the spec using the FPROF=1 tag and check the output.

From the output of factory profiler you can immediately recognize a factory cascade, by checking how many records were created in total by calling only the factory without any options.

You can also run the profiler on your full test suite, and see which factory used most time in total. This way you can prioritize the optimization of those factories that will allow you to have the highest impact on the overall execution time, and avoid loosing time working on factories that are rarely used.

How to fix a problem of “Factory Cascade” iteratively

After you identified one of your factories that is generating a lot of factory cascade, the next question is “how do I fix it”? If you are working on a large code base, and you have to do such a breaking change on a highly used factory, you will need to come up with a strategy to do it in multiple steps, because many tests will start failing once you stop creating the associated objects on cascade.

At Livestorm we came up with the following plan:

1. Duplicate the session factory to deprecated session factory
2. Move all associations from the default session factory attributes / after_create callbacks to traits
3. For all failing specs, replace session factory with deprecated session factory
4. Manually replace usage of deprecated session factory with session factory and fix specs

Let’s see this more in detail.

Duplicate the factory

If you picked a highly used factory, then it is almost impossible you will not break many specs by changing the amount of associations you create on cascade. Since with a simple create(:session) you get also an event, some participants, and more and more, developers might not always create all the data they need explicitly, but just rely on this side-effect. For this reason, before doing any change to the factory, duplicate it and give it a meaningful name, something like deprecated_session, so that you discourage people to use it when creating new specs. We will see later how to use this new (duplicated) factory.

Move associations to traits

Factory cascades are generated by associations created by default in our factories. Some of them might be necessary, e.g. a mandatory belongs_to association, others are optional. In our example, the has many associations created in the after(:create) block are all objects that might be useful for some specs, but are not needed in order to have a valid session. These are the kind of associations you do not want to create by default in your factory, but only keep as an optional trait that you can call when needed. Let’s see how to transform the session factory.

Now, testing the session factory again with fprof, we can see how this small changed started reducing the factory cascade problem.

We still have a lot of space to improve the other factories, but already with this small change we removed 24 unnecessary database inserts from the default factory, reducing the cascade percentage from 97% to 92% on this single file. If this factory is used a lot in your test suite, creating more than 20 records less on each factory creation can have a strong impact on your overall execution time.

Since we defined a trait we can still easily create the associated objects if need.

Fix failing specs with deprecated factory

After changing the default behaviour of the spec, a lot of specs will start failing, because they are expecting the associated records to exist. E.g. you could have a test that instead of creating explicitly a participant of a session in the test setup, takes it from the list of existing participants of the session (session.participants.take). Since the optimized spec does not create any participant by default anymore, those specs will start failing. In a large code base the number of such failing specs can be huge, in our case we had several hundred failing specs, and all of them need to be fixed manually. Fixing them all at the same time would require a lot of time, while the application keeps evolving together with the specs. This would generate a high number of merge conflicts at the end, making it very difficult to finalize the optimization and release the code. For this reason we created a copy of the old factory at the beginning, so that we can simply replace the factory name to the deprecated one, and have the specs passing again. Of course you don’t want to keep this reference to the deprecated factory forever in your code base, but now that you have a an optimized factory and passing specs, you can merge your optimization, and tackle the removal of the deprecated factory references in small steps.

Avoid creating factory cascades: make the problem obvious

One of the reasons why factory cascade is so frequent, is that you don’t really notice it, until your test suite becomes so slow that you need to investigate about it, and only then you will discover that you are creating maybe 100s of records when you need just 1. In order to avoid getting to this point, and notice as soon as possible that a factory is creating too much unnecessary records, you can set your development environment to automatically print the factory usage at the end of each test. E.g. in VSCode you can use the Vscode-run-rspec-file ****extension to run specs from your IDE using a keyboard shortcut, and customize the command used to run the specs to include the FPROF tag.

Summary

We only optimized one core factory in our codebase, but the result was already huge.

Our unit test suite before the optimization was creating 150.000 database records, 93% of them were created on cascade, and took around 9 hours of computation (split on 150 jobs running in parallel). After the optimization the cascade percentage went down to 67%, reducing the number of created records to 100.000 and the execution time to 7 hours. It is worth mentioning that we did not remove any spec, nor we did change the spec itself. The only change happened in the factory definition and in the way the factories are used to setup the test data.

It was not an easy journey, but looking at the result it was definitely worth it, so I encourage you to try it out yourself and benefit from a faster and more efficient test suite.

References

This test optimization project was strongly inspired by Evil Maritans’ blog article, TestProf II: Factory therapy for your Ruby tests.

Save money, optimize your test factories was originally published in @livestormapp on Medium, where people are continuing the conversation by highlighting and responding to this story.

Imagine you work on a complex product that requires multiple smaller applications to work with each other, for example an eCommerce platform made of a service for user profiles, inventory, payments, recommendations, etc… and each of those services is developed and maintained by a different team in your organization. In an environment like this, how can you make sure that a change on one application does not break another application that relies on it?

In this article I am going to to show you how you can use consumer-driven contract tests using pact.io to solve this problem, allowing you to test each service in isolation, thus in an efficient way, by stubbing external requests and at the same time ensuring that those “stubs” are an accurate representation of the external service.

Contract testing is a methodology for ensuring that two separate systems (such as two microservices) are compatible with one other. It captures the interactions that are exchanged between each service, storing them in a contract, which can then be used to verify that both parties adhere to it. (pactflow.io)

Before going into contract testing and pact.io, let’s first see other approaches and why they might be less efficient or secure than the contract testing approach.

End-to-end Testing

One way of making sure that a change does not break any other application is to test the full platform functionality end-to-end by doing the real calls between the services.

The problem with this approach is that it is slow, expensive and potentially frustrating.

In order to detect a failure on the consumer side when the provider changes, you would need to run all tests that possibly involve calling the part of application you are changing. It can be very tricky to determine subsets of functionalities that can be affected, thus you would end up running the full test suite for the whole platform. Those tests would have to perform real calls between services, create database entries, move data around, and for this reason they are very slow.

On top of that, they tend to be unstable, since they rely on networking and probably also on data being available on the different services. Whenever you have a failing test you have to figure out if it is a random failure due to some networking or service unavailability, if any data changed, or if it is really caused by the changes you want to release.

This can block or at least delay releases. One of the biggest advantages of microservices is that it allows a big organization to create small sized teams that work independently from each other on different services, so that they can implement new features and deploy them as soon as they are ready, without having to coordinate with any other team. If you can’t release your change because you get blocked by flaky tests or slow end-to-end runs, you may slow down the general development speed, and lower the developer happiness.

The main reason for microservices is speeding up development
 — Adrian Cockcroft, former cloud architect at Netflix, in Scalable Microservices with Kubernetes

Stub external requests and test each service in isolation

Since end-to-end tests are slow and not always reliable, normally the interaction with an external service is faked inside the test suite of each application. Based on the technology you are using there are different ways to do this. In Ruby on Rails for example you could use a gem like VCR, to record the real response to an external service the first time you run the test, and afterwards use the stored response (cassette) instead of calling the external service again. Or you could use Webmock to manually define the response that you want to get when you make a specific call.

This makes the test suite faster, more reliable and isolated from external factors. But that’s also the problem. The test suite would not fail if the external service suddenly changes the API. In the example above, if the user profile service removes the email field from the response, the test would still pass, but because of this change our application would start failing when interacting with the remote service for real.

So the question is: how can you make sure that those mocks actually represent the current version of the external service? Let’s see how pact.io and consumer-driven contract testing solve the problem.

Consumer-driven Contract Testing

The basic idea is this: whenever you need an interaction between a consumer and a provider, create a contract for it, which specifies that for a given request, the provider has to respond in a certain way.

The contract that defines the interaction is used by the consumer to mock the requests to the external service, and by the provider to ensure that whatever change it makes, it still respects the contract. With this in place, removing the email field from the response would make the provider be not compliant with the contract anymore, and should prevent the release of the change.

Example Application

On Pact.io you can find libraries for many different languages, which will help you to generate the contract, use it to mock the response for the consumer tests and verify changes on the provider side.

I am going to use Ruby on Rails with the pact-ruby gem and, like in my previous article, it will be about a blog application that interacts with an external service, in this case a user profile service that provides user specific data.

Consumer: Blog

Let’s start with the consumer application.

The Remote::User class defines a find class method, that calls the remote UserProfile service to fetch user data. It then uses name and email contained in the response to build a new User instance, after transforming their values to lower case. Notice that the base URL of the external service is defined as an environment variable, which will come in handy when mocking the service using pact.

The webmock_example_spec.rb we saw earlier shows how you would test this code using the Webmock gem.

Next let’s see how to we can test this code by still stubbing the remote service, but at the same time generating a contract JSON that will be used by the provider to ensure the API it implements matches the mock behavior. First of all we need to add pact to our Gemfile and bundle install. See the readme for more details.

Now we can create a mock service on localhost:1234, which will respond to our application’s queries over HTTP as if it was the real User Profile Service app. It also creates a mock provider object which we will use to set up our expectations. The method name to access the mock service provider will be whatever name we give as the service argument — in this case user_profile. The value we give to service_consumer and has_pact_with will set the Consumer and Provider values in the generated contract pact.json file.

Ok, we have a mock server running on localhost:1234, which we can access using the object user_profile in order to add expectations, basically saying how the mock server should reply given certain conditions and requests.

The blog application sends a request to the User Profile service at the URL specified in the environment variable USER_PROFILE_BASE_URL . We want to stub this request in our tests with the mock server we defined before. We can do that by using a test environment variables configuration, that sets USER_PROFILE_BASE_URL=localhost:1234. I use dot-env to define different configuration files for each of the environments, but you can also set it in your shell.

The way of specifying the expectations looks very similar to the Given-When-Then style used for example by Cucumber. First you define a precondition given('...') or ProviderState how it is called in pact, then the event upon_receiving('...').with(method: get, path: '...'), and finally the expected result will_respond_with(...).

After running bundle exec rspec you should see your specs pass, meaning that it correctly used the mocked service, and a new file should appear in spec/pacts which specifies the contract.

It is very easy to see how the definition of the mock-service in user_profile_provider_helper.rb and the endpoint mock in user_spec.rb map to this contract. At this point you might want to publish the contract to a pact-broker so that the provider service can verify that his API complies to the contract. You could run your own broker server or you could use pactflow.io which allows you to setup one in the cloud. To get more details about how to publish the contract to a pact-broker checkout the Pact Broker Client gem. For the scope of this article, we will just access the contract file locally, without using a broker.

Provider: UserProfile

Now let’s see on the provider application, how to verify that the API is compliant with the contract.

The implementation of the API endpoint is trivial. It just looks up the user by id, and returns it as JSON.

In this case pact is not used to mock any external service. Instead we need to define and link all contracts the API should comply to, so that we can run a verification task that will ensure that the consumer expectations are met.

Before that, we need to do some setup tasks: first add pact to your gems, then add require pact/tasks to your Rakefile, which will allow you to run rake pact:verify. See pact-ruby’s readme for more details.

To specify the contract to which the provider has to comply we add a new filespec/support/service_consumers/pact_helper.rb

We indicate that the User Profile application honors a pact with the Blog application, and that the contract can be reached at a specific pact_uri. Since the contract we defined earlier contains a provider state, a user with id 100 exists, we need to tell the provider application how to translate this into code: create a new user record with id 100 before running the test, and delete it after the test is over. And that’s it. Let’s run the verify task.

The task confirmed that the API honors the pact with the Blog application. Last step, let’s remove the email field from the response to be sure that the verifier actually is able to detect the change.

Let’s run the verifier task again.

As expected the verify task returns a failure and more specifically it tells us that the “email” key was not found.

Conclusion

In this article we saw how contract testing can allow you to test your application in isolation by stubbing an external service call, and at the same time translate the stubbed call into a pact contract JSON, which in turn can be used by the external service, the provider, to verify that its API meets the consumer expectations.

You can checkout the full code used in this example here.

References

• https://docs.pact.io/implementation_guides/cli
• https://github.com/pact-foundation/pact-ruby
• https://pactflow.io/blog/what-is-contract-testing/#what-is-contract-testing
• https://martinfowler.com/bliki/ContractTest.html
• https://martinfowler.com/bliki/GivenWhenThen.html
• https://classroom.udacity.com/courses/ud615/lessons/7826112332/concepts/81957124680923
• https://github.com/vcr/vcr
• https://github.com/bblimke/webmock
• https://medium.com/better-programming/dealing-with-data-and-communication-in-kubernetes-4b1c940719af
• https://github.com/domangi/pact-tutorial

Contract Testing with Pact.io was originally published in New Work Development on Medium, where people are continuing the conversation by highlighting and responding to this story.

Some time ago I did an online course on Udemy called “Microservices Software Architecture: Patterns and Techniques”. In this course, the instructor greatly describes the main challenges that come with microservice architecture and possible solutions. While going through the lessons about Service Registration, Service Discovery, Failover Mechanism, and Health API, I thought, “Thank, God, we have Kubernetes that gives us most of those solutions out of the box, so that we can focus on implementing our business logic.”

Even though you get much from Kubernetes, there are some aspects of a microservice architecture that you need to address in any case yourself. One of the most important ones being Data and Communication, which addresses the following questions:

How do you store your data?

How do services exchange this data?

In this article, I will show you how you can set up a local Kubernetes cluster using minikube. By implementing a very basic example I am going to describe how to deal with data and communication in a microservice architecture.

The example application itself will be very trivial: a blog application that allows you to create and view articles and see an impression counter displayed on the article page. The blogging logic will be implemented as a Ruby on Rails application that stores articles on a MySQL Database — while the Impressions logic will be handled by a separate application, written in Elixir, that uses Redis as a key-value store for impressions counter. We are going to use Kafka to get the impressions counter updated.

I will walk you through the setup step by step: from launching minikube on your local machine for the first time to accessing the blog application from your browser and seeing the counter increase.

What we will do in detail

1. Setup Minikube
2. Deploy MySQL
3. Deploy Redis
4. Deploy Kafka
5. Deploy the Impressions API
6. Deploy the Blog Application

After each step, I will provide a small test that you can run to make sure that everything is working for you.

MySQL, Redis, and Kafka will be deployed using Helm — a package manager for Kubernetes applications. Basically, it will allow us to install those servers with a simple command.

For the Impressions API and the Blog Application, I will provide all Kubernetes configuration files needed.

What do we need Kafka for?

Our architecture is composed of several pieces: a MySQL server to store the blog’s articles and a Redis Datastore to manage impressions counters. But why do we need also Kafka?

One database per service rule

To understand why we need Kafka, let’s speak about Data in microservices.

Usually, in a monolith application, you have a single database that can be accessed from every part of your code. This is very straightforward and fast.

But how do you store your data when you have multiple applications?

Imagine you have an eCommerce platform, where one service is responsible for the user profile and another service is responsible for sending email recommendations. The user profile service is responsible for all user data (including name and email), while the recommendation service simply associates products with user ids and uses them to send a recommendation email to the user. The recommendation service needs to access user data — like name and email.

Since the recommendation needs to read the user profile data, the easiest solution would probably be to share the same database credentials and let the recommender service query the user profile directly from the database. This solution looks easy and fast, but problems will start popping up very quickly.

Applications are constantly changing based on new requirements, and often those changes require changes in the database schema too. If many applications directly access the same database, it is very hard to make those changes while ensuring that no other application breaks. In our example, think about what happens if we need to change the “name” field into “firstname” and “lastname”. After changing the schema, the recommender application would crash, because it tries to access the “name” field which no longer exists in the database.

That’s why using a “shared database” is a known antipattern in microservice architecture.

The rule is one database per service.

In other words, every application has its own database which can be accessed directly only by itself. If other services need some information that belongs to that application, they need to ask for it. In our previous example, the recommendation service would ask the user profile service — “Can you give me the name and email of user 123?” — and the user service would return the requested data.

Having the application between the database and the data exposed to the outside allows us to implement a proper deprecation process. For example, when the user profile service changes the single field “name” to “firstname” and “lastname,” it could continue returning the “name” field to the recommender service as a concatenation of “firstname” and “lastname.” Meanwhile, it can inform all other applications to switch to the new fields. Once it knows from the requests it receives that the deprecated field is not used anymore, the old “name” field mapping can be removed.

Communication between services

We know now that every service will have its own database, and the only way to access this data is through the service that owns it. Thus, services need to communicate with each other a lot in order to get and provide data that in a monolith application would be just accessed directly. Since every service can directly access only the part of application data it is responsible for, it will need to continuously integrate this data with data from other services.

In general, there are two types of communications involved: synchronous and asynchronous.

Synchronous communication, or request-response, happens when one service is performing an action and needs information from another service immediately. For example, you have an inventory service that allows you to know the current availability of a product, and another service that acts as the eCommerce frontend. When the frontend service receives the request to render a specific product page, a synchronous call, e.g., a REST call, is made to the inventory service. This call returns the current status for that specific product, which is then presented to the end user.

There are other situations, though, in which this type of communication is not very useful. Imagine you have a search service, which is responsible for indexing all your products so that users can always find available products. This service wants to hide products that are out of stock so that users do not get lost on products that they can’t purchase. Using synchronous communication to check for product availability would mean that the search service would need to make frequent requests to the inventory service — so that it can remove products that got out of stock. This is, of course, not ideal.

In this case, we do not want to continuously ask if a change happened. We want the owner (service) of the data to inform the other applications when something changed so that whoever is interested can react to the change. In this case, we speak about asynchronous communication, or publish-subscribe, where the data owner publishes a message when something on their application changed which could be interesting to other applications.

For example, a product being purchased (→ stock updated → maybe status updated). Other applications can then consume those messages and perform their tasks. Using another example, the search service could read the message about a product being purchased and decide to reindex that product by fetching updated information from all necessary services.

Asynchronous communication can be implemented using different technologies, but the most commonly used are AMQP and Kafka. In this example, we are going to use Kafka.

To sum it up, we said each application has its own database and this data can be accessed only through the application. And since part of this data is needed or even updated by other services, we need to implement communication between services, in particular of two types: REST API endpoints for synchronous communication and Kafka Publisher and Consumers for asynchronous communication.

We will now see step by step how to add needed blocks until we have a working microservice architecture to run our blog application. We will be able to access this application from the browser so that we can see the impressions counter of articles incrementing with each refresh.

Example Application

Now, let’s get to the example application. As said before, this example application will be about a simple blog application that allows you to create and view articles, and see the number of views on each article increase. The blogging logic and the impression count logic will be implemented on two distinct services, which allows you to work on both communication types: synchronous and asynchronous.

The synchronous communication will be needed to display the number of page views of an article on the blog application. This will be done by calling the impressions service via a REST API call.

The asynchronous communication will be needed to inform the impressions service that a page was viewed. In other words, when the blog application receives a request to render an article, it publishes a new message on a Kafka Topic blog.articles.viewed. This will specify the payload of the id of the article and, more precisely, the urn (uniform resource name — a globally unique identifier in our application). The impressions service, which is subscribed to that queue, will consume the message, read the urn in the payload, and increment the associated counter.

I will now walk you through the example step by step. I will not go into too much detail about the actual application’s implementation, because it would make this article much longer, but I will provide all used code in this GitHub repo. This repo will also have the Kubernetes configuration files used to start those two applications.

Step 1. Set Up Minikube

minikube is local Kubernetess; it focuses on making it easy to learn and develop for Kubernetes.

First thing we need to do is to install minikube on our local machine by following the official guide.

If you are running macOS (else check the guide), installing minikube should be as easy as installing it via brew. Use the following command:

$ brew install minikube

After minikube is installed, we can start our first local Kubernetes cluster as follows:

$ minikube start

Test Minikube set up

As promised at the beginning, for each step we will do a test to ensure that everything is working as expected. To test that minikube works, let’s create a temporary test pod as follows:

$ kubectl run -it --rm --restart=Never test-pod --image=alpine sh

If everything works, we should see the new pod’s shell. The --rm option in the command makes sure that the pod gets deleted when we leave the pod, which you can do by typing exit in the shell.

Minikube dashboard

Another way to test that everything works is to start the Kubernetes dashboard as follows:

$ minikube dashboard

This command will open the dashboard in your default browser, from which you can see and manage your Kubernetes cluster. Here’s an image of the dashboard:

Step 2. Deploy MySQL

As you can imagine, I am not the first one trying to deploy a MySQL server on Kubernetes. And like you would look on RubyGems for a Ruby library that solves a common problem in your RubyOnRails application, you can look on https://artifacthub.io/ for Helm Charts that provide the Kubernetes Application you are looking for. In fact, Helm is the package manager for Kubernetes.

But, before we can use Helm we need to install it. On macOS, you can do so using brew:

$ brew install helm

If that’s not an option for you, please check the official guide for more details.

Now that we have Helm installed, let’s use it to install MySQL as follows:

$ helm repo add t3n https://storage.googleapis.com/t3n-helm-charts 

$ helm install mysql t3n/mysql --version 1.0.0

You can find more details about configuration options on the chart page.

After doing this step, this will deploy a MySQL server that can be accessed from the cluster using a hostname mysql. If you want to call it mysql-service, you would have to run helm install mysql-service t3n/mysql --version 1.0.0, and then you would access your MySQL server at host mysql-service.

Test MySQL server

To test that it is working we need first to get the database root password, then we will create a dummy pod running Ubuntu, install a MySQL client, and access the MySQL database.

We could get the mysql_root_passwordusing the minikube dashboard (under “secrets,” find the MySQL entry, and then click on the eye next to mysql-root-password.

Alternatively, we can get the password from the command line using the following commands:

$ kubectl get secret --namespace default mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo

# YOURPASSWORDHERE

Now that we have the password, let’s start a new temporary pod running Ubuntu, install mysql-client, and access our MySQL service using the hostname mysql — the name we gave to the MySQL installation earlier. Here are the commands:

$ kubectl run -it --rm --restart=Never test-pod --image=ubuntu:16.04 sh
# apt-get update && apt-get install mysql-client -y
...
...
... INSTALLING STUFF
...
...

# mysql -h mysql -p
Enter password:

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 22059
Server version: 5.7.32 MySQL Community Server (GPL)

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

Great. We tested that the MySQL server is running and that we can access it from a different pod using mysql as the hostname.

Step 3. Deploy Redis

As with MySQL, Redis can also be deployed using Helm. This time, we pass the installation a setting so that Redis can be accessed without a password. Use the following commands:

$ helm repo add bitnami https://charts.bitnami.com/bitnami

$ helm install redis bitnami/redis --set usePassword=false

Check the chart installation page for more details.

Test Redis server

As before, we want to make sure that the Redis server is running and that it can be accessed from another pod using the hostname redis-mater.

We can again use a temporary pod running Ubuntu, install redis-tools, and try to access our Redis server using the following commands:

$ kubectl run -it --rm --restart=Never test-pod --image=ubuntu:16.04 sh
root@ubuntu:/# apt-get update && apt-get install redis-tools -y
root@ubuntu:/# redis-cli -h redis-master
redis-master:6379> PING
PONG

Done. We ensured that our Redis server is running and that it can be accessed from other pods at the hostname redis-master.

Step 4. Add Kafka Service

The last infrastructure block we want to deploy is Kafka. Luckily, we can do it using Helm (Deploying Kafka manually can be a big headache :D). Here are the commands:

$ helm repo add bitnami https://charts.bitnami.com/bitnami

$ helm install kafka --set replicaCount=1 bitnami/kafka

I manually set the replicationCount to 1, because my local machine was suffering a little bit, and since I don’t really care about Kafka replication in this tutorial I just set it to 1.

You can find more details about all possible installation options on the helm chart page.

Test Kafka deployment

To test that our Kafka Deployment works as expected, we are going to verify that messages published to a topic get consumed correctly using a Kafka (bootstrap) server with the host kafka.

First, let’s create a pod on which we can run a Kafka-consumer or Kafka-producer. To do this, we will use the following commands:

$ kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:2.7.0-debian-10-r35 --namespace default --command -- sleep infinity

Now we can open a shell on this pod and start a consumer on a topic called test-topic using the following commands:

$ kubectl exec -it kafka-client bash
# kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test-topic --from-beginning

Now we open another shell on the pod and use it to produce the new test-topicwith the following commands:

$ kubectl exec -it kafka-client bash
# kafka-console-producer.sh --bootstrap-server kafka:9092 --topic test-topic
> This is my first message

As soon as you hit enter you should see the following message appearing on the consumer shell:

# kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test-topic --from-beginning
This is my first message

Great. We’ve finished the infrastructure setup. We have a MySQL server accessible at host mysql, a Redis server accessible at host redis-master, and a Kaka Broker accessible at kafka:9092.

Now it’s time to add our own applications.

Step 5. Add Impressions Service

The impressions service is an Elixir Application that does two simple things:

1. It consumes from the Kafka topic blog.article.viewed and increments a Redis counter with a key specified in the Kafka message payload
2. It exposes a REST endpoint to get the current count for a specific key

Without going too much into the implementation detail of the application (you can check it out on GitHub), the only thing to note here is that the application needs to interact with Kafka and with Redis. The application assumes the full Kafka URL to be defined in an environment variable called KAFKA_URL, and the Redis host and port to be defined in REDIS_HOST and REDIS_PORT. The docker image of the application is publicly available on Docker Hub as domangi/impressions:v11.

Let’s now define all Kubernetes configurations needed to run the service, which are:

• a configmap to define the environment variables
• a deployment to describe our desired state, which includes which docker image to run and how many replicas to use
• a service to make the application accessible from the cluster using a specific hostname

Impressions ConfigMap

This configmap is named impressions-configmap and will be used in the impressions deployment to access the three needed environment variables.

Impressions Deployment

This will run our impressions server. domangi/impressions is the docker image of the server. With envFrom -> configMapRef we tell the deployment to use the environment variables defined in impressions-configmap in all pods it creates.

Impressions Service

The last step is to define a service that allows other pods in the cluster to access the application “as a service”. With Kubernetes, you don’t need to modify your application to use an unfamiliar service discovery mechanism. Kubernetes gives Pods their own IP addresses and a single DNS name for a set of Pods and can load-balance across them. In this case impressions-service at port 80 will allow you to access one of the impressions pods at port 4000.

We can now apply these configuration files and test our impressions deployment as follows:

$ kubectl apply -f impressions-configmap.yml
$ kubectl apply -f impressions-deployment.yml
$ kubectl apply -f impressions-service.yml

From the Kubernetes dashboard or using the command linekubectl get deployment impressions-deployment, check that all replicas are running before proceeding with the test.

Test Impressions Service

To test the impressions service we are going to start a pod and use curl to interact with the impressions REST API. Then we will use a Kafka client to publish a new message on blog.article.viewed and verify that the counter returned from the REST API is updated as follows:

$ kubectl run -it --rm --restart=Never test-pod --image=ubuntu:16.04 sh

# apt-get update && apt-get install curl -y
...
... installing stuff
...

# curl impressions-service/impressions/blog:article:123
{"urn":"blog:article:123","count":0}

Looks like the REST API is accessible at impressions-service as expected. Now let’s open our kafka-client pod and use a kafka-console-producer to simulate a page view with urn blog:article:123.

### see kafka deployment step if you don't have a kafka-client pod
$ kubectl exec -it kafka-client bash
# kafka-console-producer.sh --bootstrap-server kafka:9092 --topic blog.article.viewed
> blog:article:123

Now let’s call the REST endpoint again for urn blog:article:123 and we should see that the counter has increased:

#  curl impressions-service/impressions/blog:article:123
{"urn":"blog:article:123","count":1}

Yep. We have ensured that our impressions application consumes messages on topic blog.article.viewed and that it increments the counter associated with the urn contained in the published Kafka message — which is returned by the rest endpoint.

Step 6. Add Blog Webapp

Last step to deploy a web application that implements the following blog logic: create and view articles, and track increasing article views through the impressions service. This time we are going to use MySQL as the database, thus we need, apart from the KAFKA_URL, all database environment variables like DATABASE_NAME, DATABASE_HOST, DATABASE_PORT,DATABASE_USERNAME, and DATABASE_PASSWORD. And since we are going to use the impressions-service REST API to fetch the article impressions count to render on the page, we need also to define IMPRESSIONS_SERVICE_URL.

DATABASE_PASSWORD will be accessed directly from the MySQL secret definition we created when adding the MySQL server through Helm. All other environment variables are going to be defined in a configMap.

The docker image of the application is publicly available on Docker Hub as domangi/blog:v5.

Like before, we will create a configMap, which will be used as a deployment and a service.

Blog configmap

This configmap is named blog-configmap and will be used in the blog deployment to access the database environment variables, along with those needed to access Kafka and the Impressions Rest API.

Blog deployment

Here we define our desired number of running instances, two replicas, which will run the image domangi/blog using all environment variables defined in blog-configmap. Then we will add a new DATABASE_PASSWORD env variable, whose value has to be taken from the secret config file called mysql , and in particular, will use the value associated with the key mysql-root-password.

Blog Service

As before, we define a service that works like a DNS for our deployment. In this case, we specify type: LoadBalancer because we want to assign an external-ip address so that we can access it from outside the Kubernetes cluster.

We can now apply these configuration files and test our blog deployment as follows:

$ kubectl apply -f blog-configmap.yml
$ kubectl apply -f blog-deployment.yml
$ kubectl apply -f blog-service.yml

If you check the status of the blog-service, you will realise that the external-ip access never gets assigned:

NAME           TYPE           CLUSTER-IP      EXTERNAL-IP
blog-service   LoadBalancer   10.96.53.11     <pending>

To get the external-ip assigned we need to run minikube tunnel in a new terminal. See the documentation for more details.

Test blog server

In this case, the test will consist of using the application from the browser and checking that the counter increments. You can get the public ip address from the Kubernetes dashboard (in the services section) or by using the command line as follows:

$ kubectl get service blog-service
NAME           TYPE           CLUSTER-IP    EXTERNAL-IP
blog-service   LoadBalancer   10.96.53.11   10.96.53.11

$ open http://10.96.53.11

This will open your browser and show you the blog application where you can create new articles and view them. And if everything worked, you should see an increasing impressions counter.

It works :)

Let’s review what we did. In this article, we did the following:

1. how to set up minikube for local Kubernetes development
2. how to use Helm to install common applications on Kubernetes, like MySQL, Redis, and Kafka
3. how to run your own docker image on Kubernetes, define configmaps, deployments, and services
4. how to make your application accessible from your browser by using a service of type LoadBalancer and how to run minikube tunnel to get the external-ip assigned

And that’s it! I hope you enjoyed this article. Please leave a comment below, and let me know what you think about it!

References

• https://minikube.sigs.k8s.io/docs/start/
• https://codingbee.net/tutorials/kubernetes/using-docker-with-minikube
• https://helm.sh/
• https://github.com/domangi/blog-impressions-k8
• https://www.udemy.com/course/microservices-software-architecture-patterns-and-techniques/

Dealing With Data and Communication in Kubernetes was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

In my experience as a Ruby on Rails Developer, I often had to deal with adding search functionality to web applications. In fact, almost all applications I worked on at some point needed search engine capabilities, while many of them had a search engine as the most important core functionality.

Many applications we use everyday would be useless without a good search engine at their core. For example, on Amazon, you can find a particular product among the more than 550 million products available on the site in a matter of a few seconds — all thanks to a fulltext search combined with category filters, facets, and a recommendation system.

On Airbnb, you can search for an apartment by combining a geospatial search with filters on house characteristics, like dimension, price, available dates, and so on.

And Spotify, Netflix, Ebay, Youtube…all of them rely heavily on a search engine.

In this article, I will describe how to develop a Ruby on Rails 5 API backend with Elasticsearch. According to DB Engines Ranking, Elasticsearch is currently the most popular open source search platform.

This article will not go into the details of Elasticsearch and how it compares to its competitors like Sphinx and Solr. Instead, it will be a step-by-step guide on how to implement a JSON API Backend with Ruby on Rails and Elasticsearch, using a Test Driven Development approach.

This article will cover:

1. Elasticsearch Setup for test, development, and production enviornments
2. Ruby on Rails Test Environment Setup
3. Model indexing with Elasticsearch
4. Search API endpoint

As in my previous article, How to boost your performance with serverless architecture, I will cover everything in a step-by-step tutorial. Then you can try it out yourself and have a simple working example on which to build something more complex.

The example application will be a Movie search engine. It will have a single JSON API endpoint that allows you to make a fulltext search on Movie titles and overviews.

1. Elasticsearch Setup

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected. — www.elastic.co/products/elasticsearch

According to DB-Engines’ Ranking of Search Engines, Elasticsearch is by far the most popular search engine platform today (as of April 2018). And it has been since the end of 2015, when Amazon announced the launch of AWS Elasticsearch Service, a way to start an Elasticsearch cluster from the AWS Management console.

Elasticsearch is opensource. You can download your preferred version from their website and run it wherever you want. While I suggest using the AWS Elasticsearch Service for production enviornments, I prefer having Elasticsearch running on my local machine for testing and developing.

Let’s begin by downloading the (currently) most recent Elasticsearch Version (6.2.3) and unzip it. Open a terminal and run

$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.zip

$ unzip elasticsearch-6.2.3.zip

Alternatively, you can download Elasticsearch from your browser here and unzip it with your preferred program.

2. Test Environment Setup

We are going to build a backend application with Ruby on Rails 5 API. It will have one model that represents Movies. Elasticsearch will index it, and that will be searchable through an API endpoint.

First of all, let’s create a new rails application. In the same folder you downloaded Elasticsearch before, run the command for generating a new rails app. If you are new to Ruby on Rails, please refer to this starting guide to setup your environment first.

$ rails new movies-search --api; cd movies-search

When using the “api” option, all the middleware used primarily for browser applications is not included. Exactly what we want. Read more about it directly on the ruby on rails guide.

Now let’s add all the Gems we will need. Open your Gemfile and add the following code:

# Gemfile

...
# Elasticsearch integration
gem 'elasticsearch-model'
gem 'elasticsearch-rails'

group :development, :test do
  ...
  # Test Framework
  gem 'rspec'
  gem 'rspec-rails'
end

group :test do
  ...
  # Clean Database between tests
  gem 'database_cleaner'
  # Programmatically start and stop ES for tests
  gem 'elasticsearch-extensions'
end
...

We are adding two Elasticsearch Gems that will provide all necessary methods to index our model and run search queries on it. rspec, rspec-rails, database_cleaner, and elasticsearch-extensions are used for testing.

After saving your Gemfile, run bundle install to install all added Gems.

Now let’s configure Rspec by running the following command:

rails generate rspec:install

This command will create a spec folder and add spec_helper.rb and rails_helper.rb to it. They can be used to customize rspec to your application needs.

In this case, we will add a DatabaseCleaner block to rails_helper.rb so that each each test will run in an empty database. Moreover we will modify spec_helper.rb in order to start an Elasticsearch test server each time the test suite is started, and shut it down again once the test suite has finished.

This solution is based on Rowan Oulton’s article Testing Elasticsearch in Rails. Many claps for him!

Let’s start with DatabaseCleaner. Inside spec/rails_helper.rb add the following code:

# spec/rails_helper.rb
...
RSpec.configure do |config|
  ...

config.before(:suite) do
    DatabaseCleaner.strategy = :transaction
    DatabaseCleaner.clean_with(:truncation)
  end

config.around(:each) do |example|
    DatabaseCleaner.cleaning do
      example.run
    end
  end
end

Next, let’s think about the Elasticsearch test server setup. We need to add some configuration files so that Rails knows where to find our Elasticsearch executable. It will also tell it on which port we want it to run, based on the current environment. To do so, add a new configuration yaml to your config folder:

# config/elasticsearch.yml

development: &default
  es_bin: '../elasticsearch-6.2.3/bin/elasticsearch'
  host: 'http://localhost:9200'
  port: '9200'
test:
  es_bin: '../elasticsearch-6.2.3/bin/elasticsearch'
  host: 'http://localhost:9250'
  port: '9250'
staging:
  <<: *default
production:
  es_bin: '../elasticsearch-6.2.3/bin/elasticsearch'
  host: 'http://localhost:9400'
  port: '9400'

If you did not create the rails application in the same folder where you downloaded Elasticsearch, or if you are using a different version of Elasticsearch, you will need to adjust the es_bin path here.

Now add a new file to your initializers folder that will read from the configuration we just added:

# config/initializers/elasticsearch.rb

if File.exists?("config/elasticsearch.yml")
   config = YAML.load_file("config/elasticsearch.yml")[Rails.env].symbolize_keys
   Elasticsearch::Model.client = Elasticsearch::Client.new(config)
end

And finally let’s change spec_helper.rb to include the Elasticsearch test setup. This means start and stop an Elasticsearch test server and create/delete Elasticsearch indexes for our Rails model.

# spec/spec_helper.rb

require 'elasticsearch/extensions/test/cluster'
require 'yaml'

RSpec.configure do |config|
  ...
  # Start an in-memory cluster for Elasticsearch as needed
  es_config = YAML.load_file("config/elasticsearch.yml")["test"]
  ES_BIN = es_config["es_bin"]
  ES_PORT = es_config["port"]

config.before :all, elasticsearch: true do
    Elasticsearch::Extensions::Test::Cluster.start(command: ES_BIN, port: ES_PORT.to_i, nodes: 1, timeout: 120)  unless Elasticsearch::Extensions::Test::Cluster.running?(command: ES_BIN, on: ES_PORT.to_i)
  end

# Stop elasticsearch cluster after test run
  config.after :suite do
    Elasticsearch::Extensions::Test::Cluster.stop(command: ES_BIN, port: ES_PORT.to_i, nodes: 1) if Elasticsearch::Extensions::Test::Cluster.running?(command: ES_BIN, on: ES_PORT.to_i)
  end

# Create indexes for all elastic searchable models
  config.before :each, elasticsearch: true do
    ActiveRecord::Base.descendants.each do |model|
      if model.respond_to?(:__elasticsearch__)
        begin
          model.__elasticsearch__.create_index!
          model.__elasticsearch__.refresh_index!
        rescue => Elasticsearch::Transport::Transport::Errors::NotFound
          # This kills "Index does not exist" errors being written to console
        rescue => e
          STDERR.puts "There was an error creating the elasticsearch index for #{model.name}: #{e.inspect}"
        end
      end
    end
  end

# Delete indexes for all elastic searchable models to ensure clean state between tests
  config.after :each, elasticsearch: true do
    ActiveRecord::Base.descendants.each do |model|
      if model.respond_to?(:__elasticsearch__)
        begin
          model.__elasticsearch__.delete_index!
        rescue => Elasticsearch::Transport::Transport::Errors::NotFound
          # This kills "Index does not exist" errors being written to console
        rescue => e
          STDERR.puts "There was an error removing the elasticsearch index for #{model.name}: #{e.inspect}"
        end
      end
    end
  end

end

We have defined four blocks:

1. a before(:all) block that starts an Elasticsearch test server, unless it is already running
2. an after(:suite) block that stops the Elasticsearch test server, if it is running
3. a before(:each) block that creates a new Elasticsearch index for each model that is configured with Elasticsearch
4. an after(:each) block that deletes all Elasticsearch indexes

Adding elasticsearch: true ensures that only tests tagged with elasticsearch will run these blocks.

I find that this setup works great when you run all your tests once, for example before a deploy. On the other hand, if you’re using a test driven development approach and you run your tests very often, then you will probably need to modify this configuration slightly. You do not want to start and stop your Elasticsearch test server at each test run.

In this case, you could comment out the after(:suite) block where the Test Server is stopped. You can shut it down manually, or using a script, whenever you don’t need it anymore.

require 'elasticsearch/extensions/test/cluster'
es_config = YAML.load_file("config/elasticsearch.yml")["test"]
ES_BIN = es_config["es_bin"]
ES_PORT = es_config["port"]
Elasticsearch::Extensions::Test::Cluster.stop(command: ES_BIN, port: ES_PORT.to_i, nodes: 1)

3. Model indexing with Elasticsearch

Now we start implementing our Movie Model with search capabilities. We use a Test Driven Development approach. This means that we write tests first, see them fail, and then write code to make them pass.

First we need to add the movie model which has four attributes: a title (String), an overview (Text), an image_url(String), and an average vote value (Float).

$ rails g model Movie title:string overview:text image_url:string vote_average:float

$ rails db:migrate

Now it’s time to add Elasticsearch to our model. Let’s write a test that checks that our model is indexed.

# spec/models/movie_spec.rb
require 'rails_helper'

RSpec.describe Movie, elasticsearch: true, :type => :model do
  it 'should be indexed' do
     expect(Movie.__elasticsearch__.index_exists?).to be_truthy
  end
end

This test will check if an elasticsearch index was created for Movie. Remember that before tests begin, we automatically create an elasticsearch index for all models that respond to the __elasticsearch__ method. That means for all models that include the elasticsearch modules.

Run the test to see it fail.

bundle exec rspec spec/models/movie_spec.rb

The first time you run this test, you should see that the Elasticsearch Test Server is starting. The test fails, because we didn’t add any Elasticsearch module to our Movie model. Let’s fix that now. Open the model and add the following Elasticsearch to include:

# app/models/movie.rb

class Movie < ApplicationRecord
  include Elasticsearch::Model
end

This will add some Elasticsearch methods to our Movie model, like the missing __elasticsearch__ method (which generated the error in the previous test run) and the search method we will use later.

Run the test again and see it pass.

bundle exec rspec spec/models/movie_spec.rb

Great. We have an indexed movie model.

By default, Elasticsearch::Model will setup an index with all attributes of the model, automatically inferring their types. Usually this is not what we want. We are now going customize the model index so that it has the following behavior:

1. Only title and overview should be indexed
2. Stemming should be used (which means that searching for “actors” should also return movies that contain the text “actor,” and vice-versa)

We also want our index to be updated each time a Movie is added, updated, or deleted.

Let’s translate this into tests by adding the following code to movie_spec.rb

# spec/models/movie_spec.rb
RSpec.describe Movie, elasticsearch: true, :type => :model do
  ...

describe '#search' do
    before(:each) do
      Movie.create(
        title: "Roman Holiday",
        overview: "A 1953 American romantic comedy films ...",
        image_url: "wikimedia.com/Roman_holiday.jpg",
        vote_average: 4.0
      )
      Movie.__elasticsearch__.refresh_index!
    end
    it "should index title" do
      expect(Movie.search("Holiday").records.length).to eq(1)
    end
    it "should index overview" do
      expect(Movie.search("comedy").records.length).to eq(1)
    end
    it "should not index image_path" do
      expect(Movie.search("Roman_holiday.jpg").records.length).to eq(0)
    end
    it "should not index vote_average" do
      expect(Movie.search("4.0").records.length).to eq(0)
    end
  end

end

We create a Movie before each test, because we configured DatabaseCleaner so that each test is isolated. Movie.__elasticsearch__.refresh_index! is needed to be sure that the new movie record is immediately available for search.

As before, run the test and see it fail.

Seems that our movie is not being indexed. That’s because we didn’t yet tell our model what to do when the movie data changes. Thankfully, this can be fixed by adding another module to our Movie model:

class Movie < ApplicationRecord
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks
end

With Elasticsearch::Model::Callbacks, whenever a movie is added, modified, or deleted, its document on Elasticsearch is also updated.

Let’s see how the test output changes.

Ok. Now the problem is that our search method also returns queries that match on the attributes vote_average and image_url. To fix that we need to configure the Elasticsearch index mapping. So we need to tell Elasticsearch specifically which model attributes to index.

# app/models/movie.rb

class Movie < ApplicationRecord
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks

# ElasticSearch Index
  settings index: { number_of_shards: 1 } do
    mappings dynamic: 'false' do
      indexes :title
      indexes :overview
    end
  end
end

Run the test again and see it pass.

Cool. Now let’s add a stemmer so that there is no difference between “actor” and “actors.” As always, we will first write the test and see it fail.

describe '#search' do
    before(:each) do
      Movie.create(
        title: "Roman Holiday",
        overview: "A 1953 American romantic comedy films ...",
        image_url: "wikimedia.com/Roman_holiday.jpg",
        vote_average: 4.0
      )
      Movie.__elasticsearch__.refresh_index!
    end

...

it "should apply stemming to title" do
      expect(Movie.search("Holidays").records.length).to eq(1)
    end

it "should apply stemming to overview" do
      expect(Movie.search("film").records.length).to eq(1)
    end

end

Note that we are testing both ways: Holidays should return also Holiday, and Film should also return Films.

To make these tests pass again, we need to modify the index mapping. We’ll do that this time by adding an English analyzer to both fields:

class Movie < ApplicationRecord
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks

# ElasticSearch Index
  settings index: { number_of_shards: 1 } do
    mappings dynamic: 'false' do
      indexes :title, analyzer: 'english'
      indexes :overview, analyzer: 'english'
    end
  end
end

Run your tests again to see them pass.

Elasticsearch is a very powerful search platform, and we could add a lot of functionalities to our search method. But this is not within the scope of this article. So we will stop here and move on to building the controller part of the JSON API through which the search method is accessed.

4. Search API endpoint

The Search API we are building should allow users to make a fulltext search on the Movies Table. Our API has a single endpoint defined as follows:

Url: 
 GET /api/v1/movies

Params:
 * q=[string] required

Example url:
 GET /api/v1/movies?q=Roma

Example response:
[{"_index":"movies","_type":"movie","_id":"95088","_score":11.549209,"_source":{"id":95088,"title":"Roma","overview":"A virtually plotless, gaudy, impressionistic portrait of Rome through the eyes of one of its most famous citizens.", "image_url":"https://image.tmdb.org/t/p/w300/rqK75R3tTz2iWU0AQ6tLz3KMOU1.jpg","vote_average":6.6,"created_at":"2018-04-14T10:30:49.110Z","updated_at":"2018-04-14T10:30:49.110Z"}},...]

Here we are defining our endpoint according to some best practices RESTful API Design:

1. The URL should encode the object or resource, while the action to take should be encoded by the HTTP method. In this case, the resource is the movies (collection) and we are using the HTTP method GET (because we are requesting data from the resource without producing any side effect). We use URL parameters to further define how this data should be obtained. In this example, q=[string], which specifies a search query. You can read more about how to design RESTful APIs on Mahesh Haldar’s article RESTful API Designing guidelines — The best practices.
2. We also add versioning to our API by adding v1 to our endpoint URL. Versioning your API is very important, because it allows you to introduce new features that are not compatible with previous releases without breaking all clients that were developed for previous versions of your API.

Ok. Let’s start implementing.

As always, we begin with failing tests. Inside the spec folder, we will create the folder structure that reflects our API endpoint URL structure. This means controllers →api →v1 →movies_spec.rb

You can do this manually or from your terminal running:

mkdir -p spec/controllers/api/v1 && 
touch spec/controllers/api/v1/movies_spec.rb

The tests we are going to write here are controller tests. They do not need to check the search logic defined in the model. Instead we will test three things:

1. A GET request to /api/v1/movies?q=[string] will call Movie.search with [string] as parameter
2. The output of Movie.search is returned in JSON format
3. A success status is returned

A controller test should test controller behavior. A controller test should not fail because of problems in the model .

(Prescription 20 — Rails 4 Test Prescriptions. Noel Rappin)

Let’s transform this into code. Inside spec/controllers/api/v1/movies_spec.rb add the following code:

# spec/controllers/api/v1/movies_spec.rb
require 'rails_helper'
RSpec.describe Api::V1::MoviesController, type: :request do
  # Search for movie with text movie-title
  describe "GET /api/v1/movies?q=" do
    let(:title) { "movie-title"}
    let(:url) { "/api/v1/movies?q=#{title}"}

it "calls Movie.search with correct parameters" do
      expect(Movie).to receive(:search).with(title)
      get url
    end

it "returns the output of Movie.search" do
      allow(Movie).to receive(:search).and_return({})
      get url
      expect(response.body).to eq({}.to_json)
    end

it 'returns a success status' do
      allow(Movie).to receive(:search).with(title)
      get url
      expect(response).to be_successful
    end
  end
end

The test will immediately fail because Api::V1::MoviesController is not defined, so let’s do that first. Create the folder structure as before and add the movies controller.

mkdir -p app/controllers/api/v1 && 
touch app/controllers/api/v1/movies_controller.rb

Now add the following code to app/controllers/api/v1/movies_controller.rb:

# app/controllers/api/v1/movies_controller.rb
module Api
  module V1
    class MoviesController < ApplicationController
      def index;end
    end
  end
end

It’s time to run our test and see it fail.

All tests fail because we still need to add a route for the endpoint. Inside config/routes.rb add the following code:

# config/routes.rb
Rails.application.routes.draw do
  namespace :api do
    namespace :v1 do
      resources :movies, only: [:index]
    end
  end
end

Rerun your tests and see what happens.

The first error tells us we need to add a call to Movie.search inside our controller. The second one complains about the response. Let’s add the missing code to the movies_controller:

# app/controllers/api/v1/movies_controller.rb
module Api
  module V1
    class MoviesController < ApplicationController
      def index
        response = Movie.search params[:q]
        render json: response
      end
    end
  end
end

Run the test and see if we are done.

Yup. That’s all. We have completed a really basic backend application that allows users to search a model through API.

You can find the complete code on my GitHub repo here. You can populate your Movie table with some data by running rails db:seed so that you can see the application in action. This will import circa 45k Movies from a Dataset downloaded from Kaggle. Take a look at the Readme for more details.

If you enjoyed this article, please recommend it by hitting the clap icon that you’ll find at the bottom of this page so that more people can see it on Medium.

How to add a powerful search engine to your Rails backend was originally published in We’ve moved to freeCodeCamp.org/news on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, I am going to describe how I moved a heavy task like image pre-processing from my application server to a completely serverless architecture on AWS in charge of storing, processing and serving images.

The Problem

Image pre-processing is a task required by many web applications. Each time an application allows a user to upload an image, it is very likely that this image needs to be pre-processed before it is served to a front-end application.

In this article I am going to describe a serverless architecture based on AWS, that is extremely scalable and cost-efficient.

But let’s start from the beginning. In one of my last projects, a marketplace web application where users have to upload an image of a product they want to sell, the original image is first cropped to the correct image ratio (4:3). It is then transformed in three different formats used in different places of the front-end application: 800x600px, 400x300px, and 200x150px.

Being a Ruby on Rails developer, my first approach was to use a RubyGem — in particular Paperclip or Dragonfly, which both make use of ImageMagick for image processing.

Although this implementation is quiet straightforward (since it it mostly just configuration), there are different drawbacks that could arise:

1. The images are processed on the application server. This could increase the general response time because of the greater workload on the CPU
2. The application server has limited computing power, which is set upfront, and is not well-suited for burst request handling. If many images need to be processed at the same time, the server capacity could be exhausted for a long period of time. Increasing the computing power on the other side would result in higher costs.
3. Images are processed in sequence. Again, if many images need to be processed at the same time, speed could be very bad.
4. If not correctly configured, these gems save processed images on disk, which could quickly make your server run out of space.

In general, based on how much image processing your application does, this solution is not scalable.

The Solution

Having a closer look to the image pre-processing task, you’ll notice that there is probably no need to run it directly on your application server. In particular this is the case if your image transformations are always the same and do not rely on other information than the image itself. This was the case for me, where I always generated different image sizes together with an image quality/weight optimization.

Once you realize that this task can be easily isolated from the rest of the application logic, thinking about a serverless solution that just takes an original image as input and generates all needed transformations is straightforward.

AWS Lambda turns out to be a perfect fit for this kind of problem. On the one side, it can handle thousands of requests per second, and on the other side, you pay only for the compute time you consume. There is no charge when your code is not running.

AWS S3 provides unlimited storage at a very low price, while AWS SNS provides an easy way of Pub/Sub messaging for microservices, distributed systems, and serverless applications. Finally, AWS Cloudfront is used as the Content Delivery Network for the images stored on S3.

The combination of these four AWS services results in a very powerful image processing solution at a very low cost.

High Level Architecture

The process of generating different image versions from an original image starts with an upload of the original image on AWS S3. This triggers, through AWS SNS, the execution of an AWS Lambda function in charge of generating the new image versions and uploading them again on AWS S3. Here is the sequence in more detail:

1. Images are uploaded to a specific folder inside an AWS S3 bucket
2. Each time a new image is uploaded to this folder, S3 publishes a message containing the S3 key of the created object on an AWS SNS topic
3. AWS Lambda, which is configured as consumer on the same SNS topic, reads the new message and uses the S3 object key to fetch the new image
4. AWS Lambda processes the new image, applying the necessary transformations, and uploads the processed image(s) to S3
5. The processed images are now served to the final users through AWS Cloudfront CDN, in order to optimize the download speed.

This architecture is very scalable, since each uploaded image will trigger a new Lambda code execution to handle just that request, so that there can be thousands of images being processed in parallel by as many code executions.

No disk space or computation power is used on the application server, because everything is stored on S3 and processed by Lambda.

Finally, configuring a CDN in front of S3 is very easy and allows you to have high download speeds from everywhere in the world.

Step-by-Step Tutorial

The implementation of this solution is relatively easy, since it is mostly configuration, except for the Lambda code that performs the image pre-processing. The rest of this article will describe in detail how to setup the AWS architecture, and will provide the code executed by AWS Lambda to resize the uploaded image in order to have a complete working example.

To try it out yourself, you will need an AWS account. If you don’t have one, you can create one for free and take advantage of the AWS Free Tier here.

Step 1: Create a Topic on AWS SNS

First of all, we need to configure a new SNS (Simple Notification Service) topic on which AWS will publish a message each time a new image is uploaded to S3. This message contains the S3 object key used later by the Lambda function to fetch the uploaded image and process it.

From your AWS console visit the SNS page, click on “Create topic,” and enter a topic name, for example “image-preprocessing.”

Next, we need to change the topic policy to allow our S3 bucket to publish messages on it.

From the topic page, click on Actions -> Edit Topic Policy, choose Advanced view, add the following JSON block (with your own arns for Resource and SourceArn) to the statement array and update the policy:

{
      "Sid": "ALLOW_S3_BUCKET_AS_PUBLISHER",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "SNS:Publish",
      ],
      "Resource": "arn:aws:sns:us-east-1:AWS-OWNER-ID:image-preprocessing",
      "Condition": {
          "StringLike": {
              "aws:SourceArn": "arn:aws:s3:*:*:YOUR-BUCKET-NAME"
          }
      }
}

You can find an example of a complete policy JSON here.

Step 2: Create AWS S3 folder structure

Now we need to prepare the folder structure on S3 that will contain the original and the processed images. In this example, we will generate two resized image versions, 800x600 and 400x300.

From your AWS console, open the S3 page and create a new bucket. I will call mine “image-preprocessing-example.” Then, inside the bucket, we need to create a folder named “originals,” a folder named “800x600,” and another named “400x300.”

Step 3: Configure AWS S3 Events

Every time a new image is uploaded to the originals folder, we want S3 to publish a message on our “image-preprocessing” SNS topic so that the image can be processed.

To do that, open your S3 bucket from the AWS console, click on Properties -> Events -> + Add notification and fill in the following fields:

Here we are telling S3 to generate an event each time a new object is created (ObjectCreate) inside the originals folder (prefix), and to publish this event on our SNS Topic “image-preprocessing.”

Step 4: Configure IAM role to allow Lambda to access the S3 folder

We want to create a Lambda function that fetches image objects from S3, processes them, and uploads the processed versions again to S3. To do that, we need first to setup an IAM role that will allow our Lambda function to access the needed S3 folder.

From the AWS Console IAM page:

1. Click on Create Policy
2. Click on JSON and type in (replace YOUR-BUCKET-NAME)

{
      "Version": "2012-10-17",
      "Statement": [
          {
              "Sid": "Stmt1495470082000",
              "Effect": "Allow",
              "Action": [
                  "s3:*"
              ],
              "Resource": [
                  "arn:aws:s3:::YOUR-BUCKET-NAME/*"
              ]
          }
      ]
}

where the resource is our bucket on S3. Click on review, enter the policy name, for example AllowAccessOnYourBucketName, and create the policy.

3. Click on Roles -> Create role
4. Choose Aws Service -> Lambda (who will use the policy)
5. Select the previously created policy (AllowAccessOnYourBucketName)
6. Finally, click on review, type in a name (LambdaS3YourBucketName), and click create role

Step 5: Create the AWS Lambda function

Now we have to setup our Lambda function to consume messages from the “image-preprocessing” SNS Topic and generate our resized image versions.

Let’s start with creating a new Lambda function.

From your AWS console, visit the Lambda page, click on “Create function,” and type in your function name, for example ImageResize, choose your runtime, in this case Node.js 6.10, and the previously created IAM role.

Next we need to add SNS to the function triggers, so that the Lambda function will be called each time a new message is published to the “image-preprocessing” topic.

To do that, click on “SNS” in the list of triggers, select “image-preprocessing” from the SNS topic list, and click “add.”

Finally we have to upload our code that will handle the S3 ObjectCreated event. That means fetching the uploaded image from the S3 originals folder, processing it, and uploading it again in the resized image folders.

You can download the code here. The only file you need to upload to your Lambda function is version1.1.zip, which contains index.js and the node_modules folder.

In order to give the Lambda function enough time and memory to process the image, we can increase the memory to 256 MB and the timeout to 10 sec. The needed resources depend on the image size and the transformation complexity.

The code itself is quiet simple, and just has the purpose of demonstrating the AWS integration.

First, a handler function is defined (exports.handler). This function is called by the external trigger, in this case the message published on SNS which contains the S3 object key of the uploaded image.

It first parses the event message JSON to extract the S3 bucket name, the S3 object key of the uploaded image, and the filename that is just the final part of the key.

Once it has the bucket and object key, the uploaded image is fetched using s3.getObject and then passed to the resize function. The SIZE variable holds the image sizes we want to generate, which correspond also to the S3 folder names where the transformed images will be uploaded.

var async = require('async');
var AWS = require('aws-sdk');
var gm = require('gm').subClass({ imageMagick: true });
var s3 = new AWS.S3();

var SIZES = ["800x600", "400x300"];

exports.handler = function(event, context) {
    var message, srcKey, dstKey, srcBucket, dstBucket, filename;
    message = JSON.parse(event.Records[0].Sns.Message).Records[0];

srcBucket = message.s3.bucket.name;
    dstBucket = srcBucket;
    srcKey    =  message.s3.object.key.replace(/\+/g, " "); 
    filename = srcKey.split("/")[1];
    dstKey = ""; 
    ...
    ...
    // Download the image from S3
    s3.getObject({
            Bucket: srcBucket,
            Key: srcKey
    }, function(err, response){
        if (err){
            var err_message = 'Cannot download image: ' + srcKey;
            return console.error(err_message);
        }
        var contentType = response.ContentType;

        // Pass in our image to ImageMagick
        var original = gm(response.Body);

        // Obtain the size of the image
        original.size(function(err, size){
            if(err){
                return console.error(err);
            }

            // For each SIZES, call the resize function
            async.each(SIZES, function (width_height,  callback) {
                var filename = srcKey.split("/")[1];
                var thumbDstKey = width_height +"/" + filename;
                resize(size, width_height, imageType, original,   
                       srcKey, dstBucket, thumbDstKey, contentType, 
                       callback);
            },
            function (err) {
                if (err) {
                    var err_message = 'Cannot resize ' + srcKey;
                    console.error(err_message);
                }
                context.done();
            });
        });
    });

}

The resize function applies some transformations on the original image using the “gm” library, in particular it resizes the image, crops it if needed, and reduces the quality to 80%. It then uploads the modified image to S3 using “s3.putObject”, specifying “ACL: public-read” to make the new image public.

var resize = function(size, width_height, imageType, 
                      original, srcKey, dstBucket, dstKey, 
                      contentType, done) {

    async.waterfall([
        function transform(next) {
            var width_height_values = width_height.split("x");
            var width  = width_height_values[0];
            var height = width_height_values[1];

            // Transform the image buffer in memory
            original.interlace("Plane")
                .quality(80)
                .resize(width, height, '^')
                .gravity('Center')
                .crop(width, height)
                .toBuffer(imageType, function(err, buffer) {
                if (err) {
                    next(err);
                } else {
                    next(null, buffer);
                }
            });
        },
        function upload(data, next) {
            console.log("Uploading data to " + dstKey);
            s3.putObject({
                    Bucket: dstBucket,
                    Key: dstKey,
                    Body: data,
                    ContentType: contentType,
                    ACL: 'public-read'
                },
                next);
            }
        ], function (err) {
            if (err) {
                console.error(err);
            }
            done(err);
        }
    );
};

Step 6: Test

Now we can test that everything is working as expected by uploading an image to the originals folder. If everything was implemented correctly, then we should find a resized version of the uploaded image in the 800x600 folder and one in the 400x300 folder.

In the video below, you can see three windows: on the left the originals folder, in the middle the 800x600 folder, and on the right the 400x300 folder. After uploading a file to the original folder, the other two windows are refreshed to check if the images were created.

And voilà, here they are ;)

(Optional) Step 6: Add Cloudfront CDN

Now that the images are generated and uploaded to S3, we can add Cloudfront CDN to deliver the images to our end users, so that download speed is improved.

1. Open the Cloudfront Page
2. Click on “Create Distribution”
3. When asked for the delivery method, choose “Web Distribution”
4. Choose your S3 bucket as “Origin Domain Name” and click on “Create Distribution”

The process of creating the distribution network is not immediate, so you will have to wait until the status of your CDN changes from “In Prog” to “Deployed.”

Once it is deployed you can use the domain name instead of your S3 bucket URL. For example if your Cloudfront domain name is “1234-cloudfront-id.cloudfront.net”, then you can access your resized image folder by “https://1234-cloudfront-id.cloudfront.net/400x300/FILENAME” and “https://1234-cloudfront-id.cloudfront.net/800x600/FILENAME”

Cloudfront has many other options that should be set, but those are out of the scope of this article. For a more detailed guide to setting up your CDN, take a look at Amazon’s getting started guide.

And that’s it! I hope you enjoyed this article. Please leave a comment below, and let me know what you think!

How to boost your performance with serverless architectures was originally published in We’ve moved to freeCodeCamp.org/news on Medium, where people are continuing the conversation by highlighting and responding to this story.