Chapter 16
The First Query Builder: Paginator
In this chapter, we are going to build the first query builder. While doing this, we’ll run into a few things that are missing, so we’ll implement those along the way.
16.1. The Paginator builder
Query builders are used to scope down a list of entities based on the parameters requested by the client. In Alexandria, we want to provide 3 ways for the client to do that: pagination, sorting and filtering.
First, pagination will allow the client to only get a subset of the results using two parameters: page and per. Sorting will let a client decide how the list of entities should be organized, following the identifier, the released date or something else. Finally, filtering will be used to only get a subset of entities that are relevant to the given parameters.
To create the pagination query builder, we don’t need to update our presenters. Since we will be using global default values for the page and per parameters, there is no need to define those values in the presenters.
Currently, if we ask for the list of books, we get the entire list back. That’s fine for now because we only have 3 books, but for 3000 it won’t be okay anymore.
To fix that, we are going to build the pagination query builder that will rely on a neat little gem known as Kaminari. Just like will_paginate, this gem takes care of paginating a list of records.
16.2. Pagination with Kaminari
Let’s add Kaminari to the Gemfile…
# Gemfile
source 'https://rubygems.org'
git_source(:github) { |repo| "https://github.com/#{repo}.git" }
ruby '2.5.0'
gem 'rails', '5.2.0'
gem 'sqlite3'
gem 'puma', '~> 3.11'
gem 'bootsnap', '>= 1.1.0', require: false
gem 'carrierwave'
gem 'carrierwave-base64'
# Add the kaminari gem
gem 'kaminari'
group :development, :test # Hidden Code
group :development # Hidden Code
group :test # Hidden Code
and run bundle install to get it installed.
Next, we need to create a bunch of folders for the query builders.
mkdir app/query_builders && mkdir spec/query_builders
Now, add the files for the Paginator builder and its tests with the command below.
touch app/query_builders/paginator.rb && \
touch spec/query_builders/paginator_spec.rb
In the paginator.rb file we just created, put the code below. It’s the skeleton of the class we are about to build.
# app/query_builders/paginator.rb
class Paginator
def initialize(scope, params, url)
end
end
Before we implement the rest, let’s write a bunch of tests to define the expectations of the Paginator. Once instantiated, this class will have the method paginate, which will return the paginated scope we passed in. To be able to write tests for it though, we first need to define some variables.
All these variables does is create a list of 3 books (the same as before) for us to play with. The other let calls are there to instantiate the Paginator class.
We don’t extensively test the pagination itself because we rely on the tests implemented in Kaminari itself to ensure that it works correctly. Here we just want to ensure that, when given specific parameters, we get the corresponding subset of entities.
# spec/query_builders/paginator_spec.rb
require 'rails_helper'
RSpec.describe Paginator do
let(:ruby_microscope) { create(:ruby_microscope) }
let(:rails_tutorial) { create(:ruby_on_rails_tutorial) }
let(:agile_web_dev) { create(:agile_web_development) }
let(:books) { [ruby_microscope, rails_tutorial, agile_web_dev] }
let(:scope) { Book.all }
let(:params) { { 'page' => '1', 'per' => '2' } }
let(:paginator) { Paginator.new(scope, params, 'url') }
let(:paginated) { paginator.paginate }
before do
books
end
describe '#paginate' do
it 'paginates the collection with 2 books' do
expect(paginated.size).to eq 2
end
it 'contains ruby_microscope as the first paginated item' do
expect(paginated.first).to eq ruby_microscope
end
it 'contains rails_tutorial as the last paginated item' do
expect(paginated.last).to eq rails_tutorial
end
end
end
Try to run the tests.
rspec spec/query_builders/paginator_spec.rb
Of course it fails, since we haven’t implemented the code yet.
Failure (RED)
...
Finished in 0.24135 seconds (files took 2.23 seconds to load)
3 examples, 3 failures
...
We can fix it by adding a bunch of code to the Paginator class. We need to ensure that the paginate method returns the paginated scope. To do so, we need to extract the page and per parameters from the query parameters. Don’t pay too much attention to the url parameter, we will be using it to generate some metadata in the next section.
# app/query_builders/paginator.rb
class Paginator
def initialize(scope, query_params, url)
@query_params = query_params
@page = @query_params['page'] || 1
@per = @query_params['per'] || 10
@scope = scope
@url = url
end
def paginate
@scope.page(@page).per(@per)
end
end
Let’s run the tests.
rspec spec/query_builders/paginator_spec.rb
Success (GREEN)
...
Paginator
#paginate
paginates the collection with 2 books
contains ruby_microscope as the first paginated item
contains rails_tutorial as the last paginated item
Finished in 0.30235 seconds (files took 2.18 seconds to load)
3 examples, 0 failures
Yes, great! However, we need to provide something else. We need to give more information to the client about the pagination and how to access the other pages.
16.3. Pagination Metadata
Instead of letting the client figure out everything on its own, we can provide it with a few useful links to go through the paginated collection. For example, things like the first page, previous page, next page and last page would be very useful. This kind of metadata is usually added somewhere in the JSON document and then sent back, so we could do it that way. But an emerging best practice is actually to put those links in the Link header, so we’ll do that now.
Let’s add more tests to the paginator_spec file. We need to ensure that the links method in the Paginator class returns a string containing the first/previous/next/last URLs formatted in the correct way:
<http://localhost:4567/api/books?page=X&per=Y>; rel="first|previous|next|last"
# spec/query_builders/paginator_spec.rb
require 'rails_helper'
RSpec.describe Paginator do
# Hidden Code: let definitions
describe '#paginate' # Hidden Code
describe '#links' do
let(:links) { paginator.links.split(', ') }
context 'when first page' do
let(:params) { { 'page' => '1', 'per' => '2' } }
it 'builds the "next" relation link' do
expect(links.first).to eq '<url?page=2&per=2>; rel="next"'
end
it 'builds the "last" relation link' do
expect(links.last).to eq '<url?page=2&per=2>; rel="last"'
end
end
context 'when last page' do
let(:params) { { 'page' => '2', 'per' => '2' } }
it 'builds the "first" relation link' do
expect(links.first).to eq '<url?page=1&per=2>; rel="first"'
end
it 'builds the "previous" relation link' do
expect(links.last).to eq '<url?page=1&per=2>; rel="prev"'
end
end
end # describe '#links' end
end
If we run the tests for this file, we get a bunch of failures.
rspec spec/query_builders/paginator_spec.rb
Failure (RED)
...
Finished in 0.39977 seconds (files took 2.2 seconds to load)
7 examples, 4 failures
...
Let’s add the missing code to the Paginator to make all those tests pass. Since there are many additions, going through each one of them before looking at the whole class should be easier.
The first thing we need is a few methods to check which links should be included. A client asking for the first page (?page=1&per=Y) doesn’t really care about the pagination link of the first page or the previous page (there are no previous pages!). However, the client would be interested in getting the next page and the last one.
def show_first_link?
@scope.total_pages > 1 && !@scope.first_page?
end
def show_previous_link?
!@scope.first_page?
end
def show_next_link?
!@scope.last_page?
end
def show_last_link?
@scope.total_pages > 1 && !@scope.last_page?
end
Next, we need a method that’s going to build a hash with each link. This method will be the only one using the checkers we created above.
def pages
@pages ||= {}.tap do |h|
h[:first] = 1 if show_first_link?
h[:prev] = @scope.current_page - 1 if show_previous_link?
h[:next] = @scope.current_page + 1 if show_next_link?
h[:last] = @scope.total_pages if show_last_link?
end
end
Finally, the links method will be a public method. This method will be called on an instantiated Paginator at the controller level to set the Link header.
def links
@links ||= pages.each_with_object([]) do |(k, v), links|
query_params = @query_params.merge({ 'page' => v, 'per' => @per }).to_param
links << "<#{@url}?#{query_params}>; rel=\"#{k}\""
end.join(', ')
end
Here is the complete updated class.
# app/query_builders/paginator.rb
class Paginator
def initialize(scope, query_params, url)
@query_params = query_params
@page = @query_params['page'] || 1
@per = @query_params['per'] || 10
@scope = scope.page(@page).per(@per)
@url = url
end
def paginate
@scope
end
def links
@links ||= pages.each_with_object([]) do |(k, v), links|
query_params = @query_params.merge({ 'page' => v, 'per' => @per }).to_param
links << "<#{@url}?#{query_params}>; rel=\"#{k}\""
end.join(", ")
end
private
def pages
@pages ||= {}.tap do |h|
h[:first] = 1 if show_first_link?
h[:prev] = @scope.current_page - 1 if show_previous_link?
h[:next] = @scope.current_page + 1 if show_next_link?
h[:last] = @scope.total_pages if show_last_link?
end
end
def show_first_link?
@scope.total_pages > 1 && !@scope.first_page?
end
def show_previous_link?
!@scope.first_page?
end
def show_next_link?
!@scope.last_page?
end
def show_last_link?
@scope.total_pages > 1 && !@scope.last_page?
end
end
If we run the tests now, everything should work properly!
rspec spec/query_builders/paginator_spec.rb
Success (GREEN)
...
Paginator
#paginate
paginates the collection with 2 books
contains ruby_microscope as the first paginated item
contains rails_tutorial as the last paginated item
#links
when first page
builds the "next" relation link
builds the "last" relation link
when last page
builds the "first" relation link
builds the "previous" relation link
Finished in 0.69567 seconds (files took 2.73 seconds to load)
7 examples, 0 failures
16.4. Pagination Helper Method
The Paginator seems to be ready. To make it easier to reuse it across different controllers, we are going to add a method in the ApplicationController that will deal with instantiating a paginator and setting the Link header.
# app/controllers/application_controller.rb
class ApplicationController < ActionController::API
protected
def paginate(scope)
paginator = Paginator.new(scope, request.query_parameters, current_url)
response.headers['Link'] = paginator.links
paginator.paginate
end
def current_url
request.base_url + request.path
end
end
16.5. Updating the Books Controller
Before we update the books controller, we can write more tests in the spec/requests/books_spec.rb file to test the pagination. In those tests, we need to check if we are correctly getting the paginated subset of entities and if the Link header is correctly set depending on the page we requested.
# spec/requests/books_spec.rb
require 'rails_helper'
RSpec.describe 'Books', type: :request do
# Hidden Code: let definitions
describe 'GET /api/books' do
before { books }
context 'default behavior' # Hidden Code
describe 'field picking' # Hidden Code
describe 'pagination' do
context 'when asking for the first page' do
before { get('/api/books?page=1&per=2') }
it 'receives HTTP status 200' do
expect(response.status).to eq 200
end
it 'receives only two books' do
expect(json_body['data'].size).to eq 2
end
it 'receives a response with the Link header' do
expect(response.headers['Link'].split(', ').first).to eq(
'<http://www.example.com/api/books?page=2&per=2>; rel="next"'
)
end
end
context 'when asking for the second page' do
before { get('/api/books?page=2&per=2') }
it 'receives HTTP status 200' do
expect(response.status).to eq 200
end
it 'receives only one book' do
expect(json_body['data'].size).to eq 1
end
end
end # describe 'pagination' end
end
end
Try to run the tests… then watch them fail miserably
rspec spec/requests/books_spec.rb
Failure (RED)
...
Finished in 0.61217 seconds (files took 2.16 seconds to load)
10 examples, 3 failures
...
Luckily, we can easily fix those tests by simply calling the paginate method (paginate(Book.all)) in the books controller.
# app/controllers/books_controller.rb
class BooksController < ApplicationController
def index
books = paginate(Book.all).map do |book|
FieldPicker.new(BookPresenter.new(book, params)).pick
end
render json: { data: books }.to_json
end
end
Let’s try running the tests again.
rspec spec/requests/books_spec.rb
Success (GREEN)
...
Books
GET /api/books
gets HTTP status 200
receives a json with the "data" root key
receives all 3 books
field picking
with the fields parameter
gets books with only the id, title and author_id keys
without the field parameter
gets books with all the fields specified in the presenter
pagination
when asking for the first page
receives HTTP status 200
receives only two books
receives a response with the Link header
when asking for the second page
receives HTTP status 200
receives only one book
Finished in 1.12 seconds (files took 1.81 seconds to load)
10 examples, 0 failures
Success, yay!
16.6. Returning Errors
What if a client starts sending incorrect values for the page or per parameters? For instance using a word instead of a number? We cannot let that happen.
In those situations, we should send back 400 Bad Request because the client made a mistake when building the request and we cannot process it. Add a small test to ensure that we get 400 when calling /api/books?page=fake&per=10.
# spec/requests/books_spec.rb
require "rails_helper"
RSpec.describe "Books", type: :request do
# Hidden Code: let definitions
describe 'GET /api/books' do
before { books }
context 'default behavior' # Hidden Code
describe 'field picking' # Hidden Code
describe 'pagination' do
context 'when asking for the first page' # Hidden Code
context 'when asking for the second page' # Hidden Code
context "when sending invalid 'page' and 'per' parameters" do
before { get('/api/books?page=fake&per=10') }
it 'receives HTTP status 400' do
expect(response.status).to eq 400
end
end
end # describe 'pagination' end
end
end
rspec spec/requests/books_spec.rb
Failure (RED)
1) Books GET /api/books pagination when sending invalid 'page' and 'per'
parameters receives HTTP status 400
Failure/Error: expect(response.status).to eq 400
expected: 400
got: 200
...
Finished in 0.78048 seconds (files took 2.22 seconds to load)
11 examples, 1 failure
Of course the new test is failing. We are currently not checking if the parameters are valid. To fix this test, we are going to create our first custom error, QueryBuilderError.
Let’s store our custom errors in the app/errors folder.
mkdir app/errors && touch app/errors/query_builder_error.rb
Here is the content of this new class that inherits from StandardError.
# app/errors/query_builder_error.rb
class QueryBuilderError < StandardError
attr_accessor :invalid_params
def initialize(invalid_params)
@invalid_params = invalid_params
super
end
end
Next, we need a method in the Paginator class to ensure that the given query parameters are valid. That’s the responsibility of the validate_param method that will take two arguments: the parameter being checked (page or per) and the default value for that specific parameter. It will check if the value sent by the client is a number using a simple regular expression and raise the QueryBuilderError error if that’s not the case.
# app/query_builders/paginator.rb
class Paginator
def initialize(scope, query_params, url)
@query_params = query_params
@page = validate_param!('page', 1)
@per = validate_param!('per', 10)
@scope = scope.page(@page).per(@per)
@url = url
end
def paginate # Hidden Code
def links # Hidden Code
private
def validate_param!(name, default)
return default unless @query_params[name]
unless (@query_params[name] =~ /\A\d+\z/)
raise QueryBuilderError.new("#{name}=#{@query_params[name]}"),
'Invalid Pagination params. Only numbers are supported for "page" and "per".'
end
@query_params[name]
end
def pages # Hidden Code
def show_first_link? # Hidden Code
def show_previous_link? # Hidden Code
def show_next_link? # Hidden Code
def show_last_link? # Hidden Code
end
How is our test doing now?
rspec spec/requests/books_spec.rb
Failure (RED)
1) Books GET /api/books pagination when sending invalid "page" and "per"
parameters receives HTTP status 400
Failure/Error: before { get('/api/books?page=fake&per=10') }
QueryBuilderError:
Invalid Pagination params. Only integers supported for 'page' and 'per'.
It’s correctly raising the error, but this is not good… It’s just going to tell the client that something went wrong in the API without many details. We should return a JSON document explaining the problem.
To be able to return a JSON document when the error is raised, we need to catch it at the controller level. The good news is that it’s pretty simple using something like this:
rescue_from QueryBuilderError, with: :handling_method
Let’s add this line to the ApplicationController with a handling method named query_builder_error that will return a response containing the error message and the invalid parameters.
# app/controllers/application_controller.rb
class ApplicationController < ActionController::API
rescue_from QueryBuilderError, with: :query_builder_error
protected
def query_builder_error(error)
render status: 400, json: {
error: {
message: error.message,
invalid_params: error.invalid_params
}
}
end
def paginate(scope)
paginator = Paginator.new(scope, request.query_parameters, current_url)
response.headers['Link'] = paginator.links
paginator.paginate
end
def current_url
request.base_url + request.path
end
end
Let’s run our tests to see if we are correctly getting 400 Bad Request now.
rspec spec/requests/books_spec.rb
Success (GREEN)
...
Finished in 0.94986 seconds (files took 2.68 seconds to load)
11 examples, 0 failures
Before we proceed, it’s probably a good idea to add a few more tests checking the error document being returned.
# spec/requests/books_spec.rb
require "rails_helper"
RSpec.describe "Books", type: :request do
# Hidden Code: let definitions
describe 'GET /api/books' do
before { books }
context 'default behavior' # Hidden Code
describe 'field picking' # Hidden Code
describe 'pagination' do
context 'when asking for the first page' # Hidden Code
context 'when asking for the second page' # Hidden Code
context "when sending invalid 'page' and 'per' parameters" do
before { get('/api/books?page=fake&per=10') }
it 'receives HTTP status 400' do
expect(response.status).to eq 400
end
it 'receives an error' do
expect(json_body['error']).to_not be nil
end
it "receives 'page=fake' as an invalid param" do
expect(json_body['error']['invalid_params']).to eq 'page=fake'
end
end
end # describe 'pagination' end
end
end
rspec spec/requests/books_spec.rb
...
Finished in 0.85853 seconds (files took 2.93 seconds to load)
13 examples, 0 failures
We are now handling errors and will be able to reuse this system for the other query builders.
16.7. Preparing For the Arrival of More Builders
We didn’t have to change our presenter for the pagination. In order to implement sorting and filtering, though, we need a place for each model to define which fields can be used for those two features.
16.7.1. Updating the BasePresenter Class
Just like for the field picking feature, we need a way in the presenters to define a list of attributes that can be used. First, let’s write some tests! They are going to be super similar to the one we wrote for the build_with class method because they will work in exactly the same way.
# spec/presenters/base_presenter_spec.rb
require 'rails_helper'
RSpec.describe BasePresenter do
class Presenter < BasePresenter; end
describe '#initialize' # Hidden code
describe '#as_json' # Hidden code
describe '.build_with' # Hidden Code
describe '.related_to' do
it 'stores the correct value' do
Presenter.related_to :author, :publisher
expect(Presenter.relations).to eq ['author', 'publisher']
end
end
describe '.sort_by' do
it 'stores the correct value' do
Presenter.sort_by :id, :title
expect(Presenter.sort_attributes).to eq ['id', 'title']
end
end
describe '.filter_by' do
it 'stores the correct value' do
Presenter.filter_by :title
expect(Presenter.filter_attributes).to eq ['title']
end
end
end
First step of the TDD cycle: run the tests.
rspec spec/presenters/base_presenter_spec.rb
Failure (RED)
...
Finished in 0.06466 seconds (files took 2.17 seconds to load)
8 examples, 3 failures
...
To make those tests pass, let’s implement the same logic used for build_attributes. This time, the new attributes are sort_attributes, filter_attributes and relations. relations will be used later for the representation builder that will take care of embedding other entities, but we can already add it.
# app/presenters/base_presenter.rb
class BasePresenter
# Add more class instance attributes
@relations = []
@sort_attributes = []
@filter_attributes = []
@build_attributes = []
class << self
# Define the accessors for the attributes created
# above
attr_accessor :relations, :sort_attributes,
:filter_attributes, :build_attributes
def build_with(*args)
@build_attributes = args.map(&:to_s)
end
# Add a bunch of methods that will be used in the
# model presenters
def related_to(*args)
@relations = args.map(&:to_s)
end
def sort_by(*args)
@sort_attributes = args.map(&:to_s)
end
def filter_by(*args)
@filter_attributes = args.map(&:to_s)
end
end
# Hidden Code
# accessors, initialize, ...
end
If we run the tests now, they are passing - awesome!
rspec spec/presenters/base_presenter_spec.rb
Success (GREEN)
...
Finished in 0.06283 seconds (files took 2.24 seconds to load)
8 examples, 0 failures
But the code is really repetitive and I don’t like that! I think we could improve it with a bit of metaprogramming. This is totally optional and won’t change any functionality, so implement it only if you want to. I know some developers dislike metaprogramming because it can make the code less readable, but, in my opinion, it makes the code cleaner and more compact.
The logic is the same but now, instead of implementing each method manually, we use a hash containing a key (the method name) and a value (the class instance variable name). All we have to do is loop through this hash to define our methods!
# app/presenters/base_presenter.rb
class BasePresenter
CLASS_ATTRIBUTES = {
build_with: :build_attributes,
related_to: :relations,
sort_by: :sort_attributes,
filter_by: :filter_attributes
}
CLASS_ATTRIBUTES.each { |k, v| instance_variable_set("@#{v}", []) }
class << self
attr_accessor *CLASS_ATTRIBUTES.values
CLASS_ATTRIBUTES.each do |k, v|
define_method k do |*args|
instance_variable_set("@#{v}", args.map(&:to_s))
end
end
end
# Hidden Code
# accessors, initialize, ...
end
Thanks to the tests we wrote, we can be sure that refactoring the code didn’t break anything.
rspec spec/presenters/base_presenter_spec.rb
Success (GREEN)
...
Finished in 0.05242 seconds (files took 2.26 seconds to load)
8 examples, 0 failures
16.7.2. Updating the BookPresenter Class
Now we can quickly update the book presenter by using the new methods we just created: related_to, sort_by and filter_by.
# app/presenters/book_presenter.rb
class BookPresenter < BasePresenter
build_with :id, :title, :subtitle, :isbn_10, :isbn_13, :description,
:released_on, :publisher_id, :author_id, :created_at, :updated_at,
:cover
related_to :publisher, :author
sort_by :id, :title, :released_on, :created_at, :updated_at
filter_by :id, :title, :isbn_10, :isbn_13, :released_on, :publisher_id,
:author_id
def cover
@object.cover.url.to_s
end
end
16.8. Pushing Our Changes
It’s now time to push our changes to GitHub. The flow is the same as in the previous chapter. First, however, let’s run our test suite to ensure that everything is still working.
rspec
Success (GREEN)
...
Finished in 1.34 seconds (files took 2.2 seconds to load)
50 examples, 0 failures
Great! Here is the list of steps to push the code.
Check the changes.
git status
On branch master
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: Gemfile
modified: Gemfile.lock
modified: app/controllers/application_controller.rb
modified: app/controllers/books_controller.rb
modified: app/presenters/base_presenter.rb
modified: app/presenters/book_presenter.rb
modified: spec/presenters/base_presenter_spec.rb
modified: spec/requests/books_spec.rb
Untracked files:
(use "git add <file>..." to include in what will be committed)
app/errors/
app/query_builders/
spec/query_builders/
no changes added to commit (use "git add" and/or "git commit -a")
Stage them.
git add .
Commit the changes.
git commit -m "Add Paginator"
Push to GitHub.
git push origin master
16.9. Wrap Up
In this chapter, we have implemented our first query builder, the Paginator. Throughout its implementation we also created useful mechanisms to make our life a little bit simpler.
We have also updated the BasePresenter class to be ready for all the future query and representation builders.